Method for measuring local application optimization space of operating system

Abstract

The invention discloses a method for measuring local application optimization space of an operating system. The method comprises the following steps: acquiring a local application; determining inter-process communication consumption time of the local application through the eBPF; determining memory operation consumption time of the local application through the eBPF; determining the time consumed for reading and writing the file by the process of the local application through the eBPF, and determining the disk falling time of the local application through the ftrace; further determining IO operation consumption time; determining scheduling operation consumption time of the local application through perf; summing the time to obtain the total consumed time of the operating system of the local application, namely the optimization space of the operating system. According to the method, time consumption of inter-process communication, memory operation, IO operation and scheduling operation generated by the operating system during operation of the application is counted, and the optimization space of the operating system in the local application can be quickly found, so that the input cost of application research and development personnel during performance optimization is greatly reduced, and the research and development pressure of enterprises is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of operating system optimization, and particularly relates to a method for measuring the optimization space of local applications in an operating system. Background Art

[0002] The Linux operating system is widely used, and common applications include databases, containers, virtual machines, scientific computing, etc. Since application developers are not familiar with various mechanisms of the Linux operating system, there are many bottlenecks in the operation of applications on the Linux system. In order to improve the performance of applications, in addition to adjusting the application's own coding, it is also necessary to optimize with the help of various Linux mechanisms. Therefore, it is necessary to evaluate the space for improving the application itself in the Linux operating system.

[0003] To improve the performance of applications on the Linux operating system, the optimization methods adopted include: CPU frequency optimization, network throughput optimization, and disabling unnecessary services, etc. However, these are all optimization contents on the operating system side and cannot be further optimized from the application side.

[0004] Regarding the improvement of application performance, there is currently a lack of a way to judge the optimization direction, resulting in bottlenecks in the optimization of local applications. Summary of the Invention

[0005] In order to overcome the above defects, the present invention is proposed to solve technical problems such as the inability to evaluate the optimization direction of local applications.

[0006] The present invention provides a method for measuring the optimization space of local applications in an operating system, including the following steps: S1, obtain the local application whose optimization space is to be measured; S2, determine the PIPE consumption time, socket consumption time, and message queue consumption time of the local application through eBPF, and determine the inter-process communication consumption time of the local application according to the PIPE consumption time, socket consumption time, and message queue consumption time; S3, determine the user-mode memory consumption time and kernel-mode memory consumption time of the local application through eBPF, and determine the memory operation consumption time of the local application according to the user-mode memory consumption time and kernel-mode memory consumption time; S4, determine the process read / write file consumption time of the local application through eBPF, and determine the disk write consumption time of the local application through ftrace; determine the IO operation consumption time of the local application according to the process read / write file consumption time and the disk write consumption time; S5, determine the scheduling operation consumption time of the local application through perf; S6. Sum up the time consumed by inter - process communication, memory operations, IO operations, and scheduling operations to obtain the total time consumed by the local application's operating system, which is the optimization space of the operating system.

[0007] A further improvement of the present invention lies in that the specific process of determining the PIPE consumption time of the local application is as follows: S201. Mount the entry and return addresses of the pipe_write and pipe_read kernel functions through eBPF. S202. Statistically record the first entry time and the first return time of each call to the pipe_write and pipe_read kernel functions, and calculate the first difference based on the first entry time and the first return time. S203. Sum up all the first differences to determine the PIPE consumption time of the local application.

[0008] A further improvement of the present invention lies in that the specific process of determining the socket consumption time of the local application is as follows: S204. Mount the entry and return addresses of the sock_sendmsg kernel function through eBPF. S205. Statistically record the second entry time and the second return time of each call to the sock_sendmsg kernel function, and calculate the second difference based on the second entry time and the second return time. S206. Sum up all the second differences to determine the socket consumption time of the local application.

[0009] A further improvement of the present invention lies in that the specific process of determining the message queue consumption time of the local application is as follows: S207. Mount the entry and return addresses of the sys_msgsnd and sys_msgrcv kernel functions through eBPF. S208. Statistically record the third entry time and the third return time of each call to the sys_msgsnd and sys_msgrcv kernel functions, and calculate the third difference based on the third entry time and the third return time. S209. Sum up all the third differences to determine the message queue consumption time of the local application.

[0010] A further improvement of the present invention lies in that the specific process of determining the user - mode memory consumption time of the local application is as follows: S301. Mount the entry and return addresses of the malloc and free user - level functions through eBPF. S302. Statistically calculate the fourth entry time and the fourth return time of each call to the user-level functions malloc and free, and calculate the fourth difference based on the fourth entry time and the fourth return time; S303. Sum all the fourth differences to determine the user-mode memory consumption time of the local application.

[0011] A further improvement of the present invention lies in that the specific process of determining the kernel-mode memory consumption time of the local application is as follows: S304. Mount the entry and return addresses of the kernel-mode functions kmalloc and kfree through eBPF; S305. Statistically calculate the fifth entry time and the fifth return time of each call to the kernel-mode functions kmalloc and kfree, and calculate the fifth difference based on the fifth entry time and the fifth return time; S306. Sum all the fifth differences to determine the kernel-mode memory consumption time of the local application.

[0012] A further improvement of the present invention lies in that the specific process of determining the time consumed by the local application for reading and writing files is as follows: S401. Mount the entry and return addresses of the kernel-mode functions sys_write and sys_read through eBPF; S402. Statistically calculate the sixth entry time and the sixth return time of each call to the kernel-mode functions sys_write and sys_read, and calculate the sixth difference based on the sixth entry time and the sixth return time; S403. Sum all the sixth differences to determine the time consumed by the local application for reading and writing files.

[0013] A further improvement of the present invention lies in that the specific process of determining the disk write consumption time of the local application is as follows: S404. Listen for the block:block_rq_issue event and the block:block_rq_complete event through ftrace; S405. Statistically calculate the seventh difference between the time of each arrival at the block:block_rq_issue event and the time of arrival at the block:block_rq_complete event; S406. Sum all the seventh differences to determine the disk write consumption time of the local application.

[0014] A further improvement of the present invention lies in that the specific process of determining the time consumed by the scheduling operation of the local application is as follows: S501. Record the sched:sched_switch event during the runtime of the local application through perf; S502, recording the eighth difference between the timestamps before and after the occurrence of the sched:sched_switch event; S503: Sum all the eighth differences to determine a set of CPU consumption time passively yielded by the local application, that is, the scheduling operation consumption time.

[0015] A further improvement of the present invention is that it also includes: when the local application is a multi-process application, the time of giving up the CPU is overlapping; deleting the overlapping part of the passively giving up the CPU consumption time set to obtain the passively giving up the CPU non-overlapping consumption time, that is, the scheduling operation consumption time.

[0016] Beneficial effects of the present invention: The eBPF tool is used to mount kernel functions such as pipe_write / read, sock_sendmsg, and sys_msgsnd / msgrcv respectively, and the total time difference of calling inter-process communication (PIPE, socket, message queue) is counted; at the same time, eBPF is used to track user-state malloc / free and kernel-state kmalloc / kfree functions to calculate the total time consumption of memory allocation and release; for IO operations, sys_read / write is mounted by eBPF to count the file reading and writing time, and the time consumption of disk placement is calculated in combination with ftrace monitoring block_rq_issue / complete events; in terms of scheduling, sched_switch events are recorded by perf to analyze the total non-overlapping time of passively giving up the CPU. The present invention can quickly find the optimization space of the operating system in local applications, that is, the performance bottleneck of the application, by counting the time consumption of inter-process communication, memory operation, IO operation and scheduling operation generated by the operating system when the application is running, thereby greatly reducing the cost invested by application developers in performance optimization and saving the R&D pressure of the enterprise. DETAILED DESCRIPTION

[0017] In order to have a further understanding of the technical solution and beneficial effects of the present invention, the technical solution of the present invention and the beneficial effects produced are described in detail below.

[0018] In an embodiment of a method for measuring the optimization space of local applications of an operating system proposed by the present invention, a domestic FT2000+ / 64-core server is used as a hardware device to install the Kylin Advanced Server Operating System V10 system. The specific steps are as follows: S1, prepare a multi-process program test. In this embodiment, the multi-process program test is used as a local application; S2, the method for obtaining the time consumed by inter-process communication is as follows, including counting the PIPE consumption time, the socket consumption time and the message queue consumption time. Among them: The specific process of counting the PIPE consumption time is as follows: S201, Mount the entry and return addresses of the pipe_write and pipe_read kernel functions through eBPF; S202, Count the first entry time and the first return time of each call to the pipe_write and pipe_read kernel functions, and calculate the first difference according to the first entry time and the first return time; S203, Sum all the first differences to obtain the PIPE consumption time of the test program.

[0019] It should be noted that the terms "first", "second", etc. here are used to distinguish similar objects, rather than representing a specific order or sequence. In appropriate cases, the usage order of similar objects can be interchanged. The subsequent "first", "second", etc. have a similar effect and will not be elaborated further.

[0020] The specific process of counting the socket consumption time is as follows: S204, Mount the entry and return addresses of the sock_sendmsg kernel function through eBPF; S205, Count the second entry time and the second return time of each call to the sock_sendmsg kernel function, and calculate the second difference according to the second entry time and the second return time; S206, Sum all the second differences to obtain the socket consumption time of the test program.

[0021] The specific process of counting the message queue consumption time is as follows: S207, Mount the entry and return addresses of the sys_msgsnd and sys_msgrcv kernel functions through eBPF; S208, Count the third entry time and the third return time of each call to the sys_msgsnd and sys_msgrcv kernel functions, and calculate the third difference according to the third entry time and the third return time; S209, Sum all the third differences to obtain the message queue consumption time of the test program.

[0022] S3, The method for obtaining the memory operation consumption time is as follows, including counting the user-mode memory consumption time and the kernel-mode memory consumption time. Among them, the user-mode memory consumption time includes the total consumption time of user-mode memory allocation and release, and the kernel-mode memory consumption time includes the total consumption time of kernel-mode memory allocation and release: The specific process of counting the user-mode memory consumption time is as follows: S301, Mount the entry and return addresses of the user-level functions malloc and free through eBPF; S302, Statistically record the fourth entry time and the fourth return time of each call to the user-level functions malloc and free, and calculate the fourth difference based on the fourth entry time and the fourth return time; S303, Sum all the fourth differences to obtain the total consumption time of user-mode memory allocation and release, that is, the user-mode memory consumption time of the test program.

[0023] The specific process of statistically recording the kernel-mode memory consumption time is as follows: S304, Mount the entry and return addresses of the kernel-mode functions kmalloc and kfree through eBPF; S305, Statistically record the fifth entry time and the fifth return time of each call to the kernel-mode functions kmalloc and kfree, and calculate the fifth difference based on the fifth entry time and the fifth return time; S306, Sum all the fifth differences to obtain the total consumption time of kernel-mode memory allocation and release, that is, the kernel-mode memory consumption time of the test program.

[0024] S4, The method for obtaining the consumption time of IO operations is as follows, including statistically recording the consumption time of the process reading and writing files and statistically recording the consumption time of disk flushing. Among them: The specific process of statistically recording the consumption time of the process reading and writing files is as follows: S401, Mount the entry and return addresses of the kernel-mode functions sys_write and sys_read through eBPF; S402, Statistically record the sixth entry time and the sixth return time of each call to the kernel-mode functions sys_write and sys_read, and calculate the sixth difference based on the sixth entry time and the sixth return time; S403, Sum all the sixth differences to obtain the consumption time of the process reading and writing files of the test program.

[0025] The specific process of statistically recording the consumption time of disk flushing is as follows: S404, Listen for the block:block_rq_issue event and the block:block_rq_complete event through ftrace; S405, Statistically record the seventh difference between the time of each arrival at the block:block_rq_issue event and the time of arrival at the block:block_rq_complete event; S406, Sum all the seventh differences to obtain the consumption time of disk flushing of the test program.

[0026] S5. The method for obtaining the time consumed by the scheduling operation is as follows: The specific process of counting the time consumed by the scheduling operation (that is, counting the total non-overlapped time of passively yielding the CPU. It should be noted that since the test in this embodiment is a multi-process program, there may be a situation where the CPU is yielded simultaneously at the same time, so "non-overlapped" is emphasized here) is as follows: S501. Record the sched:sched_switch event during the running of the test program through perf; S502. Record the eighth difference of the timestamps before and after the occurrence of the sched:sched_switch event; S503. Sum up all the eighth differences to obtain the set of time consumed by passively yielding the CPU, that is, the time consumed by the scheduling operation; S504. Considering the multi-process test program, there may be a situation where the CPU is yielded simultaneously at the same time, that is, the time of yielding the CPU is overlapped; S505. Delete the overlapping part in the set of time consumed by passively yielding the CPU to obtain the non-overlapped time consumed by passively yielding the CPU, that is, the time consumed by the scheduling operation of the multi-process test program.

[0027] S6. Execute steps S2 to S5, run the test program, obtain the time consumed by inter-process communication, the time consumed by memory operations, the time consumed by IO operations, and the time consumed by the scheduling operation. Sum up the time consumed by inter-process communication, the time consumed by memory operations, the time consumed by IO operations, and the time consumed by the scheduling operation to obtain the total operating system time of the test program, which is also the optimization space of the operating system, that is, the performance bottleneck of the test program.

[0028] Advantages of the present invention: Use eBPF tools to separately hook kernel functions such as pipe_write / read, sock_sendmsg, and sys_msgsnd / msgrcv to count the total sum of call time differences for inter-process communication (PIPE, socket, message queue); at the same time, use eBPF to trace the user-space malloc / free and kernel-space kmalloc / kfree functions to calculate the total time consumed for memory allocation and release; for IO operations, mount sys_read / write through eBPF to count the file read and write time, and combine ftrace to listen for block_rq_issue / complete events to calculate the disk write time; for scheduling, record the sched_switch event through perf and analyze the total sum of non-overlapping time for passively yielding the CPU. By adding up the total time consumed for inter-process communication, memory operations, IO operations, and scheduling operations, the total system consumption time under the local application can be obtained, which is the optimization space of the operating system and the performance bottleneck of the application. By counting the time consumption of inter-process communication, memory operations, IO operations, and scheduling operations generated by the operating system during the application runtime, the present invention can quickly find the optimization space of the operating system in the local application, that is, the performance bottleneck of the application, thereby greatly reducing the cost invested by application developers during performance optimization and saving the R & D pressure of the enterprise.

[0029] Although the present invention has been described using the above preferred embodiments, it is not intended to limit the protection scope of the present invention. Any person skilled in the art can make various changes and modifications to the above embodiments without departing from the spirit and scope of the present invention, and these still fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the claims.

Claims

1. A method for measuring the optimization space of local applications in an operating system, characterized in that, Including: S1. Obtain the local application for which the optimization space is to be measured; S2. Determine the PIPE consumption time, socket consumption time, and message queue consumption time of the local application through eBPF, and determine the inter-process communication consumption time of the local application according to the PIPE consumption time, socket consumption time, and message queue consumption time; S3. Determine the user-mode memory consumption time and kernel-mode memory consumption time of the local application through eBPF, and determine the memory operation consumption time of the local application according to the user-mode memory consumption time and kernel-mode memory consumption time; S4. Determine the process read / write file consumption time of the local application through eBPF, and determine the disk write consumption time of the local application through ftrace; Determine the IO operation consumption time of the local application according to the process read / write file consumption time and disk write consumption time; S5. Determine the scheduling operation consumption time of the local application through perf; S6. Sum up the inter-process communication consumption time, memory operation consumption time, IO operation consumption time, and scheduling operation consumption time to obtain the total operating system time consumption of the local application, that is, the optimization space of the operating system.

2. The method for measuring the optimization space of local applications in an operating system according to claim 1, wherein, The specific process of determining the PIPE consumption time of the local application is as follows: S201. Mount the entry and return addresses of the pipe_write and pipe_read kernel functions through eBPF; S202. Statistically record the first entry time and the first return time of each call to the pipe_write and pipe_read kernel functions, and calculate the first difference according to the first entry time and the first return time; S203. Sum up all the first differences to determine the PIPE consumption time of the local application.

3. The method for measuring the optimization space of local applications in an operating system according to claim 1, wherein The specific process of determining the socket consumption time of the local application is as follows: S204. Mount the entry and return addresses of the sock_sendmsg kernel function through eBPF; S205. Statistically record the second entry time and the second return time of each call to the sock_sendmsg kernel function, and calculate the second difference according to the second entry time and the second return time; S206. Sum up all the second differences to determine the socket consumption time of the local application.

4. The method for measuring the optimization space of local applications of an operating system according to claim 1, wherein, The specific process of determining the message queue consumption time of the local application is as follows: S207. Mount the entry and return addresses of the sys_msgsnd and sys_msgrcv kernel functions through eBPF; S208. Statistically record the third entry time and the third return time of each call to the sys_msgsnd and sys_msgrcv kernel functions, and calculate the third difference according to the third entry time and the third return time; S209. Sum up all the third differences to determine the message queue consumption time of the local application.

5. The method for measuring the optimization space of local applications in an operating system according to claim 1, characterized in that The specific process of determining the user-mode memory consumption time of the local application is as follows: S301. Mount the entry and return addresses of the malloc and free user-level functions through eBPF; S302. Stat the fourth entry time and the fourth return time of each call to the user-level functions malloc and free, and calculate the fourth difference based on the fourth entry time and the fourth return time. S303. Sum up all the fourth differences to determine the user-mode memory consumption time of the local application.

6. The method for measuring the optimization space of local applications in an operating system according to claim 1, characterized in that The specific process for determining the kernel-mode memory consumption time of the local application is as follows: S304. Mount the entry and return addresses of the kernel-mode functions kmalloc and kfree through eBPF. S305. Stat the fifth entry time and the fifth return time of each call to the kernel-mode functions kmalloc and kfree, and calculate the fifth difference based on the fifth entry time and the fifth return time. S306. Sum up all the fifth differences to determine the kernel-mode memory consumption time of the local application.

7. The method for measuring the optimization space of local applications of an operating system according to claim 1, characterized in that The specific process for determining the time consumed by the local application for reading and writing files is as follows: S401. Mount the entry and return addresses of the kernel-mode functions sys_write and sys_read through eBPF. S402. Stat the sixth entry time and the sixth return time of each call to the kernel-mode functions sys_write and sys_read, and calculate the sixth difference based on the sixth entry time and the sixth return time. S403. Sum up all the sixth differences to determine the time consumed by the local application for reading and writing files.

8. The method for measuring the optimization space of local applications in an operating system according to claim 1, wherein, The specific process for determining the disk write consumption time of the local application is as follows: S404. Listen for the block:block_rq_issue event and the block:block_rq_complete event through ftrace. S405. Stat the seventh difference between the time of each arrival at the block:block_rq_issue event and the time of arrival at the block:block_rq_complete event. S406. Sum up all the seventh differences to determine the disk write consumption time of the local application.

9. The method for measuring the optimization space of local applications of an operating system according to claim 1, wherein The specific process for determining the time consumed by the scheduling operations of the local application is as follows: S501. Record the sched:sched_switch event during the runtime of the local application through perf. S502. Record the eighth difference between the timestamps before and after the occurrence of the sched:sched_switch event. S503. Sum up all the eighth differences to determine the set of time consumed by the local application for passively yielding the CPU, which is the time consumed by the scheduling operations.

10. A method for measuring the optimization space of local applications in an operating system according to claim 9, characterized in that, It also includes: When the local application is a multi-process application, the time for yielding the CPU is overlapping. Delete the overlapping part of the set of time consumed by passively yielding the CPU to obtain the non-overlapping time consumed by passively yielding the CPU, which is the time consumed by the scheduling operations.