Storage diagnosis and early warning method based on ebpf technology

Abstract

The invention provides a storage diagnosis and early warning method based on an ebpf technology, and belongs to the technical field of storage equipment diagnosis analysis, and the method comprises the steps: employing the ebpf technology, employing a hook technology principle, employing a tracepoint technology to track a blockrqcompact point on an ebpf layer, tracking a blockrqcompact function in a kernel block drive, and obtaining related information of a completed request; and then processing and processing the acquired information of the storage device. The method is specially used for monitoring key data executed by I / O of the storage device in real time and can be flexibly adjusted according to specific services; on the premise that kernel codes are not modified and the kernel is not recompiled and deployed, the kernel can be dynamically debugged; through an ebpf technology, data executed by storage I / O is obtained from a kernel layer, and the sequence of storage devices, the total number of random I / Os and other data are counted through an algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of storage device diagnosis and analysis, and in particular to a storage diagnosis and early warning method, and in particular to a storage diagnosis and early warning method based on ebpf technology. Background Art

[0002] With the rapid development of the times, the performance and stability of server storage are crucial to ensuring customer experience and enterprise competitiveness. With the rapid development of cutting-edge technologies such as network storage and cloud computing, the storage environment has become more and more complex, and the deployment of cluster services has become larger and larger, which has put forward greater requirements for analyzing storage problems. Therefore, it is very important to collect data executed by storage device I / O in real time and perform fine-grained data analysis to analyze storage performance and problems.

[0003] As an innovative kernel-level monitoring tool, eBPF (Extended Berkeley Packet Filter) technology has become a hot topic in the field of storage technology. It allows monitoring code to be dynamically deployed in the operating system kernel without changing or restarting the system. With eBPF, developers can deeply explore the microscopic details of storage I / O operations, accurately monitor the dynamic changes of data flows, and promptly identify potential performance bottlenecks.

[0004] Kernel debugging tools include kdb and kgdb, and detection methods include kprobe, retprobe, ftrace, etc. These are debugging methods used for kernel performance analysis and troubleshooting. The core principle of these kernel detection methods is to use trace point instructions to detect kernel functions. The requirements for users are relatively high, and they need to understand the internal operating principles. When using them, they need to recompile the kernel and deploy it to the machine. Many customers cannot replace the kernel on site; and these methods cannot dynamically modify the variable values ​​in the trace point function body, which is very inflexible.

[0005] Currently, there are tools such as iostat and blktrace to analyze I / O performance and defects. The iostat tool mainly sees the await information at a macro level, while blktrace mainly analyzes the detailed process of I / O execution and performs offline analysis. They cannot count the I / O execution of all storage devices in real time, such as the total number of sequential I / Os executed by each disk, the total number of random I / Os, the total number of I / O executions, the total amount of I / O data, and the I / O execution status. The I / O execution status is used for alarm prediction. If the status is a negative value, it means there is an I / O error. These data can also judge the I / O performance at a macro level and intuitively see the I / O execution of all storage devices, which plays a great role in quickly locating I / O problems.

[0006] Currently, there is no tool that can perform statistics on sequential and random I / O data for storage I / O from the kernel layer. Therefore, it is very important to obtain key data of storage I / O execution from the kernel layer based on eBPF technology, and to count the total number of random and sequential I / O of storage devices in real time. It can also check the status of I / O and play the role of error warning, display it in real time through user-mode programs and save the error I / O to local files. There is also a system alarm function for error I / O. Summary of the invention

[0007] In response to the above problems, the present invention provides a storage diagnosis and early warning method based on eBPF technology, which is dedicated to real-time monitoring of key data of storage device I / O execution, including but not limited to current I / O execution data, and can be flexibly adjusted according to specific business.

[0008] To achieve the above object, the technical solution of the present invention is: A storage diagnosis and early warning method based on ebpf technology uses hook technology at the ebpf layer to track the kernel block_rq_complete function to obtain statistical information of storage devices, specifically including the following steps: S1: The ebpf layer defines a BPF map of type hash table to store I / O statistics of storage devices; S2: The ebpf layer sets the tracepoint function, obtains the I / O statistics of the storage device in the kernel layer through the SEC ("tracepoint / block / block_rq_complete") function, and then calls back to the handle_block_rq_completion function to obtain the I / O statistics of the storage device in the ebpf layer; S3: The ebpf layer adds a storage device filtering function and defines two global variables, filter_dev and targ_dev, for filtering storage devices. S4: the ebpf layer searches for or initializes the statistical information of the corresponding storage device by using the bpf_map_lookup_or_try_init function, where the statistical information is the I / O statistical information of the storage device obtained in step S2, and the corresponding device is the storage device selected after processing in step S3; S5: The ebpf layer responds to the execution parameters of the application and determines the I / O execution mode according to the sector number sector and device identifier dev of the storage device of the current I / O request and the previous I / O request, and the statistical information of the corresponding storage device obtained in step S4; then the corresponding statistical information is updated using the __sync_fetch_and_add function and stored in bpf_map_look_up; at the same time, the value of last_sector of the corresponding storage device is updated for the next comparison; S6: The front-end application uses the print_map function to print the statistics in the BPF map and determine the I / O execution status.

[0009] Furthermore, the key of the BPF map in step S1 is a storage device identifier, and the value is a counter structure.

[0010] Furthermore, the statistical information in step S2 includes the sector number sector, the number of sectors nr_sector and the device identifier dev; the parameter accepted when calling back to the handle_block_rq_completion function is void , indicating that any type of data is supported.

[0011] Furthermore, in step S3, the storage device filtering function is controlled by the application execution parameters, the global variable filter_dev determines whether to enable device filtering, and the global variable targ_dev is the target storage device identifier; if filter_dev is true and the current device is not the target storage device, it returns directly.

[0012] Furthermore, in step S4, after searching or initializing the statistical information of the corresponding storage device, the statistical information of the storage device is stored in the BPF map in real time; if there is no statistical information of the corresponding storage device in the BPF map, the zero structure is used for initialization.

[0013] Furthermore, in step S5, the I / O execution mode includes random and sequential. If the sector number sector of the current I / O request is the same as that of the previous I / O request, it is sequential. If the sector number sector of the current I / O request is different from that of the previous I / O request, it is random. While judging the I / O execution mode, the total number of random 1 / 0s, the total number of sequential I / Os, the percentage of random I / Os, the percentage of sequential I / Os, the total I / O data, the I / O status data and the total data volume of the storage device are calculated through an algorithm.

[0014] Furthermore, in step S5, while updating the value of last_sector, the total data volume of the corresponding storage device is updated, and the total data volume is calculated by multiplying the number of sectors nr_sector by the number of bytes of each sector, and the number of bytes of each sector is 512.

[0015] Furthermore, in step S6, the sleep function is used to wait for the specified time env.interval and the print_map function is used to print the statistical data in the BPF map. If an exit signal is received or the specified number of runs is reached, the loop is exited.

[0016] Furthermore, before printing the statistical information in the BPF map in step S6, the BPF map is traversed by the bpf_map_lookup_elem function to obtain the statistical information of each storage device; if the timestamp is enabled when printing the statistical information in the BPF map, the statistical information of the current time corresponding to the timestamp is printed first.

[0017] Furthermore, in step S6, the I / O execution status is determined based on the printed statistical information. If the I / O status data in the statistical information is a negative value, the system interface is called to directly alarm, indicating an I / O error, and the I / O request information corresponding to the error is transferred to a local file.

[0018] The beneficial effects of the storage diagnosis and early warning method based on eBPF technology of the present invention are: The present invention proposes a new method to address the shortcomings of debugging tools in the prior art, which enables the kernel to be dynamically debugged without modifying the kernel code, recompiling, and deploying the kernel; through the eBPF technology, the data of storage I / O execution is obtained from the kernel layer, and the total number of sequential and random I / Os of the storage device and other data are counted through an algorithm.

[0019] The present invention obtains kernel function tracking points and kernel disk information through hook technology based on ebpf technology, processes and processes disk data, and counts the total number of disk device sequences and random I / Os in real time. The method is used to analyze I / O performance bottlenecks and scenarios where I / O errors occur, to determine which I / O mode is being executed and the proportion of various I / O mode operations during the period. These data are of great help in analyzing I / O problems.

[0020] When debugging the kernel, the present invention does not need to modify the kernel code and compile and redeploy to the target machine. It only needs to modify the ebpf kernel tracking point to obtain the required kernel data and process the data according to business needs. For customers who do not want to redeploy the kernel, it is of great help in analyzing kernel problems.

[0021] In terms of program implementation effect, the present invention can count all disk I / O execution conditions in real time, including the total number of sequential I / Os executed by each disk, the total number of random I / Os, the total number of I / O executions, the total amount of I / O data, and the I / O execution status. These data can be used to determine disk performance problems and I / O execution status. If the value is negative, the system interface is directly called to sound an alarm, indicating an I / O error. The error-corresponding I / O request information is transferred to a local file, which is very helpful for quickly locating I / O problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] Figure 1 It is an overall architecture diagram of a storage diagnosis and early warning method based on eBPF technology of the present invention. DETAILED DESCRIPTION

[0024] The technical scheme in the embodiment of the present invention is described clearly and completely below in conjunction with the accompanying drawings in the embodiment of the present invention. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below. Example 1

[0025] A storage diagnosis and early warning method based on ebpf technology, such as Figure 1 As shown, ebpf technology is used, the hook technology principle is used, and the tracepoint technology is used in the ebpf layer to track the block_rq_complete point, and the block_rq_complete function in the kernel block driver is traced to obtain the relevant information of the completed request. The specific steps include: S1: The ebpf layer defines a BPF map of type hash table to store I / O statistics of storage devices; the key of the BPF map is the identifier of the storage device, and the value is a counter structure. The structure is explained in detail as follows: struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 64); __type(key, u32); __type(value, struct counter); } counters SEC(".maps") S2: The ebpf layer sets the tracepoint function, obtains the I / O statistics of the storage device in the kernel layer through the SEC ("tracepoint / block / block_rq_complete") function, and then calls back to the handle_block_rq_completion function to obtain the I / O statistics of the storage device in the ebpf layer; the statistics include the sector number sector, the number of sectors nr_sector and the device identifier dev; the accepted parameter when calling back to the handle_block_rq_completion function is void , indicating that any type of data is supported.

[0026] Among them, block_rq_complete is the corresponding function passed to the kernel layer, and handle_block_rq_completion is the processing function that block_rq_complete traces to the kernel layer and calls back to the ebpf layer.

[0027] S3: The ebpf layer adds storage device filtering function and defines two global variables filter_dev and targ_dev for storage device filtering; the storage device filtering function is controlled by the application execution parameters (that is, the startup filter parameters are passed from the application execution parameters), the global variable filter_dev determines whether to enable device filtering, and the global variable targ_dev is the identifier of the target storage device; if filter_dev is true and the current device is not the target storage device, it returns directly. This allows users to track only specific storage devices, not all devices.

[0028] S4: The ebpf layer searches for or initializes the statistical information of the corresponding storage device by using the bpf_map_lookup_or_try_init function. The statistical information is the I / O statistical information of the storage device obtained in step S2, and the corresponding device is the storage device selected after processing in step S3. After searching or initializing the statistical information of the corresponding storage device, the statistical information of the storage device is stored in the BPF map in real time. If there is no statistical information of the corresponding storage device in the BPF map, it is initialized using the zero structure.

[0029] S5: The ebpf layer responds to the execution parameters of the application and determines the I / O execution mode according to the sector number sector and device identifier dev of the storage device of the current I / O request and the previous I / O request, and the statistical information of the corresponding storage device obtained in step S4 (infinite polling to obtain the data of the kernel disk operation, and the corresponding function of the ebpf layer will process the data when the driver disk operation data is scanned); then the __sync_fetch_and_add function is used to update the corresponding statistical information and store it in bpf_map_look_up; at the same time, the value of last_sector of the corresponding storage device (that is, the sector number of the previous device used in the most recent comparison) is updated to facilitate the next comparison; The device identifier dev is used to mark a specific storage device (disk device). Each I / O operation needs to indicate which storage device, which sector, and the number of sectors. These data are calculated by the algorithm in the eBPF layer to calculate the number of sequential I / Os, random I / Os, and I / O status, and then saved in the bpf map lookup. The purpose is to provide data to the application layer (application). The application polls to obtain the data in the bpf map lookup and then performs business processing. First, it determines the I / O status and performs error warning functions. Second, it prints all temporary data required for disk I / O execution in real time. The I / O execution mode includes random and sequential. If the sector number sector of the current I / O request is the same as that of the previous I / O request, it is sequential. If the sector number sector of the current I / O request is different from that of the previous I / O request, it is random. While judging the I / O execution mode, the total number of random 1 / 0s, the total number of I / O sequences, the percentage of random I / Os, the percentage of sequential I / Os, the total I / O data, the I / O status data and the total data volume of the storage device are calculated by the algorithm. When the value of last_sector is updated, the total data volume of the corresponding storage device is updated. The total data volume is calculated by multiplying the number of sectors nr_sector by the number of bytes of each sector, and the number of bytes of each sector is 512.

[0030] S6: The front-end application uses the print_map function to print the statistics in the BPF map and determine the I / O execution status. Specifically: Use the sleep function to wait for the specified time env.interval and use the print_map function to print the statistics in the BPF map. If an exit signal is received or the specified number of runs is reached, the loop is exited; Before printing the statistics in the BPF map, the bpf_map_lookup_elem function is used to traverse the BPF map to obtain the statistics of each storage device. When printing the statistics in the BPF map, if the timestamp is enabled, the statistics of the current time corresponding to the timestamp are printed first. The I / O execution status is determined based on the printed statistical information. If the I / O status data in the statistical information is a negative value, the system interface is called to directly alarm, indicating an I / O error. At the same time, the I / O request information corresponding to the error is transferred to a local file. Example 2

[0031] On the basis of Example 1, the application and the eBPF layer transfer data through a BPF map hash table structure; the application itself sets execution parameters, such as specifying a disk (storage device), that is, only executing disk statistics I / O execution data, that is, the eBPF filter function, and setting a timer, execution count, and execution timestamp, that is, displaying real time; the execution parameters of the application are passed to the eBPF layer for execution; the end of the application depends on the parameters of the execution program, such as sudo . / biomode 1 10 means to print output once per second and last for 10 seconds.

[0032] In summary, the present invention uses eBPF technology to track kernel key I / O execution functions and obtain real-time status and data of storage devices; the eBPF layer obtains real-time data of kernel disk (storage device) for processing and processing, mainly obtaining disk sector number, sector quantity, and disk device identifier.

[0033] The ebpf layer adds a filter function. Turning on this function means that the I / O operation data of the specified disk is counted. Otherwise, the I / O operations and data of all disks are counted by default. The ebpf layer can determine whether the current request is random or sequential based on the sector number of the current I / O request and the previous I / O request. If the sector numbers of the two requests are the same, then it is sequential; otherwise, it is random. Then, the __sync_fetch_and_add function is used to update the corresponding statistics. This is an atomic operation to ensure data consistency in a concurrent environment. Update the sector number of the last I / O request and update the value of last_sector for the next comparison.

[0034] The application prints in real time the name of the disk (storage device) being tracked, the percentage of random I / O, the percentage of sequential I / O, the total number of I / O requests in a specified time interval, and the amount of data read and written in a specified time interval. The application configures execution parameters in real time and can dynamically set the execution time, execution count, timestamp, and specify the disk device being executed. Determine the I / O execution status. If it is a negative value, call the system interface to directly alarm, indicating an I / O error, and transfer the I / O request information corresponding to the error to a local file.

[0035] The present invention does not need to change the kernel, recompile the kernel, and redeploy it to the device. It only needs to change the ebpf tracking point as needed, obtain different tracepoint functions of the kernel, obtain the required kernel data, process and process it, and compile the application. It is of great help to customers who do not want to modify the kernel and redeploy the environment.

[0036] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. A storage diagnosis and early warning method based on ebpf technology, characterized in that: At the ebpf layer, the hook technology is used to track the kernel block_rq_complete function to obtain the statistical information of the storage device. The specific steps include: S1: The ebpf layer defines a BPF map of type hash table to store I / O statistics of storage devices; S2: The ebpf layer sets the tracepoint function, obtains the I / O statistics of the storage device in the kernel layer through the SEC ("tracepoint / block / block_rq_complete") function, and then calls back to the handle_block_rq_completion function to obtain the I / O statistics of the storage device in the ebpf layer; S3: The ebpf layer adds a storage device filtering function and defines two global variables, filter_dev and targ_dev, for filtering storage devices. S4: the ebpf layer searches for or initializes the statistical information of the corresponding storage device by using the bpf_map_lookup_or_try_init function, where the statistical information is the I / O statistical information of the storage device obtained in step S2, and the corresponding device is the storage device selected after processing in step S3; S5: The ebpf layer responds to the execution parameters of the application and determines the I / O execution mode according to the sector number sector and device identifier dev of the storage device of the current I / O request and the previous I / O request, and the statistical information of the corresponding storage device obtained in step S4; then the corresponding statistical information is updated using the __sync_fetch_and_add function and stored in bpf_map_look_up; at the same time, the value of last_sector of the corresponding storage device is updated for the next comparison; S6: The front-end application uses the print_map function to print the statistics in the BPF map and determine the I / O execution status.

2. According to the storage diagnosis and early warning method based on ebpf technology according to claim 1, it is characterized in that: The key of the BPF map in step S1 is the storage device identifier, and the value is a counter structure.

3. According to the storage diagnosis and early warning method based on eBPF technology according to claim 1, it is characterized in that: The statistical information in step S2 includes the sector number sector, the number of sectors nr_sector and the device identifier dev; the parameter accepted when calling back to the handle_block_rq_completion function is void , indicating that any type of data is supported.

4. According to the storage diagnosis and early warning method based on ebpf technology as described in claim 1, it is characterized in that: In step S3, the storage device filtering function is controlled by the application execution parameters. The global variable filter_dev determines whether to enable device filtering. The global variable targ_dev is the target storage device identifier. If filter_dev is true and the current device is not the target storage device, it returns directly.

5. The storage diagnosis and early warning method based on eBPF technology according to claim 1 is characterized in that: In step S4, after searching or initializing the statistical information of the corresponding storage device, the statistical information of the storage device is stored in the BPF map in real time; if there is no statistical information of the corresponding storage device in the BPF map, it is initialized using a zero structure.

6. The storage diagnosis and early warning method based on ebpf technology according to claim 1 is characterized in that: In step S5, the I / O execution mode includes random and sequential. If the sector number sector of the current I / O request is the same as that of the previous I / O request, it is sequential. If the sector number sector of the current I / O request is different from that of the previous I / O request, it is random. While judging the I / O execution mode, the total number of random 1 / 0s, the total number of sequential I / Os, the percentage of random I / Os, the percentage of sequential I / Os, the total I / O data, the I / O status data and the total data volume of the storage device are calculated through an algorithm.

7. The storage diagnosis and early warning method based on eBPF technology according to claim 6 is characterized in that: In step S5, while updating the value of last_sector, the total data volume of the corresponding storage device is updated. The total data volume is calculated by multiplying the number of sectors nr_sector by the number of bytes per sector, and the number of bytes per sector is 512.

8. The storage diagnosis and early warning method based on eBPF technology according to claim 1 is characterized in that: In step S6, the sleep function is used to wait for the specified time env.interval and the print_map function is used to print the statistical data in the BPF map. If an exit signal is received or the specified number of runs is reached, the loop is exited.

9. The storage diagnosis and early warning method based on eBPF technology according to claim 8 is characterized in that: Before printing the statistical information in the BPF map in step S6, the BPF map is traversed through the bpf_map_lookup_elem function to obtain the statistical information of each storage device; if the timestamp is enabled when printing the statistical information in the BPF map, the statistical information of the current time corresponding to the timestamp is printed first.

10. The storage diagnosis and early warning method based on eBPF technology according to claim 9 is characterized in that: In step S6, the I / O execution status is determined based on the printed statistical information. If the I / O status data in the statistical information is a negative value, the system interface is called to directly alarm, indicating an I / O error. At the same time, the I / O request information corresponding to the error is transferred to a local file.

Images