A Linux kernel event log collection method and system based on eBPF

Abstract

The present invention belongs to the field of network security and operating systems, and discloses a Linux kernel event log collection method and system based on eBPF. The method includes parsing a command line input by a user to obtain an event type, output format, and output location; initializing a collector; parsing data captured by a hook function when a mount point is triggered, filtering the parsed data, performing structural conversion on the filtered data, encoding and compressing the data after the structural conversion, storing the encoded and compressed data in a secondary buffer, and synchronizing the encoded and compressed data in the secondary buffer to the ring buffer when the free length of the ring buffer is not less than the amount of encoded and compressed data; and a processing thread obtaining newly added data in the ring buffer, converting the newly added data according to an output format, and outputting the converted data according to an output location. The present invention monitors, tracks, and observably monitors kernel events, and has strong flexibility.

Description

Technical Field

[0001] The present invention belongs to the field of network security and operating systems, and in particular relates to a Linux kernel event log collection method and system based on eBPF (extended Berkeley Packet Filter). Background Art

[0002] The kernel log is a crucial component of any operating system. It records various events, errors, and warnings that occur during kernel runtime, providing a crucial reference for system stability, performance optimization, and troubleshooting. Through the kernel log, system administrators and developers can monitor the system's operational status, understand its health, and identify and resolve potential issues. The kernel log not only records key runtime events but also helps users track the execution of specific processes or modules, analyze system performance bottlenecks, and diagnose the causes of system failures. Overall, the kernel log is an indispensable resource for system maintenance and development, playing an irreplaceable role in ensuring stable system operation and continuous optimization.

[0003] Current Linux event log collection tools primarily focus on collecting user-level data, recording the behavior and purpose of programs and software through API calls. However, data collected at the user level has inherent disadvantages, including unclear semantics and potential for malicious tampering. In addition to data incompleteness, existing log collection tools also suffer from high performance overhead. Frequent log writes can lead to significant CPU and memory resource consumption, resulting in slow operating system response or latency. Furthermore, the heterogeneous nature of the log data collected by the tools makes it difficult to read and integrate the collected log information. Finally, because kernel logs generate a large amount of data, traditional log storage mechanisms may not meet requirements and cannot guarantee the integrity and availability of log data.

[0004] In summary, kernel logs play an important role in operating system maintenance and monitoring due to their high granularity, security, and resistance to tampering. However, existing Linux log collection systems are insufficient to meet the needs of collecting and analyzing kernel logs. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a Linux kernel event log collection method based on eBPF, wherein eBPF is a kernel technology provided by Linux. Without compiling the kernel or loading kernel modules, it can be safely and efficiently attached to various kernel events to monitor, track, and observe kernel events. The method has strong flexibility and helps the present invention to more flexibly obtain highly fine-grained kernel log information. Using eBPF technology, the corresponding tracepoint of the event to be collected is mounted to obtain the corresponding kernel event.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A Linux kernel event log collection method based on eBPF, the Linux kernel event log collection method based on eBPF comprising:

[0008] Step 1: Parse the command line input by the user to obtain the event type, output format and output location;

[0009] Step 2: Initialize the collector, including:

[0010] Step 2-1: Determine the mount point based on the event type and the event selected for tracking by the user in the preloaded mount point configuration file, compile, verify, and convert the eBPF program to obtain a machine instruction set, and mount the machine instruction set as a hook function to the determined mount point;

[0011] Step 2-2, loading a filtering configuration file and an event structure configuration file, configuring data filtering rules for the collector according to the filtering configuration file, and configuring the structure of event information according to the event structure configuration file;

[0012] Step 2-3, initialize the ring buffer, the ring buffer is located in the kernel;

[0013] Step 2-4: Initialize the secondary buffer, which is located in the kernel;

[0014] Step 2-5: Create a processing thread, where the processing thread is configured to monitor data changes in the ring buffer and perform conversion and output;

[0015] Step 3. The collector performs data collection: when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules. The filtered data is structurally converted according to the structure of the configured event information, and after the structural conversion, the encoded and compressed data is stored in the secondary buffer. The secondary buffer is dynamically resized according to the amount of encoded and compressed data. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer.

[0016] Step 4: The collector outputs data: the processing thread obtains the newly added data in the ring buffer, converts the newly added data according to the output format, and outputs the converted data according to the output location.

[0017] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0018] Preferably, in step 3, parsing the data captured by the hook function includes:

[0019] Get the format file corresponding to each tracepoint, use the structure and field information defined in the format file to parse the parameter data obtained by the hook function when the kernel runs the triggered mount point, extract the value of each field from the parameter data obtained by the hook function according to the offset and data type in the format file, and convert the value of each field into a readable form.

[0020] Preferably, the structure of the event information consists of two parts, one of which is the attributes common to all events, namely BaseEvent; and the other is the parameters of each event itself, which come from the format file of the corresponding mount point.

[0021] Preferably, in step 3, the coding compression includes:

[0022] For the data stream after structural conversion, the corresponding event type is mapped to a unique integer code according to the preset event type coding table; the corresponding error message or warning message is mapped to a unique integer code according to the preset information coding table.

[0023] Preferably, in step 3, the secondary buffer is dynamically resized according to the amount of data after encoding and compression, including:

[0024] At each preset time interval, the amount of encoded and compressed data within the preset time is counted, and the average data flow within the current preset time is calculated;

[0025] If the average data flow exceeds the upper threshold, the expansion operation of the secondary buffer is triggered, and the size of the secondary buffer is expanded to the original first preset multiple to obtain a new secondary buffer, and the upper and lower thresholds are updated based on the new secondary buffer; if the average data flow is lower than the lower threshold, the reduction operation of the secondary buffer is triggered, and the size of the secondary buffer is reduced to the original second preset multiple to obtain a new secondary buffer, and the upper and lower thresholds are updated based on the new secondary buffer.

[0026] Preferably, the output formats include JSON format, text format and traceability graph format, and the output locations include output in a terminal and output in a file.

[0027] Preferably, the eBPF-based Linux kernel event log collection method further includes an ending stage, in which the ring buffer, the secondary buffer, and the eBPF Map and file descriptors related to the eBPF program are released.

[0028] The present invention provides a Linux kernel event log collection method based on eBPF, which has the following advantages compared with the existing technology:

[0029] (1) The adaptive secondary buffer pool adjustment strategy can effectively reduce the event loss rate and improve the integrity of collected data compared with similar eBPF data collection tools.

[0030] (2) The output formats are flexible and diverse, and the traceability diagram format output can directly support subsequent security analysis.

[0031] (3) Design a configuration file module to support users to customize the collection event type and improve the pertinence of the collected event data.

[0032] (4) Streamline the original data encoding, optimize the size of collected data, and save data storage space.

[0033] A second object of the present invention is to provide a Linux kernel event log collection system based on eBPF, comprising:

[0034] The command parsing module is used to parse the command line input by the user to obtain the event type, output format and output location;

[0035] The initialization module is used to initialize the collector and specifically performs the following operations: determining the mount point according to the event type and the event selected by the user to be tracked in the pre-loaded mount point configuration file, compiling, verifying and converting the eBPF program to obtain a machine instruction set, and mounting the machine instruction set as a hook function to the determined mount point; loading the filter configuration file and the event structure configuration file, configuring the data filtering rules of the collector according to the filter configuration file, and configuring the structure of the event information according to the event structure configuration file; initializing the ring buffer, which is located in the kernel; initializing the secondary buffer, which is located in the kernel; creating a processing thread, which is configured to monitor data changes in the ring buffer and convert and output them;

[0036] The acquisition module is used to collect data based on the collector, and specifically performs the following operations: when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules, the filtered data is structurally converted according to the structure of the configured event information, and after the structural conversion, the encoded and compressed data is stored in the secondary buffer. The secondary buffer is dynamically resized according to the amount of encoded and compressed data. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer;

[0037] The output module is used to output data based on the collector, and specifically performs the following operations: the processing thread obtains the new data in the ring buffer, converts the new data according to the output format, and outputs the converted data according to the output location. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the process and system framework of a Linux kernel event log collection method based on eBPF of the present invention;

[0039] Figure 2 Flowchart for initializing the collector of the present invention;

[0040] Figure 3 This is a data processing and output flow chart of the present invention;

[0041] Figure 4 This is a flow chart of the dynamic adjustment of the secondary buffer of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0044] To overcome the shortcomings of the existing technology, this paper proposes a Linux kernel event log collection method based on eBPF. This method helps improve the collection and preprocessing capabilities of kernel logs, enabling more flexible collection of desired event types. Compared with previous collection tools, it reduces CPU and memory resource consumption during operation. Furthermore, this method facilitates processing of massive amounts of kernel log data, ensuring the availability and integrity of the acquired data.

[0045] Reference Figure 1 This embodiment provides a Linux kernel event log collection method based on eBPF, including the following steps:

[0046] Step 1: Parse the command line entered by the user to obtain the event type, output format, and output location. This stage is responsible for parsing the command line entered by the user when using the tool on the terminal interface, so as to accurately understand the user's needs and perform the corresponding operation.

[0047] Step 1-1: Parsing event types: This step analyzes the commands entered by the user to determine the specific operations or event types the user wants to perform, such as viewing help documents, querying system versions, or various event types that the user wants to track.

[0048] Step 1-2: Parse the output format: In this step, the user's requirements for the output results will be checked, including the output format requirements, such as text, JSON, traceability graph format, etc., so that they can be processed and displayed according to the user's wishes.

[0049] Steps 1-3: Resolving the output location: This step identifies where the user expects the output to be, whether on the screen, in a specific file, or transmitted to another device over the network. This step ensures that the system can accurately present the results to the user or deliver them to the specified location.

[0050] More specifically, the command line command is parsed: the command line format of the execution system is: . / kellect -e parameter output mode output format, the user inputs the command line according to this format and performs corresponding parsing.

[0051] Use -h to view help content, command parameter options, and output methods. Entering "process" in the execution parameter collects process events, "file" collects file events, "network" collects network events, and "user" collects user management events. Entering "all" collects all types of events. For output methods, use "-o filename" to output to a file, and "-f output format" to select the output format.

[0052] Step 2: Collector initialization phase. Figure 2 As shown, this stage selects the required mount points based on the configuration file and command line input. The corresponding hook functions are then mounted to these mount points, enabling the successful acquisition of Linux native log information. Based on the event type parsed in step 1 and the events selected for tracking in the mount point configuration file, the corresponding tracepoints are hooked to meet user customization requirements.

[0053] Step 2-1: The written eBPF program is compiled into eBPF pseudocode, i.e., eBPF bytecode. The compiled eBPF pseudocode is loaded by the high-level language library program corresponding to eBPF, and the high-level language performs system call processing. After falling into the kernel through the system call, the kernel eBPF program first verifies (verify) to ensure that the program itself is correct (will not crash, will not have an infinite loop) and there are no permission exceptions; the verified eBPF pseudocode program is then converted into a specific machine instruction set to optimize program execution, and finally the machine instruction set is mounted as a hook function to the corresponding mount point.

[0054] The mount point is determined based on the event type parsed in step 1 and the event type selected for tracking in the preloaded mount point configuration file. Throughout the compilation, verification, and mounting process, the kernel's eBPF virtual machine provides a secure and isolated environment, ensuring that eBPF programs executed in the kernel do not pose potential risks or threats to the system.

[0055] During step 2-1, both Clang and LLVM play a crucial role. First, the Clang compiler compiles the C language into eBPF bytecode. Clang's optimization capabilities enable the generated eBPF bytecode to run more efficiently during the subsequent execution phase. Next, the LLVM (Low Level Virtual Machine) processes the eBPF bytecode and converts it into the final machine instruction set for final kernel loading.

[0056] Step 2-2: Load the filter configuration file and event structure configuration file: Loading the filter configuration file configures the collector's collection, ignoring data generated by certain processes, such as various events generated by the system itself, and user-defined process data. Furthermore, the event information structure is initialized and configured according to the event structure configuration file. This embodiment uses a custom data structure, Event, to standardize the data format, defining event information as a data structure consisting of BaseEvent and Arguments.

[0057] Step 2-3: Initialize the ringbuffer. A ring buffer is a data structure provided by BPF (Berkeley PacketFilter). It is a multi-producer, single-consumer queue that can be safely shared across multiple CPUs. This step initializes the ring buffer, which includes determining the buffer size, allocating memory space, and setting read and write pointers. This process provides an efficient data transmission channel for subsequent data collection and processing, ensuring timely and smooth data transmission and recording.

[0058] Step 2-4: Initialize the secondary buffer. In addition to the ring buffer set in step 2-3, this embodiment also initializes the secondary buffer in the kernel environment. The secondary buffer is used to cache and process data when needed and can be sized to accommodate different data flows and processing requirements. By initializing the secondary buffer, data can be temporarily stored in the secondary buffer and synchronized to the ring buffer when the ring buffer is able to process the data, ensuring data integrity and providing higher efficiency and flexibility for subsequent processing and transmission.

[0059] Because the ringbuffer, used for data exchange between the kernel and user-mode environments, is fixed in size at program startup, high data traffic can cause the ringbuffer to overflow, making it impossible to store kernel log data before it is consumed by consumers. This compromises the integrity and availability of kernel log data. Therefore, when high data traffic prevents consumers from consuming data in a timely manner, kernel log data is temporarily stored in a resizable secondary buffer using a first-in, first-out mechanism to prevent data loss. Unlike the ringbuffer, the secondary buffer can adjust itself based on the data traffic within a given event.

[0060] Step 2-5: Initialize the processing thread. This step creates a processing thread with a default number of 5 threads. The thread is used to periodically monitor data changes in the ringbuffer (the default monitoring period is 1 millisecond). The newly added data in the ringbuffer is parsed and processed according to the configured data filtering rules and the structure of the event information to ensure that the data can be correctly recorded, processed, and transmitted.

[0061] Step 3: Data collection by the collector: This is done on the kernel side, parsing the captured event information and converting the raw binary event stream into semantically clear attribute values. Some event attributes are encoded and compressed, and the secondary buffer size is dynamically adjusted based on data flow.

[0062] In this step, when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules. The filtered data is structurally converted according to the structure of the configured event information, and encoded and compressed after the structural conversion. The encoded and compressed data is stored in the secondary buffer. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer.

[0063] Step 3-1: Capture the raw data stream generated by the hook function when the kernel triggers the mount point. Parse the data using the various tracepoint format files in the Linux / sys / kernel / tracing / events folder. These format files define the structure and field information that the hook function can obtain when the corresponding tracepoint event is triggered, helping the system accurately parse and extract key data. Successfully parsed data is then treated as a valid event data stream.

[0064] The following describes the tracepoint format and fields: In Linux, tracepoint format information is stored in the / sys / kernel / tracing / events folder. Each tracepoint has a corresponding folder containing its format file. Viewing the corresponding format file reveals the structure and fields of the tracepoint.

[0065] Reading data obtained by the hook function: Each tracepoint is mounted using eBPF technology, and the parameter data obtained by the hook function when the kernel executes the triggered mount point is read from the kernel. The data read is usually a context information (ctx). ctx is a special object in the eBPF program that provides data related to the current execution environment. ctx can also provide information about the current CPU context, such as the current timestamp and CPU ID. This information can be used for performance analysis or monitoring, and also includes the values ​​of various fields defined by the tracepoint.

[0066] Parse data according to the format file: (1) Use the structure and field information defined in the format file to parse the read binary data. (2) According to the offset and data type in the format file, extract the value of each field from the read binary data and convert it into a readable form.

[0067] like Figure 3 As shown, the parsed field values ​​can be further processed as needed, such as calculation, statistics, screening, filtering, structural conversion, etc. In this embodiment, in order to extract valid data, the parsed data needs to be filtered first, and the rules based on which the filtering is performed are the data filtering rules configured in step 2-2.

[0068] The output of this embodiment is a custom data structure called Event, representing the event information structure configured in step 2-2. This structure consists of two parts: one part, representing attributes common to all events, known as BaseEvent; and the other part, representing the parameters of each event, which are derived from the format file at the corresponding mount point. BaseEvent stores basic information about the event process, such as the process ID, process name, parent process ID, and event type. It also records the start time of the process or thread, the time the process or thread has run in user mode and kernel mode, and the number of involuntary and voluntary context switches performed by the process or thread.

[0069] Taking the collected EVENT_FILE_RENAME event as an example, the structure conversion step will output the content shown in Table 1: including the basic information of BaseEvent and the two arguments oldname and newname unique to the EVENT_FILE_RENAME event.

[0070] Table 1 Custom data structure Event example

[0071]

[0072] Step 3-2: Coding and compression: Since the collected data may be relatively large, this embodiment will perform streamlined coding or compression processing on part of the data after structural conversion to reduce the data volume and retain necessary information, thereby improving storage and transmission efficiency.

[0073] Encode attributes such as event types, error messages, and warning messages that appear frequently in the kernel log data stream. For example, event types are divided into four categories in the system: User, Network, Process, and File. Under these four event categories, there are also several sub-events. For these attributes, you can choose to create an event type encoding table to map each event type to a unique integer code. In the log data, replace the original event type with the corresponding code. This can greatly reduce the space occupied by the text representation of the event type. For example, the encoding of network events is shown in Table 2.

[0074] Table 2 Network event coding examples

[0075] Event Type coding EVENT_NETWORK_CONNECT 1 EVENT_NETWORK_DISCONNECT 2 EVENT_NETWORK_SENDTO 3 EVENT_NETWORK_RECVFROM 4 EVENT_NETWORK_SENDMSG 5 EVENT_NETWORK_RECVMSG 6 EVENT_NETWORK_ACCEPT 7 EVENT_NETWORK_ACCEPT4 8

[0076] Similarly, for error messages and warning messages that are long strings, a message encoding table can be used to map them to a unique integer code to reduce the space occupied by the text. For example, the encoding of some common error and warning messages is shown in Table 3. The encoded and compressed data is stored in the secondary buffer. When the free length of the ring buffer is no less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer.

[0077] Table 3 Error message / warning message coding examples

[0078] Error messages / warning messages coding Permission Denied 1 No such file or directory 2 No such process 3 Interrupted system call 4 Input / output error 5 Argument list too long 6 Connection Refused 7 Low Disk Space 8 Resource Exhausted 9 Untrusted Certificate 10

[0079] By encoding frequent attributes such as event types and various error and warning messages, the amount of data can be reduced without losing key information. When parsing the data, the corresponding encoding table needs to be used to convert the encoded values ​​back to the original representation to obtain more readable data.

[0080] Step 3-3: Dynamically adjust the secondary buffer. This step dynamically adjusts the size of the secondary buffer based on real-time data traffic to adapt to different traffic conditions, optimize data processing performance, ensure the timeliness and stability of data transmission, and effectively avoid resource waste and data loss.

[0081] Set the initial size of the secondary buffer: When the program starts, set the initial size of the secondary buffer to 512MB.

[0082] like Figure 4 As shown, monitor data traffic and perform regular statistics: the program needs to monitor the amount of compressed and encoded data in real time, and count the total amount of data in the past time every 10S (the time length is adjustable), and calculate the average data traffic.

[0083] Dynamically adjust the buffer size: (1) When the average data flow rate during the statistical period exceeds the upper threshold of 256MB (the upper threshold is adjustable), the secondary buffer expansion operation is triggered. The secondary buffer size is expanded by 2 times the original size (the expansion multiple is adjustable), and the upper and lower thresholds are reset to half and one-quarter of the new buffer size; (2) When the average data flow rate continues to be lower than the lower threshold, the secondary buffer reduction operation is triggered. Similar to the expansion operation, the upper and lower limits of the secondary buffer size threshold need to be changed.

[0084] Continuous monitoring and feedback: After resizing the secondary buffer, continue to monitor the secondary buffer size and compressed data flow, and record historical data for analysis and optimization. Provide timely feedback on the results of secondary buffer adjustments through logs or alarm mechanisms for further optimization and adjustment.

[0085] Step 4: Output phase. The log information collected in step 3 is parsed according to the output requirements obtained after command parsing in step 1. This step standardizes the output of log information based on the requirements obtained in the command parsing phase.

[0086] There are two common output locations to choose from: (1) Output to Terminal: Log information can be output directly to the terminal for real-time viewing and monitoring; (2) Output to File: Log information can also be output to a file. This method can conveniently save log information for later analysis.

[0087] In terms of output format selection, users can choose from the following common formats according to their needs: (1) JSON format: The output log information is stored in JSON format. The JSON format is well structured and easy to read, and is suitable for data exchange and subsequent processing; (2) Text format: The output log information is stored in plain text. This format is simple and clear, easy to read, and suitable for quickly finding and searching key information. (3) Origin graph format: After certain processing, the output of this format can display the information in a graphical manner. This format is suitable for more complex system structures and dependencies, helping users to understand and analyze log information more intuitively. The event data format of the origin graph consists of three parts<Subject,Object,Arguments> The Subject is the process that executes the event, including the process ID, timestamp, process name, etc. The Object part is the object being operated in the event, which can be a file or a process. The Arguments part is the same as the Arguments of the Event structure, indicating the parameters of the process's operation on the object.

[0088] Step 5: Finalization phase. Release the ringbuffer and secondary buffer as well as the eBPFMap and file descriptors related to the eBPF program.

[0089] Release the ringbuffer: (1) Stop writing data to the ringbuffer and ensure that all data has been processed. (2) Release or destroy the ringbuffer to free up the occupied memory space.

[0090] Release the secondary buffer: When processing kernel log data, a secondary buffer may be used to store and process the data. At the end of the process, the secondary buffer needs to be released or destroyed to free up the occupied memory space.

[0091] Release the eBPF Map associated with the eBPF program: In the eBPF program, the eBPF Map is used to store and share data. At the end of the program, the eBPF Map associated with the eBPF program needs to be released or destroyed to release the occupied resources.

[0092] Closing file descriptors: During kernel log collection, some file descriptors may be opened. At the end of the process, these file descriptors need to be closed to free up system resources and ensure that the files are closed properly.

[0093] By following these steps, you can effectively release resources associated with the eBPF program, ensuring system stability and performance. Before releasing resources, ensure that all data processing operations have completed to avoid data loss or incompleteness.

[0094] In another embodiment, the present invention further provides a Linux kernel event log collection system based on eBPF, comprising:

[0095] The command parsing module is used to parse the command line input by the user to obtain the event type, output format and output location;

[0096] The initialization module is used to initialize the collector and specifically performs the following operations: determining the mount point according to the event type and the event selected by the user to be tracked in the pre-loaded mount point configuration file, compiling, verifying and converting the eBPF program to obtain a machine instruction set, and mounting the machine instruction set as a hook function to the determined mount point; loading the filter configuration file and the event structure configuration file, configuring the data filtering rules of the collector according to the filter configuration file, and configuring the structure of the event information according to the event structure configuration file; initializing the ring buffer, which is located in the kernel; initializing the secondary buffer, which is located in the kernel; creating a processing thread, which is configured to monitor data changes in the ring buffer and convert and output them;

[0097] The acquisition module is used to collect data based on the collector, and specifically performs the following operations: when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules, the filtered data is structurally converted according to the structure of the configured event information, and after the structural conversion, the encoded and compressed data is stored in the secondary buffer. The secondary buffer is dynamically resized according to the amount of encoded and compressed data. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer;

[0098] The output module is used to output data based on the collector, and specifically performs the following operations: the processing thread obtains the new data in the ring buffer, converts the new data according to the output format, and outputs the converted data according to the output location.

[0099] The specific limitations of an eBPF-based Linux kernel event log collection system can be found in the above-mentioned limitations of an eBPF-based Linux kernel event log collection method, which will not be repeated here.

[0100] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A Linux kernel event log collection method based on eBPF, characterized in that: The eBPF-based Linux kernel event log collection method includes: Step 1: Parse the command line input by the user to obtain the event type, output format and output location; Step 2: Initialize the collector, including: Step 2-1: Determine the mount point based on the event type and the event selected for tracking by the user in the preloaded mount point configuration file, compile, verify, and convert the eBPF program to obtain a machine instruction set, and mount the machine instruction set as a hook function to the determined mount point; Step 2-2, loading a filtering configuration file and an event structure configuration file, configuring data filtering rules for the collector according to the filtering configuration file, and configuring the structure of event information according to the event structure configuration file; Step 2-3, initialize the ring buffer, the ring buffer is located in the kernel; Step 2-4: Initialize the secondary buffer, which is located in the kernel; Step 2-5: Create a processing thread, where the processing thread is configured to monitor data changes in the ring buffer and perform conversion and output; Step 3. The collector performs data collection: when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules. The filtered data is structurally converted according to the structure of the configured event information, and after the structural conversion, the encoded and compressed data is stored in the secondary buffer. The secondary buffer is dynamically resized according to the amount of encoded and compressed data. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer. Step 4: The collector outputs data: the processing thread obtains the newly added data in the ring buffer, converts the newly added data according to the output format, and outputs the converted data according to the output location.

2. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: In step 3, parsing the data captured by the hook function includes: Get the format file corresponding to each tracepoint, use the structure and field information defined in the format file to parse the parameter data obtained by the hook function when the kernel runs the triggered mount point, extract the value of each field from the parameter data obtained by the hook function according to the offset and data type in the format file, and convert the value of each field into a readable form.

3. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: The structure of the event information consists of two parts. One part is the attributes common to all events, namely BaseEvent; the other part is the parameters of each event itself, which come from the format file of the corresponding mount point.

4. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: In step 3, the coding compression includes: For the data stream after structural conversion, the corresponding event type is mapped to a unique integer code according to the preset event type coding table; the corresponding error message or warning message is mapped to a unique integer code according to the preset information coding table.

5. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: In step 3, the secondary buffer is dynamically resized according to the amount of data after encoding and compression, including: At each preset time interval, the amount of encoded and compressed data within the preset time is counted, and the average data flow within the current preset time is calculated; If the average data flow exceeds the upper threshold, the expansion operation of the secondary buffer is triggered, and the size of the secondary buffer is expanded to the original first preset multiple to obtain a new secondary buffer, and the upper and lower thresholds are updated based on the new secondary buffer; if the average data flow is lower than the lower threshold, the reduction operation of the secondary buffer is triggered, and the size of the secondary buffer is reduced to the original second preset multiple to obtain a new secondary buffer, and the upper and lower thresholds are updated based on the new secondary buffer.

6. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: The output formats include JSON format, text format and traceability graph format, and the output locations include output in a terminal and output in a file.

7. The method for collecting Linux kernel event logs based on eBPF according to claim 1, wherein: The eBPF-based Linux kernel event log collection method also includes an end phase, in which the ring buffer, the secondary buffer, and the eBPF Map and file descriptors related to the eBPF program are released.

8. A Linux kernel event log collection system based on eBPF, characterized in that: The eBPF-based Linux kernel event log collection system includes: The command parsing module is used to parse the command line input by the user to obtain the event type, output format and output location; The initialization module is used to initialize the collector and specifically performs the following operations: determining the mount point according to the event type and the event selected by the user to be tracked in the pre-loaded mount point configuration file, compiling, verifying and converting the eBPF program to obtain a machine instruction set, and mounting the machine instruction set as a hook function to the determined mount point; loading the filter configuration file and the event structure configuration file, configuring the data filtering rules of the collector according to the filter configuration file, and configuring the structure of the event information according to the event structure configuration file; initializing the ring buffer, which is located in the kernel; initializing the secondary buffer, which is located in the kernel; creating a processing thread, which is configured to monitor data changes in the ring buffer and convert and output them; The acquisition module is used to collect data based on the collector, and specifically performs the following operations: when the mount point is triggered, the data captured by the hook function is parsed, and the parsed data is filtered according to the configured data filtering rules, the filtered data is structurally converted according to the structure of the configured event information, and after the structural conversion, the encoded and compressed data is stored in the secondary buffer. The secondary buffer is dynamically resized according to the amount of encoded and compressed data. When the free length of the ring buffer is not less than the amount of encoded and compressed data, the encoded and compressed data in the secondary buffer is synchronized to the ring buffer; The output module is used to output data based on the collector, and specifically performs the following operations: the processing thread obtains the new data in the ring buffer, converts the new data according to the output format, and outputs the converted data according to the output position.

Images