Service device and database event processing method
By using the batch commit mechanism of eBPF programs and array buffers in the database monitoring system, the performance problem caused by frequent kernel mode and user mode switching was solved, and performance improvement was achieved in high-concurrency scenarios.
Patent Information
- Application Number
- CN202610302092.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-11
- Publication Date
- 2026-07-14
Smart Images
Figure CN122387703A_ABST
Abstract
Description
Technical Field
[0001] This application relates to database technology. More specifically, it relates to a service device and a database event processing method. Background Technology
[0002] In database monitoring systems, every database event needs to be immediately submitted to the user space via system calls, resulting in frequent switching between kernel mode and user mode, causing huge performance overhead and seriously affecting the operating performance of the database itself. Summary of the Invention
[0003] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this application provide a service device and a database event processing method.
[0004] In a first aspect, embodiments of this application provide a service device, including: a controller configured to: enter kernel mode via a system call during the execution of SQL statements in a database process, so as to execute kernel functions through a target CPU core; trigger the corresponding eBPF mount point during the execution of the kernel function, so as to run an eBPF program on the target CPU core; capture target database events by executing the eBPF program; write target event data corresponding to the target database events into a target array buffer corresponding to the target CPU core; and, if the event attribute information corresponding to the target array buffer meets the target conditions, submit multiple event data in the target array buffer in batches to a user-mode program.
[0005] In this embodiment, when a target database event is captured by executing an eBPF program, the target event data corresponding to the target database event is written to the target array buffer corresponding to the target CPU core. If the event attribute information corresponding to the target array buffer meets the target conditions, multiple event data items in the target array buffer are submitted in batches to the user-mode program. Thus, by caching the event data of the database event captured by the eBPF program in the array buffer corresponding to the CPU core, and submitting multiple cached event data items in the array buffer in batches to the user-mode program when the event attribute information corresponding to the array buffer meets the target conditions, it is unnecessary to copy the event data from kernel mode to user mode in real time via system calls when the database event is captured by the eBPF program. This reduces the overhead of memory copying, context switching, and cache invalidation involved in cross-mode data transfer. Especially in high-concurrency scenarios, it greatly reduces the performance overhead caused by frequent kernel mode and user mode switching, improving the operating performance of the database itself.
[0006] In some embodiments of this application, the controller is further configured to: obtain a first timestamp corresponding to the target database event before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core; and determine the target array buffer from multiple array buffers of the target CPU core based on the first timestamp.
[0007] In this embodiment, BPF_PERCPU_ARRAY is used to allocate an independent event cache space for each CPU core. By improving the structure of the PERCPU array, a PERCPU array caching mechanism is implemented. By performing a modulo operation on the timestamps corresponding to database events, the index of the corresponding array buffer is determined, thereby implementing a round-robin mechanism for multiple buffers and avoiding single-point contention.
[0008] In some embodiments of this application, the event attribute information includes the number of event data stored in the target array buffer, and the target condition includes the number of event data stored in the target array buffer being a target number; the controller is specifically configured to: when the number of event data corresponding to the target array buffer is the target number, submit the multiple event data in the target array buffer to the user-mode program in batches.
[0009] In this embodiment of the application, a batch submission mechanism based on cache capacity is used to maximize the batch processing effect and reduce the number of system calls.
[0010] In some embodiments of this application, the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, and the target condition includes the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event being greater than or equal to a duration threshold; the controller is specifically configured to: when the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to the duration threshold, submit the multiple event data in the target array buffer to the user-mode program in batches.
[0011] In this embodiment of the application, by setting a maximum storage time threshold (duration threshold) for the array buffer, when the storage time of the first event data stored in the array buffer exceeds the duration threshold, a batch submission is forced, which can ensure the real-time requirements and avoid events from being delayed for a long time.
[0012] In some embodiments of this application, the controller is further configured to: write the timestamp corresponding to the target database event into the target array buffer when the target array buffer is empty before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core.
[0013] In this embodiment, only the timestamp corresponding to the first event data stored in the target array buffer needs to be recorded. There is no need to record the timestamps corresponding to the event data that is not the first to be stored in the target array buffer. In this way, storage space can be saved, more event data can be stored, the batch processing effect can be maximized, and the number of system calls can be reduced.
[0014] In some embodiments of this application, the event attribute information further includes the number of event data stored in the target array buffer, and the target condition further includes the number of event data stored in the target array buffer being greater than or equal to a target number; the controller is further configured to: when the target array buffer satisfies at least one of the following conditions: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration of the timestamp corresponding to the first event data stored in the target array buffer being greater than or equal to the duration threshold of the timestamp corresponding to the target database event is greater than or equal to the duration threshold, submit the plurality of event data in the target array buffer to the user-mode program in batches.
[0015] In this embodiment, the event attribute information includes the number of event data stored in the target array buffer, and whether the number of event data stored in the target array buffer is greater than or equal to a target number. Thus, if at least one of the following conditions is met: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to a duration threshold, the multiple event data in the target array buffer can be batch submitted to the user-space program. This achieves the goal of maximizing batch processing efficiency, reducing the number of system calls, ensuring real-time requirements, and preventing events from remaining in the buffer for extended periods.
[0016] In some embodiments of this application, the controller is further configured such that: the target CPU core corresponds to multiple array buffers, and the multiple array buffers correspond to different output channels; specifically, the controller is configured to: submit the multiple event data in the target array buffer to the user-mode program in batches through the output channel corresponding to the target array buffer.
[0017] In this embodiment of the application, the target CPU core corresponds to multiple array buffers, and each array buffer corresponds to a different output channel. By submitting the multiple event data in the target array buffer to the user-mode program in batches through the output channel corresponding to the target array buffer, load distribution can be achieved.
[0018] Secondly, embodiments of this application provide a database event processing method applied to a service device, comprising: during the execution of SQL statements by a database process, entering kernel mode via a system call to execute kernel functions through a target CPU core; during the execution of the kernel functions, triggering the corresponding eBPF mount point to run an eBPF program on the target CPU core; capturing target database events by executing the eBPF program; writing target event data corresponding to the target database events into a target array buffer corresponding to the target CPU core; and, if the event attribute information corresponding to the target array buffer meets the target conditions, batch submitting multiple event data in the target array buffer to a user-mode program.
[0019] In some embodiments of this application, before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core, the method further includes: obtaining a first timestamp corresponding to the target database event; and determining the target array buffer from multiple array buffers of the target CPU core based on the first timestamp.
[0020] In some embodiments of this application, the event attribute information includes at least one of the following: the number of event data stored in the target array buffer; the timestamp corresponding to the first event data stored in the target array buffer; when the event attribute information includes the number of event data stored in the target array buffer, the target condition includes that the number of event data stored in the target array buffer is greater than or equal to a target number; when the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, the target condition includes that the duration between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to a duration threshold.
[0021] Thirdly, embodiments of this application provide a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the database event handling method as shown in the second aspect.
[0022] Fourthly, embodiments of this application provide a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the database event processing method as shown in the second aspect. Attached Figure Description
[0023] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0024] Figure 1 A possible structural architecture diagram of a service device according to some embodiments is shown; Figure 2 The overall architecture of a high-performance event processing system based on an eBPF PERCPU array, according to some embodiments, is shown; Figure 3 A schematic diagram illustrating multi-core isolation implemented using a PERCPU array according to some embodiments is shown; Figure 4 One of the flowcharts of a database event handling method according to some embodiments is shown; Figure 5 A second schematic flowchart of a database event handling method according to some embodiments is shown; Figure 6 A third schematic flowchart of a database event handling method according to some embodiments is shown; Figure 7 A fourth schematic flowchart of a database event handling method according to some embodiments is shown; Figure 8 Fifth of a series of schematic diagrams illustrating a database event handling method according to some embodiments; Figure 9 A sixth schematic flowchart of a database event handling method according to some embodiments is shown; Figure 10 The seventh of several schematic diagrams illustrates a database event handling method according to some embodiments. Detailed Implementation
[0025] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.
[0026] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.
[0027] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.
[0028] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.
[0029] In some embodiments of this application, the service device may be a server device, a server cluster, or a data platform composed of multiple server devices or multiple server clusters (such as a high-concurrency database monitoring system, a real-time performance analysis system, a distributed system monitoring platform, or a cloud-native application monitoring system). It may also be other feasible devices, which can be determined according to the actual situation and are not limited here.
[0030] This application relates to the field of computer system performance optimization technology, specifically to a database monitoring system based on eBPF (Extended Berkeley Packet Filter) technology, particularly in the field of database event collection. This technology is mainly applied to high-concurrency database monitoring systems, real-time performance analysis systems, distributed system monitoring platforms, and cloud-native application monitoring scenarios.
[0031] In database monitoring systems, traditional event collection methods have the following core problems: every time a database event occurs, it needs to be immediately submitted to the user space via a system call, resulting in frequent switching between kernel mode and user mode, causing huge performance overhead and seriously affecting the operating performance of the database itself.
[0032] With the rapid development of cloud computing and big data technologies, database monitoring systems have become an important component of modern IT infrastructure. Currently, the industry mainly adopts the following technical solutions: Direct monitoring based on system calls: This involves intercepting system calls directly using tools like ptrace and strace, and reporting each event immediately. This is the most traditional method, which works in low-load scenarios, but incurs significant performance overhead in high-concurrency scenarios.
[0033] Kernel module-based monitoring: Event collection is achieved by writing kernel modules (LKM). Although the performance is better than user-space tools, there are stability risks and recompilation is required for different kernel versions.
[0034] eBPF-based monitoring technology: Since its introduction into the Linux kernel in 2014, eBPF technology has become a mainstream system monitoring technology. The database process runs in user space, while the eBPF program runs in kernel space. The user-space database must rely on the kernel space to complete resource operations; this is the basis for eBPF's ability to collect database events. Database function calls or system calls trigger state switching, and the kernel automatically executes the mounted eBPF program, collecting events and sending them back to user space for analysis via shared storage. eBPF leverages its "kernel-level central control" characteristic to achieve non-intrusive, high-performance event collection from the user-space database, without interfering with the database's original operating logic. Its advantages lie in security and portability, but existing eBPF monitoring solutions still have performance bottlenecks. Existing eBPF monitoring systems adopt an architecture of "full kernel-side collection + centralized user-space processing." When the frequency of monitoring events is high, a large number of probe triggers cause the eBPF program to be executed frequently, resulting in significant CPU overhead. Meanwhile, the collected data needs to be copied from kernel mode to user mode. This cross-mode data transfer involves overhead such as memory copying, context switching, and cache invalidation, which becomes a bottleneck for system performance.
[0035] Performance bottleneck caused by frequent system calls: Every database event (such as SQL execution, connection establishment, etc.) needs to be submitted to the user space immediately through perf_event or ring buffer. There is a lack of effective event caching mechanism and events cannot be processed in batches. In high-concurrency scenarios (such as tens of thousands of SQL executions per second), system call overhead can account for 30-50% of CPU utilization.
[0036] The relevant technologies have the following limitations: eBPF technology is used for application performance monitoring, but it mainly focuses on the network layer and uses the traditional direct reporting method for database event monitoring, which has a large performance overhead in high-concurrency scenarios; using the proxy mode for database monitoring requires the deployment of a proxy at the application layer, which has deployment complexity and performance overhead issues; and deep packet inspection (DPI) technology is used, but it mainly focuses on network traffic analysis and has limited monitoring capabilities for internal database events.
[0037] Related technologies describe database monitoring through kernel modules, but these suffer from stability and multi-core contention issues. They also provide a rich set of eBPF tools, but these tools use direct event reporting and lack batch caching mechanisms. Furthermore, eBPF-based security monitoring tools primarily focus on security events, with less emphasis on performance optimization.
[0038] The eBPF program runs in the Linux kernel space and is subject to the execution environment of the eBPF verifier.
[0039] eBPF is a "restricted virtual machine" running in the Linux kernel, and its restrictions stem from the verifier and security model. eBPF development has the following six main categories of restrictions: Restriction 1: Stack space limitation (512 bytes). Each eBPF program has a maximum stack space of 512 bytes, prohibiting recursion and the creation of large arrays. This results in: event queues cannot be cached on the stack, and large batch structures cannot be constructed temporarily. Restriction 2: Dynamic memory allocation is prohibited. eBPF programs are not allowed to use malloc, kmalloc, or free. This means: dynamic cache expansion is not allowed, linked lists cannot be built, and the STL / queue model cannot be used. Restriction 3: Threads and blocking are not allowed. eBPF has no thread model; sleep, schedule, spin waiting are not allowed, and mutexes cannot be used. This means: background flush threads, timer threads, and producer-consumer models are not allowed. Restriction 4: Loops must be "statically provable." The verifier requires loops to have a fixed upper bound, which must be deduceable at compile time; infinite loops are not allowed. This means: You cannot use `while(true)`, you cannot traverse dynamically long linked lists, and you cannot scan queues of indeterminate size. Fifth, execution time constraints: eBPF programs must return quickly and cannot run for long periods; otherwise, they will be refused loading or terminated. This means you cannot maintain complex data structures, perform sorting, or perform large-scale scans. Sixth, concurrency model constraints: eBPF runs on the CPU that triggered it, and cross-CPU locking is not allowed; cross-CPU synchronization is extremely costly. This means traditional globally shared queues are not feasible, and global caches will cause contention. Due to these limitations, existing eBPF monitoring programs all adopt a real-time data collection and submission mechanism and lack effective batch processing capabilities.
[0040] A typical use of the eBPF PERCPU array is as a counter. Essentially, it's an array, limited to 512KB in size. Unlike user-space arrays, this array cannot be traversed in the kernel, and the number of inserted data cannot be directly obtained. Such an array is unsuitable as a caching component. In this application, the array is fixed as a single-element array, a key-value structure. The key is a fixed number, such as 0. The value contains a data structure capable of caching database event data, and also capable of inserting event attribute information (such as the number of data entries and / or the timestamp of the first cached event). This allows for determining whether batch submission is possible based on time and / or the number of entries.
[0041] This application provides a service device, including: a controller configured to: enter kernel mode via a system call during the execution of SQL statements in a database process, so as to execute kernel functions through a target CPU core; trigger the corresponding eBPF mount point during the execution of the kernel function, so as to run an eBPF program on the target CPU core; capture target database events by executing the eBPF program; write the target event data corresponding to the target database events into a target array buffer corresponding to the target CPU core; and, if the event attribute information corresponding to the target array buffer meets the target conditions, submit multiple event data in the target array buffer to the user-mode program in batches.
[0042] One of the service devices may include one or more CPU cores, and the target CPU core is one of the one or more CPU cores.
[0043] Here, the eBPF mount point is the eBPF hook. By triggering the eBPF mount point, an eBPF program is triggered, and by executing the eBPF program, events from the target database are captured.
[0044] The target database events can be SQL execution events, including: SQL execution start, SQL execution end, SQL execution time, SQL type (SELECT / INSERT / UPDATE / DELETE), slow query events, transaction start / commit / rollback events, etc., which are not limited here.
[0045] The target event data is used to indicate target database events. The target event data can be driven by business metrics, but this is not limited here.
[0046] For example, it can include SQL, client IP, username, etc.
[0047] One CPU core can correspond to one or more array buffers, and the target array buffer can be one of the one or more array buffers.
[0048] The event attribute information and target conditions can be determined according to the actual situation, and are not limited here.
[0049] Among them, multiple event data can be all event data in the current target array buffer, or it can be a part of the event data in the current target array buffer; there is no limitation here.
[0050] The user-mode program can be any program related to a user-mode control; there are no restrictions here. For example, a user-mode monitoring program.
[0051] In this embodiment, when a target database event is captured by executing an eBPF program, the target event data corresponding to the target database event is written to the target array buffer corresponding to the target CPU core. If the event attribute information corresponding to the target array buffer meets the target conditions, multiple event data items in the target array buffer are submitted in batches to the user-mode program. Thus, by caching the event data of the database event captured by the eBPF program in the array buffer corresponding to the CPU core, and submitting multiple cached event data items in the array buffer in batches to the user-mode program when the event attribute information corresponding to the array buffer meets the target conditions, it is unnecessary to copy the event data from kernel mode to user mode in real time via system calls when the database event is captured by the eBPF program. This reduces the overhead of memory copying, context switching, and cache invalidation involved in cross-mode data transfer. Especially in high-concurrency scenarios, it greatly reduces the performance overhead caused by frequent kernel mode and user mode switching, improving the operating performance of the database itself.
[0052] In some embodiments of this application, the controller is further configured to: obtain a first timestamp corresponding to the target database event before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core; and determine the target array buffer from multiple array buffers of the target CPU core based on the first timestamp.
[0053] Specifically, BPF_PERCPU_ARRAY is used to allocate an independent event cache space for each CPU core, which means that multiple array buffers are allocated for each CPU core.
[0054] For example, if the target CPU core includes a pair of array buffers, and different array buffers correspond to different indices, the target index can be calculated by performing a modulo operation on the first timestamp. The target index is used to indicate the target array buffer. Based on the target index, the target event data is written to the target array buffer.
[0055] In this embodiment, BPF_PERCPU_ARRAY is used to allocate an independent event cache space for each CPU core. By improving the structure of the PERCPU array, a PERCPU array caching mechanism is implemented. The index of the corresponding array buffer is determined by performing a modulo operation on the timestamp corresponding to the database event (divided by the total number of array buffers), thereby implementing a round-robin mechanism for multiple buffers and avoiding single-point contention.
[0056] In some embodiments of this application, the event attribute information includes the number of event data stored in the target array buffer, and the target condition includes the number of event data stored in the target array buffer being a target number; the controller is specifically configured to: when the number of event data corresponding to the target array buffer is the target number, submit the multiple event data in the target array buffer to the user-mode program in batches.
[0057] The target number can be determined based on the actual situation and is not limited here. For example, the target number is 32.
[0058] For example, an event counter is designed in the target array buffer to record the count of event data (i.e., the number of event data). Writing an event data item to the target data buffer increments the event counter by 1. After submitting multiple event data items in a batch to the user-space program, the event counter is reset.
[0059] In this embodiment of the application, by setting a cache capacity threshold (i.e. target number), when the number of cached events reaches the target number, a batch submission is immediately executed. In this way, the batch submission mechanism based on cache capacity maximizes the batch processing effect and reduces the number of system calls.
[0060] In some embodiments of this application, the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, and the target condition includes the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event being greater than or equal to a duration threshold; the controller is specifically configured to: when the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to the duration threshold, submit the multiple event data in the target array buffer to the user-mode program in batches.
[0061] The duration threshold can be set according to the actual situation and is not limited here. For example, the duration threshold is 300 milliseconds.
[0062] The timestamp corresponding to the first event data can be either the timestamp when the eBPF program captures the corresponding database event, or the timestamp when the eBPF program writes the event data of the captured database event into the target array buffer; there is no limitation here.
[0063] The timestamp corresponding to the target database event can be either the timestamp when the eBPF program captures the target database event, or the timestamp when the eBPF program writes the event data of the target database event into the target array buffer; there is no limitation here.
[0064] It is understandable that if the difference between the timestamp corresponding to the target database event and the timestamp corresponding to the first event data stored in the target array buffer is greater than or equal to the duration threshold, the multiple event data in the target array buffer will be submitted to the user-space program in batches.
[0065] In this embodiment of the application, by setting a maximum storage time threshold (duration threshold) for the array buffer, when the storage time of the first event data stored in the array buffer exceeds the duration threshold, a batch submission is forced, which can ensure the real-time requirements and avoid events from being delayed for a long time.
[0066] In some embodiments of this application, the controller is further configured to: write the timestamp corresponding to the target database event into the target array buffer when the target array buffer is empty before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core.
[0067] In this embodiment, only the timestamp corresponding to the first event data stored in the target array buffer needs to be recorded. There is no need to record the timestamps corresponding to the event data that is not the first to be stored in the target array buffer. In this way, storage space can be saved, more event data can be stored, the batch processing effect can be maximized, and the number of system calls can be reduced.
[0068] In some embodiments of this application, the event attribute information further includes the number of event data stored in the target array buffer, and the target condition further includes the number of event data stored in the target array buffer being greater than or equal to a target number; the controller is further configured to: when the target array buffer satisfies at least one of the following conditions: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration of the timestamp corresponding to the first event data stored in the target array buffer being greater than or equal to the duration threshold of the timestamp corresponding to the target database event is greater than or equal to the duration threshold, submit the plurality of event data in the target array buffer to the user-mode program in batches.
[0069] In this embodiment, the event attribute information includes the number of event data stored in the target array buffer, and whether the number of event data stored in the target array buffer is greater than or equal to a target number. Thus, if at least one of the following conditions is met: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to a duration threshold, the multiple event data in the target array buffer can be batch submitted to the user-space program. This achieves the goal of maximizing batch processing efficiency, reducing the number of system calls, ensuring real-time requirements, and preventing events from remaining in the buffer for extended periods.
[0070] This application implements a dual control mechanism based on time windows and capacity thresholds, intelligently determining the timing of batch submissions to maximize performance while ensuring real-time performance. It resolves the conflict between real-time performance and performance, adapting to different load scenarios, controlling the average latency to within 2ms, and reducing system calls by more than 80%. The dual control mechanism ensures both real-time performance and maximum performance: capacity priority ensures rapid threshold reaching and immediate submission under high load; time priority ensures real-time applicability through time windows under low load.
[0071] In some embodiments of this application, multiple array buffers may correspond to one output channel or multiple output channels.
[0072] In some embodiments of this application, the controller is further configured such that: the target CPU core corresponds to multiple array buffers, and the multiple array buffers correspond to different output channels; specifically, the controller is configured to: submit the multiple event data in the target array buffer to the user-mode program in batches through the output channel corresponding to the target array buffer.
[0073] For example, the corresponding output channel can be selected based on the index of the array buffer (i.e., different array buffer indices correspond to different output channels), thus achieving load distribution. One array buffer can correspond to one or more output channels.
[0074] Alternatively, a modulo operation can be performed on the timestamp corresponding to the target database event (divided by the total number of output channels) to determine the index of the output channel, and then the output channel can be determined based on the index of the output channel to achieve load distribution.
[0075] In this embodiment of the application, the target CPU core corresponds to multiple array buffers, and each array buffer corresponds to a different output channel. By submitting the multiple event data in the target array buffer to the user-mode program in batches through the output channel corresponding to the target array buffer, load distribution can be achieved.
[0076] In this embodiment, each CPU core has an independent PERCPU cache space, completely eliminating multi-core contention. The multi-buffer rotation mechanism achieves load distribution, and the multi-output channel achieves user space processing load distribution.
[0077] For example, such as Figure 1The diagram shown illustrates a possible architecture of the service device provided in this application embodiment, including a user-mode space and a kernel-mode space. The user-mode space includes a database for event processing, data analysis, and storage. The database includes a database process used to execute SQL statements, trigger kernel functions, and generate events that can be hooked by eBPF. The kernel-mode space includes an eBPF program layer, which comprises a PERCPU array layer and a batch commit layer. A CPU core is a hardware execution unit responsible for actually executing binary instructions; the kernel (operating system kernel) is the system software running on the CPU, responsible for managing, scheduling, and controlling all CPU cores; a single kernel manages all CPU cores in the system, and each CPU core runs under the kernel's scheduling; any program (including database threads and eBPF programs) must be scheduled by the kernel to execute on a CPU core; kernel code, kernel functions, and eBPF programs are ultimately executed by the current CPU core.
[0078] In this embodiment, the CPU core is responsible for executing the database process, the eBPF program, the kernel function, and the perf_event output mechanism. Note: The eBPF program runs on the CPU core that triggered it (bound to the current CPU core), which is the physical basis of PERCPU.
[0079] In this embodiment, the eBPF program layer is used for event acquisition, cache management, and batch submission, specifically including database event capture, buffer selection, fixed slot writing, atomic counting, batch submission trigger judgment, and perf_event output.
[0080] In this embodiment, the PERCPU array layer is essentially a BPF map type: BPF_PERCPU_ARRAY. Physical characteristics: one independent instance per CPU core, lock-free, no cross-CPU contention, and only the current CPU can access its copy. Responsibilities: Provides fixed-slot caching, stores the EXECUTE_REQ_EVENT structure, and stores metadata such as num and timestamp. The PERCPU array layer achieves load balancing through multiple array buffers rotating in a round-robin fashion.
[0081] In this embodiment, the batch submission layer consists of conditional (time window control and / or capacity threshold control) triggering logic in the eBPF program, plus the perf_event output mechanism. The user-space application layer reads batch data through perf_event / ringbuffer and performs parsing and analysis.
[0082] For example, combined Figure 1 ,like Figure 2The diagram shows the overall architecture of a high-performance event processing system based on an eBPF PERCPU array.
[0083] For example, such as Figure 3 This demonstrates how the PERCPU array achieves multi-core isolation, with each CPU core having its own independent cache space. Each CPU core accesses its own independent PERCPU array, eliminating lock contention and fully utilizing CPU cache locality. Performance scales linearly with the number of CPU cores.
[0084] To illustrate this solution in more detail, the following will use examples to illustrate it. Figures 4 to 10 To explain, it is understandable that Figures 4 to 10 The steps involved may include more or fewer steps in actual implementation, and the order of these steps may also differ, as long as the database event processing method provided in the embodiments of this application can be achieved. The executing entity of the database event processing method can be a service device, or a functional module or functional entity within the service device that can implement the database event processing method; no limitation is made here. Furthermore, the specific description of the database event processing method provided in the embodiments of this application can be found in the relevant description of the aforementioned service device, and the same or similar technical effects can be achieved; further details are omitted here.
[0085] Figure 4 The flowchart illustrates the steps of implementing a database event handling method according to one or more embodiments of this application, applied to a service device. The database event handling method may include the following steps S401 to S405.
[0086] S401. During the execution of SQL statements in the database process, a system call is made to enter kernel mode so that kernel functions can be executed through the target CPU core.
[0087] S402. During the execution of kernel functions, the corresponding eBPF mount point is triggered to run eBPF programs on the target CPU core.
[0088] S403. Capture target database events by executing the eBPF program.
[0089] S404. Write the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core.
[0090] S405. If the event attribute information corresponding to the target array buffer meets the target conditions, submit multiple event data in the target array buffer to the user-mode program in batches.
[0091] In some embodiments of this application, combined with Figure 4 ,like Figure 5As shown, prior to S404, the database event processing method provided in this application embodiment may further include the following S406 to S407.
[0092] S406. Obtain the first timestamp corresponding to the target database event.
[0093] S407. Based on the first timestamp, determine the target array buffer from multiple array buffers of the target CPU core.
[0094] In some embodiments of this application, the event attribute information includes at least one of the following: the number of event data stored in the target array buffer; the timestamp corresponding to the first event data stored in the target array buffer; when the event attribute information includes the number of event data stored in the target array buffer, the target condition includes that the number of event data stored in the target array buffer is greater than or equal to a target number; when the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, the target condition includes that the duration between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to a duration threshold.
[0095] In some embodiments of this application, the event attribute information includes the number of event data stored in the target array buffer, and the target condition includes the number of event data stored in the target array buffer being a target number; combined with Figure 4 ,like Figure 6 As shown, the above S405 can be implemented in the following S405a.
[0096] S405a: If the number of event data corresponding to the target array buffer is the target number, submit the multiple event data in the target array buffer to the user-mode program in batch.
[0097] In some embodiments of this application, the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, and the target condition includes the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event being greater than or equal to a duration threshold; combined with Figure 4 ,like Figure 7 As shown, the above S405 can be implemented in specific form through the following S405b.
[0098] S405b: If the duration of the timestamp corresponding to the first event data stored in the target array buffer is greater than or equal to the duration threshold of the timestamp corresponding to the target database event, the multiple event data in the target array buffer are submitted in batch to the user-mode program.
[0099] In some embodiments of this application, combined with Figure 4 ,like Figure 8 As shown, prior to S404, the database event processing method provided in this application embodiment may further include the following S408.
[0100] S408. If the target array buffer is empty, write the timestamp corresponding to the target database event into the target array buffer.
[0101] In some embodiments of this application, the event attribute information further includes the number of event data stored in the target array buffer, and the target condition further includes the number of event data stored in the target array buffer being greater than or equal to a target number; combined with Figure 8 ,like Figure 9 As shown, the above S405 can be implemented in the following S405c.
[0102] S405c, if the target array buffer satisfies at least one of the following conditions: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration of the timestamp corresponding to the first event data stored in the target array buffer from the timestamp corresponding to the target database event is greater than or equal to the duration threshold, the multiple event data in the target array buffer are submitted in batch to the user-mode program.
[0103] In some embodiments of this application, the target CPU core corresponds to multiple array buffers, and each of these multiple array buffers corresponds to a different output channel; combined with Figure 4 ,like Figure 10 As shown, the above S405 can be implemented in the following S405d.
[0104] S405d: Submit the multiple event data in the target array buffer to the user-mode program in batches through the output channel corresponding to the target array buffer.
[0105] In some embodiments of this application, the target CPU core corresponds to multiple array buffers, and each of these multiple array buffers corresponds to a different output channel; combined with Figure 8 and Figure 10 Specifically, S405 can be performed when at least one of the following conditions is met: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration of the timestamp corresponding to the first event data stored in the target array buffer from the timestamp corresponding to the target database event is greater than or equal to the duration threshold is met: the multiple event data in the target array buffer are submitted in batches to the user-mode program through the output channel corresponding to the target array buffer.
[0106] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the database event handling method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0107] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0108] The present invention provides a computer program product, comprising: when the computer program product is run on a computer, causing the computer to implement the above-described database event processing method.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
[0110] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.
Claims
1. A service device, characterized in that, include: The controller is configured to enter kernel mode via a system call during the execution of SQL statements by the database process, so as to execute kernel functions through the target CPU core. During the execution of kernel functions, the corresponding eBPF mount point is triggered to run eBPF programs on the target CPU core; Events in the target database are captured by executing the eBPF program; Write the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core; If the event attribute information corresponding to the target array buffer meets the target conditions, multiple event data in the target array buffer are submitted to the user-space program in batches.
2. The service equipment according to claim 1, characterized in that, The controller is also configured to: Before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core, obtain the first timestamp corresponding to the target database event; Based on the first timestamp, the target array buffer is determined from multiple array buffers of the target CPU core.
3. The service equipment according to claim 1, characterized in that, The event attribute information includes the number of event data stored in the target array buffer, and the target condition includes the number of event data stored in the target array buffer being a target number; the controller is specifically configured as follows: If the number of event data corresponding to the target array buffer is the target number, the multiple event data in the target array buffer are submitted to the user-space program in batches.
4. The service equipment according to claim 1, characterized in that, The event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer; the target condition includes the duration of the time between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event being greater than or equal to a duration threshold; the controller is specifically configured as follows: If the duration of the timestamp corresponding to the first event data stored in the target array buffer is greater than or equal to the duration threshold of the timestamp corresponding to the target database event, the multiple event data in the target array buffer are submitted to the user-mode program in batches.
5. The service equipment according to claim 4, characterized in that, The controller is also configured to: Before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core, if the target array buffer is empty, the timestamp corresponding to the target database event is written into the target array buffer.
6. The service equipment according to claim 4, characterized in that, The event attribute information also includes the number of event data stored in the target array buffer, and the target condition also includes that the number of event data stored in the target array buffer is greater than or equal to a target number; the controller is specifically configured as follows: If at least one of the following conditions is met: the number of event data stored in the target array buffer is greater than or equal to the target number, and the duration of the timestamp corresponding to the first event data stored in the target array buffer from the timestamp corresponding to the target database event is greater than or equal to the duration threshold, the multiple event data in the target array buffer are submitted to the user-mode program in batches.
7. The service equipment according to any one of claims 1-6, characterized in that, The target CPU core corresponds to multiple array buffers, and each of the multiple array buffers corresponds to a different output channel; the controller is specifically configured as follows: The multiple event data in the target array buffer are submitted in batches to the user-mode program through the output channel corresponding to the target array buffer.
8. A database event handling method, characterized in that, Applied to service equipment, including: During the execution of SQL statements in the database process, a system call is used to enter kernel mode so that kernel functions can be executed through the target CPU core; During the execution of kernel functions, the corresponding eBPF mount point is triggered to run eBPF programs on the target CPU core; Events in the target database are captured by executing the eBPF program; Write the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core; If the event attribute information corresponding to the target array buffer meets the target conditions, multiple event data in the target array buffer are submitted to the user-space program in batches.
9. The method according to claim 8, characterized in that, Before writing the target event data corresponding to the target database event into the target array buffer corresponding to the target CPU core, the method further includes: Obtain the first timestamp corresponding to the target database event; Based on the first timestamp, the target array buffer is determined from multiple array buffers of the target CPU core.
10. The method according to claim 8 or 9, characterized in that, The event attribute information includes at least one of the following: The number of event data stored in the target array buffer; the timestamp corresponding to the first event data stored in the target array buffer; When the event attribute information includes the number of event data stored in the target array buffer, the target condition includes that the number of event data stored in the target array buffer is greater than or equal to the target number; when the event attribute information includes the timestamp corresponding to the first event data stored in the target array buffer, the target condition includes that the duration between the timestamp corresponding to the first event data stored in the target array buffer and the timestamp corresponding to the target database event is greater than or equal to a duration threshold.