A database os layer performance monitoring method, medium and device

Abstract

The present application relates to the technical field of database operation and maintenance monitoring, and particularly relates to a database OS layer performance monitoring method, medium and equipment, wherein a plurality of probe groups are respectively mounted at a plurality of kernel levels of a target server, operating system events generated at the corresponding kernel levels are collected by the probe groups, and the same-source collection of multi-dimensional operating system layer events is realized; a first process identifier of a database process identified is written into a filter mapping table, the mapping table is queried by the probe groups when an event is triggered, and events that do not match are discarded before being encapsulated and output, kernel-state targeted collection is realized, and the uncontrollable problem of the amount of full collection data is solved; the matched events are encapsulated according to a unified event structure body, then sent to a user-state program through a single output channel, and aggregated into monitoring indexes with database semantics according to database types, the semantic alignment of operating system layer events and database business is realized, and the pertinence of monitoring and the cross-database adaptation capability are improved.

Description

Technical Field

[0001] This invention relates to the field of database operation and maintenance monitoring technology, and in particular to a database OS layer performance monitoring method, medium, and device. Background Technology

[0002] Existing database performance monitoring technologies are mainly divided into two categories. The first category is internal database monitoring, which obtains session-level metrics such as wait events, structured query language execution statistics, and connection counts by querying dynamic database performance views, such as MySQL's `performance_schema` and PostgreSQL's `pg_stat_activity` view. The second category is operating system-level monitoring, which uses system tools to view the server's CPU, memory, I / O, network, and other resource usage. Commonly used system tools include `iostat`, `top`, `sar`, and `strace`.

[0003] In actual operation and maintenance, database administrators usually manually compare and analyze these two types of data. For example, when a slow query occurs in the database, the database administrator first checks the wait events within the database, then switches to the operating system to use iostat to check the input / output request queue length and disk response time, and then uses top to check the CPU usage of the database process.

[0004] However, there is a lack of correlation between internal database monitoring and operating system monitoring. Database administrators need to manually compare data from different tools, each with independent output and asynchronous timelines. Troubleshooting typically takes over an hour, and manual comparison is prone to missing clues and making it difficult to reproduce complete causal chains. Furthermore, internal database views cannot provide root cause analysis at the operating system level. For example, when the database generates a large number of disk reads and writes, it can only display "db file sequentialread" wait events, without indicating which disk partition it is, the read / write latency in milliseconds, or the length of the input / output queues. Simultaneously, directly collecting operating system events from all processes at the operating system level generates massive amounts of data; a 64-core server may generate hundreds of thousands of events per second, placing enormous pressure on storage and analysis. Moreover, existing centralized database monitoring solutions are typically strongly tied to database types, requiring separate deployment or configuration of collection terminals for different databases. The more types of databases in a data center, the more monitoring systems need to be maintained.

[0005] Therefore, how to achieve unified performance collection and semantic alignment across database types at the operating system layer without intruding on the database process or depending on the database version has become an urgent problem to be solved. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention provides a database OS layer performance monitoring method, which includes the following steps:

[0007] S1. Deploy a probe system at the operating system kernel layer of the target server. The operating system kernel layer corresponds to several kernel levels, and the probe system contains several probe groups, each probe group being mounted at a different kernel level.

[0008] S2, identify the database process in the target server, obtain the first process identifier of the database process and the database type to which the database process belongs, and write the first process identifier into the preset filter mapping table.

[0009] S3 collects operating system events generated at the corresponding kernel level through each probe group. Each operating system event carries at least the second process identifier of the process that triggered the operating system event.

[0010] S4. When an event is triggered, each probe group queries the filter mapping table. If the second process identifier carried by the current operating system event does not match the filter mapping table, the current operating system event is discarded before encapsulation and output. If the filter mapping table is matched, the current operating system event is encapsulated according to the preset event structure and sent to the user-mode program through a single preset output channel.

[0011] S5 aggregates and processes operating system events received by user-mode programs according to the database type to obtain monitoring metrics, where the monitoring metrics have database semantics corresponding to the database type.

[0012] The present invention also provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described database OS layer performance monitoring method.

[0013] The present invention also provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0014] This invention has at least the following beneficial effects: By mounting probe groups at several kernel levels within the operating system kernel layer, operating system events across five dimensions—input / output, network, CPU scheduling, file system, and memory management—are included in the collection scope. This avoids the limitation of single-dimensional collection being unable to fully characterize database performance issues, achieving co-source collection of multi-dimensional operating system layer events. Furthermore, by identifying the database process and writing its process identifier into a filtering mapping table before collection, each probe group queries this mapping table when an event is triggered and discards unmatched events before encapsulation and output. This ensures that events generated by non-database processes are terminated in kernel mode, preventing them from entering subsequent encapsulation and output. In the writing and reading stages, the system solves the problems of uncontrollable full-data collection volume and the submersion of valid events by irrelevant noise, thus improving the signal-to-noise ratio of the collected data. By encapsulating events output by each probe group using a unified event structure and sharing the same output channel, heterogeneous events generated at different kernel levels have a unified field layout when leaving the kernel, solving the problem of needing to parse events from multiple sources separately. By aggregating received operating system events according to database type, the aggregation results carry semantics corresponding to the specific database, solving the problem of inconsistent meanings of similar operating system events in different databases, and improving the cross-database comparability and interpretability of monitoring metrics. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a database OS layer performance monitoring method provided in Embodiment 1 of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the terms used to distinguish similar objects can be interchanged so that the invention can also be implemented in other embodiments besides the illustrated or described embodiments. Furthermore, the terms "including," "having," and any variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0019] Example 1

[0020] This embodiment provides a database OS layer performance monitoring method, such as... Figure 1 As shown, this database OS layer performance monitoring method includes the following steps:

[0021] S1. Deploy a probe system at the operating system kernel layer of the target server. The operating system kernel layer includes several kernel levels, and the probe system contains several probe groups, each probe group being mounted at a different kernel level.

[0022] The target server is a physical or virtual server that hosts database instances and runs database processes. As the object of performance monitoring at the operating system level, the target server can run one or more database instances.

[0023] The operating system kernel layer is the core software layer in the operating system running on the target server, responsible for resource management, process scheduling, device drivers, and system calls. The kernel layer is comprised of several sub-layers within the operating system kernel, divided according to their functions. Examples include the block device layer responsible for merging and distributing input / output requests; the network protocol stack layer responsible for network data transmission and reception and protocol processing; the CPU scheduling subsystem layer responsible for process scheduling and context switching; the file system layer responsible for file and directory metadata operations; and the memory management subsystem layer responsible for page caching and memory reclamation.

[0024] A probe system is a collection of data acquisition programs deployed at the operating system kernel level. A probe group is a collection unit within the probe system, divided according to kernel level. Each probe group is attached to a specific kernel level and is used to collect operating system events generated at that kernel level. Since each probe group is attached to a different kernel level, the collection scope covers multiple different resource dimensions.

[0025] Because a single database operation leaves traces in multiple kernel subsystems at the operating system level—for example, a transaction commit might trigger file system synchronization calls, block device layer I / O requests, potential direct memory reclamation, and subsequent scheduling switches—these traces are distributed across different kernel levels. Deploying probes at only one kernel level would only provide information on resource usage in that single dimension, failing to fully characterize the causes of performance problems. Therefore, this embodiment mounts probe groups at different kernel levels, encompassing all five resource dimensions within the collection scope.

[0026] In one specific implementation, the kernel layer includes at least a block device layer, a network protocol stack layer, a central processing unit scheduling subsystem layer, a file system layer, and a memory management subsystem layer.

[0027] Correspondingly, the block device layer is equipped with an input / output probe group, which is used to collect the latency, sector number, and device number of each input / output operation. The block device layer is the layer in the operating system kernel where input / output requests are merged and sorted by the input / output scheduler and then delivered to the device driver. Collecting data at this layer can obtain the complete input / output latency, including queuing time, and not just the device's own service time.

[0028] The network protocol stack layer carries a network probe group, which is used to collect Transmission Control Protocol (TCP) retransmission events, connection establishment delays, and congestion windows. Among these, retransmission events and connection establishment delays are key indicators for judging the connection quality between the database master-slave replication link and the client, while the congestion window reflects the degree of network link congestion.

[0029] The CPU scheduling subsystem layer is equipped with a CPU scheduling probe group, which is used to collect CPU context switching counts, wake-up latency, and run queue length. Among them, wake-up latency is the waiting time between a process being woken up and actually obtaining processor access, and run queue length reflects the queuing pressure on the processor. Together, they characterize the schedulability of database worker threads.

[0030] The file system layer mounts a file system probe group, which is used to collect file system metadata operation latency. Among them, file system metadata operations are an important factor affecting database write and persistence performance, and increased latency often directly manifests as longer database commit wait times.

[0031] The memory management subsystem layer is equipped with a memory management probe group, which is used to collect data on memory allocation latency, page reclamation pressure, and the number of direct reclamation events. Direct memory reclamation occurs on the memory allocation path and simultaneously blocks the current process, which is a common cause of sudden database lag. Therefore, this embodiment sets it as a separate collection dimension.

[0032] Specifically, the input / output probe group can be attached to the block device layer request completion trace point `block:block_rq_complete`; the network probe group can be attached to the Transmission Control Protocol retransmission trace point `tcp:tcp_retransmit_skb` and the Transmission Control Protocol connection establishment trace point `tcp:tcp_rcv_establish`; the CPU scheduling probe group can be attached to the process wake-up trace point `sched:sched_wakeup` and the process switch trace point `sched:sched_switch`; the file system probe group can be attached to the kernel probe `kprobe:sys_enter_fsync` for the file synchronization system call entry point `kprobe:sys_exit_fsync` and the kernel probe `kprobe:sys_exit_fsync` for the exit point; and the memory management probe group can be attached to the page cache get trace point `mm:page_cache_get` and the direct memory reclamation trace point `mm:vmscan_direct_reclaim`.

[0033] Alternatively, the file system probe group can also be mounted at the file system layer tracepoints ext4:ext4 and xfs:xfs to collect file system metadata operation latency; the memory management probe group can also be mounted at the kernel memory allocation tracepoint kmem:mm_page_alloc and the virtual memory scan tracepoint vmscan:mm_vmscan to collect memory allocation latency, page reclamation pressure, and the number of direct reclamation operations. The specific mount point selection can be determined by the implementer based on the actual file system type used on the target server and the availability of kernel tracepoints.

[0034] Specifically, the probe system can be implemented using extended Berkeley packet filtering technology, where each probe group is an extended Berkeley packet filtering program running in the operating system kernel. Extended Berkeley packet filtering is a mechanism for securely executing custom programs within the operating system kernel. These programs must pass security checks by the kernel verifier before being loaded, thus enabling event collection without modifying the kernel source code, loading kernel modules, or intruding on database processes.

[0035] As described above, by mounting probe groups at different kernel levels, five resource dimensions—input / output, network, CPU scheduling, file system, and memory management—are included in the collection scope. This solves the problem that single-dimensional collection cannot fully characterize the causes of database performance issues, and improves the completeness of operating system-level performance data and the reliability of subsequent analysis.

[0036] S2, identify the database process in the target server, obtain the first process identifier of the database process and the database type to which the database process belongs, and write the first process identifier into the preset filter mapping table.

[0037] The database process refers to the process on the target server that hosts the database service, including the main database process and its derived background processes, worker threads, and session processes. The first process identifier is a unique number for the database process in the operating system, used as a comparison benchmark for the whitelist in subsequent steps. The database type indicates the category of the database product, specifying which database product the process belongs to. The filter mapping table is a key-value store structure located in the operating system kernel, directly queried by the probe group. It uses the process identifier as the key to record the range of processes allowed for collection.

[0038] Since the target server runs a large number of system processes and middleware processes in addition to the database process, such as systemd, sshd, crond, dockerd, kubelet, prometheus, node-exporter, java, and nginx, these processes also trigger operating system events at the block device layer, scheduling layer, file system layer, and memory management layer. If filtering is performed after collection, these irrelevant events will have already consumed the overhead of encapsulation, channel writing, and user-space reading. Therefore, this embodiment first determines the whitelist of collection targets and writes it into the filtering mapping table before collection, so that subsequent steps can complete filtering at minimal cost.

[0039] In one specific embodiment, S2 includes the following steps:

[0040] S21. Obtain the reference process identifier and reference process name of each initial process through the process information query interface provided by the operating system of the target server.

[0041] S22, match the reference process name with the preset database process name keywords, and determine the initial process that matches successfully as a candidate process.

[0042] S23. Obtain the process resource usage information and network connection information of each candidate process through the process resource query interface provided by the operating system of the target server, and confirm the communication port number bound to each candidate process based on the process resource usage information and network connection information.

[0043] S24. The communication port number is compared and verified with the preset database port. If they match, the corresponding candidate process is determined to be the database process. The corresponding reference process identifier is used as the first process identifier, and the database type is determined according to the database process name keyword that the candidate process successfully matches.

[0044] The process information query interface is the interface through which the operating system exposes runtime information of processes in the form of files. For example, in the Linux operating system, the process name of each process can be obtained by traversing the command-line files (cmdline) in the process directories under the proc filesystem. The process resource query interface is the interface through which the operating system exposes information about the resources used by a process. For example, the file descriptor directories (fd) named after the process ID and the network connection status files (net / tcp) under the proc filesystem. Process resource usage information records the file descriptors currently opened by the process, while network connection information records the network connections currently established and listened to by the process.

[0045] Database process name keywords are a set of strings used to identify database processes, including at least oracle, mysqld, postgres, gaussdb, and sqlservr, corresponding to the main process names of Oracle, MySQL, PostgreSQL, openGauss, and SQL Server, respectively. Database port is the default port number listened to by each type of database, including at least 1521, 3306, 5432, and 1433.

[0046] It should be noted that specific keywords and port numbers can be added or deleted by the implementer based on the actual database type used.

[0047] If only process name matching is used, two types of false positives may occur: one is that a non-database process uses an executable file with the same name as the database process, and the other is that the database process has been renamed or started with a custom process name. On the other hand, matching only by port cannot distinguish between different processes listening on the same port, nor can it cover background processes and worker threads that are not listening on ports. Therefore, this embodiment uses a two-factor cross-validation method using process name and communication port. First, a rapid initial screening is performed using the process name, and then identity verification is performed using the port. Only when both match is it identified as a database process, thereby improving the recognition accuracy without intruding on the database process or querying the internal database structure.

[0048] In one specific embodiment, S23 includes the following steps:

[0049] S231, extract the socket identifier from the process resource usage information of the candidate process.

[0050] S232: Based on the socket identifier, search for the corresponding port number in the network connection information and use the found port number as the communication port number.

[0051] The socket identifier is an inode number assigned to each socket by the operating system. Specifically, by reading the link targets of each file descriptor under the file descriptor directory (fd) of the candidate process, a socket inode identifier in the form of socket:[inode number] can be obtained. Then, by reading the inode field of each connection record in the candidate process's network connection state file net / tcp, and matching the socket inode identifier with the connection record, the transmission control protocol port number being listened to by the candidate process can be located. Since the socket identifier is unique and consistent in the process's file descriptor table and the kernel network connection table, this embodiment uses the socket identifier as the association key between process resources and network connections, which can reliably complete the mapping between the two.

[0052] In one specific implementation, the database type includes at least Oracle, MySQL, PostgreSQL, and openGauss databases. The database type is determined by the database process name keyword that is successfully matched with the candidate process; for example, matching the keyword "gaussdb" determines it to be an openGauss database, and matching the keyword "mysqld" determines it to be a MySQL database.

[0053] In one specific implementation, S2 further includes: traversing the thread directory corresponding to the database process, reading the thread identifier and thread group identifier of each thread under the database process, establishing a mapping relationship between thread identifiers and thread group identifiers, and supplementing each thread identifier into the filter mapping table, so that each probe group can collect events from the main process of the database process and all its child threads. Databases generally use multi-process or multi-threaded models to handle concurrency. If only the events of the main process are collected, the worker threads that actually perform input / output and scheduling will be missed, resulting in the collection results failing to reflect the true resource consumption of the database.

[0054] In one specific implementation, the identification step is re-executed according to a preset update cycle, the first process identifier of the newly identified database process is incrementally written into the filter mapping table, and the process identifiers of exited processes are deleted from the filter mapping table.

[0055] Because the database creates new session processes, restarts background processes, or exits abnormal processes during operation, if the whitelist is written once and not updated, it will gradually deviate from the actual process state, leading to missed or incorrect sampling. Therefore, this embodiment adopts periodic incremental updates. The specific value of the preset update period can be set by the implementer according to the actual situation. For example, in this embodiment, the preset update period is 30 seconds.

[0056] The above-mentioned method identifies database processes through two-factor cross-validation using process name keywords and communication ports, and completes port location by using socket identifiers as the association key between process resources and network connections. This improves the identification accuracy without intruding on the database process or querying the internal structure of the database. By traversing the thread directory to include all child threads in the whitelist and updating it incrementally at a preset period, the method solves the problems of incomplete collection coverage and whitelist expiration over time under the multi-process and multi-threaded database model, thus improving the completeness and timeliness of the collected objects.

[0057] S3 collects operating system events generated at the corresponding kernel level through each probe group. Each operating system event carries at least the second process identifier of the process that triggered the event.

[0058] Operating system events are observable records generated at a specific kernel level during the processing of resource requests by the operating system kernel. Examples include completion events generated when an input / output request is completed at the block device layer, and switch events generated when a process switch occurs in the scheduling subsystem. The second process identifier is a unique number of the process that triggered the operating system event within the operating system, used for comparison with the first process identifier in the filter mapping table.

[0059] Since each event generated at the kernel level is triggered by a specific process, the event naturally carries the identifier of the triggering process. In this embodiment, the identifier of the triggering process is made a mandatory field of the event. This allows subsequent steps to directly determine whether the event falls within the collection range based on this field, without relying on reverse lookup of the process name or other indirect association methods, thus simplifying the filtering process to a single key-value comparison.

[0060] In one specific implementation, each operating system event is encapsulated according to a preset event structure, which is a struct os_event event structure. The struct os_event event structure includes a process identifier field, a nanosecond-level timestamp field, an event type field, and an event details field. The nanosecond-level timestamp field is generated by each probe group calling the time interface function of the same kernel clock source.

[0061] The process identifier field records the second process identifier of the process that triggered the event; the nanosecond timestamp field records the timestamp of the event, with a precision of nanoseconds; the event type field distinguishes which kernel level the event comes from; and the event details field carries parameters specific to the event at that kernel level, such as the delay, sector number, and device number for input / output events.

[0062] Using the same kernel clock source means that each probe group calls the same time interface function in the operating system kernel, such as bpf_ktime_get_ns(), when acquiring timestamps. The timestamps of events generated at different kernel levels are taken from the same time base, eliminating cross-clock source drift. Specifically, with the same kernel clock source, the time deviation between events generated at different kernel levels is less than 1 microsecond, a precision sufficient to support micro-delay analysis at the 99th percentile and 99.9th percentile.

[0063] Specifically, the event type field and the event details field can be organized in a union form, so that different types of events can share the storage space of the same structure, avoiding the memory waste caused by allocating a fixed-length structure for each kernel level event.

[0064] As described above, by ensuring that all operating system events generated at each kernel level carry the second process identifier of the triggering process, a direct comparison basis is provided for subsequent targeted filtering. By adopting a unified event structure and generating nanosecond-level timestamps from the same kernel clock source, heterogeneous events generated at different kernel levels have a unified field layout and a unified time base, solving the problem of needing to parse events from multiple sources separately and align them with manual time, thus improving the comparability of event data and the accuracy of subsequent analysis.

[0065] S4. When an event is triggered, each probe group queries the filter mapping table. If the second process identifier carried by the current operating system event does not match the filter mapping table, the current operating system event is discarded before encapsulation and output. If it matches, the current operating system event is encapsulated according to the preset event structure and sent to the user-mode program through a single preset output channel.

[0066] The filter mapping table contains the first process identifier of each database process. A miss occurs when the second process identifier carried by the current operating system event does not have a corresponding entry in the filter mapping table. In this case, subsequent processing of the event is terminated before the event is assembled into a preset event structure and written to the output channel. A single preset output channel means that all probe groups share the same data transfer channel from kernel mode to user mode, rather than each probe group using its own channel. User-mode programs are programs that run in the operating system's user mode and are responsible for receiving and processing events.

[0067] Once an event is encapsulated and written to the output channel, the encapsulation overhead, channel write overhead, and user-space read overhead have already occurred. If filtering is performed after the event reaches user space, these overheads cannot be recovered, and the amount of data transferred from kernel space to user space is not reduced. In this embodiment, however, the filtering mapping table lookup is performed before encapsulation when the event is triggered, and processing terminates if no match is found. This allows filtering to be completed at minimal cost as soon as the event occurs, significantly reducing the amount of events that actually enter the encapsulation and transmission stages.

[0068] In one specific embodiment, S4 includes the following steps:

[0069] S41, extract the second process identifier from the current operating system events.

[0070] S42, query the filter mapping table to see if there is an entry that matches the second process identifier.

[0071] S43, if it exists, it is determined to be a hit, and the current operating system event is encapsulated according to the preset event structure and sent through the preset output channel; if it does not exist, it is determined to be a miss, and the current operating system event is discarded before encapsulation and output.

[0072] In one specific implementation, the preset output channel is the Perf Event Array output channel. Perf EventArray is a kernel data structure provided by the extended Berkeley packet filter for efficiently transmitting events from kernel mode to user mode. Each probe group writes the encapsulated events into this preset output channel, and the user-mode program reads them uniformly from this preset output channel.

[0073] Specifically, the filtering mapping table uses the process identifier as the key and the value representing the database process as the value. The probe group performs a mapping table lookup at the entry point. If the key exists, it continues to collect data; otherwise, it returns immediately.

[0074] Specifically, taking a 64-core server running Oracle RAC as an example, the server has approximately 800 running processes, of which approximately 30 to 60 are database processes. During full data collection, approximately 500,000 events are generated per second. After targeted data collection using this embodiment, the number of events generated per second is approximately 500 in an idle scenario, approximately 800 in a light online transaction processing scenario (10 concurrent users), approximately 2000 in a medium online transaction processing scenario (50 concurrent users), and approximately 10,000 in a heavy online transaction processing scenario (200 concurrent users). The event reduction is calculated as (total event count - targeted event count) / total event count × 100%. Substituting the values ​​from the medium online transaction processing scenario, the event reduction is calculated as (500,000 - 2000) / 500,000 × 100% = 99.6%. The event reduction in each scenario is 99.2% to 99.9%, meaning the data volume is reduced by 2 to 3 orders of magnitude.

[0075] Specifically, the processor overhead on the target server after probe loading can be measured using the difference method. This involves first running the target load without loading any probes and measuring the baseline processor utilization. Then, after loading each probe group, the processor utilization is measured again with the same load parameters. Since the processor utilization includes the processor resources consumed by the database business load itself, directly reading the loaded processor utilization cannot distinguish between probe overhead and business overhead. Therefore, by using the baseline measurement under the same load as a benchmark, subtracting the baseline processor utilization from the loaded processor utilization yields the pure incremental overhead of the probes, eliminating interference from the business load itself.

[0076] To improve measurement stability, each measurement lasted 300 seconds, and the average value of the stable segment after removing the first and last 30 seconds was taken. Each group was repeated 3 times and the median was taken to eliminate the influence of instantaneous jitter and warm-up effect.

[0077] During the measurement period, single-core incremental overhead (%) = (post-load processor time - baseline processor time) / measurement period × 100%.

[0078] Probe callbacks execute on the processor core that triggered the event, with the overhead distributed across several cores. In a typical scenario, five probe groups have events triggered on 2 to 8 cores respectively. Therefore, system-level incremental overhead = single-core incremental overhead / number of processor cores handling the event.

[0079] Specifically, in a test environment with a server configured with two Intel Xeon Gold 6248 processors (40 cores total), 256GB of memory, an NVMe SSD, and a 25GbE network card, running Linux kernel 5.10, using libbpf v1.2 as the extended Berkeley packet filtering framework, Oracle 19c RAC as the database, and Swingbench 2.7 and sysbench 1.0.20 as the load balancing tools, the actual test results with all five probe groups enabled were as follows: idle scenario event rate 534 per second, system-level incremental overhead less than 0.05%; light online transaction processing scenario event rate 5241 per second, system-level incremental overhead 0.17%; medium online transaction processing scenario event rate 21893 per second, system-level incremental overhead 0.17%, equivalent to approximately 3.6% single-core incremental overhead; heavy online transaction processing scenario event rate 83476 per second, system-level incremental overhead 0.23%; and extreme load test scenario event rate 198234 per second, system-level incremental overhead 0.37%. The overhead of the probes for layer-by-layer loading measurement is as follows: input / output probe only 0.06%, CPU scheduling probe overlaid 0.10%, file system probe overlaid 0.12%, memory management probe overlaid 0.14%, and all five layers open 0.17%.

[0080] As described above, by querying the filter mapping table when an event is triggered and discarding the missing events before encapsulation and output, events generated by non-database processes are terminated in kernel mode and do not enter the subsequent encapsulation, channel writing and user mode reading stages. This solves the problems of uncontrollable full-volume data collection and the submersion of valid events by irrelevant noise, thereby improving collection efficiency and availability in the production environment.

[0081] S5 aggregates and processes operating system events received by user-mode programs based on the database type to obtain monitoring metrics.

[0082] Among them, monitoring metrics are quantitative results reflecting the resource usage of a target server, obtained by statistically reducing received operating system events, and possess database semantics corresponding to the database type. Database semantics means that monitoring metrics have been reduced according to the process model and business meaning of the specific database product, and can be directly understood and used by the database's operations and maintenance personnel, rather than just abstract operating system counts.

[0083] Because different database products have significantly different process models, the same operating system event can have different business meanings in different databases. For example, Oracle uses a multi-process model, where its log writer process and data writer process have different responsibilities. The same input / output event occurring in the log writer process corresponds to completely different database behaviors than when it occurs in the data writer process. MySQL, on the other hand, uses a multi-threaded model, where write operations are handled by background threads, and there are no process roles that correspond one-to-one with Oracle's. If a uniform aggregation rule is used, the resulting metrics cannot be mapped to specific database behaviors. Therefore, this embodiment reduces events according to database type, so that the aggregation results carry semantics corresponding to the specific database.

[0084] In one specific embodiment, S5 includes the following steps:

[0085] The received operating system events are sent to a type-aware aggregator corresponding to the database type to obtain monitoring metrics.

[0086] The type-aware aggregator is a computational unit that corresponds to a specific database type and reduces operating system events according to the process model and semantics of that database type. Specifically, corresponding type-aware aggregators can be configured for Oracle, MySQL, PostgreSQL, and openGauss databases, and each aggregator reduces events according to the process role and metric definitions of its respective database type.

[0087] Specifically, the type-aware aggregator calculates the average latency, peak latency, number of events per unit time, and total number of events for each received operating system event according to a preset statistical period. The preset statistical period is 1 second. The average latency is the arithmetic mean of the latency of the same type of events within the same statistical period, the peak latency is the maximum value among them, the number of events per unit time is the number of times that type of event occurs within a unit of time, and the total number of events is the cumulative number of times that type of event occurs within the statistical period.

[0088] As described above, by sending the received operating system events to the corresponding type-aware aggregator for reduction according to the database type, the aggregation results carry business semantics corresponding to the specific database product. This solves the problem of inconsistent meanings of similar operating system events in different databases and the difficulty for operation and maintenance personnel to interpret them directly, thereby improving the cross-database comparability and interpretability of monitoring indicators.

[0089] This embodiment attaches probe groups to several kernel levels within the operating system kernel layer, encompassing five dimensions of operating system events: input / output, network, CPU scheduling, file system, and memory management. This avoids the limitation of single-dimensional collection failing to fully characterize database performance issues, achieving co-source collection of multi-dimensional operating system layer events. Before collection, the database process is identified and its process identifier is written into a filtering mapping table. Each probe group queries this table when an event is triggered and discards unmatched events before encapsulation and output. This ensures that events not generated by database processes are terminated in kernel mode, preventing them from entering subsequent encapsulation, channel writing, and usage. In the user status reading stage, the problems of uncontrollable full-data collection volume and the submersion of valid events by irrelevant noise were solved, thus improving the signal-to-noise ratio of the collected data. By encapsulating the events output by each probe group with a unified event structure and sharing the same output channel, heterogeneous events generated at different kernel levels have a unified field layout when leaving the kernel, solving the problem of needing to parse events from multiple sources separately. By aggregating the received operating system events according to database type, the aggregation results carry semantics corresponding to the specific database, solving the problem of inconsistent meanings of similar operating system events in different databases, and improving the cross-database comparability and interpretability of monitoring indicators.

[0090] Example 2

[0091] Embodiment 2 of the present invention provides a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the database OS layer performance monitoring method provided in the above embodiment.

[0092] Example 3

[0093] Embodiment 3 of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium of Embodiment 2 of the present invention.

[0094] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A database OS layer performance monitoring method, characterized in that, The method includes the following steps: S1, Deploy a probe system at the operating system kernel layer of the target server. The operating system kernel layer corresponds to several kernel levels. The probe system contains several probe groups. Each probe group is mounted at a different kernel level. Each probe group is an eBPF program running in the operating system kernel. The kernel level includes at least a block device layer, a network protocol stack layer, a CPU scheduling subsystem layer, a file system layer, and a memory management subsystem layer. S2, identify the database process in the target server, obtain the first process identifier of the database process and the database type to which the database process belongs, and write the first process identifier into a preset filter mapping table, wherein S2 includes the following steps: Traverse the thread directory corresponding to the database process, read the thread identifier and thread group identifier of each thread under the database process, establish the mapping relationship between the thread identifier and the thread group identifier, and supplement each thread identifier into the filter mapping table; The identification steps are re-executed according to the preset update cycle. The first process identifier of the newly identified database process is incrementally written into the filter mapping table, and the process identifier of the exited process is deleted from the filter mapping table. S3 collects operating system events generated at the corresponding kernel level through each probe group. Each operating system event carries at least the second process identifier of the process that triggered the operating system event. S4, each probe group queries the filter mapping table when the event is triggered. If the second process identifier carried by the current operating system event does not match the filter mapping table, the current operating system event is discarded before encapsulation and output. If the filter mapping table is matched, the current operating system event is encapsulated according to the preset event structure and sent to the user-mode program through a single preset output channel. S5. Based on the database type, aggregate the operating system events received by the user-mode program to obtain monitoring metrics, wherein the monitoring metrics have database semantics corresponding to the database type.

2. The database OS layer performance monitoring method according to claim 1, characterized in that, The block device layer is equipped with an I / O probe group, which is used to collect the delay, sector number, and device number of each I / O operation. The network protocol stack layer is equipped with a network probe group, which is used to collect TCP retransmission events, connection establishment delays and congestion windows. The CPU scheduling subsystem layer is equipped with a CPU scheduling probe group, which is used to collect CPU context switching count, wake-up latency and run queue length. The file system layer is equipped with a file system probe group, which is used to collect file system metadata operation latency. The memory management subsystem layer is equipped with a memory management probe group, which is used to collect memory allocation latency, page reclamation pressure, and direct reclamation count.

3. The database OS layer performance monitoring method according to claim 1, characterized in that, S2 includes the following steps: S21, obtain the reference process identifier and reference process name of each initial process through the process information query interface provided by the operating system of the target server; S22, Match the reference process name with preset database process name keywords, and determine the initial process that successfully matches as a candidate process; S23, obtain the process resource usage information and network connection information of each candidate process through the process resource query interface provided by the operating system of the target server, and confirm the communication port number bound to each candidate process based on the process resource usage information and the network connection information; S24, compare and verify the communication port number with the preset database port. If they match, determine the corresponding candidate process as the database process, use the corresponding reference process identifier as the first process identifier, and determine the database type based on the database process name keyword that the candidate process successfully matches.

4. The database OS layer performance monitoring method according to claim 3, characterized in that, S23 includes the following steps: S231, Extract the socket identifier from the process resource usage information of the candidate process; S232, based on the socket identifier, search for the corresponding port number in the network connection information, and use the found port number as the communication port number.

5. The database OS layer performance monitoring method according to claim 1, characterized in that, The preset output channel is the Perf Event Array output channel, and the preset event structure is the struct os_event event structure. The struct os_event event structure includes a process identifier field, a nanosecond-level timestamp field, an event type field, and an event details field. The nanosecond-level timestamp field is generated by each probe group calling the time interface function of the same kernel clock source.

6. The database OS layer performance monitoring method according to claim 1, characterized in that, S4 includes the following steps: S41, Extract the second process identifier from the current operating system events; S42, query the filter mapping table to see if there is an entry that matches the second process identifier; S43, if it exists, then it is determined that a hit has occurred, and the current operating system event is encapsulated according to the preset event structure and sent through the preset output channel; If it does not exist, it is considered a miss, and the current operating system event is discarded before encapsulation and output.

7. The database OS layer performance monitoring method according to claim 1, characterized in that, The database types include at least Oracle, MySQL, PostgreSQL, and openGauss databases.

8. The database OS layer performance monitoring method according to claim 1, characterized in that, S5 includes the following steps: The received operating system events are sent to a type-aware aggregator corresponding to the database type to obtain the monitoring metrics.

9. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the database OS layer performance monitoring method as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images