Log data acquisition method and device, electronic equipment and storage medium

Abstract

According to the log data collection method and device, the electronic equipment and the storage medium provided by the embodiment of the invention, efficient data collection is realized through a user mode-kernel mode collaboration mechanism: when service query is finished, a user thread reads a dynamic configuration generation log header from a global memory and writes the log header into a shared memory; a kernel thread directly analyzes configuration after monitoring event triggering, original audit data is collected from a user thread memory, and network interception, protocol analysis and disk I / O links are omitted; performance data are directly collected in a thread memory through a kernel thread according to a configuration index type, and intrusion interference of traditional instrumentation on a user thread is avoided. According to the architecture, buffer area construction and intermediate storage are eliminated through memory direct acquisition, continuous association of long transaction data is ensured while CPU / memory resource consumption is reduced, the problem of data loss under high load is effectively solved, auditing data zero-error acquisition and performance index millisecond-level monitoring are achieved, and dual optimization of resource occupation and data reliability is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log data acquisition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Traditional database audit and performance data collection face common technical challenges, both exhibiting a dual dilemma of resource consumption and data reliability. Audit data collection typically relies on external probe tools, intercepting network traffic or parsing SQL (Structured Query Language) execution trees through side-channel monitoring mechanisms. This process not only requires additional CPU resources for packet parsing and memory buffer construction but also generates continuous disk I / O operations for log writing, causing system throughput to decrease with increasing load. Simultaneously, due to CPU resource contention, memory swapping pressure, and lock contention, the risk of data loss is significantly increased, especially in long transactions or distributed scenarios, where the difficulty in establishing logical connections across data packets makes misjudgments of transaction states more likely. Performance data collection employs program instrumentation techniques, directly inserting probe code into the database code to synchronously record execution metrics. This intrusive collection method significantly increases the CPU computational overhead of user threads, thereby impacting overall database performance. Although the two data acquisition schemes have different technical approaches, both suffer from compromised data integrity due to intensified resource competition, creating an inherent contradiction between acquisition efficiency and data accuracy. Summary of the Invention

[0003] In view of this, in order to effectively alleviate the technical problems of collection efficiency and data accuracy in existing audit data and performance data collection schemes, this application provides a log data collection method, device, electronic device and storage medium.

[0004] In a first aspect, embodiments of this application provide a log data collection method, the method being applied to a database, the method comprising: Upon detecting the end of a business query, a user thread is triggered to retrieve configuration data from global memory, generate a log header based on the configuration data, and store the log header in shared memory; wherein, the configuration data is metadata that dynamically controls log collection behavior and is used to specify the audit log strategy and the type of performance monitoring metrics; When the log header write completion event is detected, a kernel thread is started to read the log header from the shared memory and parse the log header to extract the configuration data. Log data is then collected from the user thread's thread memory based on the configuration data. The log data includes audit data and / or performance data, and the user thread's thread memory stores all the original data during the execution of the business query.

[0005] Optionally, as described above, the method further includes: When the log data collection completion event is detected, the user thread is triggered to expand the information of the log header to obtain an enhanced log header; The log data and the enhanced log header are combined into a log record according to a preset log architecture; The log records are written to the shared memory in a lock-free concurrent manner.

[0006] Optionally, as described above, the preset log architecture includes a log header range, a fixed-length audit data range, a variable-length audit data range, and a performance data storage range. The step of assembling the log data and the enhanced log header into a log record according to a preset log architecture includes: Write the enhanced log header into the log header range; If the log data contains performance data, the performance data is written to the performance data storage area. If the log data is found to include audit data, the data type field values ​​included in the audit data are obtained; Determine whether the value of the data type field falls within the fixed field value range; If it is determined that the value of the data type field falls within the range of the fixed field value, the audit data is written into the fixed length range of the audit data; If it is determined that the data type does not fall within the range of the fixed field value, the audit data is written into the variable-length audit data range.

[0007] Optionally, as described above, writing the log records into the shared memory in a lock-free concurrent manner includes: Obtain the start sequence number and end sequence number from the global memory; The physical start offset of the shared memory is determined by modulo operation based on the start sequence number, and the physical end offset of the shared memory is determined by modulo operation based on the end sequence number; Determine whether the physical end offset is less than the physical start offset; If the physical end offset is determined to be equal to or greater than the physical start offset, the log record is written to the shared memory in a single instance. If the physical end offset is determined to be less than the physical start offset, the log record is written to the shared memory in two separate transactions.

[0008] Optionally, as described above, the method further includes: The log coordination thread is started to poll the current consumption offset and the latest production offset in the global memory through atomic operations; Detect whether the current consumption offset lags behind the latest production offset; If it is detected that the current consumption offset is behind the latest production offset, retrieve the valid log records between the current consumption offset and the latest production offset from the shared memory; The valid log records are distributed to a parsing thread pool. Multiple log parsing threads in the parsing thread pool are used to call an external parsing script to parse the valid log records. The parsed valid log records are then written to a local file or sent to an external component.

[0009] Optionally, as described above, the method further includes: The log offset set parsed by the log parsing thread is collected through a timed sampling mechanism, and the consumption offset set recorded in the global memory is read synchronously. The write speed is determined based on multiple log offsets in the log offset set, and the parsing speed is determined based on multiple current consumption offsets in the consumption offset set. Detect the numerical relationship between the write speed and the parsing speed; If the write speed is detected to be greater than the parsing speed, the size of the parsing thread pool is dynamically expanded until the parsing speed reaches or exceeds the write speed. If the write speed is detected to be lower than the parsing speed, the size of the parsing thread pool is dynamically reduced until the write speed reaches the parsing speed.

[0010] Optionally, as described above, the method further includes: Start the log buffer management thread to detect in real time whether the parsed log offset is greater than the latest production offset of the valid log record; If the log offset is detected to be greater than the latest production offset of the valid log record, the log records between the log offset and the latest production offset are cleaned up and reclaimed from the shared memory; wherein, the role of the log buffer management thread also includes maintaining the memory area between the current consumption offset and the latest production offset to always satisfy physical contiguousness.

[0011] Secondly, embodiments of this application provide a log data collection device, which is applied to a database, and the device includes: The acquisition and storage module is used to trigger the user thread to retrieve configuration data from global memory when the business query is detected to be finished, generate a log header based on the configuration data, and store the log header in shared memory; wherein, the configuration data is metadata that dynamically controls the log collection behavior and is used to specify the audit log strategy and the performance monitoring metric type; The read parsing and acquisition module is used to start a kernel thread to read the log header from the shared memory when the log header write completion event is detected, and to parse the log header to extract the configuration data. Based on the configuration data, log data is collected from the thread memory of the user thread. The log data includes audit data and / or performance data, and the thread memory of the user thread stores all the original data in the process of the business query execution.

[0012] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory. The processor is used to execute a program for collecting log data stored in the memory to implement the above-described log data collection method.

[0013] Fourthly, embodiments of this application provide a storage medium, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described log data collection method.

[0014] The log data acquisition method, apparatus, electronic device, and storage medium provided in this application comprehensively solve the technical challenges faced by traditional audit data and performance data acquisition through a collaborative working mechanism between user space and kernel space. For audit data, when a business query ends, the user thread directly obtains dynamic configuration data from global memory to generate a log header and stores it in shared memory, avoiding the overhead of protocol parsing and network interception by external probes. Immediately after the log header is written, the kernel thread is started to execute data acquisition. By parsing the configuration data, the target data is accurately located, and the raw data is directly extracted from the user thread's memory, eliminating the intermediate steps of buffer construction and disk I / O. For performance data, the monitoring indicator type is accurately specified based on the dynamic configuration data, and the execution indicators in the thread's memory are directly collected by the kernel thread, replacing the intrusive interference of traditional program instrumentation techniques on user threads. This data acquisition architecture not only achieves unified and efficient acquisition of audit data and performance data, but also avoids data loss under high load through a direct memory acquisition mechanism. At the same time, it utilizes the continuity of thread memory to ensure the complete correlation between long transaction data and performance indicators. This significantly reduces system resource consumption while ensuring the accuracy of audit data and the real-time performance monitoring, ultimately achieving a dual improvement in acquisition efficiency and data reliability. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an embodiment of a log data collection method provided in this application; Figure 2 A flowchart illustrating an embodiment of another log data collection method provided in this application; Figure 3 A schematic diagram of a log architecture provided in an embodiment of this application; Figure 4 A flowchart illustrating an embodiment of another log data collection method provided in this application; Figure 5 A schematic diagram of a log parsing architecture provided in an embodiment of this application; Figure 6 A schematic diagram of a data storage architecture provided in an embodiment of this application; Figure 7 A block diagram illustrating an embodiment of a log data acquisition device provided in this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0021] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0022] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0024] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0025] This application provides a log data collection method, which is applied to a database. See [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a log data collection method provided in this application. Figure 1 The process shown may include the following steps: Step 101: Upon detecting the end of a business query, trigger the user thread to retrieve configuration data from global memory, generate a log header based on the configuration data, and store the log header in shared memory; The configuration data is the metadata that dynamically controls the log collection behavior. It is used to specify the audit log strategy (such as sensitive fields such as operation type, user permissions, username of the operation, execution time, database name, SQL execution error code, client address, etc.) and the performance monitoring metric type (such as key parameters such as SQL execution time, CPU utilization, network bandwidth, number of concurrent connections, transaction success rate, etc.).

[0026] This can be understood as precisely controlling the scope of audit and performance data collection through configuration data. Its core configuration fields include: 1. `record type`: When configured as `AUDIT_ONLY`, only audit data collection is triggered; when configured as `PERF_ONLY`, only performance metrics are collected; and when configured as `AUDIT_PERF_MIX`, both audit and performance data are collected simultaneously; 2. `perf config`: As a fine-grained control layer, it supports sub-item selection configuration for audit data (such as user operation records and system events) and performance data (such as CPU load and memory usage), enabling on-demand collection; 3. `data length`: This limits the content range of a single record by the number of bytes, avoiding redundant invalid data while ensuring the complete storage of critical information. This three-dimensional configuration system, through its hierarchical design of type-content-length, balances the flexibility of data collection with storage efficiency.

[0027] The above configuration data can be dynamically configured according to actual needs. This dynamic configuration mechanism allows administrators to flexibly adjust the collection strategy without restarting the service, based on actual security compliance requirements and performance optimization goals (such as resource bottleneck analysis in high-concurrency scenarios).

[0028] In this embodiment, after the database completes the business query operation, the log generation process is triggered immediately: first, the query end signal is captured through the event listening mechanism; then, the user thread loads the above configuration data from the global memory in an atomic access manner, and dynamically constructs a log header structure that conforms to the specification based on the configuration data; finally, the generated log header is written to the shared memory area of ​​the database kernel through atomic operations, ensuring the thread safety and data consistency of log writing in a multi-threaded environment.

[0029] Step 102: When the log header write completion event is detected, a kernel thread is started to read the log header from the shared memory and parse the log header to extract configuration data. Log data is then collected from the user thread's thread memory based on the configuration data. The log data includes audit data and / or performance data. The user thread's thread memory stores all the raw data during the business query execution process. When the log header in shared memory is written, the database immediately wakes up the kernel thread via event listening (such as semaphores or callback mechanisms). This design implements an asynchronous, non-blocking log processing flow. After the kernel thread reads the binary-formatted log header from shared memory using system calls such as `do_syslog`, it needs to parse the structured fields—including record type, performance configuration items (perf config), and data length—to determine the collection scope and specific configuration. After parsing, the kernel thread accesses the user thread's thread-local storage (TLS) via atomic pointers to precisely filter out audit data (such as SQL statements and user operation records) and / or performance data (such as execution time and resource consumption statistics) that match the configuration data from the raw data generated during the entire business query execution process stored in the thread-local storage.

[0030] The entire data acquisition process is completed autonomously by the database kernel, without relying on external probe tools or detection code. It achieves millisecond-level response times through built-in event listening mechanisms and atomic operations. When a business query is executed, the database automatically caches audit data (such as SQL statements, user identities, and operation timestamps) and performance data (such as execution plan hash values, lock wait times, and CPU consumption statistics) in thread memory. After the query is completed, the kernel thread parses the log header configuration through shared memory and directly extracts the target data from the thread memory accurately with zero copying. This deep integration design not only eliminates the performance overhead and transmission latency caused by external tools but also ensures data consistency in a multi-threaded environment through memory barriers and atomic pointer operations, ultimately achieving sub-millisecond acquisition efficiency and data integrity assurance.

[0031] The log data collection method provided in this application comprehensively solves the technical challenges faced by traditional audit and performance data collection through a collaborative working mechanism between user space and kernel space. For audit data, when a business query ends, the user thread directly obtains dynamic configuration data from global memory to generate a log header and stores it in shared memory, avoiding the overhead of external probe protocol parsing and network interception. Immediately after the log header is written, the kernel thread starts data collection, accurately locating the target data by parsing the configuration data and directly extracting the raw data from the user thread's memory, eliminating the intermediate steps of buffer construction and disk I / O. For performance data, the monitoring indicator type is accurately specified based on the dynamic configuration data, and the kernel thread directly collects the execution indicators in the thread's memory, replacing the intrusive interference of traditional program instrumentation techniques on user threads. This collection architecture not only achieves unified and efficient collection of audit and performance data but also avoids data loss under high load through a direct memory acquisition mechanism. Simultaneously, it utilizes the continuity of thread memory to ensure the complete association between long transaction data and performance indicators, thereby significantly reducing system resource consumption while ensuring the accuracy of audit data and the real-time performance monitoring, ultimately achieving a dual improvement in collection efficiency and data reliability. In traditional database log collection architectures, systems typically employ a lock contention mechanism to write log data to shared memory. When multiple threads attempt to write simultaneously, serialized access is required via mutexes or semaphores. While this design is acceptable under low load, as the volume of concurrent database queries increases, frequent thread contention for lock resources leads to severe performance bottlenecks. On one hand, business query threads incur considerable context switching overhead while waiting for lock release; on the other hand, CPU cache misses and memory barriers caused by lock contention further amplify latency. To address these technical issues, this embodiment proposes a lock-free concurrent approach to writing log data to shared memory. For detailed implementation details, please refer to [link to implementation details]. Figure 2 , Figure 2 A flowchart illustrating another embodiment of the log data collection method provided by this invention. Figure 2 The process shown may include the following steps: Step 201: Upon detecting the end of a business query, trigger the user thread to retrieve configuration data from global memory, generate a log header based on the configuration data, and store the log header in shared memory; wherein, the configuration data is metadata that dynamically controls log collection behavior and is used to specify the audit log strategy and the performance monitoring metric type; Step 202: When the log header write completion event is detected, a kernel thread is started to read the log header from the shared memory and parse the log header to extract configuration data. Log data is collected from the user thread's thread memory according to the configuration data. The log data includes audit data and / or performance data. The user thread's thread memory stores all the original data during the business query execution process. For the specific implementation logic of steps 201 to 202, please refer to the complete execution flow of steps 101 to 102 in the aforementioned embodiment, which will not be repeated here.

[0032] Step 203: When the log data collection completion event is detected, the user thread is triggered to expand the information of the log header to obtain an enhanced log header; When the database detects that log data collection is complete via a memory barrier, the user thread initiates a log header extension process: First, it verifies the magic number and version number in the original log header, and then appends the following key metadata, including but not limited to: 1) a dynamically generated nanosecond-level timestamp (calibrated based on a TSC (Time Stamp Counter) clock source); 2) the NUMA (Non-Uniform Memory Access) node ID (dentifier) ​​to which the user thread belongs; and 3) a performance context identifier containing the CPU cache line size. This extended information is written to the reserved extension area in the log header through atomic operations, ensuring that data tearing does not occur in an SMP (Symmetric Multi-Processing) architecture.

[0033] Step 204: Assemble log records by combining log data and enhanced log headers according to a preset log architecture; To facilitate understanding of the log architecture, Figure 3 The diagram illustrates a log architecture, as shown below. Figure 3 As shown, the above-mentioned preset log architecture includes a log header range 301, a variable-length audit data range 302, a fixed-length audit data range 303, and a performance data storage range 304, for storing different types of data.

[0034] Based on the above-mentioned preset log architecture, the specific process of log recording in step 204 can be implemented through steps A1 to A6: Step A1: Write the enhanced log header to the log header range; Specifically, memory-mapped file technology can be used to write the enhanced log header to a pre-allocated log header range. This log header range can be designed with double buffering (active / shadow buffer) to avoid write conflicts.

[0035] Step A2: If performance data is detected in the log data, write the performance data to the performance data storage area. If performance data has been collected, it can be directly written to the performance data storage area. The writing process uses the TLV (Type-Length-Value) format: the type field occupies 2 bytes to identify the metric category (such as CPU / IO), the length field occupies 4 bytes using network byte order, and the value field is stored aligned to the L1 cache line size (usually 64 bytes).

[0036] Step A3: If audit data is detected to be included in the log data, obtain the data type field values ​​included in the audit data; The header structure of the audit data directly encodes the data type field value, which can be extracted by parsing the header after data collection. According to the encoding rules: when the data type field value falls within the fixed range of 0x0001-0x00FF, it is determined to be a fixed field type; if the data type field value is within the variable range of 0x0100-0xFFFF, it is determined to be a variable field type. This design allows for quick identification of whether the audit data has fixed or variable attributes based on the numerical range.

[0037] Step A4: Determine whether the data type field value falls within the fixed field value range; The essence of step A4 above is to determine whether the data type field value falls within the range of x0001-0x00FF.

[0038] Step A5: If the data type field value falls within the fixed field value range, write the audit data into the fixed length range of the audit data. If a data type field value falls within a fixed range (x0001-0x00FF), indicating that the audit data is of a fixed field type, then the audit data must be written into the fixed-length audit data range. In practical applications, data type field values ​​can clearly identify SQL execution code errors, the start time of execution, the number of rows returned, the number of rows scanned, and the number of rows updated—all of which are audit data of a fixed field type. Therefore, this audit data can be written into the fixed-length audit data range. Step A6: If it is determined that the data type field value does not fall within the fixed field value range, write the audit data into the variable-length audit data range.

[0039] When it is determined that the value of the data type field does not fall within the range of fixed field values, that is, the value of the data type field is in the range of 0x0100 - 0xFFFF, it indicates that the audit data is of variable field type. Then, the audit data needs to be written into the variable-length interval of the audit data. In actual applications, through the value of the data type field, the user name of the executed operation, the executed SQL statement, the client address accessing the RDS (Relational Database Service) instance, etc., which are all audit data of variable field type, can be clearly determined. Therefore, the above audit data can be written into the variable-length interval of the audit data.

[0040] Step 205, write the log record into the shared memory in a lock-free concurrent manner. The implementation process of the above step 205 can be specifically achieved through steps B1 to B5: Step B1, obtain the starting sequence number and the ending sequence number from the global memory; Read the starting sequence number (start_seq) and the ending sequence number (end_seq) from the global memory. These two sequence numbers are usually maintained by the built-in log generation module of the database and are used to identify the logical position range of the current log record.

[0041] Step B2, determine the physical starting offset of the shared memory through modulo operation based on the starting sequence number, and determine the physical ending offset of the shared memory through modulo operation based on the ending sequence number; Calculate the physical starting offset through the formula start_offset = start_seq % shared_mem_size, where shared_mem_size is the total size of the shared memory, and the modulo operation ensures that the offset is within the valid address range of the shared memory; similarly, calculate the physical ending offset through the formula end_offset = end_seq % shared_mem_size; this step maps the logical sequence number to the physical address space of the shared memory and supports circular writing (ring buffer).

[0042] Step B3, determine whether the physical ending offset is less than the physical starting offset; Compare the physical ending offset (end_offset) with the physical starting offset (start_offset): If end_offset >= start_offset, it means that the log record is continuously stored in the shared memory; if end_offset < start_offset, it means that the log record spans the end and start positions of the shared memory (ring buffer wrap-around).

[0043] Step B4, when it is determined that the physical end offset is equal to or greater than the physical start offset, write the log record into the shared memory once. When end_offset >= start_offset, directly write the log record continuously starting from start_offset, with a length of end_offset - start_offset. This operation is completed through one memory copy, with high efficiency.

[0044] Step B5, when it is determined that the physical end offset is less than the physical start offset, write the log record into the shared memory in two parts.

[0045] When end_offset < start_offset, it needs to be written in two parts. The first write is from start_offset to the end of the shared memory, writing the first part of the data, with a length of shared_mem_size - start_offset; the second write is from the start address of the shared memory to write the remaining data, with a length of len - (shared_mem_size - start_offset), where len is the data length of the log record. This design avoids buffer overflow and supports the efficient utilization of circular storage. [[ID=X]]

[0046] This solution adopts a lock-free writing mechanism, with very little and stable performance loss to the database, eliminating the performance loss caused by traditional lock competition, so that the data writing does not continuously increase as the load increases, thus ensuring the database performance.

[0047] In actual application, a dedicated log coordination thread will be created when the database starts. The log coordination thread is responsible for monitoring the shared memory in real time to asynchronously batch send the log records written into the shared memory to external components or write them into local files. The specific implementation process can be seen in Figure 4 , Figure 4 The shown process may include the following steps: Step 401, start the log coordination thread to poll the current consumption offset and the latest production offset in the global memory through atomic operations; Among them, the current consumption offset (entire_log_write_tail) can be understood as the position of the log record that the log coordination thread has currently processed, and the latest production offset (entire_log_write_head) is the position where the user thread has latest written. The consumption offset and the latest production offset are usually designed as global memory variables to coordinate the progress of production and consumption.

[0048] In practical applications, the log coordination thread maintains the current consumption offset and the latest production offset in global memory using atomic variables of type AtomicLong, and implements lock-free polling using a CAS (Compare-And-Swap) mechanism. Each time the thread polls, it atomically reads the two offsets using the get() method.

[0049] Step 402: Check if the current consumption offset is behind the latest production offset; The essence of detecting whether the current consumption offset lags behind the latest production offset is to detect whether the current consumption offset is less than the latest production offset.

[0050] Step 403: If the current consumption offset is detected to be behind the latest production offset, extract the valid log records between the current consumption offset and the latest production offset from the shared memory; If the current consumption offset lags behind the latest production offset (i.e., the current consumption offset is less than the latest production offset), it indicates that there are unprocessed log records in shared memory. The log coordination thread then retrieves valid log records from shared memory. The range of valid log records is defined by the current consumption offset and the latest production offset.

[0051] If the current consumption offset is not behind the latest production offset (i.e., the current consumption offset is equal to or greater than the latest production offset), it indicates that there are no valid log records to read (or the writing is incomplete). In this case, the system will block and wait until the latest production offset exceeds the current consumption offset (i.e., a valid log record has been written) before reading can proceed.

[0052] Step 404: Distribute valid log records to the parsing thread pool, use multiple log parsing threads in the parsing thread pool to call external parsing scripts to parse the valid log records, and write the parsed valid log records to a local file or send them to an external component.

[0053] The entire process can be achieved through Figure 5 The illustrated architecture diagram is as follows: Figure 5As shown, the log coordination thread 500 sequentially extracts valid log records from shared memory 501 and distributes them to the parsing thread pool 502 for parsing, on a record-by-record basis (single log entry or fixed-format block). The parsing thread pool 502 dynamically allocates tasks to the log parsing thread 503 using a work-stealing algorithm. Each log parsing thread calls an external parsing script 504 (such as Lua / Python) via FFI (Foreign Function Interface) to process valid log records. The parsed valid log records are then persistently stored by being written in batches to local files 505, or sent in batches to an external component (such as Kafka) 506 for subsequent in-depth database analysis by retrieving these valid log records.

[0054] In this embodiment, since the external parsing script runs independently of the database program, the valid log records are parsed by calling the external parsing script. This design separates the standardized logic from the core business logic of the database, and the database codebase does not need to be touched when modifying the script.

[0055] The aforementioned parsing thread pool has built-in backpressure control, meaning that it will automatically expand or shrink when the parsing speed is unbalanced with the writing speed. The specific implementation process can be achieved through steps C1 to C5: Step C1: Through a timed sampling mechanism, collect the set of log offsets parsed by the log parsing thread and synchronously read the set of consumption offsets recorded in global memory. The log parsing thread is polled at fixed time intervals (e.g., every second) to collect the processed log offsets, forming an ordered set {lowest_parsed_elsn_1, lowest_parsed_elsn_2, ..., lowest_parsed_elsn_n}. The set of consumed offsets {entire_log_write_tail_1, ..., entire_log_write_tail_m} recorded in global memory is synchronized to ensure data timeliness. The aforementioned fixed time interval needs to balance real-time performance with performance overhead (typically 100ms-1s, adjustable).

[0056] Step C2: Determine the write speed based on multiple log offsets in the log offset set, and determine the parsing speed based on multiple current consumption offsets in the consumption offset set; Write speed is calculated based on the time window of the log offset set (e.g., the last 5 samples) using the following formula: Write speed = (latest_parsed_elsn - oldest_parsed_elsn) / time window span.

[0057] Similarly, the parsing speed is determined by the rate of change of the current consumption offset, calculated using the following formula: Parsing speed = (latest entire_log_write_tail - oldest entire_log_write_tail) / time window span.

[0058] In practical applications, a weighted algorithm can also be used to calculate the write speed and parsing speed, which will not be elaborated here.

[0059] Step C3: Detect the numerical relationship between write speed and parsing speed; Real-time comparison of write speed and parsing speed: If write speed > parsing speed: parsing capacity is insufficient, potentially leading to an accumulation of unprocessed logs. If write speed < parsing speed: parsing resources are idle, indicating room for optimization.

[0060] Step C4: If the write speed is detected to be greater than the parsing speed, dynamically expand the size of the parsing thread pool until the parsing speed reaches or exceeds the write speed. When the write speed consistently exceeds the parsing speed (e.g., after three consecutive samples), initiate capacity expansion. Expansion strategy: 1. Increase the number of core threads in the thread pool by a preset step size (e.g., +2 threads each time); 2. Monitor changes in parsing speed; if still insufficient, continue expansion until the upper limit is reached (e.g., maximum number of threads = number of CPU cores × 2). It is important to note that database load (e.g., CPU and memory utilization) should also be considered during expansion to avoid over-expansion.

[0061] Step C5: If the write speed is detected to be lower than the parsing speed, dynamically reduce the size of the parsing thread pool until the write speed reaches the parsing speed.

[0062] When the write speed consistently falls below the parsing speed (e.g., after 5 consecutive samples), initiate scaling down. Scaling down strategy: 1. Gradually reduce idle threads (e.g., -1 thread each time) until the parsing speed is approximately equal to the write speed; 2. Maintain the minimum number of threads (e.g., 2 threads) to handle sudden traffic surges. Note that before scaling down, wait for active tasks to complete to avoid forced interruptions.

[0063] In this embodiment, the number of parsing threads can be adjusted according to the database query load to match the consumption parsing speed in different scenarios, ultimately achieving the optimal balance between resource utilization and database stability.

[0064] In practical applications, since the shared memory space is fixed in size, it needs to be reused cyclically to ensure continuous writing of logs. This requires cleaning up and reclaiming written log records at appropriate times to ensure that the shared memory does not become unusable due to space exhaustion. The specific data reclamation process can be implemented through steps D1 to D2: Step D1: Start the log buffer management thread to detect in real time whether the parsed log offset is greater than the latest production offset of the valid log record; A dedicated log buffer management thread is also created when the database starts up, which is responsible for managing shared memory. In this embodiment, the log buffer management thread continuously compares the log offset with the latest production offset of the valid log record through atomic operations.

[0065] Step D2: If the log offset is detected to be greater than the latest production offset of the valid log record, the log records between the log offset and the latest production offset are cleaned up and reclaimed from the shared memory. When the log offset is greater than the latest production offset of a valid log record, it is determined that the log parsing thread has finished processing the log records between entire_log_write_head and lowerest_parsed_elsn. These log records no longer need to be kept in shared memory, so it is necessary to clean up and reclaim the log records between these two offsets to ensure that there is enough space in shared memory to continue writing subsequent log records.

[0066] The aforementioned log buffer management thread can not only clean up and reclaim log records in shared memory, but also maintain the physical contiguousness of the memory region between the current consumption offset and the latest production offset when writing log records.

[0067] Specifically, this is achieved by adding a link_buf loop array to advance the entire_log_write_tail, ensuring that there are no gaps in the memory address before the entire_log_write_head.

[0068] For example, such as Figure 6 Assuming session 0 writes a record of length 3 with start_elsn:1, then there is a corresponding record link[1]=3 in link_buf. Therefore, the next session 1 has link[4]=2, and link[6]=0. Figure 6 (A blank indicates that the writing is not complete, and a horizontal line indicates that the writing is complete.) If the writing is not yet complete, the process will stop until link[6] is written, so as to ensure that there are no gaps in the memory area between entire_log_write_head and entire_log_write_tail (i.e., all positions have been written).

[0069] This application provides a log data acquisition device, which is applied to a database. See [link to documentation]. Figure 7This is a block diagram illustrating an embodiment of a log data acquisition device provided in this application. Figure 7 As shown, the device includes: The acquisition and storage module 701 is used to trigger the user thread to retrieve configuration data from global memory when the business query is detected to be finished, generate log headers based on the configuration data, and store the log headers in shared memory; wherein, the configuration data is the meta-information that dynamically controls the log collection behavior and is used to specify the audit log strategy and the performance monitoring metric type; The read parsing and acquisition module 702 is used to start a kernel thread to read the log header from shared memory when the log header write completion event is detected, and to parse the log header to extract configuration data. Based on the configuration data, log data is collected from the thread memory of the user thread. The log data includes audit data and / or performance data. The thread memory of the user thread stores all the raw data in the business query execution process.

[0070] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0071] As an optional implementation, the above-described apparatus further includes: The extension module is used to trigger a user thread to extend the log header when a log data collection completion event is detected, resulting in an enhanced log header; The component module is used to assemble log data and enhanced log headers into log records according to a preset log architecture. The write module is used to write log records to shared memory in a lock-free concurrent manner.

[0072] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0073] The default log architecture includes a log header area, a fixed-length audit data area, a variable-length audit data area, and a performance data storage area; as an optional implementation, the above-mentioned components are also used for: Write the enhanced log header to the log header range; If performance data is detected in the log data, the performance data will be written to the performance data storage area. If audit data is detected to be included in the log data, retrieve the data type field values ​​included in the audit data; Determine whether the value of a data type field falls within a fixed range of field values; If the data type field value is determined to fall within a fixed field value range, the audit data will be written into a fixed-length audit data range. If the data type is determined not to fall within the fixed field value range, the audit data will be written into the variable-length audit data range.

[0074] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0075] As an optional implementation, the above-described writing module is also used for: Retrieve the start and end sequence numbers from global memory; The physical start offset of the shared memory is determined by modulo operation based on the start sequence number, and the physical end offset of the shared memory is determined by modulo operation based on the end sequence number. Determine if the physical end offset is less than the physical start offset; If the physical end offset is determined to be equal to or greater than the physical start offset, the log record will be written to the shared memory in a single transaction. If the physical end offset is determined to be less than the physical start offset, the log record is written to the shared memory in two separate steps.

[0076] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0077] As an optional implementation, the above-described apparatus further includes: The polling module is started to initiate the log coordination thread to poll the current consumption offset and the latest production offset in global memory through atomic operations; The detection module is used to detect whether the current consumption offset is behind the latest production offset; The extraction module is used to extract valid log records between the current consumption offset and the latest production offset from shared memory when the current consumption offset is detected to be lagging behind the latest production offset. The parsing module distributes valid log records to the parsing thread pool. It uses multiple log parsing threads in the parsing thread pool to call external parsing scripts to parse the valid log records and writes the parsed valid log records to a local file or sends them to an external component.

[0078] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0079] As an optional implementation, the above-described apparatus further includes: The collection and reading module is used to collect the set of log offsets parsed by the log parsing thread through a timed sampling mechanism, and synchronously read the set of consumption offsets recorded in global memory. The determination module is used to determine the write speed based on multiple log offsets in the log offset set, and to determine the parsing speed based on multiple current consumption offsets in the consumption offset set. The numerical relationship detection module is used to detect the numerical relationship between the writing speed and the parsing speed; An extension module is used to dynamically expand the size of the parsing thread pool when the write speed is detected to be greater than the parsing speed, until the parsing speed reaches or exceeds the write speed; The reduction module is used to dynamically reduce the size of the parsing thread pool when the write speed is detected to be lower than the parsing speed, until the write speed reaches the parsing speed.

[0080] Specifically, the detailed process by which each module in the device of this invention implements its function can be found in the relevant description in the method embodiment, and will not be repeated here.

[0081] As an optional implementation, the above-described apparatus further includes: The offset detection module is used to start the log buffer management thread and detect in real time whether the offset of the parsed log is greater than the latest production offset of the valid log record. The cleanup and reclamation module is used to clean up and reclaim log records between the current log offset and the latest production offset from shared memory when the log offset is detected to be greater than the latest production offset of the valid log records. The log buffer management thread also plays a role in maintaining the physical contiguousness of the memory area between the current consumption offset and the latest production offset.

[0082] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 The illustrated electronic device 1200 includes at least one processor 1201, a memory 1202, at least one network interface 1204, and other user interfaces 1203. The various components in the electronic device 1200 are coupled together via a bus system 1205. It is understood that the bus system 1205 is used to implement communication between these components. In addition to a data bus, the bus system 1205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 8 The general labeled all buses as Bus System 1205.

[0083] The user interface 1203 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0084] It is understood that the memory 1202 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1202 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0085] In some implementations, memory 1202 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 12021 and application program 12022.

[0086] The operating system 12021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 12022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this application embodiment can be included in the application program 12022.

[0087] In this embodiment of the application, the processor 1201 executes the method steps provided by each method embodiment by calling the program or instructions stored in the memory 1202, specifically the program or instructions stored in the application program 12022.

[0088] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1201. The processor 1201 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the processor 1201. The processor 1201 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software unit can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1202. Processor 1201 reads the information in memory 1202 and, in conjunction with its hardware, completes the steps of the above method.

[0089] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0090] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0091] The electronic device provided in this embodiment may be as follows: Figure 8 The electronic device shown can perform the following: Figure 1-2 This involves all the steps of the log data collection method in section 4, thereby achieving... Figure 1-2 For details on the technical effectiveness of the log data collection method shown in Figure 4, please refer to [reference needed]. Figure 1-2 The relevant descriptions in section 4 are for the purpose of brevity and will not be elaborated upon here.

[0092] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0093] The above log data collection method can be implemented when one or more programs in the storage medium can be executed by one or more processors.

[0094] The processor is used to execute the log data acquisition program stored in the memory to implement the steps of the log data acquisition method.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0097] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log data collection method characterized by, The method is applied to a database, and the method comprises: When detecting that the service query ends, triggering a user thread to obtain configuration data from a global memory, generating a log head based on the configuration data, and storing the log head into a shared memory; wherein the configuration data is meta information for dynamically controlling log collection behavior, and is used for specifying an audit log strategy and an index type of performance monitoring; When listening to the log head write completion event, starting a kernel thread to read the log head from the shared memory, and parsing the log head to extract the configuration data, and collecting log data from the user thread memory according to the configuration data; wherein the log data comprises audit data and / or performance data, and the user thread memory stores all original data in the service query execution process.

2. The method of claim 1, wherein, The method further comprises: When detecting the log data collection completion event, triggering the user thread to perform information extension on the log head to obtain an enhanced log head; The log data and the enhanced log head are combined into a log record according to a preset log architecture; The log record is written into the shared memory in a lock-free concurrent manner.

3. The method of claim 2, wherein, The preset log architecture comprises a log head interval, an audit data fixed length interval, an audit data variable length interval, and a performance data storage interval; The combination of the log data and the enhanced log head into a log record according to the preset log architecture comprises: The enhanced log head is written into the log head interval; When detecting that the log data comprises performance data, the performance data is written into the performance data storage interval; When detecting that the log data comprises audit data, a data type field value included in the audit data is obtained; It is determined whether the data type field value falls within a fixed field value range; When it is determined that the data type field value falls within the fixed field value range, the audit data is written into the audit data fixed length interval; When it is determined that the data type field value does not fall within the fixed field value range, the audit data is written into the audit data variable length interval.

4. The method of claim 2, wherein, The writing of the log record into the shared memory in a lock-free concurrent manner comprises: A starting sequence number and an ending sequence number are obtained from the global memory; A physical starting offset of the shared memory is determined by a modulo operation based on the starting sequence number, and a physical ending offset of the shared memory is determined by a modulo operation based on the ending sequence number; It is determined whether the physical ending offset is less than the physical starting offset; When it is determined that the physical ending offset is equal to or greater than the physical starting offset, the log record is written into the shared memory at a time; When it is determined that the physical ending offset is less than the physical starting offset, the log record is written into the shared memory twice.

5. The method of claim 2, wherein, The method further comprises: Starting a log coordination thread to poll a current consumption offset and a latest production offset in the global memory through an atomic operation; It is detected whether the current consumption offset lags behind the latest production offset; In a case where it is detected that the current consumption offset is behind the latest production offset, effective log records between the current consumption offset and the latest production offset are extracted from the shared memory; The effective log records are distributed to a parsing thread pool, a plurality of log parsing threads in the parsing thread pool invoke an external parsing script to parse the effective log records, and the parsed effective log records are written into a local file or sent to an external component.

6. The method of claim 5, wherein, The method further comprises: A timing sampling mechanism is used to collect a set of log offsets parsed by the log parsing threads and synchronously read a set of consumption offsets recorded in the global memory; A write speed is determined based on a plurality of log offsets in the set of log offsets, and a parsing speed is determined based on a plurality of current consumption offsets in the set of consumption offsets; The numerical relationship between the write speed and the parsing speed is detected; In a case where it is detected that the write speed is greater than the parsing speed, the size of the parsing thread pool is dynamically expanded until the parsing speed reaches or exceeds the write speed; In a case where it is detected that the write speed is less than the parsing speed, the size of the parsing thread pool is dynamically reduced until the write speed reaches the parsing speed.

7. The method of claim 6, wherein, The method further comprises: A log buffer management thread is started to detect in real time whether the log offset parsed is greater than the latest production offset of the effective log record; In a case where it is detected that the log offset is greater than the latest production offset of the effective log record, log records between the log offset and the latest production offset are cleaned up from the shared memory; the log buffer management thread also maintains a memory region between the current consumption offset and the latest production offset to always meet the requirement of physical continuity.

8. A log data collection apparatus characterized by comprising: The device is applied to a database, and the device comprises: An acquisition generation module is configured to, when a business query ends, trigger a user thread to acquire configuration data from a global memory, generate a log header based on the configuration data, and store the log header into a shared memory; the configuration data is meta information for dynamically controlling log collection behavior, and is used to specify an audit log strategy and an index type of performance monitoring; A reading and parsing collection module is configured to, when a log header write completion event is listened to, start a kernel thread to read the log header from the shared memory, parse the log header to extract the configuration data, and collect log data from a thread memory of the user thread according to the configuration data; the log data includes audit data and / or performance data, and the thread memory of the user thread stores all original data in a business query execution process.

9. An electronic device, comprising: The device comprises: A processor and a memory, the processor is configured to execute a program of log data collection stored in the memory to implement the log data collection method in any one of claims 1-7. The device comprises: A processor and a memory, the processor is configured to execute a program of log data collection stored in the memory to implement the log data collection method in any one of claims 1-7.

10. A storage medium, characterized by The storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the log data collection method in any one of claims 1-7.

Images