Automatic operation and maintenance method for audit system database

Abstract

This application relates to the field of database operation and maintenance technology, specifically to an automated operation and maintenance method for audit system databases. This method includes: periodically collecting inventory levels and thresholds at each level of an LSM-Tree, and calculating the level capacity occupancy ratio; using eBPF technology to statistically analyze background I / O latency and traffic within a sliding window, and calculating the unit latency for background disk writes; performing recursive calculations in reverse order using the level capacity occupancy ratio to obtain the end-to-end logical write multiple; analyzing the coupling relationship between the level capacity occupancy ratios of adjacent levels, determining the multi-level concurrent congestion coefficient, obtaining the estimated unit data occupancy time, and inversely deriving the allowed front-end write rate; and combining this with a token bucket algorithm to smoothly limit the write requests of the front-end audit system. The goal of this application is to achieve early detection of backlog trends within the LSM-Tree and to automatically perform predictive traffic control to optimize the operation and maintenance management of the audit system database.

Description

Technical Field

[0001] This application relates to the field of database operation and maintenance technology, specifically to an automated operation and maintenance method for audit system databases. Background Technology

[0002] Audit log systems are a core component of enterprise information security, responsible for recording all business operations. Such systems typically use a distributed database based on a log merging tree (LSM-Tree) structure as their storage engine, leveraging its ability to convert random writes into sequential writes to meet high-throughput data collection needs.

[0003] In the LSM-Tree architecture, data is first written to memory (MemTable), and then flushed to disk to form Level-0 files. A background process (Compaction) is responsible for merging shallow files level by level into deeper levels (Level-1 to Level-N) to free up space and optimize query efficiency. In real-world production environments, the write traffic of foreground audit logs often exhibits sudden spikes. When the write speed consistently exceeds the background merging speed, it can lead to a surge in the number of Level-0 files or a backlog of data in the intermediate layers.

[0004] Existing database kernels typically have a write-stop mechanism. When backlog reaches a hard threshold, it forcibly reduces the foreground write speed to extremely low levels or even halts it, causing audit data collection to be interrupted or lost. Traditional operation and maintenance methods mainly rely on static rate limiting or dynamic adjustments based on general metrics such as CPU utilization and disk I / O utilization. However, general metrics have significant lag, only reflecting the current physical load and failing to detect deep-seated backlog trends within the storage engine. Often, by the time monitoring metrics trigger alarms, severe cascading backlog has already occurred within the database, and intervention in flow control at this point is often too late to avoid kernel-level forced shutdowns. Summary of the Invention

[0005] In light of the above, it is necessary to provide an automated operation and maintenance method for the audit system database to solve the aforementioned problems.

[0006] One embodiment of this application provides an automated operation and maintenance method for an audit system database, the method comprising: Compare the input and output data volumes of backend data compression and sorting during historical operations and maintenance to determine the data hierarchy merging expansion coefficient; analyze the performance degradation of storage media under multi-threaded concurrent writing conditions compared to single-threaded writing to determine the contention sensitivity coefficient. Analyze the inventory and threshold of each level of the LSM-Tree storage engine in the audit database to determine the capacity occupancy ratio of each level at the current moment; use eBPF technology to capture the time and traffic of background I / O based on a preset sliding window to calculate the unit time of background disk write at the current moment. Using the hierarchical capacity occupancy ratio and the hierarchical merging expansion coefficient, recursive calculation is performed in reverse order of the hierarchical index to obtain the full-link logical write multiple; the coupling relationship of the hierarchical capacity occupancy ratio between adjacent hierarchical levels is analyzed, and the multi-level concurrent congestion coefficient is determined in combination with the contention sensitivity coefficient; based on the full-link logical write multiple, the multi-level concurrent congestion coefficient, and the background disk write unit time, the estimated unit data occupancy time is obtained; Based on the estimated time occupied by the unit data and the preset proportion of total disk time occupied by allowed data writing and sorting operations, the allowed write rate for the front end is deduced in reverse. Combined with the token bucket algorithm, the write requests audited by the front end are smoothly limited.

[0007] Specifically, the hierarchical merging expansion coefficient of the determined data is the ratio between the output data volume and the input data volume of all data compression and sorting tasks within a preset period.

[0008] The specific formula for determining the contention sensitivity coefficient is as follows: ;in, The value represents the contention sensitivity coefficient, and max() represents the maximum value function; This represents the maximum number of background merge threads configured in the database. This represents the number of input / output operations per second performed by a single write thread for sequential writes; express The number of input / output operations per second performed by each write thread in a concurrent sequential write operation; This indicates the preset value.

[0009] The specific process for determining the capacity occupancy ratio of each level at the current moment is as follows: For Level-0: the total number of currently resident SSTable files is used as the current actual disk space occupied; for Level-1 to Level-N: the total number of disk bytes currently occupied at each level is used as the current actual disk space occupied. The preset trigger thresholds for each level are used as the level thresholds; The ratio between the current actual occupancy of each level and its corresponding level threshold is used as the level capacity occupancy ratio at the current moment.

[0010] Specifically, the calculation of the current time's background disk write-to-disk unit time is as follows: Within the preset time window of the current time t, count the total amount of data successfully written to the storage medium by the statistical thread. and total cumulative time Preset minimum statistical sample threshold and default baseline value ; The time taken to write data to disk at the current time t. The specific formula is: Where t represents time. This represents the time taken to download data to disk in the background at time t-1.

[0011] The process of obtaining the end-to-end logical write multiple is as follows: Set the cumulative magnification of Level-N to 1; Obtain the minimum value between the level capacity occupancy ratio of Level-i at the current moment and the preset saturation upper limit constant, and calculate the product between the obtained minimum value and the level merging expansion coefficient and the cumulative amplification factor of Level-(i+1); add the obtained product to the value corresponding to the write action that must occur when data is processed at Level-i, and obtain the cumulative amplification factor of Level-i at the current moment. Execute recursive computation up to the top level. The cumulative amplification factor obtained at that time is used as the full-link logical write factor at the current moment.

[0012] The specific formula for determining the multi-level concurrent congestion coefficient is as follows: In the formula: This represents the multi-level concurrent congestion coefficient at the current time t; To compete for the sensitivity coefficient; This is the preset baseline coefficient under a no-contest state; , These represent the level capacity occupancy ratios of Level-i and Level-(i+1) at the current time t, respectively. This represents the preset upper limit constant for saturation; min() represents the minimum value function.

[0013] Specifically, the estimated time taken for unit data is the result of a positive fusion of the full-link logical write multiple, the multi-layer concurrency congestion coefficient, and the unit time taken for backend disk write at the current moment.

[0014] The specific process of reverse deducing the allowed write rate of the foreground is as follows: Get the preset percentage of total disk time used for allowed data writing and defragmentation operations, as well as the preset physical transfer bandwidth limit; Obtain the maximum value between the estimated time taken for a unit of data at the current time t and the preset minimum unit time constant. Use the ratio between the proportion of the total disk time occupied by the allowed data writing and sorting operations and the obtained maximum value as the initial calculation rate. Based on the value of the level-0 capacity occupancy ratio, a preventive attenuation coefficient is set. Obtain the minimum value between the initial calculation rate and the physical transmission bandwidth limit at the current moment. Multiply the obtained minimum value by the corresponding preventive attenuation coefficient and use it as the foreground allowed write rate at the current moment.

[0015] Specifically, the setting of the preventive attenuation coefficient is as follows: If the level-0 capacity occupancy ratio is greater than the first safety warning line, the preventive attenuation coefficient is set to the first preset value; if the level-0 capacity occupancy ratio is less than the second safety warning line, the preventive attenuation coefficient is set to the second preset value; if the level-0 capacity occupancy ratio is less than the first safety warning line but greater than the second safety warning line, the preventive attenuation coefficient remains consistent with the value of the previous moment, wherein the initial value of the preventive attenuation coefficient is 1, the second preset value is greater than the first preset value, and the second safety warning line is less than the first safety warning line.

[0016] This application has at least the following beneficial effects: First, this application addresses the fact that existing technologies only focus on the current level's backlog status and cannot predict the chain reaction after data sinks. Based on this, a recursive algorithm is proposed to derive the end-to-end logical write multiplier from bottom to top (from Level-N to Level-0). This feature can convert the potential backpressure risk at deep levels into the write cost at the current moment. Through end-to-end recursive derivation, it can overcome the time difference of data merging and identify future load risks caused by underlying backpressure in advance, realizing a shift from "passive response" to "predictive intervention." This allows for early detection and limitation of traffic before physical congestion occurs, solving the problem of control lag.

[0017] Secondly, existing technologies often overlook resource contention when multiple levels of merging tasks are executed simultaneously. This application introduces a multi-level concurrency congestion coefficient, using the product of the capacity occupancy ratios of adjacent levels to characterize the probability of physical I / O resource contention. By introducing the concurrency congestion coefficient, disk performance degradation caused by multi-level merging can be dynamically detected and compensated. In high-concurrency scenarios, flow control instructions can automatically tighten, avoiding performance avalanche caused by resource contention. This feature can map logical hierarchical coupling relationships to physical performance degradation, correcting the deviation of a single physical benchmark in concurrent scenarios.

[0018] Finally, this application establishes a "time cost" model, unifying logical workload and physical processing capacity into a time dimension. By using a preset background processing time budget ratio, the allowed write rate for the front-end is calculated in reverse. This feature ensures that the rate at which the front-end generates I / O debt is always within the budgeted range of the back-end's processing capacity, eliminating the physical conditions that trigger WriteStall at the root. The smooth flow control mechanism based on time budget replaces the rigid forced pauses of the database kernel, smoothly stretching bursts of traffic through micro-flow shaping, ensuring the continuity and completeness of audit log collection. Attached Figure Description

[0019] Figure 1 A flowchart of the automated operation and maintenance method for an audit system database provided in this application; Figure 2 This is a schematic diagram illustrating the acquisition of the front-end allowed write rate provided in this application. Detailed Implementation

[0020] In the description of the embodiments in this application, the words "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary," "or," and "for example" is intended to present the relevant concepts in a specific manner.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0022] It should also be noted that the terms "first" and "second" in this application and its accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The methods disclosed in the embodiments of this application or the methods shown in the flowcharts include one or more steps for implementing the method. Without departing from the scope of protection of this application, the execution order of multiple steps can be interchanged, and some steps can also be deleted.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] This application proposes an automated operation and maintenance method for audit system databases, applied in the field of database operation and maintenance technology. (See attached document.) Figure 1 The method includes the following steps: Step 1: Compare the input and output data volumes of the backend data compression and sorting in historical operations and maintenance to determine the data hierarchy merging expansion coefficient; analyze the performance degradation of the storage medium under multi-threaded concurrent writing conditions compared to single-threaded writing to determine the contention sensitivity coefficient.

[0025] In this application, this step aims to provide an operating environment and physical baseline for the subsequent automated operation and maintenance model. Since different hardware environments (such as SSDs and HDDs) and different business load characteristics have a significant impact on model parameters, a baseline calibration process must be performed before the audit system officially runs the flow control logic.

[0026] The executing entity of this application is an automated operation and maintenance agent deployed on a database server. In other embodiments, it can also be a proxy middleware integrated into the audit log collection terminal, hereinafter referred to as the system. In this embodiment, state interaction with the database kernel is performed through a network connection, and the block device driver layer of the operating system is monitored through an eBPF (Extended Berkeley Packet Filter) probe. In other embodiments, state interaction with the database kernel can also be performed through local IPC (Inter-Process Communication).

[0027] To ensure the accuracy of the prediction model, the system needs to obtain three core parameters: the hierarchical merging inflation coefficient. Sensitivity coefficient for competition and the percentage of total disk time allowed for data write and defragmentation operations. .

[0028] First, the hierarchical merging expansion coefficient When the representation data is merged from Level-i to Level-(i+1) of the LSM-Tree, the size of the newly generated data file in the target layer is equivalent to the average volume change factor of the input data size due to the cleaning of invalid data (GC) and sorting and merging operations.

[0029] The system utilizes historical operational data to compile metadata for all background data compression and processing tasks within a complete past maintenance cycle (set to 7 days in this embodiment). The system then sums up the input data volume for each data compression and processing task. With output data volume The ratio between the output data volume and the input data volume is used as the hierarchical merging expansion coefficient.

[0030] It should be noted that if there is no historical data, the system can perform a stress test, continuously writing random data to the database until the merge from Level-0 to Level-1 is triggered, and the input-output ratio of this merge is recorded as... The initial value (usually taking a range of values) ).

[0031] Secondly, the competition for sensitivity coefficient This value characterizes the performance degradation of storage media under multi-threaded concurrent write conditions compared to single-threaded write conditions. For mechanical hard drives with poor random read / write performance, this value is larger; for NVMe SSDs, this value is smaller. The system uses disk benchmarking tools (such as fio) to perform the following tests during idle periods: (1) Single-threaded benchmark test: Start a single write thread to perform sequential writes, and record its number of input / output operations per second (IOPS), denoted as . .

[0032] (2) Multi-threaded concurrency test: Start One write thread (in this embodiment, The value is the maximum number of background merge threads configured in the database. Concurrent sequential writes are performed, and the total IOPS is recorded. .

[0033] Sensitivity coefficient of competition The specific calculation formula is as follows: Where max() represents the maximum value function; This represents a preset value, which is 0.01 in this embodiment to prevent the denominator from being 0; this formula quantifies the loss rate relative to ideal linear scaling. For example, if concurrent writes cause a 20% decrease in total IOPS, then... .

[0034] Finally, the percentage of total disk time allowed for data write and defragmentation operations. This represents the percentage of total disk I / O time that operations and maintenance personnel are allowed to use for background merge tasks. The setting of this parameter needs to balance foreground write latency with the timeliness of background data processing. If set too high, it may crowd out the write bandwidth of the foreground WAL (Write-Ahead Log), increasing write latency; if set too low, the background merge speed will be limited, easily leading to long-term backlogs. Based on standard engineering settings, it is recommended to set this parameter to [insert value here]. In this embodiment, the value is set within the range of [0.5, 0.8]. This means that background tasks are allowed to use 70% of the disk time budget, while 30% is reserved for foreground writes and critical system processes.

[0035] After the above parameters are calibrated, the system enters a real-time monitoring and flow control cycle.

[0036] Step 2: Analyze the inventory and threshold of each level of the LSM-Tree storage engine in the audit database to determine the current level capacity occupancy ratio; use eBPF technology to capture the background I / O time and traffic based on a preset sliding window, and calculate the current background disk write unit time.

[0037] In this application, this step aims to establish a dual awareness of the system's internal logical backlog status and the underlying physical disk processing capabilities. Because the LSM-Tree storage engine used by the audit database has a hierarchical and heterogeneous nature, and the write performance of the physical disk fluctuates with the load, it is necessary to normalize the heterogeneous inventory metrics and convert dynamic hardware performance into a standardized time cost benchmark.

[0038] Because the LSM-Tree storage engine uses different merge triggering mechanisms for different levels, the top-level data (usually labeled Level-0) is in an unordered stacked state, and its merge triggering condition is typically based on the number of files; while the data in deeper levels (Level-1 to Level-N) is in an ordered state, and its merge triggering condition is based on the total size of the data. To uniformly measure the pressure trend of data being dumped from each level to the next, the system at the current moment... These heterogeneous statistical values ​​are transformed into a unified dimensionless index.

[0039] Specifically, the system calls the statistics interface provided by the database kernel (such as RocksDB's GetIntProperty) to traverse each level of the storage engine. ( ): For Level-0 ( The system reads the total number of currently resident SSTable files as the current actual usage. It reads the preset Level-0 file count trigger threshold from the configuration file as the level threshold. .

[0040] For Level-1 to Level-N ( The system reads the total number of disk bytes currently occupied at this level as the current actual usage. It reads the preset data size trigger threshold for this level from the configuration file and uses it as the level threshold. .

[0041] After obtaining the aforementioned basic data, the comparison result between the current actual occupancy of each level and the level threshold is used as the level capacity occupancy ratio of each level. In this embodiment, the level capacity occupancy ratio of Level-i at the current time t is denoted as... The specific formula in this embodiment is: This indicator It is a dimensionless numerical value characterizing the degree of hierarchical saturation. When When this occurs, it indicates that the level is saturated and a merge operation is about to be initiated or is in the process of initiating at the next lower level.

[0042] To obtain a realistic and effective computational benchmark, the system utilizes eBPF technology provided by the operating system, mounted on key functions (such as blk_account_io_done) in the block device driver layer. The system sets a non-overlapping sliding statistical time window. In this embodiment, The implementer can make adjustments according to the actual situation.

[0043] Within this time window, the probe captures all completed write I / O events and filters out operation records belonging only to the database background merge and flush threads by process identifier. The system then calculates the total amount of data successfully written to the storage medium by these threads. (Unit: MB) and total cumulative time (Unit: seconds).

[0044] To prevent computational noise caused by background I / O silence or minimal activity (i.e., inaccurate results due to insufficient sample size), the system introduces a minimum statistical sample threshold. (Standard value: 10MB) and default baseline value (The standard value is 0.005s / MB).

[0045] Calculate the time taken per unit of background disk write at the current time t. The specific formula is: Where t represents time.

[0046] It should be understood that when the backend write volume is sufficient ( When there is only a small amount of background writing (), the system updates the baseline value in real time to reflect the current true performance of the hardware; when there is only a small amount of background writing (), To avoid random errors, the system uses the calculated values ​​from the previous cycle when the system is first started; when the system starts up, it uses the preset default values. This ensures that the model has initial computational capabilities during the cold start phase.

[0047] Calculated This characterizes the actual physical time required for the back-end storage subsystem to process 1MB of data under the current hardware environment.

[0048] Step 3: Using the layer capacity occupancy ratio and the layer merging expansion coefficient, perform recursive calculations in reverse order of the layer index to obtain the full-link logical write multiple; analyze the coupling relationship of the layer capacity occupancy ratio between adjacent layers, and determine the multi-layer concurrency congestion coefficient by combining the contention sensitivity coefficient; based on the full-link logical write multiple, the multi-layer concurrency congestion coefficient, and the background disk write unit time, obtain the estimated unit data occupancy time.

[0049] In this application, this step aims to dynamically combine the static inventory indicators obtained in the previous steps with the physical baseline to derive the true cost of front-end write operations. LSM-Tree write operations exhibit significant lag and cascading characteristics; the current Level-0 write is merely the beginning of the data lifecycle, and subsequent data migration down levels will continuously consume the system's I / O bandwidth. Therefore, the system synthesizes a comprehensive cost indicator that can uniformly represent both logical workload and physical execution efficiency through recursive calculation and coupling analysis.

[0050] To accurately assess the total backend logical write volume triggered by writing 1MB of data in the foreground over its entire lifecycle, a bottom-up recursive logic must be used for derivation. This metric represents "the total amount that a current write debt will need to be repaid in the future."

[0051] The system utilizes the obtained capacity occupancy ratios at each level Combined expansion coefficient with calibrated hierarchical levels According to hierarchical index from arrive Perform recursive calculations in reverse order: (1) Determine the recursion boundary ( The lowest level (Level-N) of the storage engine is the final destination of data; this level does not merge data into deeper levels. Therefore, writing a unit of data to Level-N only generates its own write operation. System settings: .

[0052] (2) Perform reverse recursion ( For any 1 Each level has a cumulative amplification factor consisting of "1 unit of workload written to this level" and "the expected workload passed to the next level." The expected workload passed to the next level depends on the cumulative amplification factor of the next level. Hierarchical merging expansion coefficient And the probability weights for downward merging in this layer.

[0053] To prevent the calculation results from diverging due to abnormal backlog of data in a single layer, the system introduces a saturation upper limit constant. (In this embodiment, the value is 2.0) Numerical truncation is performed. The calculation formula is as follows: In this formula: This represents the cumulative amplification factor of Level-i at the current time t; This represents the write action that inevitably occurs when data enters Level-i; in this embodiment, the conventional value of 1 is used. The min() function represents the probability weights of data being fed from one layer to the next. This represents the level capacity occupancy ratio of Level-i at the current time t. This represents a preset upper limit constant for saturation; when As the size increases, the probability of data being squeezed to the next level rises, thereby activating potential debt in subsequent levels; This represents the subsequent chain of writes triggered once data enters the next layer. Indicates the expansion coefficient of hierarchical merging; This represents the cumulative amplification factor of Level-(i+1) at the current time t.

[0054] (3) Output the final index: when the recursive calculation reaches the top level At that time, the system obtains That is, the current full-link logical write multiple. .

[0055] In actual operation, when multiple adjacent layers reach saturation simultaneously, background merging tasks are triggered concurrently. This parallel operation leads to disk head contention or bandwidth congestion, resulting in a significantly higher actual physical processing time per unit of data compared to the single-threaded baseline. To compensate for this performance degradation, the system calculates a multi-layer congestion coefficient based on the coupling probability between layers. .

[0056] Calculate all adjacent level pairs The sum of the products of the layer capacity occupancy ratios. This product represents the joint probability of two adjacent layers triggering a merge simultaneously. Similarly, to ensure numerical stability, the input... Perform upper limit truncation. The calculation formula is as follows: In this formula: This represents the multi-level concurrent congestion coefficient at the current time t; The coupling strength between Level-i and Level-(i+1) in resource contention is quantified, and this term increases significantly only when both layers are close to saturation; To compete for the sensitivity coefficient, it is used to map the joint probability to a specific performance penalty value; The baseline coefficient for a pre-defined contention-free state is set; in this embodiment, a conventional value of 1 is used. It should be further noted that this embodiment sets an upper limit for congestion penalties. The value is 3.0, which is used to prevent the calculation of multi-level concurrency coefficients from diverging. Implementers can adjust it according to the actual situation.

[0057] The end-to-end logical write multiplier and multi-layer concurrency congestion coefficient obtained at the current time t are positively integrated with the unit time of backend disk write to obtain the estimated time occupied per unit of data at the current time t, denoted as . The time taken to write data to disk at the current time t is recorded as ; Specifically, this refers to the actual physical time spent by the database background thread to write 1MB of data to the disk; in this embodiment, multiple variables are positively fused using a multiplication calculation method.

[0058] Calculation results The physical meaning is: Under the current storage backlog structure and concurrency congestion trend, for every 1MB of audit log successfully written to the front end, it is expected to consume a total of [amount] of the storage subsystem in the future. Effective processing time in seconds. This metric reduces the complexity of a multidimensional system state to a single time cost, providing the sole basis for subsequent rate control.

[0059] Step 4: Based on the estimated time occupied by the unit data and the preset proportion of total disk time occupied by the allowed data writing and sorting operations, the allowed write rate for the front end is deduced in reverse. Combined with the token bucket algorithm, the write requests audited by the front end are smoothly limited.

[0060] To transform the nonlinear system congestion risk into a linear rate control command, this application introduces a "time budget" as a control benchmark. First, the preset percentage of total disk time allowed for data writing and defragmentation operations is obtained. and the preset physical transmission bandwidth limit In this embodiment, the nominal bandwidth of sequential disk write is used.

[0061] Estimated duration based on unit data Calculate the allowed write rate for the foreground. This is to prevent issues caused by extreme system idleness. Approaching zero can cause overflow in the calculation results, so a minimum unit time constant is introduced. In this embodiment, the value is taken as... s / MB; Obtain the maximum value between the estimated time per unit of data at the current time t and the preset minimum unit time constant. Use the ratio between the proportion of total disk time occupied by the allowed data writing and defragmentation operations and the obtained maximum value as the initial calculation rate, denoted as . This represents the maximum number of megabytes of new data that the foreground can generate per second, provided that the background time budget is strictly adhered to.

[0062] Furthermore, to address the risks associated with potential model response lag, such as a sudden surge in the number of Level-0 files in a very short time during the initial cold start phase, which is not yet perceived by deeper layers, this application introduces a preventative attenuation coefficient. Real-time monitoring of Level-0 capacity occupancy. :like Greater than or equal to the first safety warning line, forced setting. If the first preset value is used, fast current limiting is activated to protect Level-0 from being breached; if Less than or equal to the second safety warning line, set The second preset value is used to maintain normal flow control; in this embodiment, the preset first safety threshold value is 0.8, which means 80% of the threshold capacity is used, and the preset second safety threshold value is 0.6; the first preset value is 0.5, and the second preset value is 1; if If it is less than the first safety warning line and greater than the second safety warning line, then The value remains consistent with the previous time step, where, The initial value is 1.

[0063] Obtain the minimum value between the initial calculation rate and the upper limit of the physical transmission bandwidth at the current time t. Multiply this minimum value by the corresponding preventive attenuation coefficient, and use this product as the foreground allowed write rate at the current time t, denoted as . The diagram illustrating how to obtain the allowed write speed in the foreground is shown below. Figure 2 As shown.

[0064] This dual protection mechanism (physical bandwidth limit truncation + Level-0 emergency attenuation) ensures that flow control commands can dynamically adapt to backend capabilities and cope with sudden backlog risks.

[0065] This is a macro-level rate metric (unit: MB / s). To enforce this limit in microsecond-level audit log write operations, a token bucket algorithm is used for micro-level traffic shaping. This algorithm is deployed at the audit log collection agent or database middleware layer, directly affecting the data write entry point.

[0066] In practice, at the current moment Update the token generation rate of the token bucket to And set the capacity of the bucket to ,in, The micro-window has a corresponding time length of 0.1s, which allows for small-scale bursts without causing the burst traffic to be too large and break down the system.

[0067] For each batch of audit logs to be written to the database: first calculate the size of the batch. Check the number of tokens remaining in the token bucket: if there are enough tokens, deduct them immediately. The system generates a token and writes the batch of data to the database; if there are insufficient tokens, the batch of data is not discarded (to ensure the integrity of the audit data), but the waiting time required to obtain the gap token is calculated. Force the current write thread to suspend and put it to sleep. Seconds. After the hibernation ends, try to acquire the token again and perform the write.

[0068] Through this forced hibernation mechanism, the high-frequency write requests in the foreground are smoothly stretched, and the spikes in write traffic are smoothed out, thereby avoiding the rapid accumulation of Level-0 layer files in the underlying storage engine caused by instantaneous traffic surges, and achieving stable operation of the entire system.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0070] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An automated operation and maintenance method for audit system databases, characterized in that, The method includes the following steps: Compare the input and output data volumes of backend data compression and sorting during historical operations and maintenance to determine the data hierarchy merging expansion coefficient; analyze the performance degradation of storage media under multi-threaded concurrent writing conditions compared to single-threaded writing to determine the contention sensitivity coefficient. Analyze the inventory and threshold of each level of the LSM-Tree storage engine in the audit database to determine the capacity occupancy ratio of each level at the current moment; use eBPF technology to capture the time and traffic of background I / O based on a preset sliding window to calculate the unit time of background disk write at the current moment. Using the hierarchical capacity occupancy ratio and the hierarchical merging expansion coefficient, recursive calculation is performed in reverse order of the hierarchical index to obtain the full-link logical write multiple; the coupling relationship of the hierarchical capacity occupancy ratio between adjacent hierarchical levels is analyzed, and the multi-level concurrent congestion coefficient is determined in combination with the contention sensitivity coefficient; based on the full-link logical write multiple, the multi-level concurrent congestion coefficient, and the background disk write unit time, the estimated unit data occupancy time is obtained; Based on the estimated time occupied by the unit data and the preset proportion of total disk time occupied by allowed data writing and sorting operations, the allowed write rate for the front end is deduced in reverse. Combined with the token bucket algorithm, the write requests audited by the front end are smoothly limited.

2. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The hierarchical merging expansion coefficient of the determined data is specifically the ratio between the output data volume and the input data volume of all data compression and sorting tasks within a preset period.

3. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The specific formula for determining the competition sensitivity coefficient is as follows: ;in, The value represents the contention sensitivity coefficient, and max() represents the maximum value function; This represents the maximum number of background merge threads configured in the database. This represents the number of input / output operations per second performed by a single write thread for sequential writes; express The number of input / output operations per second performed by each write thread in a concurrent sequential write operation; This indicates the preset value.

4. The automated operation and maintenance method for audit system databases as described in claim 1, characterized in that, The specific process for determining the capacity occupancy ratio of each level at the current moment is as follows: For Level-0: The total number of currently resident SSTable files is used as the current actual usage. For Level-1 to Level-N: the total number of disk bytes currently occupied at each level is taken as the current actual amount occupied; The preset trigger thresholds for each level are used as the level thresholds; The ratio between the current actual occupancy of each level and its corresponding level threshold is used as the level capacity occupancy ratio at the current moment.

5. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The calculation of the current time's background disk write-to-disk unit time is specifically as follows: Within the preset time window of the current time t, count the total amount of data successfully written to the storage medium by the statistical thread. and total cumulative time ; Preset minimum statistical sample threshold and default baseline value ; The time taken to write data to disk at the current time t. The specific formula is: Where t represents time. This represents the time taken to download data to disk in the background at time t-1.

6. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The process of obtaining the end-to-end logical write multiple is as follows: Set the cumulative magnification of Level-N to 1; Obtain the minimum value between the level capacity occupancy ratio of Level-i at the current moment and the preset saturation upper limit constant, and calculate the product of the obtained minimum value and the level merging expansion coefficient and the cumulative amplification factor of Level-(i+1); Add the obtained product to the value corresponding to the write action that will inevitably occur when the data is processed at Level-i, and get the cumulative amplification factor of Level-i at the current moment; Execute recursive computation up to the top level. The cumulative amplification factor obtained at that time is used as the full-link logical write factor at the current moment.

7. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The specific formula for determining the multi-level concurrent congestion coefficient is as follows: In the formula: This represents the multi-level concurrent congestion coefficient at the current time t; To compete for the sensitivity coefficient; This is the preset baseline coefficient under a no-contest state; , These represent the level capacity occupancy ratios of Level-i and Level-(i+1) at the current time t, respectively. This represents the preset upper limit constant for saturation; min() represents the minimum value function.

8. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The estimated time taken for each unit of data is specifically the result of a positive fusion of the full-link logical write multiple, the multi-layer concurrency congestion coefficient, and the unit time taken for backend disk writes obtained at the current moment.

9. The automated operation and maintenance method for an audit system database as described in claim 1, characterized in that, The specific process of reverse deducing the allowed write rate of the foreground is as follows: Get the preset percentage of total disk time used for allowed data writing and defragmentation operations, as well as the preset physical transfer bandwidth limit; Obtain the maximum value between the estimated time taken for a unit of data at the current time t and the preset minimum unit time constant. Use the ratio between the proportion of the total disk time occupied by the allowed data writing and sorting operations and the obtained maximum value as the initial calculation rate. Based on the value of the level-0 capacity occupancy ratio, a preventive attenuation coefficient is set. Obtain the minimum value between the initial calculation rate and the physical transmission bandwidth limit at the current moment. Multiply the obtained minimum value by the corresponding preventive attenuation coefficient and use it as the foreground allowed write rate at the current moment.

10. The automated operation and maintenance method for an audit system database as described in claim 9, characterized in that, The setting of the preventive attenuation coefficient specifically refers to: If the level-0 capacity occupancy ratio is greater than or equal to the first safety warning line, the preventive attenuation coefficient is set to the first preset value; If the level-0 capacity occupancy ratio is less than or equal to the second safety warning line, the preventive attenuation coefficient is set to the second preset value; If the level-0 capacity occupancy ratio is less than the first safety warning line but greater than the second safety warning line, the preventive attenuation coefficient remains consistent with the value of the previous moment. The initial value of the preventive attenuation coefficient is 1, the second preset value is greater than the first preset value, and the second safety warning line is less than the first safety warning line.

Images