A system log collection dynamic regulation method, system and device

Abstract

This invention relates to the field of data processing technology and discloses a method, system, and device for dynamic control of system log collection. The method includes: collecting and storing multiple resource indicators of the system in real time at a preset sampling frequency; processing the collected resource indicators to obtain a resource pressure score and a load trend prediction signal of the system; matching the resource pressure score, the load trend prediction signal, and the resource indicators to corresponding priority levels according to a preset priority hierarchical strategy; and executing the control action corresponding to the priority level to adjust the sampling frequency. This invention proactively prevents system overload by monitoring multiple resource indicators of the system in real time and adjusting the sampling frequency of the resource indicators based on the resource pressure score and load trend prediction signal. This prevents system overload risks caused by the log collection process itself from the source, improving system stability and operational reliability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method, system, and device for dynamic control of system log collection. Background Technology

[0002] Existing log processing technologies generally focus on optimizing the storage of generated logs, hierarchical management, or reactive processing based on single events, resulting in a fundamental contradiction between computer system stability and log integrity.

[0003] For example: The core of existing log processing technologies lies in the storage compression, overwrite strategy optimization, or offline analysis of generated log data, which is a passive, reactive approach. This post-processing model cannot detect and mitigate the load pressure on computer system resources such as CPU utilization, memory bandwidth, and disk I / O during log collection. When the computer system is under high load, continuous log collection operations may become a key trigger for computer system performance degradation, leading to system jitter, response delays, or even crashes.

[0004] For example: Some existing log processing technologies can adjust log levels based on single or a few event metrics, such as specific program states or the number of abnormal events. However, the performance bottlenecks of modern computer systems are multifaceted and can be caused by any of the following: CPU, memory, disk I / O, internal data queues, etc. Making control decisions based on a single metric is prone to misjudgment and cannot accurately address the complex resource pressures of a computer system. When a computer system's memory utilization is nearing exhaustion but the CPU is idle, blindly reducing the log sampling rate based solely on CPU metrics will fail to accurately address the core resource (memory) pressure issue and will hinder comprehensive resource protection.

[0005] For example: To free up storage space, a brief write window closes when switching log files, potentially leading to log data loss. On the other hand, the complex mechanisms designed to ensure zero log data loss can further exacerbate resource contention during periods of high system load, negatively impacting system performance.

[0006] Existing log processing technologies struggle to ensure zero log data loss without placing an additional burden on the computer system, creating a dilemma where maintaining computer system performance results in lost log data, while maintaining log data compromises computer system performance. Summary of the Invention

[0007] To address the problems of management lag, lack of source control capabilities, biased decision-making, inability to cope with complex bottlenecks, and difficulty in achieving both computer system performance and log data integrity in existing log processing technologies, this invention provides a method, system, and device for dynamic control of system log collection.

[0008] According to an embodiment of the present invention, a method for dynamic control of system log collection is provided, comprising the following steps: Multiple resource indicators of the system are collected in real time and stored by means of a preset sampling frequency; The collected resource indicators are preprocessed to obtain standard resource indicators; The system's resource pressure score is obtained by weighted fusion of the standard resource indicators. The historical resource pressure scores of the standard resource indicators are predicted to obtain the load trend prediction signal of the system within a preset time period. According to a preset priority hierarchical strategy, the resource pressure score, the load trend prediction signal, and the resource indicators are matched with corresponding priority levels. Execute the control action corresponding to the priority level to adjust the sampling frequency; The execution results of the various resource indicators and the control actions are stored. The resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

[0009] In some optional implementations, the standard resource indicators are weighted and fused to obtain the resource pressure score of the system, specifically including: The weights of each resource indicator are determined based on the different degrees of influence and urgency of each resource indicator on the stability of the system. The resource pressure score of the system is obtained by weighting and summing the weights of each resource indicator using a multi-indicator weighted fusion decision algorithm.

[0010] In some optional implementations, the historical resource pressure scores of the standard resource indicators are predicted to obtain a load trend prediction signal for the system within a preset time period, specifically including: Use a fixed-duration historical resource pressure score sequence as a sliding window; By fitting the historical resource pressure score sequence within the sliding window using a trend prediction algorithm, a load trend prediction signal for the system within a preset time period is obtained. The load trend prediction signal includes rising, maintaining, and falling trends.

[0011] In some optional implementations, the hierarchical strategy with preset priorities includes: an emergency protection layer with the highest priority, a performance optimization layer of the core control layer, a resource awareness layer of the auxiliary optimization layer, and a service quality assurance layer of the data integrity layer.

[0012] In some optional implementations, the resource stress score, the load trend prediction signal, and the resource indicators are matched with corresponding priority levels, specifically including: Determine whether the real-time instantaneous values ​​of the CPU utilization and / or memory utilization exceed a first preset threshold. If so, the system triggers the highest priority emergency protection layer. When it is determined that the system has not triggered the emergency protection layer, the performance optimization layer of the core control layer continues to run. Determine whether the memory usage and / or disk space usage exceed a second preset threshold and remain so for a preset time. If so, the system triggers the resource awareness layer of the auxiliary optimization layer. Determine whether the backlog of the internal data queue exceeds a third preset threshold. If so, the system triggers the service quality assurance layer of the data integrity layer.

[0013] In some optional implementations, the control action corresponding to the priority level is executed, specifically including: When the system triggers the highest priority emergency protection layer, the sampling rate is reduced to a preset minimum threshold, the log compression function is temporarily disabled, and / or the collection of non-core resource indicators is suspended. When the system continuously runs the performance optimization layer of the core control layer, the acquisition frequency and / or buffer size are dynamically and smoothly adjusted based on the resource pressure score and the load trend prediction signal. When the system triggers the resource awareness layer of the auxiliary optimization layer, it generates early warning information and outputs targeted optimization suggestions; When the system triggers the service quality assurance layer of the data integrity layer, the system scheduling priority is increased, additional processing threads are dynamically allocated, and / or a backup buffer is temporarily enabled.

[0014] In some optional implementations, multiple resource indicators of the system will be collected and stored in real time at a preset sampling frequency, including: Configure a real-time receive buffer and a rotating backup buffer; When the system is running normally, the multiple resource indicators collected in real time are written into the real-time receiving buffer, and the rotating standby buffer is in an idle standby state.

[0015] Some alternative implementations also include: When log file switching is required to trigger rotation, the writing target is switched through atomic pointer replacement instructions, and the multiple resource indicators collected in real time are written to the rotation backup buffer. After the write target is switched, the residual cached data in the real-time receive buffer is completely written to the current old log file; Write the cached data in the rotating spare buffer to the newly created log file; By using atomic pointer replacement instructions, the write target is switched, and the multiple resource indicators collected in real time are written to the real-time receiving buffer.

[0016] According to another objective of the present invention, a system log collection dynamic control system is provided, which dynamically controls the frequency of log data acquisition using any of the above-described system log collection dynamic control methods. The system includes: a multi-dimensional resource monitoring and fusion decision module, a hierarchical resource feedback control module, and a data storage module. The multi-dimensional resource monitoring and fusion decision module is used to collect multiple resource indicators of the system in real time through a preset sampling frequency, preprocess the collected multiple resource indicators to obtain standard resource indicators, perform weighted fusion processing on the standard resource indicators to obtain the resource pressure score of the system, and predict the historical resource pressure score of the standard resource indicators to obtain the load trend prediction signal of the system within a preset time period. The hierarchical resource feedback control module is used to match the resource pressure score, the load trend prediction signal and the resource indicators with corresponding priority levels according to a hierarchical strategy with preset priorities, and to execute the control actions corresponding to the priority levels. The data storage module is used to store multiple resource indicators of the system that are collected in real time at a preset sampling frequency. The resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

[0017] According to another objective of the present invention, a computer device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform an operation as described in any of the above-mentioned system log acquisition dynamic control methods.

[0018] Compared with the prior art, the present invention has the following advantages: This invention provides a dynamic control method for system log collection. By monitoring multiple resource indicators of the system in real time, the system can perceive the status of core resources such as CPU utilization, memory utilization, disk space utilization, and internal data queue depth in real time and comprehensively. Based on resource pressure scores and load trend prediction signals, the sampling frequency of resource indicators is adjusted in real time. This makes control decisions no longer dependent on a single event or post-event analysis, but can proactively and forward-lookingly assess the system load during the log data collection process. This prevents the risk of system overload caused by the log collection behavior itself from the source, and improves the stability and reliability of the system.

[0019] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 The diagram shows a flowchart of a system log collection dynamic control method provided by an embodiment of the present invention.

[0021] Figure 2 This illustration shows a schematic diagram of the implementation process of storing multiple resource indicators collected in a system log collection dynamic control method provided by an embodiment of the present invention.

[0022] Figure 3 The diagram shows a structural block diagram of a system log collection and dynamic control system provided by an embodiment of the present invention.

[0023] Figure 4 A structural block diagram of a computer device provided by an embodiment of the present invention is shown. Detailed Implementation

[0024] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0025] In order to address the problems of management lag, lack of source control capabilities, biased decision-making, inability to cope with complex bottlenecks, and difficulty in achieving both computer system performance and log data integrity in existing log processing technologies, this invention provides a dynamic control method for system log collection.

[0026] This embodiment provides a method for dynamic control of system log collection, such as... Figure 1 As shown, it includes the following steps: S10. Collect and store multiple resource indicators of the system in real time using a preset sampling frequency.

[0027] In this step, resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

[0028] This step enables continuous and proactive monitoring and storage of multiple resource indicators of the system by calling the system interface at a fixed sampling frequency, providing a data foundation for subsequent decision-making.

[0029] S20. Preprocess the collected resource indicators to obtain standard resource indicators.

[0030] This step normalizes and smooths the collected resource indicators to obtain standard resource indicators.

[0031] In this step, the unit scale range of each resource indicator is calculated, resource indicators with different physical meanings are processed to unify their dimensions, which facilitates fusion calculation and realizes the normalization of multiple resource indicators. A smoothing algorithm is applied to the values ​​of multiple resource indicators sampled continuously to smooth out high-frequency fluctuations, highlight long-term trends, and ensure stable input to downstream algorithms.

[0032] S30. Perform weighted fusion processing on the standard resource indicators to obtain the resource pressure score of the system.

[0033] S40. Predict the historical resource pressure score of the standard resource indicators to obtain the load trend prediction signal of the system within a preset time period.

[0034] Steps S30 and S40 perform weighted fusion of multiple resource indicators to generate a comprehensive resource pressure score and load trend prediction signal, realizing the transformation from post-event judgment to pre-event prediction and providing a basis for precise regulation.

[0035] S50. According to the preset priority hierarchical strategy, match the resource pressure score, the load trend prediction signal and the resource indicators with the corresponding priority levels.

[0036] This step utilizes a hierarchical strategy to map different system states to different priority processing levels based on resource pressure scores and load trend prediction signals, ensuring the orderly and targeted nature of control actions.

[0037] The system status includes emergency overload, normal load, long-term bottleneck or data backlog.

[0038] S60. Execute the control action corresponding to the priority level to adjust the sampling frequency.

[0039] This step enables the execution of corresponding control measures based on the matched priority level, translating decisions into practical actions.

[0040] This embodiment provides a dynamic control method for system log collection. By monitoring multiple resource indicators of the system in real time, the system can perceive the status of core resources such as CPU utilization, memory utilization, disk space utilization, and internal data queue depth in real time and comprehensively. Based on resource pressure scores and load trend prediction signals, the sampling frequency of resource indicators is adjusted in real time. This makes the control decision no longer dependent on a single event or post-event analysis, but can proactively and forward-lookingly assess the system load during the log data collection process. This prevents the risk of system overload caused by the log collection behavior itself from the source, and improves the stability and reliability of the system.

[0041] In this preferred embodiment, step S10 optimizes the specific process of real-time acquisition and storage of multiple resource indicators of the system through a preset sampling frequency.

[0042] In this embodiment, as Figure 2 As shown, multiple resource indicators of the system will be collected and stored in real time at a preset sampling frequency, specifically including: Set up a real-time receiving buffer and a rotating standby buffer. The real-time receiving buffer serves as the target buffer with a fixed capacity and is used to receive multiple resource indicators collected in real time. The rotating standby buffer serves as a backup buffer with a large capacity and is used for temporary data caching during adjustment or log file switching.

[0043] When the system is running normally, it writes multiple resource indicators collected in real time into the real-time receiving buffer, while the rotating standby buffer is in an idle standby state.

[0044] When log file switching is required to trigger rotation, the write target is switched through atomic pointer replacement instructions, and multiple resource indicators collected in real time are written to the rotation backup buffer so that data writing is uninterrupted. After the write target is switched, the residual cached data in the real-time receive buffer will be completely written to the current old log file; Write the cached data in the rotation spare buffer to the newly created log file; By using atomic pointer replacement instructions, the write target is switched, and multiple resource metrics collected in real time are written to the real-time receive buffer, thus completing the log file switching.

[0045] In this embodiment, the entire log file switching process is locked by a global state flag lock, which prevents multiple concurrent triggering of rotation operations, avoids data corruption caused by multi-threaded contention, and ensures that the entire log file switching process is completely transparent to the outside world. This solves the problems of log data writing interruption and log data overflow and loss that are prone to occur during log file rotation.

[0046] This embodiment is a preferred embodiment, and the specific implementation process of step S30, which performs weighted fusion processing on the standard resource indicators to obtain the resource pressure score of the system, has been optimized.

[0047] In this embodiment, the standard resource indicators are weighted and fused to obtain the resource pressure score of the system, specifically including: The weights of each resource indicator are determined based on the different degrees of impact and urgency of each resource indicator on system stability. For example, in memory-sensitive applications, memory utilization should be given a higher weight; in I / O-intensive scenarios, disk space utilization is more critical.

[0048] The resource pressure score of the system is obtained by weighting and summing the weights of each resource indicator through a multi-indicator weighted fusion decision algorithm.

[0049] The weights of each resource metric can be set empirically or adjusted dynamically. For example, if insufficient memory causes a system OOM (Out of Memory) error, it may be more fatal than short-term CPU overload, so memory should be given a higher weight.

[0050] The weighted summation formula used in this step is as follows: S=W CPU *N CPU +W mem *N mem +W io *N io +W queue *N queue ; Where: S represents the resource stress score, W CPU N represents the CPU weight. CPU For CPU utilization, W mem For memory weights, N mem For memory usage, W io N represents the disk weight. io For disk space utilization, W queue For the depth weight of the internal data queue, N queue This represents the depth of the internal data queue.

[0051] In this embodiment, the resource stress score is a continuous value between 0 and 1 or a discrete level such as low load, medium load, high load, or danger. The resource stress score will serve as the core input signal for subsequent hierarchical regulation.

[0052] This embodiment is a preferred embodiment, and the specific implementation process of step S40, which predicts the historical resource pressure score of the standard resource index to obtain the load trend prediction signal of the system within a preset time period, has been optimized.

[0053] In this embodiment, the historical resource pressure score of the standard resource index is predicted to obtain the load trend prediction signal of the system within a preset time period, specifically including: A fixed-duration historical resource pressure score sequence is used as a sliding window.

[0054] In this step, the fixed time can be set to any duration between 40S and 80S, and the sliding window maintains a historical resource pressure score sequence for a fixed duration.

[0055] By fitting the historical resource pressure score sequence within the sliding window using a trend prediction algorithm, a system load trend prediction signal within a preset time period is obtained.

[0056] The load trend prediction signal includes rising, maintaining, and falling trends.

[0057] In this step, a linear fit is performed on the historical resource pressure score sequence within the sliding window, and the slope is used to determine whether the future trend is rising, falling, or stable.

[0058] Using exponential smoothing or a lightweight machine learning model, predict the resource pressure score S(t+10) for a short period of time in the future, such as the next 10 seconds.

[0059] In this step, the prediction results are output in the form of load trend prediction signals, such as: rapid rise, slow fall, maintaining a high level, etc.

[0060] This embodiment, as a preferred embodiment, optimizes the specific implementation process of step S50, which involves matching the resource pressure score, the load trend prediction signal, and the resource index to the corresponding priority level according to a preset priority hierarchical strategy.

[0061] In this embodiment, the hierarchical strategy with preset priorities includes: the highest priority emergency protection layer, the core control layer performance optimization layer, the auxiliary optimization layer resource awareness layer, and the data integrity layer quality of service assurance layer.

[0062] The resource stress score, load trend prediction signal, and resource indicators are matched with corresponding priority levels, specifically including: Determine whether the real-time instantaneous values ​​of CPU utilization and / or memory utilization exceed the first preset threshold. If so, the system triggers the highest priority emergency protection layer.

[0063] In this embodiment, the first preset threshold is set to 95%.

[0064] In some optional embodiments, the first preset threshold can be adaptively adjusted according to the application scenario.

[0065] When it is determined that the system has not triggered the emergency protection layer, the performance optimization layer of the core control layer continues to run.

[0066] Determine whether memory usage and / or disk space usage exceed a second preset threshold and remain so for a preset time. If so, the system triggers the resource awareness layer of the auxiliary optimization layer.

[0067] In this embodiment, the second preset threshold is set to 80%, and the preset time is set to 5 minutes.

[0068] In some optional embodiments, the second preset threshold and preset time can be adaptively adjusted according to the application scenario.

[0069] The resource awareness layer of the auxiliary optimization layer uses duration-based judgment to identify long-term, slow resource pressure.

[0070] Determine if the backlog of internal data queues exceeds the third preset threshold. If so, the system triggers the service quality assurance layer of the data integrity layer.

[0071] The Quality of Service (QoS) layer of the data integrity layer is used to directly monitor the health of the log data processing pipeline and prevent log data loss due to the processing speed not keeping up with the collection speed.

[0072] This embodiment, as a preferred embodiment, executes the control actions corresponding to the priority level, specifically including: When the system triggers the highest priority emergency protection layer, the sampling rate will be reduced to the preset minimum threshold, the log compression function will be temporarily disabled, and / or the collection of non-core resource indicators will be suspended.

[0073] The most aggressive resource release methods are adopted to reduce the load on the log subsystem as quickly as possible.

[0074] When the system continuously runs the performance optimization layer of the core control layer, the acquisition frequency and / or buffer size are dynamically and smoothly adjusted based on the resource pressure score and the load trend prediction signal to achieve smooth, adaptive fine-tuning and avoid system oscillation.

[0075] When the system triggers the resource awareness layer of the auxiliary optimization layer, it generates early warning information and outputs targeted optimization suggestions. The resource awareness layer does not perform forced control, but provides diagnostic information, such as suggesting to reduce the log buffer, leaving the decision-making power to the operation and maintenance personnel, which reflects human-machine collaboration.

[0076] When the system triggers the service quality assurance layer of the data integrity layer, the system scheduling priority is increased, additional processing threads are dynamically allocated, and / or a backup buffer is temporarily enabled.

[0077] In some optional embodiments, the present invention also discloses a system log collection dynamic control system, which uses any one of the above-described system log collection dynamic control methods to dynamically control the frequency of log data acquisition.

[0078] This embodiment provides a system log collection and dynamic control system, such as... Figure 3 As shown, it includes: a multi-dimensional resource monitoring and fusion decision-making module 100, a hierarchical resource feedback control module 200, and a data storage module 300.

[0079] The multi-dimensional resource monitoring and fusion decision module 100 is used to collect multiple resource indicators of the system in real time through a preset sampling frequency, preprocess the collected multiple resource indicators to obtain standard resource indicators, perform weighted fusion processing on the standard resource indicators to obtain the resource pressure score of the system, and predict the historical resource pressure score of the standard resource indicators to obtain the load trend prediction signal of the system within a preset time period.

[0080] The resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

[0081] In this embodiment, the specific implementation process of the multi-dimensional resource monitoring and fusion decision module 100 acquiring the resource pressure score and load trend prediction signal of the system is described in steps S10-S40 of the above-mentioned system log collection dynamic control method, and will not be repeated in this embodiment.

[0082] The hierarchical resource feedback control module 200 is used to match the resource pressure score, the load trend prediction signal and the resource indicators with corresponding priority levels according to a hierarchical strategy with preset priorities, and to execute the control actions corresponding to the priority levels.

[0083] In this embodiment, the hierarchical resource feedback control module 200 matches the resource pressure score, the load trend prediction signal, and the resource indicators to corresponding priority levels according to a preset priority hierarchical strategy, and executes the control actions corresponding to the priority levels. For the specific implementation process, please refer to steps S50-S60 in the above-mentioned dynamic control method for system log collection, which will not be repeated in this embodiment.

[0084] The data storage module 300 is used to store multiple resource indicators of the system that are collected in real time at a preset sampling frequency.

[0085] In this embodiment, the data storage module 300 stores multiple resource indicators of the system in real time through a preset sampling frequency. For the specific implementation process, please refer to step S10 in the above-mentioned dynamic control method for system log collection. This embodiment will not repeat the description.

[0086] In some alternative embodiments, the present invention also discloses a computer device.

[0087] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown.

[0088] like Figure 4 As shown, the computer device includes: a processor 410, a memory 420, a communication interface 430, and a communication bus 440. The processor 410, the memory 420, and the communication interface 430 communicate with each other through the communication bus 440.

[0089] In this embodiment, the memory 420 is used to store at least one executable instruction, which causes the processor 420 to perform the operation of a system log collection dynamic control method as described in any of the above embodiments; the communication interface 430 is used to communicate with other devices, such as clients or other server network elements. The processor 410 is used to execute the program 450, which can specifically execute the relevant steps in the above embodiments of the system log collection dynamic control method.

[0090] Specifically, program 450 may include program code, which includes computer-executable instructions.

[0091] Processor 510 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0092] Memory 420 is used to store program 450. Memory 420 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0093] Specifically, program 450 can be called by processor 410 to cause the computer device to perform the operation of a system log collection dynamic control method according to any of the above embodiments.

[0094] In some optional embodiments, the present invention also discloses a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a computer device, it causes the computer device to perform the steps of a system log collection dynamic control method in any of the above method embodiments.

[0095] The specific implementation process of the system log collection dynamic control method in this embodiment can be found in any of the above method embodiments, and will not be repeated in this embodiment.

[0096] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0097] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0098] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for dynamic control of system log collection, characterized in that, Includes the following steps: Multiple resource indicators of the system are collected in real time and stored by means of a preset sampling frequency; The collected resource indicators are preprocessed to obtain standard resource indicators; The system's resource pressure score is obtained by weighted fusion of the standard resource indicators. The historical resource pressure scores of the standard resource indicators are predicted to obtain the load trend prediction signal of the system within a preset time period. According to a preset priority hierarchical strategy, the resource pressure score, the load trend prediction signal, and the resource indicators are matched with corresponding priority levels. Execute the control action corresponding to the priority level to adjust the sampling frequency; The resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

2. The system log collection dynamic control method according to claim 1, characterized in that, The system's resource pressure score is obtained by weighted fusion of the standard resource indicators, specifically including: The weights of each resource indicator are determined based on the different degrees of influence and urgency of each resource indicator on the stability of the system. The resource pressure score of the system is obtained by weighting and summing the weights of each resource indicator using a multi-indicator weighted fusion decision algorithm.

3. The system log collection dynamic control method according to claim 1, characterized in that, The historical resource pressure scores of the standard resource indicators are used to predict the load trend prediction signal of the system within a preset time period, specifically including: Use a fixed-duration historical resource pressure score sequence as a sliding window; By fitting the historical resource pressure score sequence within the sliding window using a trend prediction algorithm, a load trend prediction signal for the system within a preset time period is obtained. The load trend prediction signal includes rising, maintaining, and falling trends.

4. The system log collection dynamic control method according to claim 1, characterized in that, The preset priority layering strategy includes: the highest priority emergency protection layer, the core control layer performance optimization layer, the auxiliary optimization layer resource awareness layer, and the data integrity layer quality of service assurance layer.

5. The system log collection dynamic control method according to claim 4, characterized in that, Matching the resource stress score, the load trend prediction signal, and the resource indicators to corresponding priority levels specifically includes: Determine whether the real-time instantaneous values ​​of the CPU utilization and / or memory utilization exceed a first preset threshold. If so, the system triggers the highest priority emergency protection layer. When it is determined that the system has not triggered the emergency protection layer, the performance optimization layer of the core control layer continues to run. Determine whether the memory usage and / or disk space usage exceed a second preset threshold and remain so for a preset time. If so, the system triggers the resource awareness layer of the auxiliary optimization layer. Determine whether the backlog of the internal data queue exceeds a third preset threshold. If so, the system triggers the service quality assurance layer of the data integrity layer.

6. The system log collection dynamic control method according to claim 5, characterized in that, Executing the control action corresponding to the priority level specifically includes: When the system triggers the highest priority emergency protection layer, the sampling rate is reduced to a preset minimum threshold, the log compression function is temporarily disabled, and / or the collection of non-core resource indicators is suspended. When the system continuously runs the performance optimization layer of the core control layer, the acquisition frequency and / or buffer size are dynamically and smoothly adjusted based on the resource pressure score and the load trend prediction signal. When the system triggers the resource awareness layer of the auxiliary optimization layer, it generates early warning information and outputs targeted optimization suggestions; When the system triggers the service quality assurance layer of the data integrity layer, the system scheduling priority is increased, additional processing threads are dynamically allocated, and / or a backup buffer is temporarily enabled.

7. The system log collection dynamic control method according to claim 1, characterized in that, Multiple resource indicators of the system will be collected in real time at a preset sampling frequency and stored, specifically including: Configure a real-time receive buffer and a rotating backup buffer; When the system is running normally, the multiple resource indicators collected in real time are written into the real-time receiving buffer, and the rotating standby buffer is in an idle standby state.

8. The system log collection dynamic control method according to claim 7, characterized in that, Also includes: When log file switching is required to trigger rotation, the writing target is switched through atomic pointer replacement instructions, and the multiple resource indicators collected in real time are written to the rotation backup buffer. After the write target is switched, the residual cached data in the real-time receive buffer is completely written to the current old log file; Write the cached data in the rotating spare buffer to the newly created log file; By using atomic pointer replacement instructions, the write target is switched, and the multiple resource indicators collected in real time are written to the real-time receiving buffer.

9. A system log acquisition and dynamic control system, characterized in that, The system log collection dynamic control method according to any one of claims 1-8 dynamically controls the frequency of log data acquisition. The system includes: a multi-dimensional resource monitoring and fusion decision module, a hierarchical resource feedback control module, and a data storage module. The multi-dimensional resource monitoring and fusion decision module is used to collect multiple resource indicators of the system in real time through a preset sampling frequency, preprocess the collected multiple resource indicators to obtain standard resource indicators, perform weighted fusion processing on the standard resource indicators to obtain the resource pressure score of the system, and predict the historical resource pressure score of the standard resource indicators to obtain the load trend prediction signal of the system within a preset time period. The hierarchical resource feedback control module is used to match the resource pressure score, the load trend prediction signal and the resource indicators with corresponding priority levels according to a hierarchical strategy with preset priorities, and to execute the control actions corresponding to the priority levels. The data storage module is used to store multiple resource indicators of the system that are collected in real time at a preset sampling frequency. The resource metrics include CPU utilization, memory utilization, disk space utilization, and internal data queue depth.

10. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation of the system log collection dynamic control method as described in any one of claims 1-8.

Images