Log sampling method and system based on adaptive adjustment mechanism
Through the log sampling method and system of the adaptive adjustment mechanism, the problems of resource waste and business demand changes in log processing are solved, and efficient log analysis and storage resources are achieved.
Patent Information
- Application Number
- CN202510539400.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-27
- Publication Date
- 2025-08-15
- Estimated Expiration
- 2045-04-27
AI Technical Summary
The existing log processing methods have problems such as massive storage leading to resource waste, inability to quickly identify the causes of abnormalities, and inability to flexibly adapt to changes in business requirements.
Adaptive adjustment mechanism is adopted to dynamically adjust the log sampling rate by dividing core systems, preset keyword characteristics, calculating storage urgent items and traffic suppression items, and real-time monitoring of storage capacity, realizing dynamic adjustment of log filtering and sampling rate.
Reduce data redundancy, improve log analysis efficiency, balance storage resource usage, quickly adapt to changes in business demands, and ensure data fidelity.
Smart Images

Figure CN120491892A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of log collection technology, and in particular to a log sampling method and system based on an adaptive adjustment mechanism. Background Art
[0002] With the large-scale deployment and application of microservice architectures in the internet sector, efficient processing of massive log data in a large number of microservice architectures, middleware, and corresponding systems has become a key challenge in ensuring system stability. The current log processing method usually stores all logs in the system and then analyzes them, which has the following drawbacks: 1. When a microservice architecture or a large number of middleware are used, massive logs are generated. Storing all of these logs would take up a lot of storage space, and storing useless logs would also waste storage resources. Second, existing log processing methods fail to filter and differentiate logs, resulting in an inability to quickly identify the cause of anomalies or locate fault points during subsequent anomaly analysis through logs. This is time-consuming, labor-intensive, and inefficient. 3. The existing log processing method stores all logs in full, with a fixed sampling strategy. The sampling frequency cannot be adjusted according to real-time business indicators, making it difficult to flexibly respond to emergencies or adapt to changing business needs.
[0003] Therefore, we need to develop a log sampling method and system based on an adaptive adjustment mechanism that can screen logs before storage, reduce data redundancy, and improve the efficiency of subsequent log analysis; at the same time, it can dynamically adjust the log sampling frequency to achieve the goal of balancing storage resource usage and quickly adapting to changes in business needs. Summary of the Invention
[0004] The purpose of the present invention is to provide a log sampling method and system based on an adaptive adjustment mechanism to solve the problems mentioned in the above background technology that the existing log processing methods are inefficient, occupy a large amount of storage space, and cannot quickly adapt to changes in business needs.
[0005] To achieve the above object, the present invention adopts the following technical solutions: According to one aspect of the present invention, a log sampling method based on an adaptive adjustment mechanism is provided, the method comprising the following steps: Divide and obtain the core system, preset system keyword features, and mark the logs according to the preset rules; Calculating a storage urgency item and a flow suppression item, and calculating a dynamic sampling rate according to the storage urgency item and the flow suppression item; Performing log sampling based on the feature marker and the dynamic sampling rate; The sampled logs are stored, the capacity status of the storage system is monitored in real time, and the dynamic sampling rate is adjusted.
[0006] According to one aspect of the present invention, a log sampling system based on an adaptive adjustment mechanism is provided, the system comprising: a log marking module, a sampling rate calculation module, a log sampling module, and a storage monitoring module. The log tagging module is used to divide and obtain the core system, preset system keyword features, and perform feature tagging on the log according to preset rules; The sampling rate calculation module is used to calculate the storage urgency item and the flow suppression item, and calculate the dynamic sampling rate according to the storage urgency item and the flow suppression item; The log sampling module is configured to perform log sampling based on the feature marker and the dynamic sampling rate; The storage monitoring module is used to store sampled logs, monitor the capacity status of the storage system in real time, and adjust the dynamic sampling rate.
[0007] Based on the above-mentioned scheme, the division and obtaining of the core system refers to dividing the system based on the system's key business dimension and key infrastructure dimension to obtain the core system; among them, the key business is the current company's main business, and the key infrastructure is the key central control node through which the system calls flow.
[0008] Based on the above-mentioned scheme, the system keyword features include system error codes and abnormal metadata; wherein, the system error code adopts a four-segment specification: the first segment is the response code type, and the error response code starts with E; the second segment is the error source, the gateway type response is 0, and other business systems are coded from 1-Z; the third segment is the information type, including data information errors, business system errors, system information errors, file information errors, database information errors and other categories of errors; the fourth segment is the response code sequence number, and the coding range is 00001-zzzzz; therefore, the response code can be parsed to identify whether it is a system error code: when it is identified that the response code starts with E, it can be determined that the current response code is a system error code.
[0009] Based on the above solution, the feature tagging of logs according to preset rules specifically includes: Determine whether the current log belongs to the log in the core system; If yes, mark the current log as fully stored; If not, determine whether the current log contains the system keyword feature; If included, the current log will be marked as fully stored; If not included, no marking is performed.
[0010] Based on the above solution, the storage urgency item is calculated based on the current usage of the storage system and the upper capacity limit of the storage system. The specific formula is: ; Wherein, S is the storage urgency item; The current usage of the storage system; is the upper limit of the capacity of the storage system.
[0011] Based on the above solution, the flow suppression term is calculated based on the flow fluctuation change rate and flow sensitivity, specifically: Determine the traffic fluctuation rate based on the traffic in the current sliding window and the traffic in the previous sliding window: ; Wherein, σ is the flow fluctuation change rate; As the basic traffic fluctuation value, after each sliding window, the traffic fluctuation change rate of the previous sliding window is used as the basic traffic fluctuation value of the current sliding window; is the traffic sequence in the current sliding window; is the traffic sequence in the previous sliding window; Based on the traffic fluctuation change rate and the storage urgency item, traffic sensitivity is calculated: ; Wherein, β(t) is the flow sensitivity at time t, and β(t)>0; is the basic flow sensitivity, and the flow sensitivity at the previous moment is used as the basic flow sensitivity at the current moment in each calculation; S is the storage urgency item; Based on the traffic sensitivity, the traffic suppression item is obtained: ; Wherein, L is the traffic suppression term; λ(t) is the log input rate at time t.
[0012] Based on the above solution, the dynamic sampling rate is calculated as follows: ; Wherein, r(t) is the dynamic sampling rate at time t; α is the preset basic sampling rate, and 0<α<1; S is the storage urgency item; and L is the flow suppression item.
[0013] Based on the above solution, the log sampling is performed based on the feature marker and the dynamic sampling rate, specifically: Determine whether the current log has the full storage mark; If yes, the sampling executor is called directly to perform full log sampling; If not, the dynamic sampling rate is sent to the sampling executor, and the log is sampled according to the dynamic sampling rate.
[0014] Based on the above solution, when the capacity status of the storage system is monitored in real time, if the capacity of the storage system reaches a preset upper limit, the dynamic sampling rate controller is notified to amplify the storage weight by the square of the storage urgency term and adjust the sampling rate.
[0015] According to one aspect of the present invention, an electronic device is provided, comprising a processor and a memory; wherein the memory is used to store executable instructions of the processor, and the processor is configured to execute the log sampling method based on the adaptive adjustment mechanism described in the first aspect by executing the executable instructions.
[0016] According to one aspect of the present invention, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the log sampling method based on the adaptive adjustment mechanism described in the first aspect is implemented.
[0017] It can be seen from the above technical solutions that compared with the prior art, the present invention has at least the following advantages and positive effects: (1) The present invention divides the core system and presets system keyword features, and features the logs, which can be screened before log sampling and storage, reducing data redundancy and improving the efficiency of subsequent log analysis.
[0018] (2) The present invention calculates the dynamic sampling rate based on the storage urgency item and the flow suppression item, thereby constructing a dual adaptive adjustment mechanism including storage-driven vertical adjustment and flow-driven horizontal adjustment, which can quickly adapt to different business needs during the log sampling process and reduce the occupancy of storage resources.
[0019] (3) The present invention performs log sampling based on log feature tags and dynamic sampling rate, monitors the capacity status of the storage system in real time after log storage and adjusts the dynamic sampling rate, which can ensure the data fidelity of log sampling while balancing storage capacity. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below only relate to some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0021] Figure 1A flow chart of a log sampling method based on an adaptive adjustment mechanism provided by an embodiment of the present invention is shown; Figure 2 A flow chart showing a method for calculating a dynamic sampling rate provided by an embodiment of the present invention is shown; Figure 3 A flow chart of a method for dynamic log sampling and storage provided by an embodiment of the present invention is shown; Figure 4 A schematic diagram of the structure of a log sampling system based on an adaptive adjustment mechanism provided by an embodiment of the present invention is shown; in, Figure 4 The reference numerals in the figures are described as follows: 400 - A log sampling system based on an adaptive adjustment mechanism; 401 - Log marking module, 4011 - Preliminary configuration unit, 4012 - Feature marking unit; 402 - Sampling rate calculation module, 4021 - Capacity assessment unit, 4022 - Traffic perception unit, 4023 - Sampling rate calculation unit; 403 - Log sampling module, 4031 - Full sampling unit, 4032 - Dynamic sampling unit; 404 - Storage monitoring module, 4041 - Capacity monitoring unit, 4042 - Capacity alarm unit. DETAILED DESCRIPTION
[0022] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The example implementation methods can be implemented in various forms and should not be understood as being limited to the examples described herein. On the contrary, these implementation methods are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the example implementation methods to those skilled in the art.
[0023] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, it will be appreciated by those skilled in the art that the technical solutions of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.
[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0025] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0026] The present invention is described in detail below with reference to specific embodiments.
[0027] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a log sampling method based on an adaptive adjustment mechanism. The specific steps of the method are as follows: S1: Divide and obtain the core system, preset system keyword features, and feature-mark the logs according to preset rules; In this embodiment, the division and obtaining of the core system refers to dividing the system based on the key business dimension and the key infrastructure dimension to obtain the core system; wherein, the key business is the main business of the current company; the key infrastructure is the key central control node through which the system call flows; the core system includes the gateway system, the membership system and the points system.
[0028] Furthermore, the system keyword features include system error codes and abnormal metadata; wherein, the system error code adopts a four-segment specification: the first segment is the response code type, and the error response code starts with E; the second segment is the error source, the gateway type response is 0, and other business systems are coded from 1-Z; the third segment is the information type, including data information error, business system error, system information error, file information error, database information error and other categories of errors; the fourth segment is the response code sequence number, and the coding range is 00001-zzzzz; therefore, the response code can be parsed to identify whether it is a system error code: when it is identified that the response code starts with E, it can be determined that the current response code is a system error code.
[0029] Furthermore, the logs are marked with features according to preset rules, specifically including: Determine whether the current log belongs to the log in the core system; If yes, mark the current log as fully stored; If not, determine whether the current log contains the system keyword feature; If included, the current log will be marked as fully stored; If not included, no marking is performed.
[0030] In this embodiment, by dividing the core system and presetting system keyword features, the logs are marked with features, which can be screened before log sampling and storage. This not only reduces data redundancy, reduces storage resource usage, and avoids the waste of storage resources caused by storing irrelevant logs, but also improves the efficiency of subsequent log analysis, quickly locates the specific cause of the abnormality, and finds the desired fault point.
[0031] S2: Calculate a storage urgency item and a flow suppression item, and calculate a dynamic sampling rate according to the storage urgency item and the flow suppression item; Preferably, in this embodiment, the dynamic sampling rate is calculated by storing the urgency item and the flow suppression item, so that the adaptive adjustment of log sampling can be achieved under the dual adjustment of storage drive and flow drive. Figure 2 As shown, the calculation method of the dynamic sampling rate specifically includes: S201: Calculating a storage urgency item based on the current usage of the storage system and the upper capacity limit of the storage system; Specifically, the specific calculation formula for storing the urgent item is: ; Wherein, S is the storage urgency item; The current usage of the storage system; is the upper limit of the capacity of the storage system; S202: Determine a flow fluctuation change rate based on the flow of the current sliding window and the flow of the previous sliding window; Specifically, the specific calculation formula for the flow fluctuation change rate is: ; Wherein, σ is the flow fluctuation change rate; As the basic traffic fluctuation value, after each sliding window, the traffic fluctuation change rate of the previous sliding window is used as the basic traffic fluctuation value of the current sliding window; is the traffic sequence in the current sliding window; is the traffic sequence in the previous sliding window; S203: Calculating traffic sensitivity based on the traffic fluctuation change rate and the stored urgency item; Specifically, the flow sensitivity determines the system's response speed to flow changes, and a higher flow sensitivity will make the effect of suppressing flow growth by adjusting the sampling rate more significant. When the traffic sensitivity is high, the sampling rate will be adjusted to decrease rapidly under high traffic conditions, which is beneficial to balancing storage capacity, but may result in more logs being lost. When the traffic sensitivity is small, the sampling rate will be adjusted smoothly when the traffic change is small, thereby ensuring higher log data fidelity, but may increase the risk of storage overflow.
[0032] Specifically, the specific calculation formula of the flow sensitivity is: ; Wherein, β(t) is the flow sensitivity at time t, and β(t)>0; is the basic flow sensitivity, and the flow sensitivity at the previous moment is used as the basic flow sensitivity at the current moment in each calculation; S is the storage urgency item; S204: Obtaining a flow suppression item based on the flow sensitivity; In this embodiment, the flow suppression term is used to achieve a nonlinear response in which the sampling rate decreases as the flow rate increases. The specific calculation formula of the flow suppression term is: ; Wherein, L is the traffic suppression term; λ(t) is the log input rate at time t.
[0033] S205: Calculating a dynamic sampling rate according to the storage urgency item and the traffic suppression item; Specifically, the specific calculation formula of the dynamic sampling rate is: ; Wherein, r(t) is the dynamic sampling rate at time t; α is the preset basic sampling rate, and 0<α<1; S is the storage urgency item; and L is the flow suppression item.
[0034] S3: performing log sampling based on the feature marker and the dynamic sampling rate; Furthermore, after completing the log feature marking and sampling rate calculation, log sampling is started; specifically, Figure 3 As shown in the figure, the specific process of dynamic log sampling and storage includes: S301: Determine whether the current log has a full storage mark; S302: If yes, directly call the sampling executor to perform full log sampling; S303: If not, the dynamic sampling rate is sent to the sampling executor, and the log is sampled according to the dynamic sampling rate; S304: Store the sampled logs and adjust the sampling rate in real time according to traffic changes and storage capacity.
[0035] S4: Stores sampled logs, monitors the storage system's capacity status in real time, and adjusts the dynamic sampling rate.
[0036] Furthermore, after the sampled logs are stored, the present invention sets up a real-time monitoring mechanism to monitor the capacity status of the storage system in real time. When the capacity of the storage system reaches a preset upper limit, the dynamic sampling rate controller is promptly notified to amplify the storage weight by the square of the storage urgency term, thereby reducing the sampling rate and avoiding storage overflow. The preset upper limit can be set according to actual conditions. For example, the maximum capacity upper limit of the storage system can be used as the preset upper limit, or a certain position that is about to reach the maximum capacity upper limit of the storage system can be used as the preset upper limit, etc. This embodiment does not impose any restrictions.
[0037] The log sampling method based on the adaptive adjustment mechanism described in the embodiment of the present invention can not only perform screening before log sampling and storage, reduce data redundancy, and improve the efficiency of subsequent log analysis; it can also quickly adjust the sampling rate under the dual adjustment of storage drive and traffic drive to adapt to different business needs; at the same time, by real-time monitoring of the capacity status of the storage system and adjusting the dynamic sampling rate, it can ensure the data fidelity of log sampling while balancing storage capacity.
[0038] Example 2 like Figure 4 As shown, an embodiment of the present invention provides a log sampling system 400 based on an adaptive adjustment mechanism, the system comprising: a log marking module 401, a sampling rate calculation module 402, a log sampling module 403 and a storage monitoring module 404; wherein: Log tagging module 401 is used to divide and obtain core systems, preset system keyword features, and perform feature tagging on logs according to preset rules; The log marking module 401 includes: a preliminary configuration unit 4011 and a feature marking unit 4012; wherein: The aforementioned preliminary configuration unit 4011 is configured to: divide the system based on the key business dimension and the key infrastructure dimension to obtain the core system, and preset system keyword features; wherein the system keyword features include system error codes and abnormal metadata; The feature marking unit 4012 is configured to perform feature marking on the log according to a preset rule, specifically including the following steps: Determine whether the current log belongs to the log in the core system; If yes, mark the current log as fully stored; If not, determine whether the current log contains the system keyword feature; If included, the current log will be marked as fully stored; If not included, no marking is performed.
[0039] A sampling rate calculation module 402 is configured to calculate a storage urgency item and a flow suppression item, and calculate a dynamic sampling rate based on the storage urgency item and the flow suppression item; The sampling rate calculation module 402 includes: a capacity evaluation unit 4021, a traffic sensing unit 4022, and a sampling rate calculation unit 4023; wherein: The capacity evaluation unit 4021 is configured to: calculate the storage urgency item based on the current usage of the storage system and the upper capacity limit of the storage system; The traffic sensing unit 4022 is configured to determine the traffic fluctuation change rate and traffic sensitivity based on the traffic in the current sliding window, and calculate a traffic suppression item. The specific calculation process of the traffic suppression item includes: Determine the traffic fluctuation change rate based on the traffic of the current sliding window and the traffic of the previous sliding window; Calculating flow sensitivity based on the flow fluctuation change rate and the storage urgency item; The flow suppression item is obtained based on the flow sensitivity.
[0040] The sampling rate calculation unit 4023 is configured to calculate a dynamic sampling rate based on the storage urgency item calculated by the capacity evaluation unit 4021 and the traffic suppression item calculated by the traffic sensing unit 4022 .
[0041] A log sampling module 403, configured to perform log sampling based on the feature marker and the dynamic sampling rate; The log sampling module 403 includes: a full sampling unit 4031 and a dynamic sampling unit 4032; wherein: The full sampling unit 4031 is configured to: perform full sampling on the logs with the full storage mark based on the sampling executor; The dynamic sampling unit 4032 is configured to: based on the dynamic sampling rate calculated by the sampling rate calculation unit 4023, control the sampling executor to dynamically sample the logs according to the dynamic sampling rate.
[0042] The storage monitoring module 404 is used to store the sampled logs, monitor the capacity status of the storage system in real time, and adjust the dynamic sampling rate; The storage monitoring module 404 includes a capacity monitoring unit 4041 and a capacity alarm unit 4042; wherein: The capacity monitoring unit 4041 is configured to: monitor the capacity status of the storage system in real time after the log is stored, and notify the capacity alarm unit 4042 when the capacity of the storage system reaches a preset upper limit; The capacity alarm unit 4042 is configured to receive the capacity upper limit notification sent by the capacity monitoring unit 4042 and feed it back to the dynamic sampling rate controller so that the dynamic sampling rate controller amplifies the storage weight by the square of the storage urgency item and adjusts the sampling rate.
[0043] In this embodiment, the log is marked with features by the log marking module 401 to implement screening before log sampling and storage, thereby reducing data redundancy; the dynamic sampling rate is calculated by the sampling rate calculation module 402, and log sampling is performed based on the log feature marks and the dynamic sampling rate; and the storage monitoring module 404 monitors the capacity status of the storage system in real time after the log is stored and adjusts the dynamic sampling rate, which can ensure the data fidelity of the log sampling while balancing the storage capacity.
[0044] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present invention are indicated by the claims. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.
Claims
1. A log sampling method based on an adaptive adjustment mechanism, characterized in that: The steps include: Divide and obtain the core system, preset system keyword features, and mark the logs according to the preset rules; Calculating a storage urgency item and a flow suppression item, and calculating a dynamic sampling rate according to the storage urgency item and the flow suppression item; Performing log sampling based on the feature marker and the dynamic sampling rate; The sampled logs are stored, the capacity status of the storage system is monitored in real time, and the dynamic sampling rate is adjusted.
2. The log sampling method according to claim 1, characterized in that: The feature marking of the log according to the preset rules specifically includes: Determine whether the current log belongs to the log in the core system; If yes, mark the current log as fully stored; If not, determine whether the current log contains the system keyword feature; If included, the current log will be marked as fully stored; If not included, no marking is performed.
3. The log sampling method according to claim 1, characterized in that: The storage urgency item is calculated based on the current usage of the storage system and the upper capacity limit of the storage system. The specific formula is: ; Wherein, S is the storage urgency item; The current usage of the storage system; is the upper limit of the capacity of the storage system.
4. The log sampling method according to claim 1, characterized in that: The calculation of the traffic suppression item specifically includes: Determine the traffic fluctuation rate based on the traffic in the current sliding window and the traffic in the previous sliding window: ; Wherein, σ is the flow fluctuation change rate; As the basic traffic fluctuation value, after each sliding window, the traffic fluctuation change rate of the previous sliding window is used as the basic traffic fluctuation value of the current sliding window; is the traffic sequence in the current sliding window; is the traffic sequence in the previous sliding window; Based on the traffic fluctuation change rate and the storage urgency item, traffic sensitivity is calculated: ; Wherein, β(t) is the flow sensitivity at time t, and β(t)>0; is the basic flow sensitivity, and the flow sensitivity at the previous moment is used as the basic flow sensitivity at the current moment in each calculation; S is the storage urgency item; Based on the traffic sensitivity, the traffic suppression item is obtained: ; Wherein, L is the traffic suppression term; λ(t) is the log input rate at time t.
5. The log sampling method according to claim 1, characterized in that: The calculation of the dynamic sampling rate is specifically as follows: ; Wherein, r(t) is the dynamic sampling rate at time t; α is the preset basic sampling rate, and 0<α<1; S is the storage urgency item; and L is the flow suppression item.
6. The log sampling method according to claim 1, characterized in that: The log sampling is performed based on the feature mark and the dynamic sampling rate, specifically: Determine whether the current log has full storage; If yes, the sampling executor is called directly to perform full log sampling; If not, the dynamic sampling rate is sent to the sampling executor, and the log is sampled according to the dynamic sampling rate.
7. The log sampling method according to claim 1, characterized in that: When the capacity status of the storage system is monitored in real time, if the capacity of the storage system reaches a preset upper limit, the dynamic sampling rate controller is notified to amplify the storage weight by the square of the storage urgency term and adjust the sampling rate.
8. A log sampling system based on an adaptive adjustment mechanism, characterized in that: include: The log tagging module is used to divide and obtain the core system, preset system keyword features, and perform feature tagging on the log according to preset rules; a sampling rate calculation module, configured to calculate a storage urgency item and a flow suppression item, and calculate a dynamic sampling rate based on the storage urgency item and the flow suppression item; A log sampling module, configured to perform log sampling based on the feature marker and the dynamic sampling rate; The storage monitoring module is used to store sampled logs, monitor the capacity status of the storage system in real time, and adjust the dynamic sampling rate.
9. An electronic device, characterized in that: include: one or more processors; The memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the log sampling method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the log sampling method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method and device for controlling system logs
CN103838659A
Log output control method and device
CN108897663A
Log processing method and device, electronic equipment and storage medium
CN113791946A
Link sampling number distribution method, device, equipment and medium
CN113949624A
Multi-stage feature extraction for effective ML-based anomaly detection on structured log data
US20220292304A1