Lightweight log filing method and system
Through the lightweight log archiving method, automatic rotation, compression and multi-dimensional triggering conditions are used, combined with event-driven model and logical marking mechanism, the problem of low efficiency in log data management is solved, efficient log storage and data security is achieved, and the Operation and maintenance costs.
Patent Information
- Application Number
- CN202411009873.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
AI Technical Summary
The amount of enterprise-level application log data is growing rapidly, and unmanaged log files will consume valuable storage resources, increase operation and maintenance costs and lead to system performance degradation.
Provide a lightweight log archiving method, combining event-driven models and multi-dimensional triggering conditions through automatic rotation, compression, deletion and remote archiving functions, using compression algorithms and logical marking mechanisms to ensure data integrity, and through pre-/post-processing scripts Perform data desensitization and encryption and decryption processing.
Effectively control log storage space usage, improve system stability and operation and maintenance efficiency, ensure timely processing of key log data, reduce learning and management costs, and enhance the flexibility and security of data processing.
Smart Images

Figure CN119088757A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a method and a system, and relates to the technical field of data management, in particular to a lightweight log archiving method and a system. Background Art
[0002] With the rapid development of information technology, enterprise-level applications and services are becoming increasingly complex, and the amount of log data generated is growing exponentially. These log files not only contain key information about the system's operating status, but are also an important basis for troubleshooting, performance optimization, and compliance audits. However, if unmanaged log files accumulate indefinitely, they will quickly consume valuable storage resources, increase operation and maintenance costs, and even cause system performance to deteriorate. Summary of the invention
[0003] In view of the problems of the prior art, the present invention provides a lightweight log archiving method and system, which realizes functions such as automatic rotation, compression, deletion and remote archiving of log files, effectively controls the log storage space occupancy, and improves the stability and operation and maintenance efficiency of the system.
[0004] The specific scheme proposed by the present invention is:
[0005] The present invention provides a lightweight log archiving method, comprising:
[0006] Step 1: Configure the log archiving task: register the service or component log according to the relevant parameters entered by the user, specify the storage location for the log archive, and rename the log with the service name and timestamp to facilitate identification based on the trigger conditions.
[0007] Step 2: Preset the triggering conditions for the log archiving task: customize the fixed archiving period or time point, log file size, and log storage days of the log archiving task; set the triggering hard limit and soft limit. The hard limit needs to meet all the set rules to trigger the log archiving task, and the soft limit only needs to meet one of all the rules to trigger the log archiving task;
[0008] Step 3: Use an event-driven model and monitoring tools to track changes in log files in real time. Once the trigger conditions are met, execute the corresponding log archiving task immediately.
[0009] Step 4: Use compression algorithms to compress log data based on the characteristics of log data and storage requirements, and perform log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
[0010] Furthermore, in step 1 of the lightweight log archiving method, a customized pre-processing and post-processing script is written and executed through an API interface according to a specific business scenario, and the pre-processing and post-processing scripts are used to perform data desensitization, log parsing, or encryption and decryption on the log files before and after log archiving.
[0011] Furthermore, in the described lightweight log archiving method, in step 4, a gzip, bzip2, lz4 or zstd compression algorithm is selected to compress the log data according to the characteristics of the log data and the storage requirements, and a verification comparison is performed before and after the log data is decompressed to identify whether the log data is damaged during transmission or storage, thereby ensuring the integrity and consistency of the log data.
[0012] Furthermore, in the aforementioned lightweight log archiving method, performing log verification during log archiving in step 4 includes:
[0013] Establish a logical marking mechanism: Before log data is selected for archiving, first mark the log data in the database or metadata storage as to-be-archived or processed, track the processing progress of each log data, and ensure that in subsequent archiving cycles, only the log data that is not marked as archived is processed.
[0014] Perform physical archiving: transfer the log data marked as to be archived to the storage location. At the same time as physical archiving, update the archiving status of the log data and mark it as archived.
[0015] Perform status synchronization and verification: After completing physical archiving, verify the consistency between the logical mark label and the physical storage status, review the archive logs, confirm all successfully migrated log status changes, compare the logical mark label with the actual content of the physical storage, and check whether there are any omissions or incorrect markings.
[0016] Furthermore, the lightweight log archiving method also includes step 5: performing error detection and processing logic, wherein a log archiving task processing exception or logic error is detected, the preset retry logic is automatically executed, and the error recording process is triggered immediately, and the time of the error, context information, and stack trace are recorded in detail for later analysis, and an error report is generated.
[0017] The present invention also provides a lightweight log archiving system, including a configuration module, a condition trigger module, a monitoring module, a compression module and a verification module.
[0018] The configuration module configures log archiving tasks: register service or component logs according to the relevant parameters entered by the user, specify the storage location for log archives, rename logs using service names and timestamps to facilitate identification based on trigger conditions,
[0019] The conditional trigger module presets the trigger conditions for log archiving tasks: customize the fixed archiving period or time point of log archiving tasks, log file size, and log storage days; set the hard limit and soft limit of the trigger. The hard limit needs to meet all the set rules to trigger the log archiving task, and the soft limit only triggers the log archiving task as long as one of all the rules is met;
[0020] The monitoring module adopts an event-driven model and cooperates with monitoring tools to track the changes of log files in real time. Once the trigger conditions are met, the corresponding log archiving tasks are executed immediately.
[0021] The compression module uses a compression algorithm to compress log data according to the characteristics of the log data and storage requirements. The verification module performs log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
[0022] Furthermore, the configuration module of the lightweight log archiving system writes and executes customized pre-processing and post-processing scripts according to specific business scenarios through an API interface, and performs data desensitization, log parsing or encryption and decryption on log files before and after log archiving through the pre-processing and post-processing scripts.
[0023] Furthermore, the compression module of the lightweight log archiving system selects gzip, bzip2, lz4 or zstd compression algorithm to compress log data according to the characteristics of the log data and storage requirements, and performs verification and comparison before and after the log data is decompressed to identify whether the log data is damaged during transmission or storage, thereby ensuring the integrity and consistency of the log data.
[0024] Furthermore, the verification module of the lightweight log archiving system performs log verification when archiving logs, including:
[0025] Establish a logical marking mechanism: Before log data is selected for archiving, first mark the log data in the database or metadata storage as to-be-archived or processed, track the processing progress of each log data, and ensure that in subsequent archiving cycles, only the log data that is not marked as archived is processed.
[0026] Perform physical archiving: transfer the log data marked as to be archived to the storage location. At the same time as physical archiving, update the archiving status of the log data and mark it as archived.
[0027] Perform status synchronization and verification: After completing physical archiving, verify the consistency between the logical mark label and the physical storage status, review the archive logs, confirm all successfully migrated log status changes, compare the logical mark label with the actual content of the physical storage, and check whether there are any omissions or incorrect markings.
[0028] Furthermore, the lightweight log archiving system also includes an error detection module, which performs error detection and processing logic. When an abnormality or logical error is detected in the log archiving task processing, the preset retry logic is automatically executed, and the error recording process is triggered immediately. The time, context information, and stack trace of the error are recorded in detail for later analysis, and an error report is generated at the same time.
[0029] The benefits of the present invention are:
[0030] The present invention realizes the automatic rotation, intelligent compression, flexible archiving and efficient management of logs. Its multi-dimensional trigger condition setting and event-driven mode enable the archiving strategy to closely fit various business scenarios, whether it is a fixed cycle or real-time response, to ensure that key log data is processed in a timely and proper manner. Through the pre- / post-processing script interface, users can perform data desensitization, parsing and other operations according to specific needs, which enhances the flexibility and security of data processing. Efficient compression and decompression functions optimize the use of storage resources. The combination of logical and physical archiving strategies, as well as a comprehensive error handling and reporting mechanism, further consolidate the robustness and reliability of data management, and ensure the efficient and stable operation of the system. In addition, the lightweight design makes the system easy to deploy and respond quickly, reducing learning and management costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a schematic diagram of the application framework of the present invention.
[0032] Figure 2 It is a schematic diagram of log archiving task configuration of the present invention.
[0033] Figure 3 It is a schematic diagram of the scheduling of the log archiving event-driven model of the present invention.
[0034] Figure 4 It is a schematic diagram of the filing verification process of the present invention. DETAILED DESCRIPTION
[0035] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0036] Example 1
[0037] The present invention provides a lightweight log archiving method, comprising:
[0038] Step 1: Configure the log archiving task: Register the service or component log according to the relevant parameters entered by the user, specify the storage location for the log archive, and rename the log with the service name and timestamp to facilitate identification based on the trigger conditions.
[0039] In step 1, you can also use the API interface to write and execute custom pre-processing and post-processing scripts according to specific business scenarios. The pre-processing and post-processing scripts can be used to perform data desensitization, log parsing, or encryption and decryption on log files before and after log archiving.
[0040] The API interface meets the user's highly customized needs for log processing, allowing users to write and execute customized pre-processing and post-processing scripts according to specific business scenarios. Perform tasks such as data desensitization, log analysis, log simplification or addition, key content extraction, and backup. With this API capability, users can easily implement encryption and decryption operations at key nodes of log transmission, encrypt and protect logs before archiving, and decrypt logs after archiving, ensuring that sensitive data is strictly protected during network transmission and effectively resisting potential information leakage risks.
[0041] This function not only greatly enhances the flexibility, adaptability and security of the system, but also enables the system to better serve diverse business needs.
[0042] Step 2: Preset the triggering conditions for the log archiving task: customize the fixed archiving period or time point of the log archiving task. The time point is set using Cron expressions, log file size, and number of days for log storage. Set the triggering hard limit and soft limit. The hard limit needs to meet all the set rules to trigger the log archiving task. The soft limit triggers the log archiving task as long as one of all the rules is met.
[0043] Step 3: Use an event-driven model and monitoring tools to track changes in log files in real time. Once the trigger conditions are met, execute the corresponding log archiving task immediately. This not only avoids resource waste, but also ensures the complete preservation of key log data.
[0044] Step 4: Use compression algorithms to compress log data based on the characteristics of log data and storage requirements, and perform log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
[0045] You can further select gzip, bzip2, lz4 or zstd compression algorithms to compress log data according to the characteristics of log data and storage requirements, and perform verification and comparison before and after the log data is decompressed to identify whether the log data is damaged during transmission or storage, so as to ensure the integrity and consistency of the log data.
[0046] When performing log verification during log archiving, the following can be further included:
[0047] Establish a logical marking mechanism: Before log data is selected for archiving, first mark the log data in the database or metadata storage as to-be-archived or processed, track the processing progress of each log data, and ensure that in subsequent archiving cycles, only the log data that is not marked as archived is processed.
[0048] Perform physical archiving: transfer the log data marked as to be archived to the storage location. At the same time as physical archiving, update the archiving status of the log data and mark it as archived.
[0049] Perform status synchronization and verification: After completing physical archiving, verify the consistency between the logical tag and the physical storage status, recheck the archive logs, confirm all successfully migrated log status changes, and compare the logical tag with the actual content of the physical storage to check whether there are any omissions or incorrect tags. This allows timely corrections to avoid duplicate archiving problems caused by inconsistent data.
[0050] The method of the present invention also provides a cluster log management solution for complex distributed deployment environments. For example, once the user completes a simple registration process for the cluster, the integration and unified management of all server logs within the cluster can be automatically realized. These log data scattered on different nodes will be efficiently and orderly collected to the central log server for centralized storage and archiving. This significantly reduces the tedious work of manual archiving and classification in large-scale deployment scenarios.
[0051] The method of the present invention performs lightweight log archiving, significantly improving the automation level and resource utilization efficiency of log management. Through a series of innovative designs, it effectively responds to the challenge of massive growth of log data brought about by the rapid development of information technology, and provides high-efficiency, low-maintenance log management.
[0052] Example 2
[0053] Based on Example 1, the present invention also includes step 5: performing error detection and processing logic, wherein a log archiving task processing exception or logic error is detected, the preset retry logic is automatically executed, and the error recording process is immediately triggered, and the time of the error, context information, and stack trace are recorded in detail for later analysis, and an error report is generated.
[0054] It performs error detection and processing logic, and if problems are encountered during log processing, appropriate measures can be taken, such as automatic retry and error log recording. In terms of practical operations, first of all, once a task processing exception or logic error is detected, the preset retry logic is automatically executed to solve problems caused by common reasons such as temporary network fluctuations and resource competition. The retry mechanism is designed with an intelligent backoff algorithm to avoid frequent retries in a short period of time that may cause system resource overload, ensuring that the retry behavior is both efficient and does not affect the overall service stability. For complex errors that cannot be solved by simple retries, the error recording process is triggered immediately to record the time, context information, and stack trace of the error in detail for later analysis.
[0055] You can also configure whether to enable the error reporting function. When enabled, the configured error reporting policy will be connected to an external notification service, such as an email system or a cloud monitoring platform. When an error occurs, the system will automatically send a report to the specified mailbox or monitoring platform to ensure that the problem is discovered and handled in a timely manner.
[0056] Example 3
[0057] The present invention also provides a lightweight log archiving system, including a configuration module, a condition trigger module, a monitoring module, a compression module and a verification module.
[0058] The configuration module configures log archiving tasks: register service or component logs according to the relevant parameters entered by the user, specify the storage location for log archives, rename logs using service names and timestamps to facilitate identification based on trigger conditions,
[0059] The conditional trigger module presets the trigger conditions for log archiving tasks: customize the fixed archiving period or time point of log archiving tasks, log file size, and log storage days; set the hard limit and soft limit of the trigger. The hard limit needs to meet all the set rules to trigger the log archiving task, and the soft limit only triggers the log archiving task as long as one of all the rules is met;
[0060] The monitoring module adopts an event-driven model and cooperates with monitoring tools to track the changes of log files in real time. Once the trigger conditions are met, the corresponding log archiving tasks are executed immediately.
[0061] The compression module uses a compression algorithm to compress log data according to the characteristics of the log data and storage requirements. The verification module performs log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
[0062] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.
[0063] Similarly, the system of the present invention realizes automatic rotation, intelligent compression, flexible archiving and efficient management of logs. Its multi-dimensional trigger condition setting and event-driven mode enable the archiving strategy to closely fit various business scenarios, whether it is a fixed cycle or real-time response, to ensure that key log data is processed in a timely and proper manner. Through the pre- / post-processing script interface, users can perform data desensitization, parsing and other operations according to specific needs, which enhances the flexibility and security of data processing. Efficient compression and decompression functions optimize the use of storage resources. The combination of logical and physical archiving strategies, as well as comprehensive error handling and reporting mechanisms, further consolidate the robustness and reliability of data management, and ensure the efficient and stable operation of the system. In addition, the lightweight design makes the system easy to deploy and respond quickly, reducing learning and management costs.
[0064] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.
[0065] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A lightweight log archiving method, Its characteristics are include: Step 1: Configure the log archiving task: register the service or component log according to the relevant parameters entered by the user, specify the storage location for the log archive, and rename the log with the service name and timestamp to facilitate identification based on the trigger conditions. Step 2: Preset the triggering conditions for the log archiving task: customize the fixed archiving period or time point, log file size, and log storage days of the log archiving task; set the triggering hard limit and soft limit. The hard limit needs to meet all the set rules to trigger the log archiving task, and the soft limit only needs to meet one of all the rules to trigger the log archiving task; Step 3: Use an event-driven model and monitoring tools to track changes in log files in real time. Once the trigger conditions are met, execute the corresponding log archiving task immediately. Step 4: Use compression algorithms to compress log data based on the characteristics of log data and storage requirements, and perform log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
2. According to a lightweight log archiving method described in claim 1, it is characterized in that in step 1, a customized preprocessing and post-processing script is written and executed according to a specific business scenario through an API interface, and the preprocessing and post-processing scripts are used to perform data desensitization, log parsing or encryption and decryption on the log files before and after log archiving.
3. A lightweight log archiving method according to claim 1, characterized in that in step 4, gzip, bzip2, lz4 or zstd compression algorithm is selected to compress the log data according to the characteristics of the log data and storage requirements, and verification and comparison are performed before and after the log data is decompressed to identify whether the log data is damaged during transmission or storage, thereby ensuring the integrity and consistency of the log data.
4. A lightweight log archiving method according to claim 1 or 3, Its characteristics are In step 4, log verification is performed when the log is archived, including: Establish a logical marking mechanism: Before log data is selected for archiving, first mark the log data in the database or metadata storage as to-be-archived or processed, track the processing progress of each log data, and ensure that in subsequent archiving cycles, only the log data that is not marked as archived is processed. Perform physical archiving: transfer the log data marked as to be archived to the storage location. At the same time as physical archiving, update the archiving status of the log data and mark it as archived. Perform status synchronization and verification: After completing physical archiving, verify the consistency between the logical mark label and the physical storage status, review the archive logs, confirm all successfully migrated log status changes, compare the logical mark label with the actual content of the physical storage, and check whether there are any omissions or incorrect markings.
5. A lightweight log archiving method according to claim 1, Its characteristics are It also includes step 5: error detection and processing logic, in which an exception or logic error in the log archiving task processing is detected, the preset retry logic is automatically executed, and the error recording process is triggered immediately, recording the time, context information, and stack trace of the error in detail for later analysis, and generating an error report.
6. A lightweight log archiving system, Its characteristics are Including configuration module, condition trigger module, monitoring module, compression module and verification module, The configuration module configures log archiving tasks: register service or component logs according to the relevant parameters entered by the user, specify the storage location for log archives, rename logs using service names and timestamps to facilitate identification based on trigger conditions, The conditional trigger module presets the trigger conditions for log archiving tasks: customize the fixed archiving period or time point of log archiving tasks, log file size, and log storage days; set the hard limit and soft limit of the trigger. The hard limit needs to meet all the set rules to trigger the log archiving task, and the soft limit only triggers the log archiving task as long as one of all the rules is met; The monitoring module adopts an event-driven model and cooperates with monitoring tools to track the changes of log files in real time. Once the trigger conditions are met, the corresponding log archiving tasks are executed immediately. The compression module uses a compression algorithm to compress log data according to the characteristics of the log data and storage requirements. The verification module performs log verification when archiving logs to ensure data integrity and avoid duplicate archiving.
7. A lightweight log archiving system according to claim 6, Its characteristics are The configuration module uses the API interface to write and execute customized pre-processing and post-processing scripts according to specific business scenarios. The pre-processing and post-processing scripts perform data desensitization, log parsing, or encryption and decryption on log files before and after log archiving.
8. A lightweight log archiving system according to claim 6, Its characteristics are The compression module selects gzip, bzip2, lz4 or zstd compression algorithm to compress log data according to the characteristics and storage requirements of log data, and performs verification and comparison before and after decompression of log data to identify whether the log data is damaged during transmission or storage, so as to ensure the integrity and consistency of log data.
9. A lightweight log archiving system according to claim 6 or 8, Its characteristics are The verification module performs log verification when archiving logs, including: Establish a logical marking mechanism: Before log data is selected for archiving, first mark the log data in the database or metadata storage as to-be-archived or processed, track the processing progress of each log data, and ensure that in subsequent archiving cycles, only the log data that is not marked as archived is processed. Perform physical archiving: transfer the log data marked as to be archived to the storage location. At the same time as physical archiving, update the archiving status of the log data and mark it as archived. Perform status synchronization and verification: After completing physical archiving, verify the consistency between the logical mark label and the physical storage status, review the archive logs, confirm all successfully migrated log status changes, compare the logical mark label with the actual content of the physical storage, and check whether there are any omissions or incorrect markings.
10. A lightweight log archiving system according to claim 6, Its characteristics are It also includes an error detection module, which performs error detection and processing logic. If an exception or logical error is detected in the log archiving task processing, the preset retry logic is automatically executed, and the error recording process is triggered immediately. The time, context information, and stack trace of the error are recorded in detail for later analysis, and an error report is generated at the same time.
Citation Information
Patent Citations
Data disaster backup control system and data recovery method
CN103226502A
Log management method and server
CN107180051A
Log file roll storage method and device, storage medium and terminal
CN111597158A
Iotify file monitoring management method and system, electronic equipment and storage medium
CN117667862A
Cited By
Distribution automation operation and maintenance work log intelligent recording platform based on CS architecture
CN119692728A
