Log full life cycle management method based on object storage

Abstract

The invention discloses a log full-life-cycle management method based on object storage, and is suitable for the field of communication software and electricity. The method comprises the steps that a client side monitors changes of log files, file compression is triggered when writing of the log files is stopped through an inotify mechanism of an operating system, then the file compression is uploaded to object storage, and file consistency is verified through HASH. And the server divides the log file into common storage, low-frequency access and archiving storage according to the storage time and the access frequency through a life cycle management strategy so as to reduce the storage cost. Under a specific storage condition, the system automatically deletes useless log files. A user can recover the archived log file through a request, and the server side executes unfreezing operation according to the current storage state and sends a recovery instruction. According to the method, full-automatic management of the log file from generation to storage, recovery and deletion is realized, and the safety and the cost efficiency of data storage are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication software technology, and in particular to a log full life cycle management method based on object storage. Background Art

[0002] During the continuous operation of Internet applications, application servers will generate a large number of log files, which are used for system monitoring, troubleshooting and data analysis. However, due to the limited disk space of application servers, these log files are difficult to save for a long time and are prone to excessive disk usage, affecting the normal operation of the system. In addition, manual management of log files is not only complex and inefficient, but also difficult to cope with the growing amount of data and regulatory compliance requirements.

[0003] In current solutions, manual backup or simple scheduled backup methods are inefficient, prone to errors, and cannot meet the efficiency and security requirements of enterprise data management. In contrast, object storage has become an ideal choice for log file backup and storage due to its advantages such as high reliability, low cost and elastic expansion. The present invention divides the storage of log files into three categories of management: hot data, cold data and archived data through the reasonable configuration of the object storage system, thereby achieving efficient management and intelligent storage of log data. Summary of the invention

[0004] The present invention relates to a log full life cycle management method based on object storage, which belongs to the technical field of communication software and electrical engineering, and aims to solve the problems of low storage efficiency, complex operation and high risk of data loss in the traditional log management process. By combining the client, server and object storage system, the functions of automatic compression, uploading, storage, recovery and cleaning of log files are realized, so as to achieve the goal of efficient, safe and low-cost log full life cycle management.

[0005] Technical Solution

[0006] The present invention proposes a log full life cycle management method based on object storage, which specifically includes the following steps:

[0007] Real-time log monitoring and compression

[0008] The client monitors the log directory in real time through the inotify mechanism of the operating system. The log file is truncated by the application or logrotate tool and a new log file is generated. When it is detected that the log file stops writing and generates a close event (close_write), the client immediately starts the compression process. The compression algorithm can be selected from gzip, LZMA, etc. The most suitable algorithm is dynamically selected according to the characteristics of the file content to ensure the best compression ratio and compression speed.

[0009] Compressed file consistency check and upload

[0010] After compression is completed, the client verifies the consistency of the compressed file by calculating the HASH value of the file (such as MD5, SHA-256) to ensure that no data is lost or damaged during the file upload process. After verification, the client uploads the compressed log file to the object storage system. After the upload is successful, the client notifies the server through the API and deletes the log file on the server after confirming that the file backup is successful to free up disk space.

[0011] Log file lifecycle management

[0012] The server receives the notification from the client and records the path, upload time, and storage server information of the log file. The server stores the log file in stages according to the preset lifecycle management rules:

[0013] Hot data stage: Newly uploaded log files are first stored as ordinary storage types for quick access and are usually used for application performance monitoring and troubleshooting.

[0014] Low-frequency access stage: When a log file is uploaded for more than 30 days and there is no access record, the system automatically converts it to a low-frequency access storage type to reduce storage costs.

[0015] Archiving stage: If a log file has been stored for more than 60 days without any access record, the system will convert it to the archive storage type to further reduce storage costs, but the reading delay will be relatively long.

[0016] Deletion stage: For log files that have been archived for more than 6 months, the system will automatically delete them if there is no special need to save storage space.

[0017] Log file recovery process

[0018] When a user needs a specific log file, he can initiate a restore request through the server, and the server selects the appropriate restore operation based on the current storage status of the log file:

[0019] If the file is located in the hot data area, it is downloaded directly to the client;

[0020] If the file is in a low-frequency or archive storage area, the server sends a thawing request to the object storage to temporarily restore the file to a readable state, and after completion sends a download instruction to the client.

[0021] After receiving the restore instruction, the client downloads the specified file from the object storage and restores the decompressed file to the original path to ensure normal access to the application.

[0022] Two-way communication mechanism

[0023] The present invention establishes a two-way communication mechanism between the client and the server. The client interacts with the server through the API during the compression and upload process, and the server is responsible for the status recording and life cycle management of the log file to ensure the integrity and consistency of the data throughout the life cycle.

[0024] Exception handling

[0025] During the log recovery process, if the client encounters file corruption or download failure, it will feed back the error information to the server. The server will take corresponding measures based on the error type, such as retrying the recovery operation or notifying the system administrator for manual intervention to ensure the accuracy and completeness of data recovery.

[0026] Beneficial effects:

[0027] Efficient automated management

[0028] The present invention utilizes the inotify mechanism of the operating system and preset compression, upload, and deletion rules to realize automatic management of the entire life cycle of log files, greatly reducing the complexity and error rate of manual operations.

[0029] The balance between cost and data security

[0030] Object storage is low-cost and highly scalable. Through a multi-stage lifecycle management strategy, it can reasonably configure storage types under different access requirements and avoid unnecessary storage overhead. In addition, the two-way communication mechanism between the client and the server ensures the traceability and security of data throughout its lifecycle.

[0031] Fast data recovery

[0032] Through the multi-layer storage architecture of object storage and the thawing and recovery process on the server, users can quickly restore the required log files at any stage, improving the efficiency of system troubleshooting and data analysis.

[0033] Scalability and flexibility

[0034] The present invention makes full use of the elastic expansion characteristics of object storage, and can easily cope with storage needs as the amount of business data grows, without worrying about storage capacity limitations. At the same time, flexible lifecycle management strategies support different storage and deletion condition settings to adapt to different application scenarios.

[0035] The innovations of the present invention are mainly reflected in the following aspects:

[0036] Real-time monitoring and automated operation

[0037] The inotify mechanism is used to monitor the writing status of log files. After the file is written, compression, upload, and deletion operations can be started immediately to ensure real-time backup and persistent storage of data.

[0038] Multi-stage life cycle management

[0039] Log files are stored in layers according to access frequency and time, implementing multi-stage management such as normal, low-frequency, archiving and deletion to achieve cost optimization and maximize storage efficiency.

[0040] Two-way communication mechanism and recovery process

[0041] The two-way communication mechanism between the server and the client ensures the efficiency and security of log file management and recovery. The thawing operation of the recovery process ensures the readability of archived data, effectively supporting data analysis and troubleshooting needs. The present invention is suitable for the data management needs of large-scale Internet applications, especially the automated storage and management of log files. It has broad application prospects in log backup and recovery management in industries such as finance, telecommunications, and e-commerce. DETAILED DESCRIPTION

[0042] Example 1

[0043] The present invention provides a log full life cycle management method based on object storage. Specifically, through the collaborative work of client programs, server management systems and object storage resources, the full life cycle management of log file monitoring, compression, uploading, archiving, recovery and deletion is realized.

[0044] Log file monitoring

[0045] The client program uses the inotify mechanism of the operating system to monitor changes in files in the log directory. When the log file is written and a file close event is generated, the client captures the path of the file and starts the compression process.

[0046] Log file compression and upload

[0047] The client compresses the log file according to the pre-configured compression algorithm (such as gzip, LZMA, etc.). After compression, the file consistency is verified by calculating the HASH value of the file (such as MD5, SHA-256), and the compressed file is uploaded to the object storage after confirmation.

[0048] Log file deletion

[0049] After the file is successfully uploaded and the backup is confirmed to be correct, the client deletes the log file on the server to save disk space.

[0050] Example 2

[0051] Log lifecycle management

[0052] The server divides log files into three stages for management in object storage according to the access frequency and time of log files: general storage, low-frequency access, and archive storage.

[0053] Normal storage: Log files uploaded within one month are set to normal storage for quick access.

[0054] Low-frequency access: Files that have not been accessed for 30 days after being uploaded are automatically transferred to low-frequency storage.

[0055] Archive storage: Files that have not been accessed after 60 days are further transferred to archive storage.

[0056] Deletion stage: The system automatically deletes useless log files after 6 months of archiving.

[0057] Example 3

[0058] Log recovery process

[0059] When a user requests to restore a log file, the server processes it based on the current storage status of the log file:

[0060] If it is a common storage file, restore it directly;

[0061] If it is a low-frequency or archived storage file, perform the thawing operation and then restore it.

[0062] Two-way communication mechanism

[0063] The communication between the client and the server is completed through the API interface. The server records the path, upload time and server information of the log file, and accepts the client's query instructions at any time.

[0064] Exception handling

[0065] During the recovery process, if problems such as download failure or file corruption occur, the client will feed back the error information to the server, and the server will take measures such as retrying recovery and notifying the administrator to ensure data integrity and a smooth recovery process.

[0066] Technical advantages

[0067] Real-time monitoring: The inotify mechanism is used to achieve real-time monitoring and response of log files.

[0068] Efficient storage and recovery: Through the lifecycle management strategy of object storage, storage costs and data reading efficiency are optimized.

[0069] Flexible data management: Provides convenient data recovery and deletion operations to meet the needs of different scenarios.

[0070] Summarize

[0071] This invention realizes efficient, economical and reliable full life cycle management of log files based on object storage, effectively solves the problems of complex and error-prone traditional log backup operations, and has wide practicality.

[0072] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0073] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A log full life cycle management method based on object storage, characterized in that: The method comprises a client, a server and an object storage system, wherein the method comprises the following steps: The client monitors changes in log files. When log files stop being written, the client triggers file compression and uploads it to the object storage system. The server manages the storage type of files according to the life cycle rules of log files, supporting general storage, low-frequency access, and archive storage; Automatically delete log files that are no longer needed under specified conditions.

2. According to the object storage-based log full life cycle management method of claim 1, it is characterized in that: The client uses the inotify mechanism of the operating system to monitor the changes of the log file in real time, capture the file closing event, and start the compression and upload process.

3. The method according to claim 1 or 2, characterized in that: The client compresses the log file using a preset compression algorithm (such as gzip or LZMA) and verifies the consistency of the uploaded file using a HASH value.

4. The method according to claim 1, characterized in that The server converts the log files from ordinary storage to low-frequency access and then to archive storage in sequence according to the access frequency and storage time of the log files.

5. The method according to claim 4, characterized in that The conversion basis of the common storage, low-frequency access and archive storage is the storage time after the log file is uploaded: one month for common storage, 30 days for low-frequency access and 60 days for archive storage.

6. The method according to claim 1, characterized in that The server automatically deletes log files that are no longer needed after six months of archiving to save storage space.

7. The method according to claim 1, characterized in that Users can request to restore archived log files through the server. The server performs the thawing operation according to the file storage type and sends the recovery instruction to the client.

8. The method according to claim 7, characterized in that When the server thaws the low-frequency or archived log files, it will temporarily restore them to a readable state.

9. The method according to claim 1, characterized in that: The client interacts with the server through a two-way communication mechanism, receives instructions such as restore and delete, and feeds back the processing results to the server.

10. The method according to claim 1, characterized in that After the log file recovery operation is completed, the client automatically updates the file status to the server to maintain the integrity and consistency of the data.