Log backup method and device based on cloud platform

Abstract

The invention discloses a log backup method and device based on a cloud platform, and relates to the field of data storage and data security. Comprising the steps of (1) collecting a log file in a container, (2) storing the log file, (3) backing up a log: deploying a log backup service Elasticdump tool, and exporting data in an Elasticsearch index as a JSON (JavaScript Object Notation) file; a timed task is set, a backups.sh script file is executed regularly every day through the timed task, an Elasticdump tool is called, and data in an index of the previous day is backed up; the backup data is packed and compressed into a data file for uploading, and step 4, uploading the backup data file: uploading the data file to a cloud for storage.

Description

Technical Field

[0001] The present invention discloses a log backup method and device based on a cloud platform, and relates to the field of data storage and data security. Background Art

[0002] With the rapid development of cloud computing technology, more and more enterprises and individuals choose to transfer data storage and processing tasks to cloud platforms. However, as an important basis for system operation and troubleshooting, the security and reliability of log data on cloud platforms are particularly important. Existing log backup methods mostly rely on local storage, which has the risk of single point failure and is not easy to manage and expand. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a log backup method and device based on a cloud platform, which improves the security and reliability of log data and reduces management costs.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a log backup method based on a cloud platform, comprising:

[0006] Step 1: Collect log files inside the container: mount the log files in the Kubernetes container to the specified location of the host through the configMap configuration. On each node of the Kubernetes cluster, deploy the log collection agent program in the form of DaemonSet. Use the agent program to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time.

[0007] Step 2: Store log files: Store log file data through ElasticSearch, set the Elasticsearch index template and preset the index strategy, automatically create the corresponding index according to the index strategy and index template, and store the collected log files in the corresponding index.

[0008] Step 3: Back up logs: Deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; set up a scheduled task to execute the backups.sh script file every day through the scheduled task, call the Elasticdump tool, and back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading.

[0009] Step 4: Upload backup data files: Upload data files to the cloud for storage.

[0010] Furthermore, in step 1 of the log backup method based on a cloud platform, an agent program is used to configure different log parsing rules for different types of log files, and the log files are filtered, cleaned, formatted, and encrypted to verify the accuracy, integrity, and security of the data; for the collected log files, the agent program is used to calculate the hash value of the log file, and the hash value is recorded in the pos_file log file, which is used to compare with the hash value of the current log file during the next collection to see whether they are consistent, ensuring that the data reading maintains consistency and integrity when the log file is rotated or the log file is modified externally.

[0011] Furthermore, when setting the Elasticsearch index template in step 2 of the cloud platform-based log backup method, the format of the index name is set in the index template to the format of alias + date, and indexes with the same log type but different dates are defined to have the same alias, wherein the format of the index name is set to xxx.%Y%m%d, the prefix xxx represents the name of the alias, the suffix Y% represents the year, m% represents the month, and d% represents the date.

[0012] Furthermore, in step 2 of the cloud platform-based log backup method, a timed deletion strategy for the index is set, and indexes older than a preset number of days are automatically deleted according to the timed deletion strategy, thereby releasing system memory and optimizing query efficiency.

[0013] Furthermore, in step 4 of the log backup method based on a cloud platform, a RESTful API interface or SDK package of the object storage service OSS in the cloud service is called to upload the backed-up data files to the specified path at a regular interval, wherein different types of log backup data files are uploaded to different paths, and after the upload is completed, it is verified whether the hash value of the data file is consistent with the hash value of the data file before uploading. If they are consistent, the upload is successful, otherwise the upload fails, and the file upload process is re-executed until the data file is uploaded successfully.

[0014] The present invention also provides a log backup device based on a cloud platform, comprising a collection module, a storage module, a backup module and an upload module.

[0015] The collection module collects log files inside the container: the log files in the Kubernetes container are mounted to the specified location of the host through the configuration of configMap. On each node of the Kubernetes cluster, the log collection agent program is deployed in the form of DaemonSet. The agent program is used to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time.

[0016] The storage module stores log files: The data of log files is stored through ElasticSearch, in which the Elasticsearch index template is set and the index strategy is preset. The corresponding index is automatically created according to the index strategy and index template, and the collected log files are stored in the corresponding index.

[0017] Backup module backs up logs: deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; set up a scheduled task to execute the backups.sh script file every day through the scheduled task, call the Elasticdump tool, and back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading.

[0018] The upload module uploads backup data files: uploads data files to the cloud for storage.

[0019] Furthermore, the acquisition module of the log backup device based on a cloud platform uses an agent program to configure different log parsing rules for different types of log files, and performs filtering, cleaning, formatting, and encryption operations on the log files to verify the accuracy, integrity, and security of the data; for the collected log files, the agent program is used to calculate the hash value of the log file, and the hash value is recorded in the pos_file log file, which is used to compare with the hash value of the current log file during the next collection to see whether they are consistent, ensuring that the data reading maintains consistency and integrity when the log file is rotated or the log file is modified externally.

[0020] Furthermore, when the storage module of the log backup device based on a cloud platform sets the Elasticsearch index template, the format of the index name is set in the index template to the format of alias + date, and indexes with the same log type but different dates are defined to have the same alias, wherein the format of the index name is set to xxx.%Y%m%d, the prefix xxx represents the name of the alias, the suffix Y% represents the year, m% represents the month, and d% represents the date.

[0021] Furthermore, the storage module of the log backup device based on the cloud platform sets a timed deletion strategy for the index, and automatically deletes the index older than a preset number of days according to the timed deletion strategy, thereby releasing system memory and optimizing query efficiency.

[0022] Furthermore, the upload module of the log backup device based on the cloud platform calls the RESTful API interface or SDK package of the object storage service OSS in the cloud service, and regularly uploads the backed-up data files to the specified path, wherein different types of log backup data files are uploaded to different paths, and after the upload is completed, it is verified whether the hash value of the data file is consistent with the hash value of the data file before uploading. If they are consistent, the upload is successful, otherwise the upload fails, and the file upload process is re-executed until the data file is uploaded successfully.

[0023] The benefits of the present invention are:

[0024] Improve the reliability of backup: The present invention adopts a distributed storage architecture to back up log data to multiple cloud storage nodes, realize redundant storage and load balancing of data, effectively avoid the risk of single point failure, and ensure the reliability and availability of backup data.

[0025] Enhanced data security: The present invention uses data encryption technology during data transmission and storage to ensure the security of data in the cloud and prevent data from being illegally stolen or tampered with. At the same time, strict access control and identity authentication mechanisms are used to prevent unauthorized access and operation.

[0026] Simplify the management process: The present invention provides an automated log backup mechanism, and users do not need to manually configure and manage multiple backup tasks, which greatly simplifies the management process and improves work efficiency. At the same time, it provides flexible backup strategies and recovery methods to meet the needs of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic flow chart of the method of the present invention.

[0028] Figure 2 It is a schematic diagram of the application framework of the method of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0030] Example 1

[0031] The present invention provides a log backup method based on a cloud platform, comprising:

[0032] Step 1: Collect log files inside the container: mount the log files in the Kubernetes container to the specified location of the host through the configMap configuration. On each node of the Kubernetes cluster, deploy the log collection agent program using DaemonSet. Use the agent program to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time.

[0033] In step 1, the agent program is also used to configure different log parsing rules for different types of log files, and the log files are filtered, cleaned, formatted, and encrypted to verify the accuracy, integrity, and security of the data. For the collected log files, the agent program is used to calculate the hash value of the log file, and the hash value is recorded in the pos_file log file, which is used to compare with the hash value of the current log file during the next collection to see if they are consistent, ensuring that the data reading remains consistent and complete when the log file is rotated or the log file is modified externally.

[0034] Step 2: Store log files: Store log file data through ElasticSearch, set the Elasticsearch index template and preset the index strategy, automatically create the corresponding index according to the index strategy and index template, and store the collected log files in the corresponding index.

[0035] When setting the Elasticsearch index template in step 2, set the index name format in the index template to the format of alias + date, and define the same alias for indexes with the same log type but different dates. The format of the index name is set to xxx.%Y%m%d, where the prefix xxx represents the name of the alias, and the suffix Y% represents the year, m% represents the month, and d% represents the date. For example, the alias of the indexes test_log.20230101 and test_log.20240102 is "test_log".

[0036] A scheduled deletion policy for the index is also set, such as: {\"delete\":{\"min_age\":\"180d\",\"actions\":{\"delete\":{}}}}, which automatically deletes indexes older than the preset number of days according to the scheduled deletion policy, freeing up system memory and optimizing query efficiency.

[0037] Step 3: Back up logs: Deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; set up a scheduled task, such as "0 1***root sh / export / servers / backups.sh> / dev / null 2>&1">> / etc / crontab, and execute the backups.sh script file every day through the scheduled task, calling the Elasticdump tool to back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading,

[0038] Step 4: Upload backup data files: Upload data files to the cloud for storage.

[0039] In step 4, the RESTful API interface or SDK package of the object storage service OSS in the cloud service is also called to upload the backup data files to the specified path at a regular interval, where different types of log backup data files are uploaded to different paths, and after the upload is completed, the hash value of the data file is checked to see if it is consistent with the hash value of the data file before uploading. If they are consistent, the upload is successful, otherwise the upload fails, and the file upload process is re-executed until the data file is uploaded successfully.

[0040] The log collection and storage method of the present invention is targeted at the cloud platform environment, brings remarkable beneficial effects, significantly improves the log collection efficiency, enhances the scalability and flexibility of the system, ensures data security, and optimizes the system memory utilization.

[0041] Example 2

[0042] The present invention also provides a log backup device based on a cloud platform, comprising a collection module, a storage module, a backup module and an upload module.

[0043] The collection module collects log files inside the container: the log files in the Kubernetes container are mounted to the specified location of the host through the configuration of configMap. On each node of the Kubernetes cluster, the log collection agent program is deployed in the form of DaemonSet. The agent program is used to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time.

[0044] The storage module stores log files: The data of log files is stored through ElasticSearch, in which the Elasticsearch index template is set and the index strategy is preset. The corresponding index is automatically created according to the index strategy and index template, and the collected log files are stored in the corresponding index.

[0045] Backup module backs up logs: deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; set up a scheduled task to execute the backups.sh script file every day through the scheduled task, call the Elasticdump tool, and back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading.

[0046] The upload module uploads backup data files: uploads data files to the cloud for storage.

[0047] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0048] Likewise, the benefits of the device of the present invention are:

[0049] Improve the reliability of backup: The present invention adopts a distributed storage architecture to back up log data to multiple cloud storage nodes, realize redundant storage and load balancing of data, effectively avoid the risk of single point failure, and ensure the reliability and availability of backup data.

[0050] Enhanced data security: The present invention uses data encryption technology during data transmission and storage to ensure the security of data in the cloud and prevent data from being illegally stolen or tampered with. At the same time, strict access control and identity authentication mechanisms are used to prevent unauthorized access and operation.

[0051] Simplify the management process: The present invention provides an automated log backup mechanism, and users do not need to manually configure and manage multiple backup tasks, which greatly simplifies the management process and improves work efficiency. At the same time, it provides flexible backup strategies and recovery methods to meet the needs of different users.

[0052] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0053] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A log backup method based on a cloud platform, characterized by: include: Step 1: Collect log files inside the container: mount the log files in the Kubernetes container to the specified location of the host through the configMap configuration. On each node of the Kubernetes cluster, deploy the log collection agent program in the form of DaemonSet. Use the agent program to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time. Step 2: Store log files: Store log file data through ElasticSearch, set the Elasticsearch index template and preset the index strategy, automatically create the corresponding index according to the index strategy and index template, and store the collected log files in the corresponding index. Step 3: Back up logs: Deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; Set up a scheduled task to execute the backups.sh script file every day through the scheduled task, call the Elasticdump tool, and back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading. Step 4: Upload backup data files: Upload data files to the cloud for storage.

2. A log backup method based on a cloud platform according to claim 1, characterized in that In step 1, the agent program is used to configure different log parsing rules for different types of log files, and the log files are filtered, cleaned, formatted, and encrypted to verify the accuracy, integrity, and security of the data. For the collected log files, the agent program is used to calculate the hash value of the log file and record the hash value in the pos_file log file, which is used to compare with the hash value of the current log file during the next collection to see if they are consistent, ensuring that the data reading remains consistent and complete when the log file is rotated or modified externally.

3. The log backup method based on a cloud platform according to claim 1, characterized in that When setting the Elasticsearch index template in step 2, set the index name format in the index template to the format of alias + date, and define indexes with the same log type but different dates to have the same alias. The format of the index name is set to xxx.%Y%m%d, where the prefix xxx represents the name of the alias, and the suffix Y% represents the year, m% represents the month, and d% represents the date.

4. The log backup method based on a cloud platform according to claim 1, characterized in that In step 2, set the scheduled deletion policy for the index. According to the scheduled deletion policy, the index older than the preset number of days will be automatically deleted to free up system memory and optimize query efficiency.

5. The log backup method based on a cloud platform according to claim 1 is characterized in that In step 4, the RESTful API interface or SDK package of the object storage service OSS in the cloud service is called to upload the backup data files to the specified path at a fixed time. Different types of log backup data files are uploaded to different paths, and after the upload is completed, the hash value of the data file is verified to be consistent with the hash value of the data file before uploading. If they are consistent, the upload is successful, otherwise the upload fails, and the file upload process is re-executed until the data file is uploaded successfully.

6. A log backup device based on a cloud platform, characterized in that Including acquisition module, storage module, backup module and upload module, The collection module collects log files inside the container: the log files in the Kubernetes container are mounted to the specified location of the host through the configuration of configMap. On each node of the Kubernetes cluster, the log collection agent program is deployed in the form of DaemonSet. The agent program is used to collect the required log files from all nodes in the cluster and send the log files to the log storage system ElasticSearch in real time. The storage module stores log files: The data of log files is stored through ElasticSearch, in which the Elasticsearch index template is set and the index strategy is preset. The corresponding index is automatically created according to the index strategy and index template, and the collected log files are stored in the corresponding index. Backup module backs up logs: deploy the log backup service Elasticdump tool to export the data in the Elasticsearch index as a JSON file; Set up a scheduled task to execute the backups.sh script file every day through the scheduled task, call the Elasticdump tool, and back up the data in the index of the previous day; package and compress the backed up data into a data file for uploading. The upload module uploads backup data files: uploads data files to the cloud for storage.

7. A cloud platform-based log backup device according to claim 6, characterized in that The collection module uses the agent program to configure different log parsing rules for different types of log files, and performs filtering, cleaning, formatting, and encryption operations on the log files to verify the accuracy, integrity, and security of the data. For the collected log files, the agent program is used to calculate the hash value of the log file, and the hash value is recorded in the pos_file log file, which will be compared with the hash value of the current log file during the next collection to see if they are consistent, ensuring that the data reading remains consistent and complete when the log file is rotated or modified externally.

8. The log backup device based on a cloud platform according to claim 6, characterized in that When the storage module sets the Elasticsearch index template, the format of the index name is set to the format of alias + date in the index template, and indexes with the same log type but different dates are defined to have the same alias, where the format of the index name is set to xxx.%Y%m%d, where the prefix xxx represents the name of the alias, and the suffix Y% represents the year, m% represents the month, and d% represents the date.

9. The cloud platform-based log backup device according to claim 6, characterized in that The storage module sets a scheduled deletion policy for the index, and automatically deletes indexes older than a preset number of days based on the scheduled deletion policy, freeing up system memory and optimizing query efficiency.

10. The log backup device based on a cloud platform according to claim 6, characterized in that The upload module calls the RESTful API interface or SDK package of the object storage service OSS in the cloud service, and regularly uploads the backup data files to the specified path. Different types of log backup data files are uploaded to different paths, and after the upload is completed, it verifies whether the hash value of the data file is consistent with the hash value of the data file before uploading. If they are consistent, the upload is successful, otherwise the upload fails, and the file upload process is re-executed until the data file is uploaded successfully.

Images