Log collection and retrieval method and system based on cloud native

Through DaemonSet mode and distributed storage system, the problem of log management complexity in cloud-native environments is solved, efficient log collection, storage and retrieval is realized, and the operation and maintenance of cloud-native applications is supported, ensuring the security of log data and user experience.

CN118964296APending Publication Date: 2024-11-15SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411156214.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Traditional log collection and retrieval methods cannot meet the dynamic and complex needs of log data in cloud-native environments, resulting in insufficient log management efficiency and accuracy, making it difficult to support the operation and maintenance of cloud-native applications.

Method used

Deploy log collection agents using DaemonSet mode to realize global log collection; convert the original log into structured data through log field analysis, filtering, standardization and encryption, and efficient storage and retrieval using distributed storage systems and Elasticsearch, providing a web interface Carry out multi-dimensional search and backup management.

Benefits of technology

It realizes efficient, global collection and rapid retrieval of log data, improves the accuracy and efficiency of troubleshooting, dynamically adjusts system resources, and ensures the security and user experience of log data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964296A_ABST
    Figure CN118964296A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data storage, in particular to a log collection and retrieval method and system based on cloud native, and the method comprises the following steps: log collection, log preprocessing, log storage and log retrieval. Performing data backup; the log collection and retrieval method and system based on cloud native have the advantages that efficient and global collection of log data is achieved through a Daemon Set mode, and it is ensured that no important log is omitted; the original log is converted into the structured data through the intelligent analysis engine, and the index optimization of the distributed storage system is combined, so that the multi-dimensional quick retrieval is supported, and the accuracy and efficiency of troubleshooting are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a cloud-native-based log collection and retrieval method and system. Background Art

[0002] With the widespread application of cloud-native technologies, microservice architecture and containerized deployment have become the mainstream of modern applications. This transformation has brought greater flexibility and scalability, but it has also increased the complexity of log management. In a cloud-native environment, log data is an important basis for operation and maintenance and troubleshooting. It is large in scale, has diverse sources, and is updated frequently. Therefore, how to efficiently and accurately collect, store, and retrieve log data has become a key challenge for the operation and maintenance of cloud-native applications.

[0003] Traditional log collection and retrieval methods often rely on a single data source or centralized storage solution, which cannot meet the dynamic and complex requirements of log data in cloud-native environments. Therefore, a new log collection and retrieval method is needed that can adapt to the characteristics of cloud-native environments and achieve efficient collection, intelligent preprocessing, reliable storage, and fast retrieval of log data.

[0004] Based on the above background, the present invention proposes a cloud-native-based log collection and retrieval method, which aims to solve the shortcomings of traditional methods in cloud-native environments, improve the efficiency and accuracy of log management, and provide strong support for the operation and maintenance and troubleshooting of cloud-native applications. Summary of the invention

[0005] The purpose of the present invention is to provide a cloud-native based log collection and retrieval method and system to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a cloud-native-based log collection and retrieval method, the method comprising the following steps:

[0007] Log collection: deploy a log collector in the cloud-native environment. The log collector is configured to automatically collect log data from multiple service nodes in the cloud-native environment.

[0008] Log preprocessing, including log field parsing, filtering, standardization, and encryption operations, converts raw logs into structured log data to facilitate subsequent storage and retrieval;

[0009] Log storage, sending the collected log data to the cloud storage system, which is used to store and manage log data;

[0010] Log retrieval, providing a log retrieval interface so that users can query and retrieve log data stored in the cloud storage system through the interface;

[0011] Data backup: compress and back up the collected logs in a timely manner.

[0012] Preferably, the specific operations of log collection include:

[0013] On each node of the Kubernetes cluster, a log collection agent is deployed through DaemonSet. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. For real-time log data, an incremental update method is used to pre-process only newly generated logs. For each line of log collected, the corresponding pos_file will record which line of the monitored log file has been read, avoiding repeated processing of the entire log data set, saving computing resources, and achieving real-time collection.

[0014] Preferably, the specific operations of log preprocessing include:

[0015] Parse the log data and use regular expressions to automatically parse the logs; secondly, clean the data to remove irrelevant information in the logs, such as timestamps, repeated parts in stack traces, correct or fill missing data fields, and standardize data formats, such as date and time formats and field naming; finally, ensure that appropriate data security requirements are followed when processing, storing, and transmitting log data, including encryption and data desensitization security measures.

[0016] Preferably, the specific operations of log storage include:

[0017] A distributed storage system is used to store structured log data. Efficient storage and query are achieved through reasonable index design and sharding strategies. The index strategy is defined through the Elasticsearch index template, such as alias, scheduled cleanup, etc. The index is named in the format of index.%Y%m%d, so that each day's log is stored in a separate index. The same log file is stored in different indexes but has the same index alias. When users retrieve logs, they can use the index alias to query the data in all indexes of this type of log.

[0018] Preferably, the specific operation of log retrieval includes: a web-based log retrieval interface, where the user logs in to the system and performs log retrieval by using keywords, time range, Pod tags and other conditions. The retrieval results are displayed to the user in the form of a table or chart, which facilitates the user to quickly locate the problem and perform troubleshooting. The system provides an export button for the user to download and save the log data file;

[0019] The specific operations of data backup include: using the log cutting function of logrotate, writing log cutting expressions, limiting the size of log documents, cutting and compressing documents when the size exceeds the limit, and using the CronJob type job function of Kubernetes to write scheduled task expressions, exporting and backing up indexes in Elasticsearch that are more than 180 days old, compressing and saving them, and deleting the corresponding Elasticsearc indexes.

[0020] A cloud-native-based log collection and retrieval system, the system comprising the following modules:

[0021] The log collection module deploys a log collector in the cloud native environment. The log collector is configured to automatically collect log data from multiple service nodes in the cloud native environment.

[0022] The log preprocessing module includes log field parsing, filtering, standardization, and encryption operations to convert raw logs into structured log data for easy subsequent storage and retrieval;

[0023] The log storage module sends the collected log data to the cloud storage system, which is used to store and manage the log data;

[0024] The log retrieval module provides a log retrieval interface, allowing users to query and retrieve log data stored in the cloud storage system through the interface;

[0025] The data backup module compresses and backs up the collected logs in a timely manner.

[0026] Preferably, the log collection module deploys a log collection agent on each node of the Kubernetes cluster through DaemonSet. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. For the log data generated in real time, an incremental update method is adopted, and only the newly generated logs are pre-processed. For each line of log collected, the corresponding pos_file file records which line of the monitored log file has been read, avoiding repeated processing of the entire log data set, saving computing resources, and realizing real-time collection.

[0027] Preferably, the log preprocessing module parses the log data and uses regular expressions to automatically parse the logs; secondly, the data is cleaned to remove irrelevant information in the logs, such as timestamps, repeated parts in stack traces, correct or fill missing data fields, and standardize data formats, such as date and time formats and field naming; finally, ensure that appropriate data security requirements are followed when processing, storing and transmitting log data, including encryption and data desensitization security measures.

[0028] Preferably, the log storage module uses a distributed storage system to store structured log data, and realizes efficient storage and query through reasonable index design and sharding strategy; defines index strategies through Elasticsearch index templates, such as aliases, scheduled cleanup and other strategies, and the index is named in the format of index.%Y%m%d, so that daily logs are stored in a separate index, and the same log files are stored in different indexes but have the same index alias. When users retrieve logs, they can use the index alias to query the data in all indexes of the log of this type.

[0029] Preferably, the log retrieval module is a web-based log retrieval interface. Users log in to the system and perform log retrieval by keywords, time range, Pod tags and other conditions. The retrieval results are displayed to users in the form of tables or charts, which facilitates users to quickly locate problems and perform troubleshooting. The system provides an export button for users to download and save log data files;

[0030] The data backup module uses the log cutting function of logrotate to write log cutting expressions, limit the size of log documents, cut and compress the documents when the size exceeds the limit, and uses the CronJob type job function of Kubernetes to write scheduled task expressions, export and back up the indexes in Elasticsearch that are more than 180 days old, compress and save them, and delete the corresponding Elasticsearc indexes.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The cloud-native-based log collection and retrieval method and system proposed in the present invention realize efficient and global collection of log data through the DaemonSet mode to ensure that no important logs are missed; the original logs are converted into structured data through the intelligent parsing engine, and combined with the index optimization of the distributed storage system, multi-dimensional rapid retrieval is supported to improve the accuracy and efficiency of troubleshooting; cloud-native characteristics are utilized, such as the automatic expansion mechanism of Kubernetes, to realize dynamic adjustment of system resources to meet the growing demand for log data; a user-friendly Web interface and visual analysis tools are provided to simplify the process of log retrieval and troubleshooting, reduce the difficulty of operation and maintenance, and improve user experience; and security policies such as encrypted transmission and access control are used to ensure the security of log data during transmission and storage, and protect sensitive information from being leaked. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2The system and architecture diagram of the present invention;

[0035] Figure 3 This is a log collection mode diagram of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] For example, see Figures 1 to 3 The present invention provides a technical solution: a cloud-native-based log collection and retrieval method, the method comprising the following steps:

[0038] 1. Log Collection

[0039] The present invention adopts the DaemonSet mode log collection method. On each node of the Kubernetes cluster, a log collection agent is deployed in DaemonSet mode. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. This method can realize global log collection to ensure that no Pod logs are missed. For the log data generated in real time, we adopt the incremental update method and only pre-process the newly generated logs. Each line of log collected will record the number of lines of the monitored log file that have been read in the corresponding pos_file file, avoiding repeated processing of the entire log data set, which can not only save computing resources, but also realize real-time collection.

[0040] Deploy the log collection agent: Configure a Fluentd agent log collection program on each node through the DaemonSet mode of the Kubernetes cluster. Fluentd is an open source, cross-platform, and extensible log collector. Its core functions include collecting, transmitting, and processing various types of data in real time. It can collect logs from various sources (such as files, TCP / UDP, Syslog, Apache, etc.) and can send these logs to various targets (such as Elasticsearch, Hadoop, MongoDB, etc.).

[0041] Configure the log collection strategy: Configure the log file path, log filtering conditions, field parsing strategy, data encryption strategy, and Elasticsearch address and index name for data transmission in the program.

[0042] 2. Log preprocessing

[0043] In the log collection system, the collected logs are preprocessed. This includes operations such as log field parsing, filtering, standardization, and encryption, which convert the original logs into structured log data for subsequent storage and retrieval. First, the log data is parsed, and regular expressions are used to automatically parse the logs; secondly, the data is cleaned to remove irrelevant information in the logs, such as timestamps, repeated parts in stack traces, etc., correct or fill in missing data fields, and standardize data formats, such as date and time formats, field naming, etc.; finally, ensure that appropriate data security requirements are followed when processing, storing, and transmitting log data, including encryption, data desensitization, and other security measures.

[0044] 3. Log storage

[0045] The present invention uses a distributed storage system (such as Elasticsearch) to store structured log data. Through reasonable index design and sharding strategy, efficient storage and query are achieved. First, the index strategy is defined through the Elasticsearch index template, such as alias, scheduled cleanup and other strategies. The index is named in the format of index.%Y%m%d, so that the daily log is stored in a separate index. The same log file is stored in different indexes, but has the same index alias. When the user retrieves the log, the index alias can be used to query the data in all indexes of the log of this type.

[0046] Deploy log storage tools: By clustering the Elasticsearch data storage program, a data storage solution with advantages such as high performance, scalability, real-time data processing, high availability and fault tolerance, distributed storage and load balancing, flexibility and ease of use, and scalability is implemented. Index template information is written through scripts. The template includes index aliases, field types, scheduled index deletion strategies, and other policies.

[0047] 4. Log retrieval

[0048] The present invention provides a web-based log retrieval interface, and users can log in to the system to perform log retrieval by multiple conditions such as keywords, time range, Pod tags, etc. The retrieval results are displayed to users in the form of tables or charts, which is convenient for users to quickly locate problems and perform troubleshooting. The system provides an export button for users to download and save log data files.

[0049] 4. Data retrieval and analysis:

[0050] a) Web interface: Provides a user-friendly web interface. By assigning user permissions, different users can query different log data according to their permissions. It also supports log retrieval by keywords, request names, time ranges, Pod labels and other conditions. Users can click the export button to export the queried logs.

[0051] b) Visual analysis: Combined with charts and statistical analysis tools, the search and analysis results are displayed to users in a visual way, helping users to more intuitively understand the system status, potential problems and log development trends.

[0052] c) Log export: Users can export required logs according to customized query strategies, choose to export Excel or Word files, and export comprehensive log reports according to time periods, which can analyze log trends more intuitively.

[0053] 5. Log backup

[0054] Since the amount of logs will continue to increase, long-term accumulation will lead to an excessive amount of logs, occupying a large amount of memory space. The present invention will cut and compress the collected log files for storage, and then export and compress the logs in Elasticsearch that exceed a certain time limit and delete the corresponding indexes to maintain an efficient query speed.

[0055] In order to prevent data loss and data occupying a large amount of memory space, the collected logs are compressed and backed up in time. This solution uses the log cutting function of logrotate to write the log cutting expression, limit the size of log documents, cut and compress the documents when the size exceeds the limit, and write the scheduled task expression through the CronJob type job function of Kubernetes to export and back up the index in Elasticsearch for more than 180 days and compress and save it, and delete the corresponding Elasticsearch index.

[0056] Embodiment 2, based on embodiment 1, proposes a cloud-native-based log collection and retrieval system, the system comprising the following modules:

[0057] The log collection module deploys a log collector in the cloud native environment. The log collector is configured to automatically collect log data from multiple service nodes in the cloud native environment. On each node of the Kubernetes cluster, a log collection agent is deployed through DaemonSet. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. For real-time generated log data, an incremental update method is used to pre-process only newly generated logs. For each collected log line, the corresponding pos_file will record which line of the monitored log file has been read, avoiding repeated processing of the entire log data set, saving computing resources, and realizing real-time collection.

[0058] The log preprocessing module, including log field parsing, filtering, standardization, and encryption operations, converts raw logs into structured log data to facilitate subsequent storage and retrieval; parses log data and automatically parses logs using regular expressions; secondly, cleans the data to remove irrelevant information in the log, such as timestamps, duplicate parts in stack traces, corrects or fills in missing data fields, and standardizes data formats, such as date and time formats and field naming; finally, ensures that appropriate data security requirements are followed when processing, storing, and transmitting log data, including encryption and data desensitization security measures.

[0059] The log storage module sends the collected log data to the cloud storage system, which is used to store and manage log data. It uses a distributed storage system to store structured log data, and achieves efficient storage and query through reasonable index design and sharding strategies. It defines index strategies through Elasticsearch index templates, such as aliases, scheduled cleanup, etc. The index is named in the format of index.%Y%m%d, so that daily logs are stored in a separate index. The same log files are stored in different indexes but have the same index alias. When users retrieve logs, they can use the index alias to query the data in all indexes of this type of log.

[0060] The log retrieval module provides a log retrieval interface, allowing users to query and retrieve log data stored in the cloud storage system through the interface; based on the Web-based log retrieval interface, users log in to the system to search for logs based on keywords, time ranges, Pod tags and other conditions. The search results are displayed to users in the form of tables or charts, making it easy for users to quickly locate problems and perform troubleshooting. The system provides an export button for users to download and save log data files.

[0061] The data backup module compresses and backs up the collected logs in a timely manner, writes log cutting expressions through the log cutting function of logrotate, limits the size of log documents, cuts and compresses the documents when the size exceeds the limit, and writes scheduled task expressions through the CronJob type job function of Kubernetes, exports and backs up the indexes in Elasticsearch that are more than 180 days old, compresses and saves them, and deletes the corresponding Elasticsearc indexes.

[0062] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud-native log collection and retrieval method, characterized by: The method comprises the following steps: Log collection: deploy a log collector in the cloud-native environment. The log collector is configured to automatically collect log data from multiple service nodes in the cloud-native environment. Log preprocessing, including log field parsing, filtering, standardization, and encryption operations, converts raw logs into structured log data to facilitate subsequent storage and retrieval; Log storage, sending the collected log data to the cloud storage system, which is used to store and manage log data; Log retrieval, providing a log retrieval interface so that users can query and retrieve log data stored in the cloud storage system through the interface; Data backup: compress and back up the collected logs in a timely manner.

2. According to the cloud-native log collection and retrieval method of claim 1, it is characterized by: The specific operations of log collection include: On each node of the Kubernetes cluster, a log collection agent is deployed through DaemonSet. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. For real-time log data, an incremental update method is used to pre-process only newly generated logs. For each line of log collected, the corresponding pos_file will record which line of the monitored log file has been read, avoiding repeated processing of the entire log data set, saving computing resources, and achieving real-time collection.

3. According to a cloud-native log collection and retrieval method according to claim 1, it is characterized in that: The specific operations of log preprocessing include: Parse the log data and use regular expressions to automatically parse the logs; secondly, clean the data to remove irrelevant information in the logs, such as timestamps, repeated parts in stack traces, correct or fill missing data fields, and standardize data formats, such as date and time formats and field naming; finally, ensure that appropriate data security requirements are followed when processing, storing, and transmitting log data, including encryption and data desensitization security measures.

4. According to a cloud-native log collection and retrieval method according to claim 1, it is characterized in that: The specific operations of log storage include: A distributed storage system is used to store structured log data. Efficient storage and query are achieved through reasonable index design and sharding strategies. The index strategy is defined through the Elasticsearch index template, such as alias, scheduled cleanup, etc. The index is named in the format of index.%Y%m%d, so that each day's log is stored in a separate index. The same log file is stored in different indexes but has the same index alias. When users retrieve logs, they can use the index alias to query the data in all indexes of this type of log.

5. According to a cloud-native log collection and retrieval method according to claim 1, it is characterized in that: The specific operations of log retrieval include: based on the web-based log retrieval interface, users log in to the system and perform log retrieval by keywords, time range, Pod tags and other conditions. The retrieval results are displayed to users in the form of tables or charts, which facilitates users to quickly locate problems and conduct troubleshooting. The system provides an export button for users to download and save log data files; The specific operations of data backup include: using the log cutting function of logrotate, writing log cutting expressions, limiting the size of log documents, cutting and compressing documents when the size exceeds the limit, and using the CronJob type job function of Kubernetes to write scheduled task expressions, exporting and backing up indexes in Elasticsearch that are more than 180 days old, compressing and saving them, and deleting the corresponding Elasticsearc indexes.

6. A cloud-native-based log collection and retrieval system according to the cloud-native-based log collection and retrieval method according to any one of claims 1 to 5, characterized in that: The system includes the following modules: The log collection module deploys a log collector in the cloud native environment. The log collector is configured to automatically collect log data from multiple service nodes in the cloud native environment. The log preprocessing module includes log field parsing, filtering, standardization, and encryption operations to convert raw logs into structured log data for easy subsequent storage and retrieval; The log storage module sends the collected log data to the cloud storage system, which is used to store and manage the log data; The log retrieval module provides a log retrieval interface, allowing users to query and retrieve log data stored in the cloud storage system through the interface; The data backup module compresses and backs up the collected logs in a timely manner.

7. A cloud-native based log collection and retrieval system according to claim 6, characterized in that: The log collection module deploys a log collection agent on each node of the Kubernetes cluster through DaemonSet. The agent is responsible for collecting logs from all Pods on the node and sending the logs to the log collection system. For real-time log data, an incremental update method is used to pre-process only newly generated logs. For each line of log collected, the corresponding pos_file will record which line of the monitored log file has been read, avoiding repeated processing of the entire log data set, saving computing resources, and realizing real-time collection.

8. A cloud-native based log collection and retrieval system according to claim 6, characterized in that: The log preprocessing module parses the log data and uses regular expressions to automatically parse the logs. Secondly, the data is cleaned to remove irrelevant information in the logs, such as timestamps, repeated parts in stack traces, correct or fill missing data fields, and standardize data formats, such as date and time formats and field naming. Finally, ensure that appropriate data security requirements are followed when processing, storing, and transmitting log data, including encryption and data desensitization security measures.

9. A cloud-native based log collection and retrieval system according to claim 6, characterized in that: The log storage module uses a distributed storage system to store structured log data, and achieves efficient storage and query through reasonable index design and sharding strategies. It defines index strategies through Elasticsearch index templates, such as aliases, scheduled cleanup, etc. The index is named in the format of index.%Y%m%d, so that daily logs are stored in a separate index. The same log files are stored in different indexes but have the same index alias. When users retrieve logs, they can use the index alias to query the data in all indexes of this type of log.

10. A cloud-native based log collection and retrieval system according to claim 6, characterized in that: Log retrieval module, based on the web-based log retrieval interface, users log in to the system and perform log retrieval by keywords, time range, Pod tags and other conditions. The retrieval results are displayed to users in the form of tables or charts, which facilitates users to quickly locate problems and perform troubleshooting. The system provides an export button for users to download and save log data files; The data backup module uses the log cutting function of logrotate to write log cutting expressions, limit the size of log documents, cut and compress the documents when the size exceeds the limit, and uses the CronJob type job function of Kubernetes to write scheduled task expressions, export and back up the indexes in Elasticsearch that are more than 180 days old, compress and save them, and delete the corresponding Elasticsearc indexes.

Citation Information

Cited By

  • Cloud native scene-based log vector retrieval system and method and medium

    CN120804182A