Microservice log cleaning algorithm optimization method based on time sliding window

Abstract

The invention relates to the technical field of data information, in particular to a method for optimizing a micro-service log cleaning algorithm based on a time sliding window. The method is applied to micro-service architecture, log management and data cleaning. Comprising the following steps of log data collection, data retention strategy definition, time sliding window segmentation, cleaning strategy execution and data indexing and retrieval. Through time sliding window segmentation and optimization of a data retention strategy, an effective micro-service log data management and cleaning solution is provided, the cost is reduced, the efficiency is improved, the compliance requirement is met, and the requirement for supporting a large-scale micro-service application program is met.

Description

Technical Field

[0001] The present invention relates to the field of data information technology, and relates to a method for optimizing a microservice log cleaning algorithm based on a time sliding window, which is applied to microservice architecture, log management, and data cleaning. Background Art

[0002] Microservice architecture has become the mainstream model of modern software development, which provides a modular and distributed way to build applications. Microservice applications usually generate a large amount of log data, which is used for monitoring, troubleshooting, and performance analysis. However, as the scale of applications continues to grow, the amount of log data will also expand rapidly, making it complex and expensive to store and manage this data. Summary of the invention

[0003] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method for optimizing a microservice log cleaning algorithm based on a time sliding window, which solves the problems existing in the prior art.

[0004] 1. Storage cost issue: The large amount of log data generated by microservice applications needs to be effectively stored and managed. Traditional methods may lead to expensive storage costs, especially in cloud environments.

[0005] 2. Data retention policy issues: Different types of log data (such as error logs, information logs, and debug logs) usually require different data retention policies. How to define and manage these policies based on actual needs is a challenge.

[0006] 3. Data cleaning efficiency: Regular cleaning of expired log data is necessary, but traditional methods may lead to inefficiency and performance degradation. How to perform data cleaning operations efficiently is an important issue.

[0007] 4. Data retrieval problem: Rapid retrieval of log data related to a specific time period or event is crucial for troubleshooting and performance analysis. How to build an effective data index to improve data retrieval efficiency is a key issue.

[0008] 5. Data security and compliance issues: Some application areas need to comply with data security and compliance standards, and may require specific data retention and deletion policies. How to ensure that data management operations comply with relevant regulations and standards is an important issue.

[0009] 6. Microservice scale problem: As the scale of microservice applications continues to expand, log data management issues will become more complicated. How to handle log data of large-scale microservice applications is a challenge.

[0010] The present invention solves the existing technical problems by adopting the following technical solution: a microservice log cleaning algorithm optimization method based on a time sliding window, comprising the following steps:

[0011] Log data collection: First, microservice applications generate log data, each of which contains the following information: timestamp information, application ID, log level, message content, user ID, and other custom fields; these log data are collected and stored in a distributed log storage system;

[0012] Data retention policy definition: Users or system administrators define data retention policies, which specify how long different types of log data should be retained; these policies are configured according to the needs of the application, and the following data retention policies are defined: Flexible configuration, the system should support users to configure flexible retention times for different types of log data according to specific needs; Typed retention, the data retention policy should have typed characteristics, and be able to set different retention periods for different log types such as errors, warnings, information, and debugging; Default policy, the system should have a default data retention policy to ensure that users can enable reasonable data retention rules without manual configuration; Configuration effectiveness, the system should be able to ensure that the user-configured data retention policy takes effect, that is, perform data cleanup operations according to the user-defined period;

[0013] Time sliding window segmentation: The log data storage system will divide the data into different time periods or blocks according to the timestamp information to form a time sliding window. The specific division requirements are as follows: Division frequency, the system should allow the administrator to configure the division frequency of the time sliding window to meet the needs of different applications; Window size, the system should support setting the size of the time window according to user needs to ensure that it can meet the data management needs in different scenarios; Data block identification, each time window should have a unique identifier so that the system can identify and manage different data blocks; Division frequency, divide the time window once a day; Window size, each window contains a time span of one day; Data block identification, use the date as the unique identifier of each time window;

[0014] Cleaning policy execution: According to the defined data retention policy, the system will periodically perform cleanup operations on log data; this operation will delete expired log data, and the system should perform the following cleanup steps: configure the cleanup frequency, the administrator sets the execution frequency of the cleanup operation, which can be performed daily, weekly or monthly; identify expired data, the system checks the timestamp information of each log, identifies and marks expired data according to the user-defined data retention policy; execute the cleanup algorithm, the system uses a suitable algorithm to traverse the log data, and deletes data that no longer meets the retention policy based on the expiration mark; handle abnormal situations, implement appropriate error handling mechanisms in the cleanup operation, record abnormalities and notify the administrator; verify the data retention policy, the system verifies whether each log data meets the data retention policy before cleaning, and only deletes expired data; record the cleanup log, the system generates a log after each cleanup operation is completed, and records the detailed information of the cleanup for monitoring and analyzing system performance;

[0015] Data indexing and retrieval: Build an index to quickly locate log data for a specific time period. Users can retrieve the required log data based on time range, keywords, or other conditions.

[0016] The method of the present invention aims to solve these problems. By optimizing the time sliding window segmentation and data retention policy, an effective microservice log data management and cleaning solution is provided to reduce costs, improve efficiency, meet compliance requirements, and support the needs of large-scale microservice applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the log cleaning process of the present invention. Specific implementation plan

[0018] The present invention is based on the microservice log data management of the time sliding window, specifically comprising the following steps:

[0019] Log data collection: First, taking iZunyi app as an example, each microservice application will generate log data, each of which will contain the following information: timestamp information (timestamp format should comply with international standards), application ID, log level, message content, user ID (optional), other custom fields (optional). These log data will be collected and stored in the distributed log storage system.

[0020] Data retention policy definition: Users or system administrators can define data retention policies that specify the time period for which different types of log data should be retained. These policies can be configured according to the needs of the application. For example, the following data retention policies can be defined: Flexible configuration: The system should support users to configure flexible retention time for different types of log data according to specific needs; Typed retention: The data retention policy should have typed characteristics and be able to set different retention periods for different log types (such as errors, warnings, information, and debugging); Default policy: The system should have a default data retention policy to ensure that users can enable reasonable data retention rules without manual configuration; Configuration effectiveness: The system should be able to ensure that the user-configured data retention policy takes effect, that is, perform data cleanup operations according to the user-defined period.

[0021] Time sliding window segmentation: The log data storage system will divide the data into different time periods or blocks according to the timestamp information to form a time sliding window. The specific segmentation requirements are as follows: the frequency of segmentation, the system should allow the administrator to configure the frequency of segmentation of the time sliding window to meet the needs of different applications; the window size, the system should support the setting of the time window size according to user needs to ensure that it can meet the data management needs in different scenarios; the data block identifier, each time window should have a unique identifier so that the system can identify and manage different data blocks. Suppose there is an online e-commerce platform whose microservice applications generate a large amount of log data every day. The administrator configures the system to divide the time window once a day, and the size of each window is one day. In this way, the system will divide the log data generated every day into different time blocks, and each block contains one day's log. Example configuration: the frequency of segmentation, the time window is divided once a day; the window size, each window contains a time span of one day; the data block identifier, the date is used as the unique identifier of each time window.

[0022] Cleanup strategy execution: According to the defined data retention policy, the system will periodically perform log data cleanup operations. This operation will delete expired log data to reduce the use of storage resources. In order to implement the above log cleanup strategy, the system should perform the following cleanup steps: Configure the cleanup frequency. The administrator sets the execution frequency of the cleanup operation, such as once a day, week, or month; Identify expired data. The system checks the timestamp information of each log, identifies and marks expired data according to the user-defined data retention policy; Execute the cleanup algorithm. The system uses a suitable algorithm to traverse the log data and deletes the data that no longer meets the retention policy according to the expiration mark; Handle abnormal situations. Implement appropriate error handling mechanisms in the cleanup operation, record abnormalities and notify the administrator; Verify the data retention policy. The system verifies whether each log data meets the data retention policy before cleaning, and only deletes expired data; Record the cleanup log. The system generates a log after each cleanup operation is completed, recording the detailed information of the cleanup for monitoring and analyzing system performance.

[0023] Data indexing and retrieval: To improve retrieval efficiency, the system builds an index to quickly locate log data for a specific time period. Users can retrieve the required log data based on time range, keywords, or other conditions.

[0024] Implementation 1:

[0025] In an embodiment, step one may specifically include: classifying the logs on a time axis using a log index, where the index includes a timestamp and other relevant information of each log.

[0026] Step 2 can be specifically as follows: using NLP (natural language processing) technology to parse each log, identify and extract key information such as operation type, operation time, and operation result.

[0027] Step three may specifically include: evaluating the importance of each log based on the extracted key information and preset evaluation rules (such as an evaluation model based on machine learning).

[0028] Step 4 may be specifically as follows: based on the evaluation results, clean up the logs with lower importance, which may be achieved by using technologies such as data compression and deletion.

[0029] Implementation method 2:

[0030] In another embodiment, step one may specifically include: using a time series index to classify the microservice logs on a time axis, where the index includes a timestamp and other relevant information of each log.

[0031] Step 2 can be specifically as follows: use a deep learning model to parse each log, identify and extract key information such as operation type, operation time, and operation results.

[0032] Step three may specifically include: evaluating the importance of each log based on the extracted key information and preset evaluation rules (such as an evaluation model based on a rule base).

[0033] Step 4 may be specifically as follows: based on the evaluation results, clean up the logs with lower importance, which may be achieved by using technologies such as data compression and encryption.

[0034] Technical points

[0035] Time sliding window size formula:

[0036] T sw =n*T p

[0037] Among them, T sw represents the size of the time sliding window, n represents the number of time periods stored in the sliding window, T p Indicates the length of the time period.

[0038] The formula for the storage space occupied by logs is:

[0039] S log =N*L

[0040] Among them, S log Indicates the storage space occupied by logs, N indicates the number of log data, and L indicates the size of a single log data.

[0041] Threshold formula:

[0042] T=S max -S min

[0043] Where T represents the threshold, S max Indicates the maximum value of storage space, S min Indicates the minimum value of storage space.

[0044] Log retention period formula:

[0045] T r =T sw -T r_m

[0046] Among them, T r Represents the log retention period, T sw Indicates the size of the time sliding window, T r_m Indicates the manually set minimum retention period.

[0047] Expired log cleaning formula:

[0048]

[0049] Among them, N d Indicates the number of expired logs that need to be cleaned up, N p Indicates the total number of log data. If the log data is saved according to a time period, and the time period is 1 day, then N p Indicates the amount of log data saved in the time sliding window.

[0050] Log compression ratio formula:

[0051]

[0052] Among them, C represents the compression ratio of the log, S c Indicates the storage space occupied by the compressed log, S u Indicates the storage space occupied by uncompressed logs.

[0053] Log deletion probability formula:

[0054]

[0055] Among them, P represents the probability of deleting logs, N d Indicates the number of expired logs that need to be deleted, N t represents the total number of logs, α represents an adjustable parameter, and C represents the compression ratio of the logs.

[0056] Log compression efficiency formula:

[0057]

[0058] Among them, E represents the efficiency of log compression, N u Indicates the number of uncompressed log data, L indicates the size of a single log data, T c Indicates the average time to compress a log data set.

[0059] The main advantage of the present invention is that through the division of time sliding windows and the definition of data retention policies, microservice log data can be managed more effectively, storage costs can be reduced, and retrieval efficiency can be improved. In addition, the method also has the following innovative features:

[0060] Adaptability: Users can customize different data retention policies according to actual needs.

[0061] Efficiency: Time sliding window segmentation and data index construction make data management operations more efficient.

[0062] Scalability: Suitable for microservice applications of all sizes, from small applications to large distributed systems.

Claims

1. A microservice log cleaning algorithm optimization method based on a time sliding window, characterized by The steps include: Log data collection: First, microservice applications generate log data, each of which contains the following information: timestamp information, application ID, log level, message content, user ID, and other custom fields; these log data are collected and stored in a distributed log storage system; Data retention policy definition: The user or system administrator defines the data retention policy, which specifies how long different types of log data should be retained; these policies are configured according to the needs of the application, and the following data retention policies are defined: Flexible configuration, the system should support users to configure flexible retention time for different types of log data according to specific needs; Typed retention: data retention policies should be typed, and different retention periods should be set for different log types such as errors, warnings, information, and debugging. Default policy: the system should have a default data retention policy to ensure that users can enable reasonable data retention rules without manual configuration. Configuration effectiveness: the system should be able to ensure that the user-configured data retention policy takes effect, that is, perform data cleanup operations according to the user-defined period. Time sliding window segmentation: The log data storage system divides the data into different time periods or blocks according to the timestamp information to form a time sliding window; The specific division requirements are as follows: Division frequency: the system should allow administrators to configure the division frequency of the time sliding window to meet the needs of different applications; Window size: the system should support setting the size of the time window according to user needs to ensure that it can meet the data management needs in different scenarios; Data block identification: each time window should have a unique identification so that the system can identify and manage different data blocks; Division frequency: divide the time window once a day; window size: each window covers a time span of one day; data block identification: use the date as the unique identification of each time window; Cleanup strategy execution: According to the defined data retention policy, the system will periodically perform cleanup operations on log data; this operation will delete expired log data, and the system should perform the following cleanup steps: configure the cleanup frequency, the administrator sets the execution frequency of the cleanup operation, which can be performed daily, weekly or monthly; identify expired data, the system checks the timestamp information of each log, identifies and marks expired data according to the user-defined data retention policy; execute the cleanup algorithm, the system uses a suitable algorithm to traverse the log data, and deletes data that no longer meets the retention policy based on the expiration mark; handle abnormal situations, implement appropriate error handling mechanisms in the cleanup operation, record abnormalities and notify the administrator; verify the data retention policy, the system verifies whether each log data meets the data retention policy before cleaning, and only deletes expired data; Record cleaning logs. The system generates logs after each cleaning operation is completed, recording the detailed information of the cleaning, which is used to monitor and analyze system performance; Data indexing and retrieval: Build an index to quickly locate log data for a specific time period. Users can retrieve the required log data based on time range, keywords, or other conditions.

Images