Log cleaning method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a log cleaning method and device, electronic equipment and a medium, and relates to the technical field of computers.The method comprises the steps that logs generated by all components in a system within a preset time period are obtained, and the importance score of each log in each preset scene is calculated; a target scene where the system is located currently is determined based on the monitored system key event, a preset cleaning strategy corresponding to the target scene is obtained, and the target scene is one of the preset scenes; based on the preset cleaning strategy and the importance score of each log in the target scene, a task execution plan is determined, and the task execution plan comprises the execution sequence and the execution opportunity of cleaning tasks corresponding to each to-be-cleaned log; and clearing logs in the system according to the preset clearing strategy, the task execution plan and the system load state. The method can meet the requirements of complex and changeable system environments on log cleaning.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log cleaning method, apparatus, electronic device, and medium. Background Technology

[0002] In a digital operating environment, the massive amounts of logs generated by various systems are crucial for system management. Current systems use fixed rules based on time or file size to clean up logs, such as deleting log files generated a week ago every Sunday morning, or cleaning up when a single log file exceeds 100MB. However, different types of logs have different retention periods, and the current log cleaning methods are difficult to meet the requirements of the complex and ever-changing system environment. Summary of the Invention

[0003] The purpose of this application is to provide a log cleanup method, apparatus, electronic device, and medium to meet the log cleanup requirements of complex and ever-changing system environments. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a log cleanup method, the method comprising:

[0005] Obtain logs generated by each component in the system within a preset time period, and calculate the importance score of each log in each preset scenario;

[0006] Based on the monitored key system events, the current target scenario of the system is determined, and a preset cleanup strategy corresponding to the target scenario is obtained, wherein the target scenario is one of the preset scenarios;

[0007] Based on the preset cleanup strategy and the importance score of each log in the target scenario, a task execution plan is determined. The task execution plan includes the execution order and timing of the cleanup tasks corresponding to each log to be cleaned.

[0008] Clean up the logs in the system according to the preset cleanup strategy, the task execution plan, and the system load status.

[0009] In one possible implementation, determining the current target scenario of the system based on monitored key system events and obtaining a preset cleanup strategy corresponding to the target scenario includes:

[0010] When a critical event is detected in the system, the metadata information of the critical event is extracted. The metadata information includes the time of the event, the component information involved in the event, and the business information involved in the event.

[0011] The system identifies the target scene it is currently in using preset scene recognition rules and the metadata information;

[0012] Based on the preset correspondence between each preset scenario and the cleaning strategy, the preset cleaning strategy corresponding to the target scenario is obtained.

[0013] In one possible implementation, the method further includes:

[0014] Periodically acquire the system's performance metrics data;

[0015] The system load status is determined based on the performance index data.

[0016] In one possible implementation, determining the system load status based on the performance metric data includes:

[0017] The overall utilization rate is obtained by weighted summation of the performance index data;

[0018] If the overall utilization rate is less than a preset threshold, the system load status is determined to be a low load status.

[0019] If the overall utilization rate is greater than or equal to a preset threshold, then the system load state is determined to be a high load state.

[0020] In one possible implementation, the preset cleanup strategy includes a cleanup time period; the step of determining the task execution plan based on the preset cleanup strategy and the importance score of each log in the target scenario includes:

[0021] Based on the importance score of each log in the target scenario, determine the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy;

[0022] Add the cleanup tasks corresponding to the logs to be cleaned that conform to the preset cleanup strategy to the task queue;

[0023] Based on the number of cleanup tasks in the task queue and the cleanup time period, the cleanup tasks in the task queue are divided into multiple batches. The cleanup tasks in the same batch are executed at the same time, and the cleanup tasks in different batches are executed in chronological order within the cleanup time period.

[0024] In one possible implementation, cleaning up the logs in the system according to the preset cleanup strategy, the task execution plan, and the system load status includes:

[0025] If the system load is low, the cleanup task will be executed according to the cleanup strategy and the execution timing of each batch of cleanup tasks.

[0026] If the system load is high and the cleanup task has not yet started, the cleanup task will be postponed until the system load returns to low, and then executed according to the cleanup strategy and the execution timing of each batch of cleanup tasks.

[0027] If the system load is high and a cleanup task is being performed, the cleanup task is paused. Once the system load returns to low, the cleanup task is executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0028] In one possible implementation, the method further includes:

[0029] The system's disk is periodically scanned using multiple paths, and the information from the scanned log files is recorded.

[0030] When a change in the version of third-party software is detected, the log storage information of the third-party software is recorded;

[0031] The preset cleanup strategy is adjusted based on the scanned log file information and the log storage information of the third-party software.

[0032] In one possible implementation, calculating the importance score of each log entry in each preset scenario includes:

[0033] The acquired logs are cleaned and formatted.

[0034] A log association model is constructed for the formatted logs, where the nodes of the log association model are key log information and the edges are the logical relationships between logs;

[0035] For each log entry, based on its position in the log association model, its correlation with other log entries, and its importance configuration coefficient in each preset scenario, an importance score for that log entry in each preset scenario is determined. Secondly, embodiments of this application provide a log cleaning apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire logs generated by various components in the system within a preset time period;

[0037] The calculation module is used to calculate the importance score of each log in each preset scenario;

[0038] The acquisition module is also used to determine the target scenario in which the system is currently located based on the monitored key system events, and to acquire the preset cleanup strategy corresponding to the target scenario, wherein the target scenario is one of the preset scenarios;

[0039] The determination module is used to determine the task execution plan based on the preset cleanup strategy and the importance score of each log in the target scenario. The task execution plan includes the execution order and timing of the cleanup tasks corresponding to each log to be cleaned.

[0040] The cleaning module is used to clean up the logs in the system according to the preset cleaning strategy, the task execution plan, and the system load status.

[0041] In one possible implementation, the acquisition module is specifically used for:

[0042] When a critical event is detected in the system, the metadata information of the critical event is extracted. The metadata information includes the time of the event, the component information involved in the event, and the business information involved in the event.

[0043] The system identifies the target scene it is currently in using preset scene recognition rules and the metadata information;

[0044] Based on the preset correspondence between each preset scenario and the cleaning strategy, the preset cleaning strategy corresponding to the target scenario is obtained.

[0045] In one possible implementation, the acquisition module is further configured to periodically acquire performance index data of the system;

[0046] The determining module is also used to determine the system load status based on the performance index data.

[0047] In one possible implementation, the determining module is specifically used for:

[0048] The overall utilization rate is obtained by weighted summation of the performance index data;

[0049] If the overall utilization rate is less than a preset threshold, the system load status is determined to be a low load status.

[0050] If the overall utilization rate is greater than or equal to a preset threshold, then the system load state is determined to be a high load state.

[0051] In one possible implementation, the preset cleanup strategy includes a cleanup time period; the determining module is specifically used for:

[0052] Based on the importance score of each log in the target scenario, determine the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy;

[0053] Add the cleanup tasks corresponding to the logs to be cleaned that conform to the preset cleanup strategy to the task queue;

[0054] Based on the number of cleanup tasks in the task queue and the cleanup time period, the cleanup tasks in the task queue are divided into multiple batches. The cleanup tasks in the same batch are executed at the same time, and the cleanup tasks in different batches are executed in chronological order within the cleanup time period.

[0055] In one possible implementation, the cleanup module is specifically used for:

[0056] If the system load is low, the cleanup task will be executed according to the cleanup strategy and the execution timing of each batch of cleanup tasks.

[0057] If the system load is high and the cleanup task has not yet started, the cleanup task will be postponed until the system load returns to low, and then executed according to the cleanup strategy and the execution timing of each batch of cleanup tasks.

[0058] If the system load is high and a cleanup task is being performed, the cleanup task is paused. Once the system load returns to low, the cleanup task is executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0059] In one possible implementation, the device further includes:

[0060] The scanning module is used to periodically perform multi-path scanning on the system's disk and record the information of the scanned log files;

[0061] The recording module is used to record the log storage information of the third-party software when a change in the third-party software version is detected.

[0062] The adjustment module is used to adjust the preset cleanup strategy based on the scanned log file information and the log storage information of the third-party software.

[0063] In one possible implementation, the computing module is specifically used for:

[0064] The acquired logs are cleaned and formatted.

[0065] A log association model is constructed for the formatted logs, where the nodes of the log association model are key log information and the edges are the logical relationships between logs;

[0066] For each log, the importance score of the log in each preset scenario is determined based on its position in the log association model, its degree of association with other logs, and its importance configuration coefficient in each preset scenario.

[0067] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0068] Memory, used to store computer programs;

[0069] When a processor executes a program stored in memory, it implements the method described in the first aspect above.

[0070] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the first aspect above.

[0071] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0072] By adopting the above technical solution, when performing log cleanup, for logs generated by each component in the system within a preset time period, the importance score of each log in each preset scenario can be calculated to determine the current target scenario of the system, and then the preset cleanup strategy corresponding to the target scenario can be obtained. In this way, the preset cleanup strategy matches the current target scenario of the system, ensuring that the preset cleanup strategy closely aligns with the current scenario requirements of the system. Then, based on the preset cleanup strategy and the importance score of each log in the target scenario, a task execution plan can be determined. Logs are then cleaned according to the preset cleanup strategy, the task execution plan, and the system load status. Because the system load status is considered, log cleanup can avoid impacting normal system operations, and because the importance score of each log in the target scenario is considered, cleanup can be performed according to the importance of each log in the target scenario. Therefore, the embodiments of this application can achieve log cleanup in a manner closely aligned with the current system environment, dynamically meeting the log cleanup requirements of various system environments.

[0073] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0075] Figure 1A flowchart illustrating a log cleanup method provided in this application embodiment;

[0076] Figure 2 A flowchart illustrating another log cleanup method provided in this application embodiment;

[0077] Figure 3 This is a schematic diagram of the structure of a log cleaning device provided in an embodiment of this application;

[0078] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0080] To meet diverse log cleanup needs, embodiments of this application provide a log cleanup method, such as... Figure 1 As shown, the method includes:

[0081] S101. Obtain the logs generated by each component in the system within a preset time period, and calculate the importance score of each log in each preset scenario.

[0082] In this application embodiment, logs generated by each component in the system can be acquired periodically. For example, logs generated by each component in the system within a preset time period can be collected every preset time period, or logs generated by each component in the system within a preset time period can be received every preset time period.

[0083] The preset scenarios include various scenarios of the system. For example, the system scenarios include peak business periods, off-peak periods, troubleshooting periods, and security audit periods.

[0084] S102. Based on the monitored key system events, determine the current target scenario of the system and obtain the preset cleanup strategy corresponding to the target scenario.

[0085] The target scenario is one of the preset scenarios, and a preset cleanup strategy has been pre-configured for each preset scenario.

[0086] It should be noted that the execution order between S101 and S102 is not limited in the embodiments of this application. Figure 1 Let's take S101 as an example.

[0087] S103. Based on the preset cleanup strategy and the importance score of each log in the target scenario, determine the task execution plan. The task execution plan includes the execution order and timing of the cleanup tasks corresponding to each log to be cleaned.

[0088] Each log to be cleaned corresponds to a cleanup task, which is used to clean up the log to be cleaned.

[0089] S104. Clean up the logs in the system according to the preset cleanup strategy, task execution plan and system load status.

[0090] This method, when performing log cleanup, calculates the importance score of each log generated by various components within a preset time period in each preset scenario. This determines the current target scenario of the system and yields a corresponding preset cleanup strategy. This strategy matches the target scenario, ensuring it closely aligns with the system's current needs. Then, a task execution plan is determined based on the preset cleanup strategy and the importance score of each log in the target scenario. Logs are cleaned according to the preset cleanup strategy, the task execution plan, and the system load status. By considering the system load status, log cleanup avoids impacting normal system operations. Furthermore, by considering the importance score of each log in the target scenario, cleanup is performed according to the log's importance within that scenario. Therefore, this embodiment of the application enables log cleanup in a manner closely aligned with the current system environment, dynamically meeting the log cleanup requirements of various system environments.

[0091] In execution Figure 1 Before using the methods shown, the system needs to be initialized and configured in advance. Specifically, the following five configurations are required:

[0092] Configuration 1: The system management interface provides settings options for various types of logs, allowing system administrators to configure parameters such as default retention periods, cleanup strategies for different preset scenarios, and system performance thresholds. After completing these configurations, a policy rule base is obtained, which includes preset mappings between preset scenarios and preset cleanup strategies.

[0093] For example, the default retention period for system logs can be set to 30 days, and the cleanup strategy during peak business scenarios can be to clean up system logs with an importance score of less than 0.1. A CPU utilization threshold of 80% and a memory utilization threshold of 90% can be configured, which means that if the CPU utilization exceeds 80% and the memory utilization exceeds 90%, the current system load status is high load.

[0094] Configuration 2, the manual annotation and configuration interface provides options for manually annotating and configuring various types of system logs. In this interface, system administrators or log analysts can annotate key information of each log or each type of log, the logical relationship between logs, and the importance level of logs under each preset scenario, thereby achieving refined management of logs.

[0095] The key information in the log includes the operation subject, the operation object, the operation time, and the operation result.

[0096] For example, in an order processing system, for an order creation log, the operation subject is labeled as "order creation module," the operation object as "order information," the operation time as the timestamp at the time of the operation, and the operation result as "success / failure." If this order creation log is related to subsequent payment logs, the logical relationship can be labeled as "payment operation triggered after order creation." Furthermore, for daily operational scenarios, the importance level of this order creation log is configured as "medium," and for promotional activity scenarios, the importance level is configured as "high."

[0097] For example, for user login logs, the operation subject is labeled as "user login module," the operation object as "user account information," the operation time as the timestamp at the time of the operation, and the operation result as "success / failure." If the user login log is associated with subsequent permission verification logs, the logical relationship is labeled as "permission verification is triggered after successful login." Furthermore, for daily operational scenarios, the importance level of this user login log is configured as "medium," and for security audit scenarios, the importance level is configured as "high."

[0098] Configuration 3: Deploy time monitoring programs in advance at each key business point and system component for subsequent monitoring of critical system events.

[0099] For example, deploy monitoring programs at business points such as order creation and payment, as well as at system components such as server startup and shutdown.

[0100] Configuration 4: Pre-configure log acquisition rules, log transmission methods, and relevant parameters for log preprocessing.

[0101] The log retrieval rules specify the log sources to be retrieved, such as system operation logs, business logs, and system performance logs. System operation logs can include server startup logs and server shutdown logs; business logs can include order creation logs and order payment logs; system performance logs can include CPU usage logs and memory usage logs; and security audit logs can include user login logs and permission change logs.

[0102] Log transmission methods can include rsyslog and Flume. For Linux-based servers, system logs can be transmitted via syslog. For distributed application systems, Flume can be used to collect logs generated by each node, enabling comprehensive log collection from different sources.

[0103] The parameters related to log preprocessing can include data cleaning parameters and formatting parameters. For example, they can include the data types to be deleted, the regular expression rules for data cleaning, the date and time format specifications used for formatting, and the log level mapping rules. The date and time format specifications are used to unify the time format, and the log level mapping rules are used to unify the log level representation rules of different systems.

[0104] Configuration 5: Set the scan cycle for multi-path scanning of the system disk, for example, once a day. You also need to set the initial information for the third-party software version tracking library, including the log storage path, naming rules, and log format for the third-party software versions.

[0105] In this embodiment, a configuration interface is provided for system administrators and professional log analysts. This allows them to leverage their deep understanding of the system architecture and business processes to accurately identify the relationships between logs and precisely assess the importance of logs in each preset scenario. Compared to automatic identification methods, manual configuration fully considers the specificity and complexity of the business, greatly reducing the risk of important logs being mistakenly deleted, while avoiding the long-term retention of unnecessary logs. For example, in complex financial transaction systems, manual annotation ensures that critical logs involving fund flows are properly retained under all circumstances, providing strong support for subsequent auditing and risk assessment, and making log management more accurate.

[0106] After completing the above configuration, you can proceed with log cleanup.

[0107] In S101 above, logs generated by each component in the system can be obtained according to the above configuration 4. After obtaining the logs, the logs can be stored in a temporary log database. Then, the importance score of each log in each preset scenario is calculated through the following steps.

[0108] Step 1: Clean and format the obtained logs.

[0109] The log collection and preprocessing module in the electronic device acquires the logs and performs data cleaning. During the data cleaning process, it can delete irrelevant system prompts, duplicate records, and garbled data from the logs based on the pre-configured data types that need to be deleted.

[0110] Among them, irrelevant system prompts are those that are not related to business or critical system operation, such as "[System] Disk space is sufficient (System prompt: Disk space is sufficient)".

[0111] Duplicate log entries occur when the same log information is recorded multiple times due to a program error, resulting in multiple identical log entries.

[0112] Garbled data refers to garbled characters caused by data transmission or encoding problems.

[0113] When performing data cleaning, the obtained logs can be matched with preset regular expressions, and logs that match the preset regular expressions can be deleted, thereby removing noise information.

[0114] After data cleaning, logs of different formats are uniformly converted into a standard format that the system can recognize, such as JSON. For example, the log message "2023-10-01 10:00:00INFO Order created" can be formatted as {"timestamp":"2023-10-01 10:00:00","level":"INFO","message":"Ordercreated"}.

[0115] Step 2: Construct a log association model for the formatted logs. The nodes in the log association model are key log information, and the edges are the logical relationships between logs.

[0116] Specifically, the log association configuration and analysis module can, based on configuration 2 above, use graph theory to construct a directed graph with key log information as nodes and logical relationships as edges, thereby building a log association model. For example, in an order processing system, the order creation log node is connected to the payment log node via a directed edge, representing their triggering relationship. Simultaneously, based on the importance level pre-configured for each node in each preset scenario, a corresponding importance configuration coefficient is assigned to each log node, forming a complete log association model.

[0117] Step 3: For each log, based on its position in the log association model, the degree of association with other logs, and the importance configuration coefficient in each preset scenario, determine the importance score of the log in each preset scenario.

[0118] The log association configuration and analysis module calculates the importance score for each log node as follows:

[0119] The log node is identified as log node A. A log association model is used to determine that log node A has association edges with the other n log nodes, and each edge is assigned a certain weight W. i Let (i = 1, 2, ..., n) represent the degree of correlation between log node A and other logs, and let k be the importance coefficient of this log. Then, the importance score of log A is...

[0120] For example, during peak business periods, order creation logs are strongly correlated with three logs: payment logs, inventory update logs, etc. (weight W). i If the importance configuration coefficient k is set to 1.5 in peak business scenarios (set to 0.8), then the importance score S of the order creation log is 1.5 × (0.8 + 0.8 + 0.8) = 3.6.

[0121] For example, in a troubleshooting scenario, logs generated by components related to the fault are assigned a higher importance level. If the fault is caused by the database service, then the importance of database operation logs, database connection logs, and other database-related logs will significantly increase. If the database operation logs are closely related to the other four fault-related logs (weight W...), then... i If the importance configuration coefficient k is 2 in the fault diagnosis scenario (all are 0.9), then the importance score of the database operation log in the fault diagnosis scenario is S=2×(0.9+0.9+0.9+0.9)=7.2.

[0122] For example, during periods of low business activity, the importance level of most business-related logs decreases compared to peak periods. Taking order creation logs as an example again, the correlation weight W between the order creation log and the other three logs during low business activity periods is lower. i The importance score remains unchanged at 0.8, but the importance allocation coefficient k is 1, so the importance score S = 1 × (0.8 + 0.8 + 0.8) = 2.4.

[0123] Optionally, the importance score of each log node can be stored in a log association database to provide an accurate basis for formulating subsequent cleanup strategies.

[0124] This application embodiment can collect logs of different formats from different system components and calculate the importance score of each log under each preset scenario. This multi-source data fusion processing capability enables the system to have good scalability, allowing the log cleaning method provided by this application embodiment to adapt to system environments of different sizes and architectures. Whether it is a small enterprise information system or a large cloud computing platform, this method can be applied to log cleaning. By constructing a log association model and calculating the importance score of each log, the importance of logs under each preset scenario can be fully considered during the log cleaning process, so that the log cleaning strategy matches the target scenario of the system and the importance of the logs.

[0125] Based on any of the above embodiments, such as Figure 2 As shown, S102 above, determining the current target scenario of the system based on the monitored key system events and obtaining the preset cleanup strategy corresponding to the target scenario, can be specifically implemented as follows:

[0126] S1021. When a critical system event is detected, extract the metadata information of the critical system event. The metadata information includes the event occurrence time, the component information involved in the event, and the business information involved in the event.

[0127] Because event monitoring programs are pre-deployed in key business nodes and components, the scenario awareness and policy generation module can monitor critical system events in real time. The critical system events to be monitored are pre-configured and include system operation events, system state change events, and performance anomaly events. System operation events include, for example, order creation, payment completion, and product listing. System state change events include, for example, server startup, shutdown, online, and offline events. Performance anomaly events include high CPU utilization events and memory overflow events.

[0128] S1022. Use preset scene recognition rules and metadata information to identify the target scene in which the system is currently located.

[0129] In this step, the scene perception and strategy generation module can use preset scene recognition rules to comprehensively determine the target scene in which the system is currently located by integrating information such as event type, occurrence frequency, and relevant business data. For example, the target scene can be a business peak period, a low period, a fault investigation period, or a security audit period.

[0130] The preset scenario identification rules include: In an e-commerce system, if the number of order creation events exceeds a first threshold X and the number of payment completion events exceeds a second threshold Y within a preset time period, while performance indicators such as CPU utilization and memory utilization are within normal ranges, then the target scenario is determined to be a peak business period scenario. For example, if within one hour, there are more than 1,000 order creation events, more than 800 payment completion events, and CPU utilization remains at 60% and memory utilization remains at 70%, then the target scenario is determined to be a peak business period scenario.

[0131] The preset scenario identification rules can also include: if an abnormal system event is detected, the target scenario is determined to be a troubleshooting scenario. Abnormal system events can include CPU utilization exceeding a third threshold for a continuously preset duration, memory overflow, or failure of critical services (e.g., database connection interruption, message queue congestion). These abnormal system events can be flexibly set according to the actual system. For example, if CPU utilization is detected to be consistently above 85% for 10 minutes, and the database connection experiences more than 5 timeout errors, the target scenario can be determined to be a troubleshooting scenario.

[0132] The preset scene recognition rules may also include: in an e-commerce system, if the frequency of business operations within a specific time period is less than the corresponding threshold and the system performance index is less than the corresponding threshold, then the target scene is determined to be a business off-peak scene.

[0133] The specific time period can be a pre-set period with low business volume, such as 2-4 AM. The business operations can be order creation and order payment. The system performance indicators can be CPU utilization and memory utilization. For example, if, during 2-4 AM, there are fewer than 10 order creation events per hour, fewer than 5 payment completion events per hour, CPU utilization is below 30%, and memory utilization is below 40%, then the target scenario is determined to be a low-business period.

[0134] S1023. Based on the preset correspondence between each preset scenario and the cleaning strategy, obtain the preset cleaning strategy corresponding to the target scenario.

[0135] The scene perception and strategy generation module can obtain the preset cleanup strategy corresponding to the target scene. The preset cleanup strategy includes the cleanup time period, the range of cleanup files, and the directory location of the logs.

[0136] The cleanup time period is a pre-set time frame, such as 2 AM to 4 AM. The cleanup file range includes pre-set cleanup conditions; files within this range are log files that meet those conditions and are to be cleaned. The log directory location is used to locate the log files to be cleaned when the cleanup task is executed.

[0137] During peak business periods, the scope of files to be cleaned under the pre-defined cleanup strategy includes logs with an importance score below 0.2 that are not directly related to the current transaction process. For example, some periodic cached cleanup logs generated during system operation, which have low relevance to the current transaction process, can be cleaned if their importance score is below 0.2. Order-related logs with higher importance scores, such as order creation, payment, and shipping logs, are retained to meet subsequent needs for financial reconciliation and customer dispute resolution.

[0138] During the troubleshooting phase, the scope of files to be cleaned in the pre-defined cleanup strategy is as follows: all logs generated by components related to the fault are retained. Other routine logs unrelated to the fault, such as logs from periodically performed system health checks, are cleaned if their importance score is below a set threshold (e.g., 0.5) and they are deemed unhelpful for troubleshooting. This reduces disk space usage and allows maintenance personnel to focus on critical logs for fault analysis. During subsequent cleanup, all fault-related log files, such as database log files and logs from the application server's database interaction section, are identified and retained for further analysis of the fault's cause by maintenance personnel.

[0139] During periods of low business activity, the pre-defined cleanup strategy covers log files with an importance score below 0.5 and created more than 30 days ago. Some earlier business operation logs that have no direct impact on current operations, but have an importance score below 0.5 and were created more than 30 days ago, can be cleaned up. Logs with high importance scores or those generated recently, such as business operation logs from the past week, are retained to meet the needs of daily business queries and analysis.

[0140] This method enables scenario identification based on monitored key system events, allowing for rapid and accurate capture of changes in system operation. It then selects a matching pre-defined cleanup strategy based on the identified target scenario, ensuring the chosen strategy closely aligns with the cleanup needs of the target scenario and improving the accuracy of log management. Taking an e-commerce system as an example, key system events can be business operation events. Based on these events, the start and end of promotional activities can be accurately determined, and a pre-defined cleanup strategy for the target scenario can be identified. This ensures that key transaction logs are retained during promotional activities and that logs are promptly cleaned up after the promotion ends, thus reasonably releasing storage resources.

[0141] In this embodiment of the application, the performance monitoring and task scheduling module needs to periodically monitor the system performance to determine the system load status. Specifically, it can periodically acquire the system's performance index data and determine the system load status based on the performance index data.

[0142] The performance monitoring cycle can be 5 minutes, that is, collecting performance indicator data once every 5 minutes, or receiving performance indicator data reported by the system once every 5 minutes. The performance indicator data includes CPU utilization, memory utilization, and disk read / write speed.

[0143] The performance index data can be weighted and summed to obtain the overall utilization rate; if the overall utilization rate is less than the preset threshold, the system load status is determined to be low load; if the overall utilization rate is greater than or equal to the preset threshold, the system load status is determined to be high load.

[0144] The performance metrics include at least two of the following: CPU utilization, memory utilization, disk read / write speed, and network traffic.

[0145] For example, if performance metrics include CPU utilization, memory utilization, and disk read / write speed, then the overall utilization rate is P = a × CPU utilization + b × memory utilization + c × disk read / write speed. Here, a, b, and c are weighting coefficients.

[0146] Based on this, the above-mentioned S103, based on the preset cleanup strategy and the importance score of each log in the target scenario, determines the task execution plan, specifically as follows:

[0147] The performance monitoring and task scheduling module determines the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy based on the importance score of each log in the target scenario; adds the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy to the task queue; and divides the cleanup tasks in the task queue into multiple batches according to the number of cleanup tasks in the task queue and the cleanup time period. The cleanup tasks in the same batch are executed at the same time, and the cleanup tasks in different batches are executed in chronological order within the cleanup time period.

[0148] For example, if the preset cleanup strategy specifies that logs with an importance score below 0.2 should be cleaned up, then logs with an importance score below 0.2 are logs to be cleaned up, and the cleanup tasks for these logs to be cleaned up can be added to the task queue.

[0149] Then, the total duration required for all cleanup tasks in the cleanup task queue can be predicted according to the pre-set predicted cleanup duration for a single log. Based on the total duration and the duration of the cleanup period, the first number of cleanup tasks that should be executed concurrently is determined. Based on the first number, the cleanup tasks in the task queue are divided into multiple batches, and each batch includes the first number of cleanup tasks.

[0150] For example, if the cleanup period is 2 hours and the task queue contains 1200 cleanup tasks, assuming the predicted cleanup time for a single cleanup task is 1 minute, and these cleanup tasks need to be completed within 2 hours (120 minutes), then it can be determined that 10 cleanup tasks should be executed concurrently. These 1200 cleanup tasks can be divided into 120 batches, with each batch containing 10 cleanup tasks, and the 10 cleanup tasks in each batch will be executed concurrently.

[0151] Based on this, the above-mentioned S104, which cleans up the logs in the system according to the preset cleanup strategy, task execution plan, and system load status, can be specifically implemented as follows:

[0152] If the system load is low, the cleanup tasks will be executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0153] If the system load is high and the cleanup task has not yet started, the cleanup task will be postponed until the system load returns to low, and then executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0154] If the system load is high and a cleanup task is in progress, the cleanup task will be paused. Once the system load returns to low, the cleanup task will be executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0155] After the cleanup execution module cleans up the logs in the system according to the preset cleanup strategy, task execution plan and load status, it can record the number of log files cleaned, their size, cleanup time and other execution information. This execution information can then be used to adjust the preset cleanup strategy and optimize the task execution plan.

[0156] The cleanup threshold in the preset cleanup strategy can be optimized based on the number and size of the log files cleaned. For example, based on the number and size of the log files cleaned, the proportion of logs cleaned can be calculated. If the proportion of cleaned logs is lower than a preset threshold, the cleanup threshold is relaxed; if the proportion of cleaned logs is higher than the preset threshold, the cleanup threshold is tightened. This cleanup threshold is a threshold set in the cleanup strategy, such as a threshold for importance score. For example, if the preset cleanup strategy includes cleaning logs with an importance score lower than 0.2, and it is determined after cleanup that the proportion of cleaned logs is lower than the preset threshold, the importance score threshold can be lowered by a preset step size, for example, cleaning logs with an importance score lower than 0.1 instead; if it is determined after cleanup that the proportion of cleaned logs is higher than the preset threshold, the importance score threshold can be increased by a preset step size, for example, cleaning logs with an importance score lower than 0.3 instead.

[0157] In addition, in this embodiment, the log growth rate can be determined based on the number of log files recorded for cleanup. If the log growth rate is less than a preset first rate threshold, it indicates that the log growth rate is slow, and the log collection interval can be lengthened, that is, the preset time period in S101 above can be lengthened. This can be done by lengthening according to a preset step size, for example, increasing the preset time period by 6 hours each time. If the log growth rate is greater than a preset second rate threshold, it indicates that the log growth rate is fast, and the log collection interval can be shortened, that is, the preset time period in S101 above can be shortened. This can be done by shortening according to a preset step size, for example, shortening the preset time period by 6 hours.

[0158] This method intelligently schedules log cleanup tasks based on periodically monitored performance metrics, executing them when system load is low. This effectively avoids interference with normal business operations, significantly improving overall system performance and stability. Furthermore, precise pre-defined cleanup strategies ensure that only truly unnecessary logs are cleaned, preventing resource waste caused by over- or inappropriate cleanup. The optimized cleanup timing also reduces the additional resource consumption of cleanup operations, improving resource utilization and further ensuring stable system operation.

[0159] In some embodiments of this application, the method further includes: the log location and tracking module periodically performing multi-path scanning of the system's disk and recording the scanned log file information; when a change in the version of third-party software is detected, recording the log storage information of the third-party software; and adjusting a preset cleanup strategy based on the scanned log file information and the log storage information of the third-party software.

[0160] Specifically, the directory location of logs in the preset cleanup strategy can be adjusted according to the scanned log file information and the log storage information of third-party software to ensure that the cleanup task can find the correct storage directory location of the logs when cleaning up the logs.

[0161] Multi-path scanning can be performed according to a preset scanning cycle, such as scanning every morning. During multi-path scanning, all disk partitions and directories of the system are traversed, and files are identified as log files based on their naming rules, file formats, and content characteristics. For example, by recognizing common log file extensions (such as .log, .txt, etc.) and specific log format information in the file content (such as timestamps, log level identifiers, etc.), log files can be accurately identified regardless of whether they are in a specified directory. For identified log files, their file path, file name, file size, creation time, and other relevant feature information are further extracted and stored. For example, if the file name is "order_20231001.log" and the content contains "[INFO][2023-10-01 10:00:00]Ordercreated", the file can be identified as an order-related log file.

[0162] Multi-path scanning identifies all log files in the system, regardless of whether they are in a specified directory, effectively solving the problem of missed cleanup due to logs not being printed to the designated directory. By periodically scanning the entire disk space and intelligently identifying log files based on their characteristics, all logs are included in the cleanup scope, avoiding disk space waste caused by non-standard log locations. This not only helps free up disk space and improve storage resource utilization, but also makes the system cleaner and more organized, reducing the risk of system failures that may be caused by a large accumulation of useless logs.

[0163] In addition, this application embodiment maintains a third-party software version tracking library, which records key information such as log storage paths, naming rules, and log formats for different versions of each third-party software. When a third-party software version update is detected, the relevant information of the new version's logs can be automatically obtained, and the version tracking library can be updated.

[0164] A version tracking library for third-party software logs can track the impact of software version changes on the logs in real time. When a software version is updated, the system can automatically update the log cleanup strategy to ensure that the new version's logs are effectively cleaned. This solves the problem of difficulty in tracking and cleaning third-party software logs due to version changes, ensures reasonable use of disk space, and prevents excessive disk usage caused by log accumulation. It also ensures that the system's cleanup of various software logs remains consistent and effective. Regardless of software updates, the cleanup strategy can be adjusted promptly to avoid impacting the overall system performance and stability due to poor log management.

[0165] In summary, the log cleanup method provided in this application can improve the efficiency and quality of log management while enhancing the overall performance and stability of the system, thus ensuring the efficient operation of various computer systems.

[0166] Corresponding to the above method embodiments, this application also provides a log cleaning device, such as... Figure 3 As shown, the device includes:

[0167] The acquisition module 301 is used to acquire logs generated by each component in the system within a preset time period.

[0168] Calculation module 302 is used to calculate the importance score of each log in each preset scenario;

[0169] The acquisition module 301 is also used to determine the target scenario of the system based on the monitored key system events, and to acquire the preset cleanup strategy corresponding to the target scenario, wherein the target scenario is one of the preset scenarios;

[0170] The determination module 303 is used to determine the task execution plan based on the preset cleanup strategy and the importance score of each log in the target scenario. The task execution plan includes the execution order and timing of the cleanup tasks corresponding to each log to be cleaned.

[0171] The cleaning module 304 is used to clean up the logs in the system according to the preset cleaning strategy, task execution plan and system load status.

[0172] Optionally, module 301 is used specifically for:

[0173] When a critical system event is detected, the metadata information of the critical system event is extracted. The metadata information includes the time of the event, the component information involved in the event, and the business information involved in the event.

[0174] The system identifies the target scene it is currently in by using preset scene recognition rules and metadata information;

[0175] Based on the preset correspondence between each preset scenario and the cleaning strategy, the preset cleaning strategy corresponding to the target scenario is obtained.

[0176] Optionally, the acquisition module 301 is also used to periodically collect system performance index data;

[0177] The determination module 303 is also used to determine the system load status based on performance index data.

[0178] Optionally, module 303 is specifically used for:

[0179] The overall utilization rate is obtained by weighted summation of the performance index data;

[0180] If the overall utilization rate is less than the preset threshold, the system load status is determined to be low load.

[0181] If the overall utilization rate is greater than or equal to the preset threshold, the system load status is determined to be a high load status.

[0182] Optionally, the preset cleaning strategy includes a cleaning time period; the determination module 303 is specifically used for:

[0183] Based on the importance score of each log in the target scenario, determine the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy;

[0184] Add the cleanup tasks corresponding to the logs to be cleaned that meet the preset cleanup strategy to the task queue;

[0185] Based on the number of cleanup tasks in the task queue and the cleanup time period, the cleanup tasks in the task queue are divided into multiple batches. Cleanup tasks in the same batch are executed at the same time, and the execution times of cleanup tasks in different batches are arranged in chronological order within the cleanup time period.

[0186] Optionally, the cleanup module 304 is specifically used for:

[0187] If the system load is low, the cleanup tasks will be executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0188] If the system load is high and the cleanup task has not yet started, the cleanup task will be postponed until the system load returns to low, and then executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0189] If the system load is high and a cleanup task is in progress, the cleanup task will be paused. Once the system load returns to low, the cleanup task will be executed according to the cleanup strategy and the timing of each batch of cleanup tasks.

[0190] Optionally, the device further includes:

[0191] The scanning module is used to periodically perform multi-path scans of the system's disks and record the scanned log file information;

[0192] The logging module is used to record the log storage information of the third-party software when a change in the third-party software version is detected.

[0193] The adjustment module is used to adjust the preset cleanup strategy based on the scanned log file information and the log storage information of third-party software.

[0194] Optionally, the calculation module 302 is specifically used for:

[0195] The acquired logs are cleaned and formatted.

[0196] A log association model is constructed for the formatted logs. The nodes of the log association model are the key information of the logs, and the edges are the logical relationships between the logs.

[0197] For each log, the importance score of the log is determined based on its position in the log association model, its association with other logs, and its importance configuration coefficient in each preset scenario.

[0198] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0199] Memory 403 is used to store computer programs;

[0200] When the processor 401 executes the program stored in the memory 403, it implements the method steps in the above method embodiments.

[0201] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0202] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0203] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0204] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0205] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described log cleanup methods.

[0206] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the log cleanup methods described above.

[0207] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0208] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0209] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0210] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A log cleaning method characterized by, The method comprises: acquiring logs generated by each component in the system within a preset time period, and calculating, for each log, an importance score of the log in each preset scenario; determining a target scenario in which the system currently is based on a monitored system key event, acquiring a preset cleaning strategy corresponding to the target scenario, and the target scenario being one of the preset scenarios; determining a task execution plan based on the preset cleaning strategy and the importance score of each log in the target scenario, the task execution plan comprising an execution order and an execution time of a cleaning task corresponding to each log to be cleaned; cleaning the logs in the system according to the preset cleaning strategy, the task execution plan and a system load state.

2. The method of claim 1, wherein, The method further comprises: periodically acquiring performance index data of the system; determining the system load state based on the performance index data. The method further comprises:

3. The method of claim 1, wherein, performing weighted summation on the performance index data to obtain a comprehensive usage rate; if the comprehensive usage rate is less than a preset threshold, determining that the system load state is a low load state; if the comprehensive usage rate is greater than or equal to the preset threshold, determining that the system load state is a high load state.

4. The method of claim 3, wherein, The preset cleaning strategy comprises a cleaning time period; and the method further comprises: determining a cleaning task corresponding to a log to be cleaned that meets the preset cleaning strategy based on the importance score of each log in the target scenario; adding the cleaning task corresponding to the log to be cleaned that meets the preset cleaning strategy to a task queue; dividing the cleaning tasks in the task queue into multiple batches according to the number of cleaning tasks in the task queue and the cleaning time period, the execution time of cleaning tasks in a same batch being the same, and the execution time of cleaning tasks in different batches being arranged in a sequential order within the cleaning time period.

5. The method of claim 4, wherein, The method further comprises: if the system load state is a low load state, executing the cleaning tasks according to the cleaning strategy and the execution time of each batch of cleaning tasks; if the system load state is a high load state and the cleaning tasks have not been started to be executed, postponing the execution of the cleaning tasks according to the cleaning strategy and the execution time of each batch of cleaning tasks until the system load state returns to a low load state. ​ 6. The method of claim 5, wherein, ​ ​ ​ If the system load state is a high load state and a cleaning task is being executed, the execution of the cleaning task is suspended, and after the system load state returns to a low load state, the cleaning task is executed according to the cleaning strategy and the execution time of each batch of cleaning tasks.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: periodically performing a multi-path scan on the disks of the system to record scanned log file information; when a change in the version of the third-party software is monitored, recording log storage information of the third-party software; adjusting the preset cleaning strategy based on the scanned log file information and the log storage information of the third-party software.

8. The method of claim 1, wherein, The method further comprises: for each log, calculating an importance score of the log in each preset scenario, comprising: performing data cleaning and formatting processing on the obtained logs; constructing a log association model for the logs after the formatting processing, wherein the nodes of the log association model are log key information, and the edges are logical relationships between logs; 9. A log cleaning apparatus characterized by comprising: for each log, determining an importance score of the log in each preset scenario based on a position of the log in the log association model, an association degree between the log and other logs, and an importance configuration coefficient in each preset scenario. The device comprises: an acquisition module configured to acquire logs generated by components in a system in a preset time period; a calculation module configured to calculate, for each log, an importance score of the log in each preset scenario; the acquisition module is further configured to determine a target scenario in which the system currently is based on monitored system key events, acquire a preset cleaning strategy corresponding to the target scenario, and the target scenario is one of the preset scenarios; a determination module configured to determine a task execution plan based on the preset cleaning strategy and the importance scores of the logs in the target scenario, wherein the task execution plan comprises an execution order and an execution time of a cleaning task corresponding to each log to be cleaned; 10. An electronic device, comprising: a cleaning module configured to clean logs in the system according to the preset cleaning strategy, the task execution plan, and a system load state. The device comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is configured to store a computer program; 11. A computer readable storage medium, characterized in that, the processor is configured to execute the program stored in the memory to implement the method of any one of claims 1-8. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-8. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-8.

Images