Timing log cleaning method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a log timed cleaning method and device, equipment and a storage medium, and the method comprises the following steps: by presetting key field information, constructing a log management strategy by using JSON Schema; optimizing the log management strategy by using a pre-constructed business calendar and a dynamic window adjustment algorithm to obtain an optimization strategy; when the cleaning time included in the optimization strategy arrives, generating a log cleaning task in combination with the load model and the real-time performance index, and optimizing the log cleaning task by adopting a round-robin algorithm to obtain an optimized cleaning task; executing log cleaning operation based on the master-slave architecture of the to-be-cleaned database and the optimized cleaning task; according to the method disclosed by the invention, automation and high efficiency of log cleaning are realized, and tests show that the system processing capability is improved by 40%-65%, and the response time is also shortened from the hour level to the second; in addition, the influence of the log cleaning operation on the system performance is remarkably reduced, and the stability and reliability of the system are greatly enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, device, equipment and storage medium for periodically cleaning logs. Background Art

[0002] With the continuous evolution of information technology, the system architecture has become increasingly complex; the continuous expansion of systems and service modules has led to service splitting, which in turn has caused a sharp increase in the number of log record tables; this scattered log recording method results in the logs of different services being distributed in multiple tables, undoubtedly increasing the difficulty of management and maintenance.

[0003] Current logistics enterprises usually run multiple different systems, and these systems may use different databases and multiple log tables; existing log management methods often have difficulty in achieving unified management and periodic cleaning of logs for multiple systems, multiple databases, and multiple tables, which not only leads to waste of storage space but also may cause a decline in system performance.

[0004] In addition, different business scenarios have different requirements for log retention, which requires the ability to dynamically and real-time adjust the cleaning SQL statements and log retention period; however, current systems often lack sufficient flexibility to meet these needs; further, when dealing with logs, if a large number of log files are deleted at one time, it is very likely to have a serious impact on the performance of the database.

[0005] Therefore, how to effectively manage and clean logs while ensuring system performance has become an urgent technical problem to be solved; it can be seen that the existing technology still needs to be improved. Summary of the Invention

[0006] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method for periodically cleaning logs, which realizes the automation and high efficiency of log cleaning and greatly enhances the stability and reliability of the system.

[0007] The first aspect of the present invention provides a method for periodically cleaning logs, including: obtaining preset keyword field information, and based on the preset keyword field information, constructing a log management policy using JSON Schema; obtaining a pre-constructed business calendar, and based on the pre-constructed business calendar, optimizing the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy; when the cleaning time included in the optimized policy is reached, obtaining a pre-constructed load model and real-time performance indicators corresponding to the database to be cleaned, generating a log cleaning task based on the pre-constructed load model and real-time performance indicators; optimizing the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task; and performing a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task.

[0008] Optionally, in the first implementation manner of the first aspect of the present invention, the obtaining of the preset keyword field information and the construction of the log management policy based on the preset keyword field information by using JSON Schema include: obtaining the preset keyword field information, where the preset keyword field information includes the log retention period, the cleaning frequency, and the range of log tables used; constructing a policy template based on the preset keyword field information by using JSON Schema; verifying the constructed policy template by using the verification engine of JSON Schema, and taking the policy template passing the verification as the log management policy.

[0009] Optionally, in the second implementation manner of the first aspect of the present invention, the obtaining of the pre-constructed business calendar and the optimization of the log management policy by using the dynamic window adjustment algorithm based on the pre-constructed business calendar to obtain the optimized policy include: obtaining the pre-constructed business calendar, where the pre-constructed business calendar includes working days, holidays, and enterprise-specific periods, as well as the cleaning rules corresponding to the working days, holidays, and enterprise-specific periods; confirming the cleaning date by using the dynamic window adjustment algorithm based on the cleaning frequency in the log management policy and the pre-constructed business calendar; integrating the confirmed cleaning date and the constructed log management policy to obtain the optimized policy.

[0010] Optionally, in the third implementation manner of the first aspect of the present invention, after the obtaining of the pre-constructed business calendar and the optimization of the log management policy by using the dynamic window adjustment algorithm based on the pre-constructed business calendar to obtain the optimized policy, it includes: when a policy change request is received, obtaining the change field information; adjusting the constructed log management policy based on the change field information, and optimizing the adjusted log management policy by combining the dynamic window adjustment algorithm and the business calendar to obtain the changed policy; obtaining the preselected database type, and converting the changed policy by using the abstract syntax tree based on the preselected database type to obtain multiple conversion policies corresponding to the database type; broadcasting the multiple conversion policies respectively by using the Kafka message queue.

[0011] Optionally, in the fourth implementation manner of the first aspect of the present invention, when the cleaning time included in the optimization strategy is reached, obtaining a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generating a log cleaning task based on the pre-constructed load model and real-time performance metrics, includes: when the cleaning time included in the optimization strategy is reached, obtaining a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, the real-time performance metrics including real-time CPU usage rate, real-time disk I / O read and write rate, and real-time memory occupancy, the pre-constructed load model including CPU usage rate, disk I / O read and write rate, memory occupancy, and their corresponding weights; confirming the real-time load of the database to be cleaned based on the real-time performance metrics and the pre-constructed load model; obtaining a preset load threshold, comparing the real-time load with the preset load threshold, and generating a log cleaning task according to the comparison result, the log cleaning task including the batch quantity to be cleaned and the number of log records to be cleaned in each batch.

[0012] Optionally, in the fifth implementation manner of the first aspect of the present invention, optimizing the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task, includes: obtaining preset time slice information, the time slice information including the granularity of the time slice and the time slice class, the time slice class including the start time, end time, and priority of the time slice; sorting each time slice included in the preset time slice information according to the priority to obtain a priority queue; optimizing the log cleaning task based on the priority queue to obtain an optimized cleaning task.

[0013] Optionally, in the sixth implementation manner of the first aspect of the present invention, performing a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task, includes: constructing a data synchronization mechanism for the database to be cleaned by using the MySQL master-slave replication mechanism, and constructing a historical log archive table in the slave database; when performing a log cleaning operation on the database to be cleaned, obtaining the latest log data on the master database by using the slave database; based on the optimized cleaning task, transferring the expired log data in the latest log data to the historical log archive table; during the transfer process, when the cleaning task of any time slice is completed, obtaining the real-time performance metrics of the database to be cleaned to determine whether there are any abnormal problems; if there are no abnormal problems, performing the cleaning task of the next time slice until the cleaning tasks of all time slices are completed.

[0014] In the second aspect of the present invention, a log timing cleaning device is provided, including: a construction module, configured to obtain preset key field information, and based on the preset key field information, construct a log management policy using JSON Schema; a first optimization module, configured to obtain a pre-constructed business calendar, and based on the pre-constructed business calendar, optimize the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy; a generation module, configured to, when the cleaning time included in the optimized policy arrives, obtain a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generate a log cleaning task based on the pre-constructed load model and real-time performance metrics; a second optimization module, configured to optimize the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task; a cleaning module, configured to perform a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task.

[0015] Optionally, in the first implementation manner of the second aspect of the present invention, the construction module includes: a first acquisition unit, configured to obtain preset key field information, where the preset key field information includes a log retention period, a cleaning frequency, and a range of log tables used; a first construction unit, configured to construct a policy template using JSON Schema based on the preset key field information; a verification unit, configured to verify the constructed policy template using the verification engine of JSON Schema, and use the policy template that passes the verification as the log management policy.

[0016] Optionally, in the second implementation manner of the second aspect of the present invention, the first optimization module includes: a second acquisition unit, configured to obtain a pre-constructed business calendar, where the pre-constructed business calendar includes working days, holidays, and enterprise-specific periods, and cleaning rules corresponding to the working days, holidays, and enterprise-specific periods; a first confirmation unit, configured to confirm the cleaning date using the dynamic window adjustment algorithm based on the cleaning frequency in the log management policy and the pre-constructed business calendar; an integration unit, configured to integrate the confirmed cleaning date and the constructed log management policy to obtain an optimized policy.

[0017] Optionally, in the third implementation manner of the second aspect of the present invention, the generation module includes: a third acquisition unit, configured to acquire a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned when the cleaning time included in the optimization policy is reached, where the real-time performance metrics include real-time CPU usage rate, real-time disk I / O read / write rate, and real-time memory occupancy, and the pre-constructed load model includes CPU usage rate, disk I / O read / write rate, memory occupancy, and their corresponding weights; a second confirmation unit, configured to confirm the real-time load of the database to be cleaned based on the real-time performance metrics and the pre-constructed load model; a generation unit, configured to acquire a preset load threshold, compare the real-time load with the preset load threshold, and generate a log cleaning task according to the comparison result, where the log cleaning task includes the batch quantity to be cleaned and the number of log records to be cleaned in each batch.

[0018] Optionally, in the fourth implementation manner of the second aspect of the present invention, the second optimization module includes: a fourth acquisition unit, configured to acquire preset time slice information, where the time slice information includes the granularity of the time slice and the time slice class, and the time slice class includes the start time, end time, and priority of the time slice; a sorting unit, configured to sort each time slice included in the preset time slice information according to the priority to obtain a priority queue; an optimization unit, configured to optimize the log cleaning task based on the priority queue to obtain an optimized cleaning task.

[0019] Optionally, in the fifth implementation manner of the second aspect of the present invention, the cleaning module includes: a second construction unit, configured to construct a data synchronization mechanism for the database to be cleaned by using the MySQL master-slave replication mechanism, and construct a historical log archive table in the slave library; a fifth acquisition unit, configured to acquire the latest log data on the master library by using the slave library when performing a log cleaning operation on the database to be cleaned; a transfer unit, configured to transfer the expired log data in the latest log data to the historical log archive table based on the optimized cleaning task; a judgment unit, configured to acquire the real-time performance metrics of the database to be cleaned during the transfer process to judge whether there are any abnormal problems when the cleaning task of any time slice is completed; an execution unit, configured to execute the cleaning task of the next time slice if there are no abnormal problems until the cleaning tasks of all time slices are completed.

[0020] The third aspect of the present invention provides a log timing cleaning device, where the log timing cleaning device includes: a memory and at least one processor, and instructions are stored in the memory; at least one of the processors invokes the instructions in the memory so that the log timing cleaning device executes each step of the log timing cleaning method described in any one of the above.

[0021] The fourth aspect of the present invention provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, each step of the log timing cleaning method described in any one of the above is implemented.

[0022] In the technical solution of the present invention, by presetting keyword field information, a log management policy is constructed using JSON Schema; the log management policy is optimized by using a pre-constructed business calendar and a dynamic window adjustment algorithm to obtain an optimized policy; when the cleaning time included in the optimized policy is reached, a log cleaning task is generated in combination with a load model and real-time performance indicators, and the log cleaning task is optimized by using a time slice rotation algorithm to obtain an optimized cleaning task; based on the master-slave architecture of the database to be cleaned and the optimized cleaning task, a log cleaning operation is performed; the method disclosed in this application realizes the automation and high efficiency of log cleaning. After testing, the system processing capacity has been increased by 40% to 65%, and the response time has also been shortened from the hour level to seconds; in addition, the impact of the log cleaning operation on the system performance is significantly reduced, and the operation loss is reduced to less than 5%, greatly enhancing the stability and reliability of the system. Description of the Drawings

[0023] Figure 1 The first flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0024] Figure 2 The second flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0025] Figure 3 The third flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0026] Figure 4 The fourth flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0027] Figure 5 The fifth flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0028] Figure 6 The sixth flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0029] Figure 7 The seventh flow chart of the log timing cleaning method provided by the embodiment of the present invention;

[0030] Figure 8 A structural schematic diagram of a log timing cleaning device provided by an embodiment of the present invention;

[0031] Figure 9 A structural schematic diagram of a log timing cleaning device provided by an embodiment of the present invention. Detailed implementation manners

[0032] The present invention provides a method, device, equipment and storage medium for periodically cleaning logs. In the present invention, terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the term "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or equipment comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0033] For easy understanding, the specific processes of the embodiments of the present invention are described below. Please refer to Figure 1 , an embodiment of the method for periodically cleaning logs in the embodiments of the present invention includes:

[0034] 101. Obtain preset keyword field information, and based on the preset keyword field information, construct a log management policy using JSON Schema;

[0035] In this embodiment, by constructing a log management policy, the automation and standardization of log management are realized, which not only improves the efficiency of log management, but also ensures the standardization and consistency of log data.

[0036] 102. Obtain a pre-constructed business calendar, and based on the pre-constructed business calendar, optimize the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy;

[0037] In this embodiment, by combining the business calendar and the dynamic window adjustment algorithm to optimize the log management policy, an optimized policy that better meets the actual business needs is obtained; the optimized policy can manage log data more effectively, reduce unnecessary log storage, and thus reduce storage costs.

[0038] 103. When the cleaning time included in the optimized policy is reached, obtain a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generate a log cleaning task based on the pre-constructed load model and real-time performance metrics;

[0039] In this embodiment, when the cleaning time specified by the optimization strategy is reached, the system can automatically obtain the pre-built load model and real-time performance metrics corresponding to the database to be cleaned, and then generate a log cleaning task, ensuring the timeliness and accuracy of log cleaning and avoiding the degradation of system performance caused by excessive log data.

[0040] 104. Optimize the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task;

[0041] In this embodiment, optimizing the log cleaning task based on the time slice rotation algorithm further improves the efficiency of log cleaning, enabling the system to complete the log cleaning work faster while ensuring performance.

[0042] 105. Perform a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task;

[0043] In this embodiment, according to the master-slave architecture of the database to be cleaned and the optimized cleaning task, the system can perform a log cleaning operation on the database to be cleaned, not only ensuring the timely cleaning of log data but also guaranteeing the stability and reliability of the database.

[0044] This application discloses a method for scheduled log cleaning. By presetting keyword field information, a log management strategy is constructed using JSON Schema; the log management strategy is optimized using a pre-built business calendar and a dynamic window adjustment algorithm to obtain an optimized strategy; when the cleaning time included in the optimized strategy is reached, a log cleaning task is generated in combination with the load model and real-time performance metrics, and the log cleaning task is optimized using the time slice rotation algorithm to obtain an optimized cleaning task; a log cleaning operation is performed based on the master-slave architecture of the database to be cleaned and the optimized cleaning task; the method disclosed in this application realizes the automation and high efficiency of log cleaning. After testing, the system processing capacity has been increased by 40% to 65%, and the response time has also been shortened from the hour level to seconds; in addition, the impact of the log cleaning operation on system performance has been significantly reduced, and the operation loss has been reduced to less than 5%, greatly enhancing the stability and reliability of the system.

[0045] Please refer to Figure 2 , the second embodiment of the method for scheduled log cleaning in the embodiments of the present invention includes:

[0046] 201. Obtain preset keyword field information, where the preset keyword field information includes the log retention period, cleaning frequency, and the range of log tables used;

[0047] 202. Based on the preset keyword field information, construct a policy template using JSON Schema;

[0048] In this embodiment, JSON Schema is used to precisely describe the structure and data types of the log management policy. By formulating a set of standard JSON Schema, the policy template can cover key information. When setting the cleaning policy, the administrator must follow the structure of the template to ensure the standardization and integrity of the configuration information, which is convenient for subsequent system parsing and execution; for example, the log retention period, cleaning frequency, and the applicable log table range, etc.; taking the log retention period field as an example, its data type is set to an integer and limited within a reasonable range (such as 1 to 365 days); at the same time, for the applicable log table range, it is defined in the form of an array, and each element in the array is the unique identifier of the log table.

[0049] 203. Use the validation engine of JSON Schema to verify the constructed policy template, and use the policy template that passes the verification as the log management policy;

[0050] In this embodiment, by using the validation engine of JSON Schema to verify the constructed policy template, it can ensure that the format and content of the policy template conform to the preset specifications, thereby improving the accuracy and reliability of the policy template; the verified policy template is used as the log management policy, which can further improve the efficiency and security of log management, and ensure the compliance and traceability of log data.

[0051] Please refer to Figure 3 , the third embodiment of the log scheduled cleaning method in the embodiment of the present invention includes:

[0052] 301. Obtain a pre-constructed business calendar, where the pre-constructed business calendar includes working days, holidays, enterprise-specific periods, and cleaning rules corresponding to the working days, holidays, and enterprise-specific periods;

[0053] 302. Based on the cleaning frequency in the log management policy and the pre-constructed business calendar, use the dynamic window adjustment algorithm to confirm the cleaning date;

[0054] 303. Integrate the confirmed cleaning date and the constructed log management policy to obtain an optimized policy;

[0055] In this embodiment, the log management strategy is optimized by combining the business calendar and the dynamic window adjustment algorithm. The business calendar not only contains regular date information but also takes into account factors such as enterprise-specific holidays and working day arrangements. For example, an enterprise may not want to perform log cleaning operations during certain special holidays to avoid affecting the normal operation of the business. The dynamic window adjustment algorithm automatically adjusts the cleaning time point according to the business calendar and in combination with the cleaning frequency (such as daily, weekly, etc.) in the log management strategy. Algorithmically, first, the attribute of the current date in the business calendar (working day or holiday) is determined. If it is a holiday and the holiday non-cleaning rule is set, the cleaning time point is postponed to the next working day. For the case of weekly cleaning, according to the business calendar, the weekly cleaning day is determined. For example, if it is set to the early morning of every Sunday, the dynamic window adjustment algorithm will automatically calculate the specific time point of each Sunday as the cleaning time to ensure that the cleaning operation is executed within an appropriate time window, minimizing the impact on the business to the greatest extent.

[0056] In this embodiment, the log management strategy is optimized by combining the business calendar and the dynamic window adjustment algorithm to ensure that log cleaning not only meets the established frequency requirements but also avoids periods that interfere with enterprise operations, not only improving the cleaning efficiency but also maintaining the continuity and stability of the business.

[0057] Please refer to Figure 4 , the fourth embodiment of the log timing cleaning method in the embodiment of the present invention includes:

[0058] 401. When a policy change request is received, obtain the change field information;

[0059] 402. Adjust the constructed log management strategy based on the change field information, and optimize the adjusted log management strategy in combination with the dynamic window adjustment algorithm and the business calendar to obtain a changed policy;

[0060] 403. Obtain the preselected database type, and based on the preselected database type, use an abstract syntax tree to transform the changed policy to obtain multiple transformed policies corresponding to the database type;

[0061] In this embodiment, by constructing an Abstract Syntax Tree (AST) transformation, dedicated SQL applicable to various types of databases is generated. In a multi-database environment, the syntax differences of SQL statements are a common problem. This application uses an abstract syntax tree to overcome this problem. First, the general log cleaning SQL statement generated based on a unified policy template, that is, generated based on the log management policy, is parsed into an abstract syntax tree. For example, for a simple delete statement "DELETE FROM log_table WHERE timestamp<'specific time'", the parser will disassemble it into a node structure, including a DELETE operation node, a table name node, a WHERE condition node, etc. Next, for different types of databases (such as MySQL, Oracle, SQL Server, etc.), corresponding AST transformation rules are written. Taking MySQL and Oracle as examples, in MySQL, date comparison uses the "<" operator, while in Oracle, specific date functions may be required for comparison. Based on these differences, the transformation rules will make corresponding adjustments and replacements to the nodes in the AST, thereby generating a dedicated SQL statement applicable to the target database. This transformation method based on the abstract syntax tree can flexibly adapt to the syntax requirements of different databases, realizing the automatic generation and adaptation of log cleaning SQL in a multi-database environment.

[0062] 404. Use Kafka message queues to broadcast multiple transformation strategies respectively.

[0063] In this embodiment, the Kafka message queue technology is adopted. When the administrator modifies the log management policy in the unified configuration center, the system serializes the new configuration information into JSON format and sends it to a specific topic of Kafka. The execution node (i.e., the server or process responsible for the actual log cleaning operation) subscribes to this topic. Once it receives a new message, it immediately parses the configuration information and updates the locally cached configuration data. This push mechanism based on the message queue can ensure that configuration changes are propagated to each execution node in a short time, ensuring the consistency and timeliness of the log cleaning strategy of the entire system.

[0064] Please refer to Figure 5 , the fifth embodiment of the log scheduled cleaning method in the embodiment of the present invention includes:

[0065] 501. When the cleaning time included in the optimization strategy is reached, obtain the pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned. The real-time performance metrics include real-time CPU usage, real-time disk I / O read and write rate, and real-time memory occupancy. The pre-constructed load model includes CPU usage, disk I / O read and write rate, and memory occupancy and their corresponding weights.

[0066] 502. Confirm the real-time load of the database to be cleaned based on real-time performance metrics and a pre-built load model;

[0067] In this embodiment, the load model can be CPU usage rate (weight 0.4), disk I / O (weight 0.3), and memory occupancy (weight 0.3). Then, the real-time load = 0.4 * real-time CPU usage rate + 0.3 * real-time disk I / O read / write rate + 0.3 * real-time memory occupancy.

[0068] 503. Obtain a preset load threshold, compare the real-time load with the preset load threshold, and generate a log cleaning task according to the comparison result. The log cleaning task includes the batch quantity to be cleaned and the number of log records to be cleaned in each batch;

[0069] In this embodiment, by obtaining the preset load threshold and comparing it with the real-time load, the execution strategy of the log cleaning task can be dynamically adjusted according to the current load situation of the system, which can improve the flexibility and efficiency of log management and ensure the stability and performance of the system. The preset load threshold includes a high load threshold and a low load threshold. Among them, the high load threshold can be 0.8, and the low load threshold can be 0.5. Assume that the number of log records to be cleaned in each batch is 10,000 records. When the system is in a high load state (the real-time load threshold is greater than 0.8), reduce the number of log records to be cleaned in each batch (adjusted to 0.2 * 10,000 records) to reduce the occupation of system resources and avoid further increasing the system burden. When the system load is moderate (0.5 ≤ real-time load threshold ≤ 0.8), adjust the number of log records to be cleaned in each batch according to the real-time load threshold (10,000 * (1 - real-time load threshold) records) to achieve a balance between resource utilization and system performance. When the system is in a low load state (the real-time load threshold is less than 0.5), keep the number of log records to be cleaned in each batch as the set value (10,000 records) to ensure the efficiency of the log cleaning task.

[0070] Please refer to Figure 6 , the sixth embodiment of the log scheduled cleaning method in the embodiment of the present invention includes:

[0071] 601. Obtain preset time slice information, where the time slice information includes the granularity of the time slice and the time slice class. The time slice class includes the start time, end time, and priority of the time slice;

[0072] In this embodiment, the time slice information is determined by comprehensively considering factors such as the generation frequency of business data, storage cost, and system performance.

[0073] 602. Sort each time slice included in the preset time slice information according to the priority to obtain a priority queue;

[0074] In this embodiment, the priority queue processes time slices in order of priority, giving priority to processing time slices with higher priorities. For the data within a high-priority time slice, it can be regarded as hot data and processed preferentially, such as real-time analysis, backup, etc. For the data within a low-priority time slice, it can be regarded as cold data and archived or deleted.

[0075] 603. Optimize the log cleaning task based on the priority queue to obtain an optimized cleaning task;

[0076] In this embodiment, the round-robin time slice algorithm is adopted to decompose large transactions into multiple micro-transactions. To avoid a large impact on the database performance caused by deleting a large number of logs at one time, the round-robin time slice algorithm is used to divide the entire log cleaning task into multiple time slices, and one micro-transaction is executed within each time slice, that is, a certain number of log records are deleted. For example, each time slice is set to 100 milliseconds, and within each time slice, a deletion operation is performed to delete 100 log records. In this way, the large transaction that may originally occupy database resources for a long time is decomposed into multiple short-time and small-grained micro-transactions, reducing the impact on database concurrent operations and improving the response speed and stability of the database.

[0077] Please refer to Figure 7 , the seventh embodiment of the log scheduled cleaning method in the embodiment of the present invention includes:

[0078] 701. Build a data synchronization mechanism for the database to be cleaned by using the MySQL master-slave replication mechanism, and build a historical log archive table in the slave database;

[0079] 702. When performing a log cleaning operation on the database to be cleaned, obtain the latest log data on the master database from the slave database;

[0080] 703. Based on the optimized cleaning task, transfer the expired log data in the latest log data to the historical log archive table;

[0081] In this embodiment, by using the database to be cleaned with a master-slave architecture, the historical log archiving operation is arranged in the slave database. The master database focuses on processing the daily read and write tasks of the business system, while the slave database mainly undertakes data backup and some operations with low real-time requirements, such as log archiving. When performing historical log archiving on the slave database, it is first necessary to synchronize the latest log data from the master database to ensure data consistency. Therefore, it is necessary to ensure data synchronization between the master database and the slave database. Next, according to the configured log management policy, the outdated log data is migrated from the log table of the slave database to a dedicated historical log archive table. By performing historical log archiving on the slave database, we not only reduce the workload of the master database but also ensure the safe storage and management of historical log data, providing convenience for subsequent query and auditing work.

[0082] 704. During the transfer process, when the cleaning task for any time slice is completed, obtain the real-time performance metrics of the database to be cleaned to determine whether there are any abnormal problems;

[0083] 705. If there are no abnormal problems, execute the cleaning task for the next time slice until all time slice cleaning tasks are completed;

[0084] In this embodiment, after each time slice ends, check the status and performance metrics of the database to be cleaned. If an abnormality is found (such as too high CPU usage, too long transaction waiting time, etc.), the execution parameters for the next time slice can be dynamically adjusted, such as reducing the operation volume of micro-transactions or extending the time slice interval, to avoid unnecessary impacts on business operations; in this way, the system can adaptively adjust the rhythm of log cleaning to ensure that the process of log management is both efficient and secure.

[0085] Furthermore, if there are abnormal problems, suspend the log cleaning task and send an alarm message so that the administrator can intervene and handle it in a timely manner; in this embodiment, once an abnormal problem is detected, the system immediately suspends the log cleaning task and sends an alarm message to the administrator through a preset alarm channel (such as email, text message, or system log), reminding the administrator to pay attention to and handle the abnormal situation; the administrator can quickly locate the cause of the problem based on the content of the alarm message and take corresponding measures to solve it; this abnormal handling mechanism can ensure that when problems are encountered during the log cleaning process, they can be quickly responded to and handled, avoiding the expansion of problems and ensuring the stability and reliability of the system.

[0086] The log scheduled cleaning method in the embodiment of the present invention has been described above. Next, the log scheduled cleaning device in the embodiment of the present invention will be described. Please refer to Figure 8 , an embodiment of the log scheduled cleaning device in the embodiment of the present invention includes:

[0087] A construction module 801, configured to obtain preset keyword field information, and based on the preset keyword field information, construct a log management policy using JSON Schema; a first optimization module 802, configured to obtain a pre-constructed business calendar, and based on the pre-constructed business calendar, optimize the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy; a generation module 803, configured to when the cleaning time included in the optimized policy arrives, obtain a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generate a log cleaning task based on the pre-constructed load model and real-time performance metrics; a second optimization module 804, configured to optimize the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task; a cleaning module 805, configured to perform a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task.

[0088] In this embodiment, the building module 801 includes: a first acquisition unit 8011, configured to acquire preset keyword field information, where the preset keyword field information includes a log retention period, a cleaning frequency, and a range of log tables used; a first building unit 8012, configured to build a policy template using a JSON Schema building strategy based on the preset keyword field information; and a verification unit 8013, configured to verify the built policy template using a verification engine of JSON Schema, and use the policy template that passes the verification as a log management policy.

[0089] In this embodiment, the first optimization module 802 includes: a second acquisition unit 8021, configured to acquire a pre-built business calendar, where the pre-built business calendar includes working days, holidays, and enterprise-specific periods, as well as cleaning rules corresponding to the working days, holidays, and enterprise-specific periods; a first confirmation unit 8022, configured to confirm a cleaning date using a dynamic window adjustment algorithm based on the cleaning frequency in the log management policy and the pre-built business calendar; and an integration unit 8023, configured to integrate the confirmed cleaning date and the built log management policy to obtain an optimized policy.

[0090] In this embodiment, the generation module 803 includes: a third acquisition unit 8031, configured to, when the cleaning time included in the optimized policy is reached, acquire a pre-built load model and real-time performance metrics corresponding to the database to be cleaned, where the real-time performance metrics include real-time CPU usage rate, real-time disk I / O read / write rate, and real-time memory occupancy, and the pre-built load model includes CPU usage rate, disk I / O read / write rate, and memory occupancy and their corresponding weights; a second confirmation unit 8032, configured to confirm the real-time load of the database to be cleaned based on the real-time performance metrics and the pre-built load model; and a generation unit 8033, configured to acquire a preset load threshold, compare the real-time load with the preset load threshold, and generate a log cleaning task according to the comparison result, where the log cleaning task includes a batch quantity to be cleaned and the number of log records to be cleaned in each batch.

[0091] In this embodiment, the second optimization module 804 includes: a fourth acquisition unit 8041, configured to acquire preset time slice information, where the time slice information includes the granularity of the time slice and the time slice class, and the time slice class includes the start time, end time, and priority of the time slice; a sorting unit 8042, configured to sort each time slice included in the preset time slice information according to the priority to obtain a priority queue; and an optimization unit 8043, configured to optimize the log cleaning task based on the priority queue to obtain an optimized cleaning task.

[0092] In this embodiment, the cleaning module 805 includes: a second construction unit 8051, configured to construct a data synchronization mechanism for the database to be cleaned by using the MySQL master-slave replication mechanism, and construct a historical log archive table in the slave database; a fifth acquisition unit 8052, configured to, when performing a log cleaning operation on the database to be cleaned, obtain the latest log data on the master database by using the slave database; a transfer unit 8053, configured to transfer the expired log data in the latest log data to the historical log archive table based on an optimized cleaning task; a judgment unit 8054, configured to, during the transfer process, when the cleaning task of any time slice is completed, obtain the real-time performance metrics of the database to be cleaned to determine whether there are any abnormal problems; and an execution unit 8055, configured to, if there are no abnormal problems, execute the cleaning task of the next time slice until the cleaning tasks of all time slices are completed.

[0093] Based on the same idea as the method in the above embodiment, the device provided in this application can implement the method in the above embodiment.

[0094] Above Figure 8 The log timing cleaning device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. Next, the log timing cleaning device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0095] Figure 9 FIG. is a schematic structural diagram of a log timing cleaning device provided by an embodiment of the present invention. The log timing cleaning device 900 may vary greatly due to configuration or performance, and may include one or more processors (central processing units, CPUs) 910 (for example, one or more processors) and a memory 920, and one or more storage media 930 (for example, one or more mass storage devices) for storing application programs 933 or data 932. Among them, the memory 920 and the storage media 930 may be transient storage or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the log timing cleaning device 900. Further, the processor 910 may be configured to communicate with the storage media 930 and execute a series of instruction operations in the storage media 930 on the log timing cleaning device 900 to implement the steps of the log timing cleaning method provided in the above method embodiments.

[0096] The log scheduled cleaning device 900 may further include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that Figure 9 The shown structure of the log scheduled cleaning device does not constitute a limitation on the log scheduled cleaning device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0097] The present invention also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or it may also be a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is caused to execute the steps of the log scheduled cleaning method.

[0098] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system or device and unit can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, etc., which can store program codes.

[0100] Finally, it should be noted that the above are only preferred examples of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for periodically clearing logs, characterized in that, Including: Obtain preset keyword field information, and based on the preset keyword field information, construct a log management policy using JSON Schema; Obtain a pre-constructed business calendar, and based on the pre-constructed business calendar, optimize the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy; When the cleaning time included in the optimized policy is reached, obtain a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generate a log cleaning task based on the pre-constructed load model and real-time performance metrics; Optimize the log cleaning task based on the time slice rotation algorithm to obtain an optimized cleaning task; Based on the master-slave architecture of the database to be cleaned and the optimized cleaning task, perform a log cleaning operation on the database to be cleaned.

2. The log timing cleaning method according to claim 1, wherein The obtaining of the preset keyword field information, and based on the preset keyword field information, constructing a log management policy using JSON Schema includes: Obtain preset keyword field information, where the preset keyword field information includes a log retention period, a cleaning frequency, and a range of log tables used; Based on the preset keyword field information, construct a policy template using JSON Schema; Use the validation engine of JSON Schema to verify the constructed policy template, and use the policy template that passes the verification as the log management policy.

3. The log timing cleaning method according to claim 2, characterized in that The obtaining of the pre-constructed business calendar, and based on the pre-constructed business calendar, optimizing the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy includes: Obtain a pre-constructed business calendar, where the pre-constructed business calendar includes working days, holidays, and enterprise-specific periods, as well as cleaning rules corresponding to the working days, holidays, and enterprise-specific periods; Based on the cleaning frequency in the log management policy and the pre-constructed business calendar, use a dynamic window adjustment algorithm to confirm the cleaning date; Integrate the confirmed cleaning date and the constructed log management policy to obtain an optimized policy.

4. The log timing cleaning method according to claim 1, wherein After the obtaining of the pre-constructed business calendar, and based on the pre-constructed business calendar, optimizing the log management policy using a dynamic window adjustment algorithm to obtain an optimized policy, it includes: When a policy change request is received, obtain change field information; Adjust the constructed log management policy based on the change field information, and optimize the adjusted log management policy by combining the dynamic window adjustment algorithm and the business calendar to obtain a change policy; Obtain a preselected database type, and based on the preselected database type, use an abstract syntax tree to transform the change policy to obtain multiple transformation policies corresponding to the database type; Broadcast multiple transformation policies using a Kafka message queue respectively.

5. The log timing cleaning method according to claim 1, characterized in that The when the cleaning time included in the optimized policy is reached, obtaining a pre-constructed load model and real-time performance metrics corresponding to the database to be cleaned, and generating a log cleaning task based on the pre-constructed load model and real-time performance metrics includes: When the cleaning time included in the optimization strategy is reached, obtain the pre-built load model and real-time performance metrics corresponding to the database to be cleaned. The real-time performance metrics include real-time CPU usage, real-time disk I / O read / write rate, and real-time memory occupancy. The pre-built load model includes CPU usage, disk I / O read / write rate, memory occupancy, and their corresponding weights; Confirm the real-time load of the database to be cleaned based on the real-time performance metrics and the pre-built load model; Obtain the preset load threshold, compare the real-time load with the preset load threshold, and generate a log cleaning task according to the comparison result. The log cleaning task includes the batch volume to be cleaned and the number of log records to be cleaned in each batch.

6. The log timing cleaning method according to claim 5, wherein The optimization of the log cleaning task based on the round-robin time slice algorithm to obtain an optimized cleaning task includes: Obtain the preset time slice information. The time slice information includes the granularity of the time slice and the time slice class. The time slice class includes the start time, end time, and priority of the time slice; Sort each time slice included in the preset time slice information according to the priority to obtain a priority queue; Optimize the log cleaning task based on the priority queue to obtain an optimized cleaning task.

7. The log timing cleaning method according to claim 6, wherein The execution of the log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task includes: Adopt the MySQL master-slave replication mechanism to construct a data synchronization mechanism for the database to be cleaned, and construct a historical log archive table in the slave database; When performing the log cleaning operation on the database to be cleaned, obtain the latest log data on the master database using the slave database; Based on the optimized cleaning task, transfer the expired log data in the latest log data to the historical log archive table; During the transfer process, when the cleaning task of any time slice is completed, obtain the real-time performance metrics of the database to be cleaned to determine whether there are any abnormal problems; If there are no abnormal problems, execute the cleaning task of the next time slice until the cleaning tasks of all time slices are completed.

8. A log timing cleaning device, characterized in that, It includes: A construction module for obtaining preset keyword field information and constructing a log management strategy using JSONSchema based on the preset keyword field information; A first optimization module for obtaining a pre-built business calendar and optimizing the log management strategy using a dynamic window adjustment algorithm based on the pre-built business calendar to obtain an optimization strategy; A generation module for obtaining the pre-built load model and real-time performance metrics corresponding to the database to be cleaned when the cleaning time included in the optimization strategy is reached, and generating a log cleaning task based on the pre-built load model and real-time performance metrics; A second optimization module for optimizing the log cleaning task based on the round-robin time slice algorithm to obtain an optimized cleaning task; A cleaning module for performing a log cleaning operation on the database to be cleaned based on the master-slave architecture of the database to be cleaned and the optimized cleaning task.

9. A log timing cleaning device, characterized in that, The log scheduled cleaning device includes: a memory and at least one processor, and instructions are stored in the memory; At least one of the processors invokes the instructions in the memory to cause the log timing cleaning device to execute each step of the log timing cleaning method according to any one of claims 1-7.

10. A computer-readable storage medium, on which instructions are stored, characterized in that, When the instructions are executed by the processor, each step of the log timing cleaning method according to any one of claims 1-7 is implemented.

Images