Log cleaning and filing method and system, readable storage medium and equipment

Abstract

The invention provides a log cleaning and archiving method and system, a readable storage medium and device.The method responds to a trigger signal, whether redundancy exists in log data in a log data list or not is judged, and if yes, redundant data is filtered; dividing the filtered log data according to the data size, obtaining the log data smaller than a threshold value in batches, and obtaining the log data larger than or equal to the threshold value one by one; judging whether the obtained log data belongs to a big data field exceeding a set value or not, and if yes, performing truncation or keyword replacement on the big data field, and storing the processed log data into an archiving table or an archiving file; and deleting the corresponding data in the current log data list in batches. By optimizing a log storage mechanism, a data truncation processing strategy, a reduction and retention strategy, a storage strategy and the like, the data security is improved, the processing efficiency is improved, and high flexibility is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of log processing, and particularly relates to a log cleaning and archiving method, system, readable storage medium and device. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of informatization, the amount of data processed by business systems has increased sharply. Correspondingly, the scale of their log data is also growing day by day, and it is necessary to regularly clean and archive the logs stored in the database table.

[0004] Traditional modes include manual processing and processing using timed scripts or programs.

[0005] However, these two processing methods have the following defects:

[0006] The manual processing method often requires manual intervention, with low efficiency, resulting in increased management and maintenance costs. There may be a risk of memory overflow under a large amount of data, and there may be security risks such as data leakage during the data archiving process. The processing method of scripts or programs has a relatively mechanical processing cycle and is generally simple batch processing, without considering the flexibility of business requirements, data security, stability and efficiency of processing logic. Moreover, both processing methods will generate data redundancy, resulting in large storage space occupation, being unable to adapt to the flexible changes of log archiving requirements, having low processing efficiency for ultra-long and extra-large fields, and being prone to problems such as accidental deletion or incorrect retention. Summary of the Invention

[0007] In order to solve the above problems, the present invention provides a log cleaning and archiving method, system, readable storage medium and device. By optimizing the log storage mechanism, data truncation processing strategy, reduction retention strategy and storage strategy, etc., the present invention improves data security, processing efficiency and has great flexibility.

[0008] According to some embodiments, the present invention adopts the following technical solutions:

[0009] A log cleaning and archiving method, in response to a trigger signal, starts the log cleaning and archiving process, including the following steps:

[0010] Judge whether there is redundant log data in the log data list. If so, filter the redundant data;

[0011] Divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one;

[0012] Determine whether the acquired log data belongs to a large data field exceeding a set value. If so, truncate or perform keyword replacement on the large data field, and store the processed log data in an archive table or an archive file;

[0013] Batch delete the corresponding data in the current log data list.

[0014] As an alternative implementation, the following steps are further included:

[0015] Determine whether the log data in the log data list is abnormal log data. If so, perform delayed archiving on the abnormal log data according to the configured delay time length.

[0016] As an alternative implementation, the trigger signal is a timed trigger signal, an event trigger signal, or a trigger request signal.

[0017] As an alternative implementation, the process of filtering redundant data when determining whether there is redundancy in the log data in the log data list includes: determining whether the log data is main log data or detailed log data. If it is detailed log data, then determine whether the detailed log data covers a preset cycle. If it covers the preset cycle, only retain some log data and delete other log data.

[0018] As a further step, the specific process of only retaining some log data includes retaining a set number of log data at the front, retaining a randomly set number of log data, retaining log data in a linear cycle, retaining log data in a range cycle, or retaining / deleting log data according to a configured keyword.

[0019] As an alternative implementation, the log cleaning and archiving process is executed cyclically until the log data in the log data list is empty.

[0020] As an alternative implementation, log data is stored in different log data lists according to pre-configured service partitioning clusters and writing periods. Different service partitioning clusters are used to store log data of different business systems or business modules;

[0021] Each service partitioning cluster includes several tables, and each table is switched in sequence according to the configured writing period to store the corresponding log data.

[0022] A log cleaning and archiving system includes:

[0023] A trigger module, configured to start the log cleaning and archiving process in response to a trigger signal;

[0024] A redundancy processing module, configured to determine whether there is redundancy in the log data in the log data list, and if so, filter the redundant data;

[0025] A classification module, configured to divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one;

[0026] An archiving module, configured to determine whether the obtained log data belongs to a large data field exceeding a set value, and if so, truncate or replace keywords for the large data field, and store the processed log data in an archive table or an archive file;

[0027] A deletion module, configured to batch delete the corresponding data in the current log data list.

[0028] A computer-readable storage medium for storing computer instructions, which when executed by a processor, complete the steps in the above method.

[0029] An electronic device, including a memory, a processor, and computer instructions stored on the memory and running on the processor, which when executed by the processor, complete the steps in the above method.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] The present invention first provides a flexible configuration of a circular log table storage mechanism, which can balance the storage pressure and retrieval requirements of logs. The tables required for storing data can be divided into a group of circularly writable log table clusters. The tables in the same cluster are used to store data of the same business module. Each table in each cluster can write data in sequence according to the cycle configuration, that is, it can customize the configuration of business division clusters, the number of circular log tables, and the log writing cycle according to the data volume of the business and the maximum historical data required by the business, which is also beneficial to analyze and compare the log data of the same cycle, and to the greatest extent ensure the retrieval requirements of valid data and reduce the storage pressure of log data.

[0032] The present invention secondly provides a delayed archiving processing strategy for exception logs, so that exception logs with greater log value or higher viewing frequency can be retained according to the configured maximum retention days. After exceeding the above maximum retention days, the exception log data will enter the archive table, which not only ensures the orderliness of the data and the traceability of business execution, but also facilitates problem analysis and troubleshooting.

[0033] The present invention provides a data truncation processing strategy for ultra-large field log data, which not only supports configuring the truncation threshold of the data, but also supports configuring the maximum search level to achieve a balance between truncation efficiency and data validity.

[0034] The present invention also provides a reduction and retention strategy for redundant or non-critical logs. For unimportant logs or identical or similar data within a cycle, it flexibly reduces and retains them, effectively improving the data processing efficiency, reducing the amount of log data processed, accelerating the transmission and processing speed, and reducing the bandwidth requirement.

[0035] The present invention supports multiple triggering mechanisms such as manual triggering, event triggering, and timed triggering, which can be flexibly configured according to the requirements of the task scenario and can meet the log cleaning and archiving requirements in different scenarios.

[0036] The present invention clears and archives logs in batches or one by one according to the data size, taking both memory and efficiency into account.

[0037] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given in conjunction with the accompanying drawings and are described in detail as follows. Description of the Drawings

[0038] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0039] Figure 1 It is a schematic flow chart of a log cleaning and archiving method of an embodiment;

[0040] Figure 2 It is a schematic structural diagram of a log cleaning and archiving system of an embodiment;

[0041] Figure 3 It is a schematic structural diagram of an electronic device of an embodiment. Detailed Embodiments

[0042] The present invention will be further described below in conjunction with the drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are all illustrative and are intended to provide a further description of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] In the case of no conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0046] Embodiment 1

[0047] A log cleaning and archiving method, in response to a trigger signal, starts the log cleaning and archiving process, as Figure 1 shown, including the following steps:

[0048] S1. Determine whether there is redundant log data in the log data list. If so, filter the redundant data;

[0049] S2. Divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one;

[0050] S3. Determine whether the obtained log data belongs to a large data field exceeding a set value. If so, truncate or replace the keywords of the large data field, and store the processed log data in an archive table or an archive file;

[0051] S4. Batch delete the corresponding data in the current log data list.

[0052] First, the trigger signal is a timed trigger signal, an event trigger signal or a trigger request signal. In this embodiment, multiple trigger mechanisms such as manual trigger, event trigger and timed trigger are supported. The timed trigger can define a simple time period or define a Cron expression to define the execution rule of the timed task, and the log cleaning and archiving task will be automatically executed at a fixed time point.

[0053] Of course, if the log accumulation is relatively serious, the log archiving can also be quickly processed through manual trigger.

[0054] In addition, a certain event can also be defined, such as the occurrence of a certain type of exception log, or the log data volume exceeds a preset threshold. When the defined event occurs, the log archiving task will be automatically triggered.

[0055] Through the above flexible trigger configuration mechanism, this embodiment can meet the requirements of log cleaning and archiving in different scenarios and various scenarios. The above trigger configuration mechanism can be pre-configured or switched according to requirements.

[0056] In some embodiments, it further includes: determining whether the log data in the log data list is abnormal log data. If so, the abnormal log data is archived with a delay according to the configured delay time length.

[0057] In view of the fact that the original intention of usually recording logs is, on the one hand, to trace the business execution and, on the other hand, to facilitate problem analysis and troubleshooting when the business execution fails. Therefore, abnormal logs are often more valuable or have a higher viewing frequency than normal execution logs in most scenarios. When archiving logs, abnormal logs and normal logs should not be treated equally. Although abnormal logs may also have the property of querying after archiving, for data query of the running table and the archived table, their complexities and efficiencies are different.

[0058] The delayed archiving in this embodiment means configuring the maximum retention days of abnormal log data, and the abnormal log data after exceeding this number of days will be stored in the archived table according to the above process.

[0059] In this way, the integrity of abnormal logs can be ensured within a certain period of time, which is convenient for viewing and querying, and also ensures the orderliness of data and the traceability of business execution.

[0060] In this embodiment, the log data is stored in different log data lists according to the pre-configured business division clusters and writing cycles. Different business division clusters are used to store log data of different business systems or business modules;

[0061] A number of tables are included in the same business division cluster, and each table is sequentially switched according to the configured writing cycle to store the corresponding log data.

[0062] This strategy can well balance the storage pressure and retrieval requirements of historical logs.

[0063] In some embodiments, the tables that need to store data can be divided into a group of log table clusters that can be written cyclically. The tables in the same cluster are used to store data of the same business module. Each table in each cluster can write data in sequence according to the cycle configuration (such as quarterly, monthly, weekly or daily).

[0064] For example, the tables of the same business module are written into the same log table cluster, and there are 12 tables in this log table cluster. If stored monthly, assuming the current writing is to table 12, then next month it will be written to table 1.

[0065] Before writing to Table 1, the data in Table 1 will be archived in another dimension, such as archiving it into a file and storing it on other media, and then clearing the data in Table 1. Since the data stored in the file no longer has the flexible query of a database table, it is generally considered that these data have been deleted from the log data list.

[0066] On the other hand, since the next cycle will overwrite the log data of the previous cycle, the business partition cluster, the number of circular log tables, the log writing period, etc. can be customized according to the business data volume and the maximum historical data required by the business.

[0067] In some embodiments, the log writing period can also be switched according to the log analysis requirements. For example, if the abnormal data during this period is greater than the set value and needs to be analyzed in detail, the original monthly writing can be changed to daily writing.

[0068] In some embodiments, the number of pre-configured tables and the writing period can also be used to dynamically calculate the table to be written. For example, if the writing period is in days and there are 10 tables in this cluster, then the log time can be obtained. The data on December 25, 2024 will be automatically stored in Table 5, the data on December 30, 2024 will be automatically stored in Table 10, and the data on December 31, 2024 will be automatically stored in Table 1.

[0069] Of course, the solutions in this embodiment are not exhaustive. Those skilled in the art can flexibly configure the business partition cluster, the number of circular log tables, the log writing period, etc. according to the configuration mechanism given in this embodiment. However, such changes / configurations are easily conceivable by those skilled in the art and should reasonably fall within the protection scope of the present invention.

[0070] Through this configuration mechanism, the pressure of log data storage is reduced to the greatest extent while ensuring the retrieval requirements of valid data, and it also helps to a certain extent in analyzing and comparing the log data of the same cycle. For example, whether there are differences between the data in January 2024 and January 2025, etc.

[0071] Considering that there may often be main logs and detailed logs in log records, the data volume of the main logs is generally within the acceptable range of the system, while the detailed logs may generate a large number of log entries due to characteristics such as business cycles. The data of these logs within the cycle are the same or similar, so it is not necessary to archive and retain all of them. In this embodiment, it is determined whether the log data is main log data or detailed log data. If it is detailed log data, it is determined whether the detailed log data covers a preset cycle. If it covers the preset cycle, only part of the log data is retained, and other log data is deleted.

[0072] This embodiment supports multiple log retention algorithms, which can be selected or configured according to requirements. For example, retain the first N log data, retain randomly selected M log data, where N and M are both positive integers. N and M can be set according to experience or requirements, or obtained by taking the modulus of the total number of entries using a random number. It is also possible to retain log data in a linear cycle (such as the 1st, 3rd, 5th, ……, 2n + 1st entries, where n is a positive integer), or retain log data within a set range or set period (such as the 1 - 5th, 12 - 19th…126 - 237th entries, or the log data for the 2nd - 4th days).

[0073] It is also possible to determine whether to delete or retain the corresponding part of the log data through keyword redundancy judgment. For example, redundant data can be removed according to the configured keywords.

[0074] Similarly, the above cases are not exhaustive, and those skilled in the art can configure flexibly according to the above rules.

[0075] In this embodiment, considering that the business may process file data in string form during execution, such data will eventually form a log data with an extremely large field when it enters the log. However, this extremely large file data is actually not readable in the log and has little value for business analysis or exception troubleshooting.

[0076] In addition, such extremely large fields often may be in a complex data structure. When processing such fields, it is necessary to consider retaining the data structures and business readability of other fields.

[0077] Therefore, this embodiment configures truncation for the data of extremely large fields, that is, configures that when the size of the field data exceeds a certain threshold, this field will be truncated or keyword - replaced. When judging the data size, if the size of this field exceeds the threshold, then its parent - node field must also exceed the threshold. Therefore, while configuring the truncation threshold of the data, the maximum search level can also be configured to achieve a balance between truncation efficiency and data validity.

[0078] In some embodiments, when there is an archive table or archive file for the log data, compression storage algorithms such as Snappy, PPM, LZMA, LZ4, DEFLATE, Zstd, etc. can be used for compression storage (not exhaustively listed here). Data serialization processing can also be performed.

[0079] In this way, not only can the required storage capacity be significantly reduced, the size of data packets in network transmission be reduced, thereby accelerating the transmission speed and reducing the bandwidth requirement, but also for large databases that need to be frequently read and written, small data volumes or compression can reduce the time of disk I / O operations, thereby improving the overall performance.

[0080] In this embodiment, the above log cleaning and archiving process is executed cyclically until the log data in the log data list is empty.

[0081] Embodiment Two

[0082] A log cleaning and archiving method includes the following steps:

[0083] Step 1: Read user configuration data userConfig;

[0084] Step 2: According to the configuration, clean the data table truncateTablexx within the non-retained period, and the actually cleaned table is stored in a compressed file;

[0085] Step 3: According to the configuration, determine whether the data table to be written currently is the main table MainTable or the detail table DetailTable. Generally, the data in the main table is less, and the number of data entries in the detail table is large and the space occupied by each entry is large;

[0086] Step 4: Query the total number needHandleCount of the main table data to be processed;

[0087] Step 5: Execute the following process cyclically:

[0088] (1) Obtain the main table log data list mainLogList(id, status) to be processed in batches;

[0089] (2) Process the data with abnormal status according to the retention policy;

[0090] (3) Process the detail table cyclically:

[0091] (3-1) Obtain the detail table log data list detailLogList(id, dataSize) to be processed in batches;

[0092] (3-2) Determine and filter redundant data according to the redundancy configuration policy and containers;

[0093] (3-3) Split the filtered detailLogList according to the total data size into a data set greaterThanThresholdNodes larger than the threshold and a data set lessThanThresholdNodes smaller than the threshold;

[0094] (3-4) Batch obtain the small threshold detail data set smallNodeLogs;

[0095] (3-4-1) Judge and truncate the large data fields of the small threshold detail data set smallNodeLog data according to the configuration;

[0096] (3 - 4 - 1) Save the data of the small threshold details dataset smallNodeLog to the archive table DetailTable;

[0097] (3 - 5) Batch delete the corresponding data of smallNodeLogs in the current table;

[0098] (3 - 6) Loop through the large threshold dataset greaterThanThresholdNodes;

[0099] (3 - 6 - 1) Retrieve the detail data bigNodeLog of each data node with a large threshold one by one;

[0100] (3 - 6 - 2) Judge and truncate the big data fields of the bigNodeLog data according to the configuration;

[0101] (3 - 6 - 3) Save the data of bigNodeLog to the archive table DetailTable;

[0102] (3 - 7) Batch delete the corresponding data of greaterThanThresholdNodes in the current table.

[0103] In some embodiments, the following steps are further included:

[0104] (3 - 8) Record the currently processed data into the redundant data determination container containers;

[0105] (3 - 9) Judge whether the records in containers reach the maximum number of retained records configured. If so, sort the dataset pageLogList whose data offset pageIndex is greater than or equal to the set value (in this embodiment, 100000 is used as an example) and the data volume / page size pageSize is greater than or equal to the set value (in this embodiment, 10 is used as an example) by id;

[0106] If the data volume / page size of the dataset is less than the set value pageLogList.size < 10, determine the data volume size. If the data volume is within the controllable range (in this embodiment, 0 - 100010 is used as an example, that is, within a certain range of the set value), the remaining data can be directly deleted in batches. If the data volume is greater than or equal to the controllable range, it needs to be judged in the next round. In this round, only the values with id less than pageLogList[size - 1].Id are deleted.

[0107] Repeat the above process until detailLogList is empty.

[0108] (3-10) Save the data of mainLog to the archive table MainTable;

[0109] (3-11) Clean up the data corresponding to mainLog in the current table.

[0110] Embodiment III

[0111] A log cleaning and archiving system, as Figure 2 shown, includes:

[0112] A trigger module, configured to start a log cleaning and archiving process in response to a trigger signal;

[0113] A redundancy processing module, configured to determine whether there is redundancy in the log data in the log data list, and if so, filter the redundant data;

[0114] A classification module, configured to divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one;

[0115] An archiving module, configured to determine whether the obtained log data belongs to a large data field exceeding a set value, and if so, truncate or replace keywords for the large data field, and store the processed log data in an archive table or an archive file;

[0116] A deletion module, configured to batch delete the corresponding data in the current log data list.

[0117] Embodiment IV

[0118] A computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the electronic device and, of course, the extended storage medium supported by the electronic device. The computer-readable storage medium provides a storage space, and this storage space stores the processing system of the electronic device.

[0119] And, in this storage space, there are also stored one or more instructions suitable for being loaded and executed by a processor, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer-readable storage medium located far from the aforementioned processor.

[0120] In one embodiment, one or more instructions are stored in the computer-readable storage medium; the one or more instructions stored in the computer-readable storage medium are loaded and executed by a processor to implement the following process:

[0121] Determine whether there is redundant log data in the log data list. If so, filter the redundant data;

[0122] Divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one;

[0123] Determine whether the obtained log data belongs to a large data field exceeding the set value. If so, truncate or replace keywords for the large data field, and store the processed log data in an archive table or an archive file;

[0124] Batch delete the corresponding data in the current log data list.

[0125] For the detailed process, see the introduction in Embodiment 1, which will not be elaborated here.

[0126] Embodiment 5

[0127] An electronic device, as Figure 3 shown, the electronic device includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. Among them, the processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 can be connected through a bus or other means.

[0128] Among them, the communication interface 1002 is used to receive and send data. The computer-readable storage medium 1003 can be stored in the memory of the electronic device. The computer-readable storage medium 1003 is used to store a computer program. The computer program includes program instructions. The processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0129] The processor 1001 (or CPU (Central Processing Unit, central processor)) is the computing core and control core of the electronic device, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0130] The processor 1001 is configured to execute the following process:

[0131] Determine whether there is redundant log data in the log data list. If so, filter the redundant data;

[0132] Partition the filtered log data according to the data size, batch retrieve the log data smaller than the threshold, and retrieve the log data greater than or equal to the threshold one by one;

[0133] Determine whether the retrieved log data belongs to a large data field exceeding the set value. If so, truncate or replace the keywords of the large data field, and store the processed log data in an archive table or an archive file;

[0134] Batch delete the corresponding data in the current log data list.

[0135] For the detailed process, see the introduction in Embodiment 1, which will not be elaborated here.

[0136] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0138] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps of the process Figure 1 in one process or a plurality of processes and / or blocks Figure 1 or steps for implementing the functions specified in one block or a plurality of blocks.

[0140] The foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modifications, equivalent replacements, improvements, etc. made by those skilled in the art without creative efforts within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A log cleaning and archiving method, characterized in that, In response to a trigger signal, start the log cleaning and archiving process, including the following steps: Determine whether there is redundant log data in the log data list. If so, filter the redundant data; Divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one; Determine whether the obtained log data belongs to a large data field exceeding the set value. If so, truncate or replace keywords for the large data field, and store the processed log data in the archive table or archive file; Batch delete the corresponding data in the current log data list.

2. The method for log cleaning and archiving according to claim 1, characterized in that, It also includes the following steps: Determine whether the log data in the log data list is abnormal log data. If so, delay the archiving of the abnormal log data according to the configured delay time length.

3. The log cleaning and archiving method according to claim 1, characterized in that, The trigger signal is a timed trigger signal, an event trigger signal, or a trigger request signal.

4. The log cleaning and archiving method according to claim 1, characterized in that The process of determining whether there is redundant log data in the log data list and filtering the redundant data if so includes: determining whether the log data is main log data or detailed log data. If it is detailed log data, determine whether the detailed log data covers a preset cycle. If it covers the preset cycle, only retain part of the log data and delete other log data.

5. A log cleaning and archiving method according to claim 4, characterized in that The specific process of only retaining part of the log data includes retaining the first set number of log data, retaining a randomly set number of log data, retaining log data in a linear cycle, retaining log data in a range cycle, or retaining / deleting log data based on configured keywords.

6. The log cleaning and archiving method according to claim 1, characterized in that, The log cleaning and archiving process is executed cyclically until the log data in the log data list is empty.

7. The log cleaning and archiving method according to claim 1, characterized in that, Different business division clusters are used to store log data of different business systems or business modules; Each business division cluster includes several tables, and each table is switched in sequence according to the configured writing cycle to store the corresponding log data.

8. A log cleaning and archiving system, characterized in that, It includes: A trigger module configured to start the log cleaning and archiving process in response to a trigger signal; A redundancy processing module configured to determine whether there is redundant log data in the log data list and filter the redundant data if so; A classification module configured to divide the filtered log data according to the data size, batch obtain the log data smaller than the threshold, and obtain the log data greater than or equal to the threshold one by one; An archiving module configured to determine whether the obtained log data belongs to a large data field exceeding the set value. If so, truncate or replace keywords for the large data field, and store the processed log data in the archive table or archive file; A deletion module configured to batch delete the corresponding data in the current log data list.

9. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions are executed by a processor, the steps in the method according to any one of claims 1-7 are completed.

10. An electronic device, characterized in that, It includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps in the method according to any one of claims 1-7 are completed.

Images