A log compression method, device, apparatus and medium

Abstract

The application discloses a log compression method and device, equipment and medium, and relates to the technical field of log management. The scheme realizes positioning of repeated single logs by extracting target repeated single logs appearing multiple times in a target log. Based on the position of the target repeated single log in the target log, when it is confirmed that there are multiple same single logs in each single log after the target repeated single log, it is confirmed that the target repeated single log and the multiple same single logs constitute a repeated log paragraph, and positioning of the repeated log paragraph is realized. On this basis, by obtaining the repeated row number of the first log of the repeated log paragraph appearing in the target log, the repeated log paragraph in the target log is marked and replaced based on the repeated row number, the compression of the target log is realized, the size of the target log file is greatly reduced, the analysis efficiency of the user on the log content is improved, and the user can more conveniently find the key information in the log.

Description

Technical Field

[0001] This invention relates to the field of log management technology, and in particular to a log compression method, apparatus, device, and medium. Background Technology

[0002] Linux systems offer powerful and flexible logging capabilities, recording error messages, warnings, and other alerts generated by the system kernel and critical programs in log files. These logs are crucial for system auditing, monitoring, tracing, and statistical analysis, helping administrators understand the system's operational status.

[0003] However, processing Linux logs also presents some challenges. First, log files are typically stored in text format, and they grow increasingly large over time, making it difficult to find crucial information. Second, due to the function call relationships within the system and applications, a large amount of duplicate log information may be generated when a low-level problem occurs, further increasing the complexity of analysis. For users, carefully reading and analyzing log files is crucial for ensuring normal system operation and timely problem resolution; however, the large amount of duplicate log entries in Linux logs increases the difficulty for users to find key information within them.

[0004] Given the above problems, how to solve the problem of the large number of duplicate logs in the current Linux logs, which makes it difficult for users to find key information in the logs, is an urgent problem for technicians in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a log compression method, apparatus, device, and medium to solve the problem that current Linux logs contain a large number of duplicate logs, which makes it difficult for users to find key information in the logs.

[0006] To address the aforementioned technical problems, this invention provides a log compression method, comprising:

[0007] Obtain the target log and extract the single log entry that appears multiple times in the target log;

[0008] Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs in each single log following the target duplicate single log;

[0009] If multiple identical single log entries exist, then the target duplicate single log entry and the multiple identical single log entries constitute a duplicate log segment.

[0010] Obtain the number of duplicate lines in the target log where the first line of the duplicate log segment appears;

[0011] The duplicate log segments in the target log are marked and replaced according to the number of duplicate lines to obtain the compressed target log.

[0012] On the one hand, before extracting the target duplicate single log that appears multiple times in the target log, the process also includes:

[0013] Create a list of duplicate single log entries;

[0014] Based on the target log, a preset number of duplicate single log entries are obtained; wherein each of the duplicate single log entries is different, and each of the duplicate single log entries appears multiple times in the target log;

[0015] Based on each of the aforementioned duplicate single log entries, determine the duplicate single log entry that appears most frequently in the target log, and determine the occurrence count and position of the duplicate single log entry that appears most frequently in the target log;

[0016] The information, frequency, and location of the most frequently occurring duplicate single log entry in the target log are stored in the duplicate single log list;

[0017] Remove the duplicate single logs that have been stored in the duplicate single log list from each of the duplicate single logs, and return to the step of determining the duplicate single log that appears most frequently in the target log based on each of the duplicate single logs, until the duplicate single log list contains a preset number of duplicate single logs;

[0018] The duplicate single log entries in the duplicate single log list are sorted from highest to lowest frequency of occurrence.

[0019] Correspondingly, extracting the target duplicate single log entry that appears multiple times in the target log includes:

[0020] When the target duplicate single log is extracted for the first time, the first duplicate single log in the list of duplicate single logs is extracted.

[0021] On the other hand, determining whether there are multiple identical single log entries among the subsequent single log entries based on the position of the target duplicate single log entry in the target log includes:

[0022] Obtain the maximum number of lines in a duplicate log segment; wherein, the maximum number of lines in a duplicate log segment represents the maximum number of lines in a single log entry within the duplicate log segment;

[0023] Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs among the single logs in each of the maximum number of lines in the duplicate log segment following the target duplicate single log.

[0024] On the other hand, confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment includes:

[0025] Obtain the minimum number of lines threshold for a duplicate log segment; wherein, the minimum number of lines threshold for a duplicate log segment represents the minimum number of lines in a single log entry within the duplicate log segment; the minimum number of lines threshold for a duplicate log segment is less than the maximum number of lines threshold for a duplicate log segment;

[0026] Determine the number of identical single log entries;

[0027] Determine whether the number of identical single log entries is not less than the minimum line count threshold for the duplicate log segment;

[0028] If it is confirmed that the number of identical single log entries is less than the minimum number of lines in the duplicate log segment, then the next duplicate single log entry in the duplicate single log list is extracted as the new target duplicate single log entry, and the process returns to the step of determining whether there are multiple identical single log entries in each log entry after the target duplicate single log entry based on the position of the target duplicate single log entry in the target log.

[0029] If it is confirmed that the number of identical single log entries is not less than the minimum number of rows in the duplicate log segment, then the target duplicate single log entry and the identical single log entries constitute a duplicate log segment.

[0030] On the other hand, before obtaining the number of duplicate lines appearing in the target log of the first line of the duplicate log segment, and after confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment, the method further includes:

[0031] Obtain the minimum threshold for the number of duplicate log segments; wherein, the minimum threshold for the number of duplicate log segments represents the minimum number of times the duplicate log segment appears in the target log;

[0032] Determine the number of times the duplicate log segment is repeated in the target log;

[0033] Determine whether the number of repetitions is not less than the minimum threshold for the number of repetitive log segments;

[0034] If so, proceed to the step of obtaining the number of duplicate lines in the target log where the first line of the duplicate log segment appears;

[0035] If not, then extract the next duplicate single log in the duplicate single log list as the new target duplicate single log, and return to the step of determining whether there are multiple identical single logs in each single log after the target duplicate single log based on the position of the target duplicate single log in the target log.

[0036] On the other hand, after confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment, the method further includes:

[0037] Determine whether the single log entry exists in the list of duplicate single log entries among the identical single log entries;

[0038] If it is confirmed that the same log entry exists in the duplicate log entry list, then the log entry in the duplicate log entry list will be deleted.

[0039] On the other hand, after obtaining a preset number of duplicate single log entries based on the target log, the method further includes:

[0040] Determine whether the number of occurrences of the duplicate single log entry is less than the minimum threshold for the number of duplicate log segments;

[0041] If such duplicate log entries exist, those that appear less than the minimum threshold for the number of duplicate log segments will be removed.

[0042] To address the aforementioned technical problems, the present invention also provides a log compression device, comprising:

[0043] The first acquisition module is used to acquire the target log and extract the target duplicate single log that appears multiple times in the target log;

[0044] The judgment module is used to determine whether there are multiple identical single logs in each subsequent single log based on the position of the target duplicate single log in the target log; if multiple identical single logs exist, the confirmation module is triggered.

[0045] The confirmation module is used to confirm that the target duplicate single log and multiple identical single logs constitute a duplicate log segment;

[0046] The second acquisition module is used to acquire the number of duplicate lines in the target log where the first line of the duplicate log segment appears.

[0047] A marking module is used to mark and replace the duplicate log segments in the target log according to the number of duplicate lines, so as to obtain the compressed target log.

[0048] To address the aforementioned technical problems, the present invention also provides a log compression device, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor is used to implement the steps of the log compression method described above when executing the computer program.

[0051] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the log compression method described above.

[0052] The log compression method provided by this invention involves: acquiring a target log and extracting a single target log entry that appears multiple times; determining, based on the position of the single target log entry in the target log, whether there are multiple identical single log entries in the subsequent log entries; if multiple identical single log entries exist, confirming that the single target log entry and the multiple identical single log entries constitute a duplicate log segment; obtaining the number of repeated lines of the first line of the duplicate log segment in the target log; and marking and replacing the duplicate log segment in the target log according to the number of repeated lines to obtain the compressed target log.

[0053] The beneficial effects of this invention are as follows: By extracting multiple duplicate single log entries from the target log, the location of duplicate single log entries in the target log is achieved; furthermore, based on the position of the duplicate single log entry in the target log, when it is confirmed that multiple identical single log entries exist in the subsequent single log entries, it is confirmed that the duplicate single log entry and the multiple identical single log entries constitute a duplicate log segment, thus achieving the location of the duplicate log segment in the target log. On this basis, by obtaining the number of duplicate lines of the first line of the duplicate log segment in the target log, and marking and replacing the duplicate log segment in the target log based on this number of duplicate lines, compression of the target log is achieved, greatly reducing the file size of the target log, while improving the efficiency of user analysis of log content, making it easier for users to find key information in the log.

[0054] On the other hand, this invention specifically creates a list of duplicate single log entries and stores each duplicate single log entry in this list, thus enabling better preservation of duplicate single log entries. By obtaining the maximum line count threshold for duplicate log segments, and based on the position of the target duplicate single log entry in the target log, it determines whether multiple identical single log entries exist in the subsequent single log entries. By comparing the minimum line count threshold for duplicate log segments with the number of single log entries with identical content after the target duplicate single log entry, duplicate log segments are identified, facilitating subsequent compression of duplicate log segments. By comparing the number of times a duplicate log segment is repeated in the target log with the minimum number threshold for duplicate log segments, duplicate log segments that do not meet the minimum number threshold can be discarded, improving the analysis and compression efficiency of the target log. By determining whether any identical single log entries exist in the duplicate single log list, if it is confirmed that an identical single log entry exists in the duplicate single log list, the single log entry in the duplicate single log list is deleted, avoiding redundant analysis. It determines whether there are duplicate log entries whose occurrence count is less than the minimum threshold for duplicate log segments; if so, it removes the duplicate log entries whose occurrence count is less than the minimum threshold for duplicate log segments, further improving the efficiency of analysis and compression of target logs.

[0055] In addition, the present invention also provides a log compression device, equipment and medium, with the same effect as above. Attached Figure Description

[0056] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart of a log compression method provided in an embodiment of the present invention;

[0058] Figure 2 A flowchart of a log compression device provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of a log compression device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0061] The core of this invention is to provide a log compression method, apparatus, device, and medium to solve the problem that current Linux logs contain a large number of duplicate logs, which makes it difficult for users to find key information in the logs.

[0062] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Linux systems log data in text format, which consumes relatively little disk space, so log files are typically accumulated. However, as the system runs longer, the log files grow larger, making it challenging to extract key information.

[0064] The complexity of logs stems from the multi-layered function call mechanisms within the system and applications. When a specific situation occurs in a lower-level function, it triggers a series of upward function calls, each generating corresponding log information. Therefore, from the perspective of system logs, when a certain underlying problem or situation occurs, the logs will contain a large amount of continuous log content related to that problem. Furthermore, some problems may recur even with unchanged hardware configurations. This causes the system kernel and applications to frequently generate the same log content, further increasing the length of log files. For log analysis, accurately locating key information from these lengthy log files becomes more difficult. Therefore, to avoid the significant impact of a large amount of duplicate logs in Linux logs on user log analysis, this invention provides a log compression method.

[0065] Figure 1 A flowchart illustrating a log compression method provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0066] S10: Obtain the target log and extract the single log entry that appears multiple times in the target log.

[0067] It is understandable that a Linux log, as a whole, contains multiple individual log entries; and these individual log entries together constitute log segments; the repeated log segments in the target log are actually composed of multiple repeated individual log entries. Therefore, in this invention, to compress the repeated log segments in the target log, it is first necessary to identify the target repeated individual log entries that appear multiple times in the target log. Specifically, the target repeated individual log entries that appear multiple times in the target log are extracted.

[0068] It should be noted that the target log generally contains multiple repeated single log entries, and the target repeated single log entry is actually one of multiple repeated single log entries. This embodiment does not limit the method for extracting the target repeated single log entry; it depends on the specific implementation.

[0069] S11: Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs in the subsequent single logs. If multiple identical single logs exist, proceed to step S12; if no multiple identical single logs exist, return to step S10.

[0070] S12: Confirm that the target duplicate single log and multiple identical single logs constitute a duplicate log segment.

[0071] Furthermore, based on the position of the target duplicate single log in the target log, it is determined whether there are multiple identical single logs in each single log following the target duplicate single log, that is, whether the content of multiple single logs following the target duplicate single log is the same.

[0072] If there are no multiple identical log entries after the target duplicate log entry, then the current target duplicate log entry is considered not to form a duplicate log segment with other log entries, and we need to return to step S10 to extract the target duplicate log entry again. If there are multiple identical log entries, then it is confirmed that the target duplicate log entry and the multiple identical log entries form a duplicate log segment.

[0073] It should be noted that this embodiment does not limit the specific process of determining whether there are multiple identical log entries in each log entry after the target duplicate log entry; it depends on the specific implementation.

[0074] S13: Get the number of duplicate lines in the target log where the first line of the duplicate log segment appears.

[0075] S14: Mark and replace duplicate log segments in the target log according to the number of duplicate lines to obtain the compressed target log.

[0076] After identifying duplicate log segments, compression is achieved by obtaining the number of times the first line of each segment appears in the target log, thus determining the segment's position within the target log. Finally, the duplicate segments in the target log are marked and replaced based on their number of duplicate lines. For example, markers like P1 and P2 are used to replace the duplicate segments, significantly reducing the target log's size and achieving compression.

[0077] In this embodiment, by extracting multiple duplicate single log entries from the target log, the location of duplicate single log entries in the target log is achieved. Furthermore, based on the position of the duplicate single log entry in the target log, when it is confirmed that multiple identical single log entries exist in subsequent single log entries, the duplicate single log entry and the multiple identical single log entries constitute a duplicate log segment, thus locating the duplicate log segment in the target log. Based on this, by obtaining the number of duplicate lines of the first line of the duplicate log segment in the target log, and marking and replacing the duplicate log segment in the target log based on this number of duplicate lines, compression of the target log is achieved, greatly reducing the file size of the target log, while improving the efficiency of user analysis of log content, making it easier for users to find key information in the log.

[0078] To better extract target duplicate single log entries, based on the above embodiments, in some embodiments, before extracting target duplicate single log entries that appear multiple times in the target log, the following steps are also included:

[0079] S15: Create a list of duplicate single log entries;

[0080] S16: Based on the target log, obtain a preset number of duplicate single log entries;

[0081] Among them, each duplicate log entry is different, and each duplicate log entry appears multiple times in the target log;

[0082] S17: Based on each duplicate single log, determine the duplicate single log that appears most frequently in the target log, and determine the occurrence count and position of the duplicate single log that appears most frequently in the target log;

[0083] S18: Store the information, frequency, and location of the most frequently occurring duplicate log entry in the target log into the duplicate log entry list;

[0084] S19: Remove duplicate single logs that have been stored in the duplicate single log list from each duplicate single log, and return to step S17 until the duplicate single log list contains a preset number of duplicate single logs.

[0085] Among them, the duplicate single log entries in the duplicate single log list are sorted from highest to lowest frequency of occurrence.

[0086] In practice, the first step is to create a list of duplicate single log entries. It should be noted that the list of duplicate single log entries consists of three columns: the first column is a character type and stores the information of the duplicate single log entry; the second column is an integer type and stores the number of times the duplicate single log entry appears; the third column is a character type and stores the position of the duplicate single log entry in the target log.

[0087] Further, based on the target log, a preset number of duplicate single log entries are obtained. In this embodiment, there is no limit to the preset number. For example, a minimum threshold p for the number of duplicate log segments (representing the minimum number of times a duplicate log segment appears in the target log) and a minimum threshold n for the number of lines in a single log entry within a duplicate log segment can be preset, with the preset number set to p*n. It is worth noting that each obtained duplicate single log entry is different, and each duplicate single log entry appears multiple times in the target log.

[0088] Then, based on each duplicate log entry, the most frequent duplicate log entry in the target log is identified, along with its occurrence count and position. This information, occurrence count, and position are stored in a duplicate log entry list. Further, duplicate log entries already stored in the duplicate log entry list are removed from each existing duplicate log entry list, returning to the step of identifying the most frequent duplicate log entry in the target log based on each duplicate log entry. This process continues until the duplicate log entry list contains a predetermined number of duplicate log entries. It should be noted that the duplicate log entries in the duplicate log entry list are sorted from highest to lowest frequency; that is, the first row of the duplicate log entry list has the highest frequency, and the frequency of each duplicate log entry gradually decreases as the number of rows increases. This achieves better preservation of each duplicate log entry.

[0089] Correspondingly, extract the single log entry that appears multiple times in the target log, including:

[0090] S101: When extracting the target duplicate single log for the first time, extract the first duplicate single log in the duplicate single log list.

[0091] In specific implementation, based on the list of duplicate single log entries created in the above embodiments, in order to extract the target duplicate single log entry, when extracting the target duplicate single log entry for the first time, the relevant information of the first duplicate single log entry in the list of duplicate single log entries is extracted as the target duplicate single log entry.

[0092] Based on the above embodiments, in some embodiments, determining whether there are multiple identical single log entries among the subsequent single log entries according to the position of the target duplicate single log entry in the target log includes:

[0093] S111: Get the maximum number of lines in a duplicate log segment threshold;

[0094] Among them, the maximum number of lines in a duplicate log segment represents the maximum number of lines in a single log entry within a duplicate log segment;

[0095] S112: Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs in each single log of the maximum number of lines in the duplicate log segment following the target duplicate single log.

[0096] In practice, the first step is to obtain the pre-set maximum number of lines threshold m for a duplicate log segment. It should be noted that the maximum number of lines threshold m for a duplicate log segment represents the maximum number of lines in a single log entry within a duplicate log segment; that is, the number of lines in a single log entry within a duplicate log segment will not exceed m.

[0097] Further, based on the position of the target duplicate log entry within the target log, it is determined whether there are multiple identical log entries among the log entries following the target duplicate log entry, within the maximum number of lines in the duplicate log segment. In other words, it is determined whether the content of log entries 1 to m following the target duplicate log entry is identical. This achieves the determination of whether there are multiple identical log entries among the log entries following the target duplicate log entry.

[0098] Based on the above embodiments, in some embodiments, it is confirmed that a target duplicate single log entry and multiple identical single log entries constitute a duplicate log segment, including:

[0099] S121: Obtain the minimum number of lines required for duplicate log segments;

[0100] Among them, the minimum number of lines in a duplicate log segment represents the minimum number of lines in a single log entry within a duplicate log segment; the minimum number of lines in a duplicate log segment is less than the maximum number of lines in a duplicate log segment.

[0101] S122: Determine the number of identical single log entries;

[0102] S123: Determine whether the number of identical single log entries is not less than the minimum number of lines in the duplicate log segment threshold; if not, proceed to step S124; if yes, proceed to step S125.

[0103] S124: Extract the next duplicate single log from the duplicate single log list as the new target duplicate single log, and return to step S11;

[0104] S125: Confirm that the target duplicate single log and the same single logs constitute a duplicate log segment.

[0105] In practice, the first step is to obtain the minimum line count threshold n for a duplicate log segment. It's important to note that the minimum line count threshold n represents the minimum number of lines in a single log entry within a duplicate log segment; that is, the number of lines in a single log entry within a duplicate log segment will not be less than n. It's easy to understand that the minimum line count threshold n for a duplicate log segment is less than the maximum line count threshold m for a duplicate log segment.

[0106] Further determine the number 'a' of identical single log entries, which is to determine the number 'a' of single log entries with identical content among the 1 to m single log entries following the target duplicate single log entry. Determine whether the number 'a' of identical single log entries is not less than the minimum threshold 'n' for duplicate log segments.

[0107] If the number 'a' of identical log entries is less than the minimum line count threshold 'n' for a duplicate log segment, then the content of the duplicate log segment to which the current target duplicate log entry belongs is considered too small, making replacement and compression of it meaningless, and no processing is required for the duplicate log segment. Further steps can be taken: extract the next duplicate log entry from the duplicate log entry list as the new target duplicate log entry, and return to the step of determining whether multiple identical log entries exist in subsequent log entries based on the target duplicate log entry's position in the target log, thus proceeding to the next round of judgment. If the number 'a' of identical log entries is not less than the minimum line count threshold 'n' for a duplicate log segment, then the target duplicate log entry and the identical log entries constitute a duplicate log segment, which needs to be replaced and compressed.

[0108] In this embodiment, the duplicate log segment is determined by comparing the minimum number of lines in the duplicate log segment with the number of logs with the same content after the target duplicate log segment, so as to compress the duplicate log segment in the future.

[0109] To improve the efficiency of target log analysis and compression, based on the above embodiments, in some embodiments, before obtaining the number of duplicate lines of the first line of the duplicate log segment in the target log, and after confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment, the method further includes:

[0110] S20: Obtain the minimum threshold for the number of duplicate log segments;

[0111] Among them, the minimum number of duplicate log segments threshold represents the minimum number of times a duplicate log segment appears in the target log;

[0112] S21: Determine the number of times a duplicate log segment is repeated in the target log;

[0113] S22: Determine whether the number of repetitions is not less than the minimum threshold for the number of repetitive log segments; if yes, proceed to step S13; if no, proceed to step S23.

[0114] S23: Extract the next duplicate single log from the duplicate single log list as the new target duplicate single log, and return to step S11.

[0115] Specifically, the minimum number of duplicate log segments, p, is obtained. It should be noted that the minimum number of duplicate log segments, p, represents the minimum number of times a duplicate log segment appears in the target log. After obtaining the duplicate log segments, the number of times each segment appears in the target log is determined, and it is judged whether the number of appearances is not less than the minimum number of duplicate log segments, p.

[0116] If the number of repetitions is not less than the minimum threshold p for duplicate log segments, then the current duplicate log segment is considered to meet the condition and has value for analysis and compression. The process then proceeds to the step of obtaining the number of duplicate lines of the first line of the duplicate log segment in the target log. If the number of repetitions is less than the minimum threshold p, then the current duplicate log segment is considered to not meet the condition, has too few repetitions, and has no value for analysis and compression. Therefore, no further processing is performed on the current duplicate log segment. Instead, the next duplicate single log in the duplicate single log list is extracted as the new target duplicate single log. The process then returns to the step of determining whether there are multiple identical single logs in the subsequent single logs based on the position of the target duplicate single log in the target log.

[0117] In this embodiment, by comparing the number of times a duplicate log segment is repeated in the target log with the minimum number threshold for duplicate log segments, duplicate log segments that do not meet the minimum number threshold for duplicate log segments can be discarded, thereby improving the analysis efficiency and compression efficiency of the target log.

[0118] To avoid redundant analysis of duplicate single log entries, based on the above embodiments, in some embodiments, after confirming that the target duplicate single log entry and multiple identical single log entries constitute a duplicate log segment, the following steps are also included:

[0119] S24: Determine whether there is a single log in the duplicate single log list among the identical single logs; if it is confirmed that there is a single log in the duplicate single log list among the identical single logs, proceed to step S25;

[0120] S25: Delete the single log from the list of duplicate single log entries.

[0121] In practice, after identifying duplicate log segments, it is determined whether any of the identical individual log entries are also in the duplicate individual log list. Specifically, it checks whether any of the 1 to m individual log entries following the target duplicate individual log entry are also in the duplicate individual log list. If it is confirmed that no individual log entry (a log entries in total) is in the duplicate individual log list, the current determination process ends. If it is confirmed that an individual log entry exists in the duplicate individual log list, that individual log entry (the individual log entry in the duplicate individual log list out of a log entries) is deleted from the duplicate individual log list, thus preventing further checks on these identical individual log entries (a log entries in total) and avoiding redundant analysis.

[0122] To further improve the analysis and compression efficiency of the target log, based on the above embodiments, in some embodiments, after obtaining a preset number of duplicate single log entries based on the target log, the method further includes:

[0123] S26: Determine if there is a duplicate single log entry whose occurrence count is less than the minimum threshold for duplicate log segments; if so, proceed to step S27.

[0124] S27: Remove duplicate log entries whose occurrence count is less than the minimum threshold for duplicate log segments.

[0125] When populating the list of duplicate single log entries, after obtaining a preset number of duplicate single log entries based on the target log, it can be further determined whether there are any duplicate single log entries whose occurrence count is less than the minimum threshold p for duplicate log segments. If not, the process ends. If there are, it is assumed that the occurrence count of this duplicate single log entry is less than the minimum threshold p for duplicate log segments, and the occurrence count of the duplicate log segment to which this duplicate single log entry belongs must also be less than the minimum threshold p for duplicate log segments. Therefore, this duplicate log segment does not need to be tracked because its occurrence count is too low. Duplicate single log entries with occurrence counts less than the minimum threshold p for duplicate log segments are then removed, thereby further improving the analysis and compression efficiency of the target log.

[0126] In addition, after compressing the target logs, the following methods can be used to analyze the key information: First, use some built-in Linux commands to help users analyze log files, such as grep, awk, and sed; these commands can be used to search for specific keywords, extract specific fields, or format log content. For example, the grep command can quickly find log entries containing specific error codes. Furthermore, besides built-in commands, there are many specialized log analysis tools available, such as journalctl (for analyzing systemd logs), logwatch (for generating log summary reports), and the ELK Stack (a combination of Elasticsearch, Logstash, and Kibana for log searching, analysis, and visualization). These tools typically offer richer functionality, such as log aggregation, real-time monitoring, and advanced queries. Meanwhile, for system issues requiring immediate response, users can use the tail command to monitor the latest contents of log files in real time. By combining this with filtering commands like grep, users can focus only on log entries relevant to specific issues. Furthermore, for large systems or distributed environments, aggregating logs into a central location for management can improve analysis efficiency. This can be achieved by configuring a log forwarding service (such as rsyslog or fluentd) to collect logs from various nodes onto a unified log server. Finally, for scenarios requiring a more intuitive display of log data, users can use log visualization tools such as Kibana or Grafana to present log data in the form of charts, dashboards, etc., for better understanding and analysis. Through these methods, users can choose appropriate tools and technologies based on their actual needs to efficiently analyze Linux logs, thereby quickly locating problems, optimizing system performance, or meeting compliance requirements.

[0127] In the above embodiments, the log compression method has been described in detail. The present invention also provides embodiments of the log compression device.

[0128] Figure 2 A flowchart of a log compression device provided in an embodiment of the present invention. Figure 2 As shown, the device includes:

[0129] The first acquisition module 10 is used to acquire the target log and extract the target duplicate single log that appears multiple times in the target log.

[0130] The judgment module 11 is used to determine whether there are multiple identical single logs in each single log after the target duplicate single log based on the position of the target duplicate single log in the target log; if there are multiple identical single logs, the confirmation module 12 is triggered.

[0131] The confirmation module 12 is used to confirm that a target duplicate single log and multiple identical single logs constitute a duplicate log segment.

[0132] The second acquisition module 13 is used to obtain the number of duplicate lines in the target log where the first line of the duplicate log segment appears.

[0133] The marking module 14 is used to mark and replace duplicate log segments in the target log according to the number of duplicate lines, so as to obtain the compressed target log.

[0134] In some embodiments, it also includes:

[0135] The list creation submodule is used to create lists of repeating single log entries;

[0136] The first acquisition submodule is used to acquire a preset number of duplicate single log entries based on the target log; wherein each duplicate single log entry is different and each duplicate single log entry appears multiple times in the target log.

[0137] The first determination submodule is used to determine the most frequently occurring single log in the target log based on each duplicate single log, and to determine the occurrence count and position of the most frequently occurring single log in the target log;

[0138] The save submodule is used to store the information, frequency, and location of the most frequently occurring duplicate log entry in the target log into the duplicate log entry list;

[0139] The removal submodule is used to remove duplicate single logs that have been stored in the duplicate single log list from each duplicate single log, triggering the first determination submodule, and ending when the duplicate single log list contains a preset number of duplicate single logs;

[0140] Among them, the duplicate single log entries in the duplicate single log list are sorted from highest to lowest frequency of occurrence;

[0141] Correspondingly, the first acquisition module 10 includes:

[0142] The first extraction submodule is used to extract the first duplicate single log from the duplicate single log list when the target duplicate single log is extracted for the first time.

[0143] In some embodiments, the determining module 11 includes:

[0144] The second acquisition submodule is used to acquire the maximum number of lines threshold for duplicate log segments; wherein, the maximum number of lines threshold for duplicate log segments represents the maximum number of lines in a single log entry within a duplicate log segment;

[0145] The first judgment submodule is used to determine whether there are multiple identical log entries among the log entries in the maximum number of lines of the duplicate log segment following the target duplicate log entry, based on the position of the target duplicate log entry in the target log.

[0146] In some embodiments, the confirmation module 12 includes:

[0147] The third acquisition submodule is used to obtain the minimum number of lines threshold for duplicate log segments; wherein, the minimum number of lines threshold for duplicate log segments represents the minimum number of lines in a single log entry in a duplicate log segment; the minimum number of lines threshold for duplicate log segments is less than the maximum number of lines threshold for duplicate log segments;

[0148] The second determination submodule is used to determine the number of identical single log entries;

[0149] The second judgment submodule is used to determine whether the number of identical single log entries is not less than the minimum number of lines in a duplicate log segment threshold; if it is confirmed that the number of identical single log entries is less than the minimum number of lines in a duplicate log segment threshold, the second extraction submodule is triggered; if it is confirmed that the number of identical single log entries is not less than the minimum number of lines in a duplicate log segment threshold, the first confirmation submodule is triggered.

[0150] The second extraction submodule is used to extract the next duplicate single log from the duplicate single log list as the new target duplicate single log, triggering the judgment module 11;

[0151] The first confirmation submodule is used to confirm that the target duplicate single log and the same single logs constitute a duplicate log segment.

[0152] In some embodiments, it also includes:

[0153] The fourth submodule is used to obtain the minimum threshold for the number of duplicate log segments; where the minimum threshold for the number of duplicate log segments represents the minimum number of times a duplicate log segment appears in the target log.

[0154] The third determination submodule is used to determine the number of times a duplicate log segment is repeated in the target log.

[0155] The third judgment submodule is used to determine whether the number of repetitions is not less than the minimum threshold for the number of repetitive log segments; if yes, the second acquisition module 13 is triggered; if no, the third extraction submodule is triggered.

[0156] The third extraction submodule is used to extract the next duplicate single log from the duplicate single log list as the new target duplicate single log, triggering the judgment module 11.

[0157] In some embodiments, it also includes:

[0158] The fourth judgment submodule is used to determine whether there is a single log in the duplicate single log list among the identical single log entries; if it is confirmed that there is a single log in the duplicate single log list among the identical single log entries, the deletion submodule is triggered;

[0159] The delete submodule is used to delete a single log entry from the list of duplicate single log entries.

[0160] In some embodiments, it also includes:

[0161] The fifth judgment submodule is used to determine whether there is a duplicate single log entry whose occurrence count is less than the minimum threshold for duplicate log segments; if so, the removal submodule is triggered.

[0162] The removal submodule is used to remove duplicate single log entries whose occurrence count is less than the minimum threshold for duplicate log segments.

[0163] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0164] In addition, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described log compression method.

[0165] Figure 3 This is a schematic diagram of a log compression device provided in an embodiment of the present invention. Figure 3 As shown, the log compression device includes:

[0166] Memory 20 is used to store computer programs;

[0167] The processor 21 is used to implement the steps of the log compression method mentioned in the above embodiments when executing a computer program.

[0168] The log compression device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0169] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0170] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the log compression method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the log compression method.

[0171] In some embodiments, the log compression device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0172] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the log compression device and may include more or fewer components than illustrated.

[0173] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0174] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The log compression method, apparatus, device, and medium provided by the present invention have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0176] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A log compression method, characterized in that, include: Obtain the target log and extract the single log entry that appears multiple times in the target log; Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs in each single log following the target duplicate single log; If multiple identical single log entries exist, then the target duplicate single log entry and the multiple identical single log entries constitute a duplicate log segment. Obtain the number of duplicate lines in the target log where the first line of the duplicate log segment appears; The duplicate log segments in the target log are marked and replaced according to the number of duplicate lines to obtain the compressed target log; The step of determining whether there are multiple identical single log entries among subsequent single log entries based on the position of the target duplicate single log entry in the target log includes: Obtain the maximum number of lines in a duplicate log segment; wherein, the maximum number of lines in a duplicate log segment represents the maximum number of lines in a single log entry within the duplicate log segment; Based on the position of the target duplicate single log in the target log, determine whether there are multiple identical single logs in each of the single logs of the maximum number of lines in the duplicate log segment after the target duplicate single log; The confirmation that the target duplicate single log and multiple identical single logs constitute a duplicate log segment includes: Obtain the minimum number of lines threshold for a duplicate log segment; wherein, the minimum number of lines threshold for a duplicate log segment represents the minimum number of lines in a single log entry within the duplicate log segment; the minimum number of lines threshold for a duplicate log segment is less than the maximum number of lines threshold for a duplicate log segment; Determine the number of identical single log entries; Determine whether the number of identical single log entries is not less than the minimum line count threshold for the duplicate log segment; If it is confirmed that the number of identical single log entries is less than the minimum number of lines in the duplicate log segment, then the next duplicate single log entry in the duplicate single log list is extracted as the new target duplicate single log entry, and the process returns to the step of determining whether there are multiple identical single log entries in each log entry after the target duplicate single log entry based on the position of the target duplicate single log entry in the target log. If it is confirmed that the number of identical single log entries is not less than the minimum number of rows in the duplicate log segment, then the target duplicate single log entry and the identical single log entries constitute a duplicate log segment.

2. The log compression method according to claim 1, characterized in that, Before extracting the target duplicate single log entry that appears multiple times in the target log, the method further includes: Create a list of duplicate single log entries; Based on the target log, a preset number of duplicate single log entries are obtained; wherein each of the duplicate single log entries is different, and each of the duplicate single log entries appears multiple times in the target log; Based on each of the aforementioned duplicate single log entries, determine the duplicate single log entry that appears most frequently in the target log, and determine the occurrence count and position of the duplicate single log entry that appears most frequently in the target log; The information, frequency, and location of the most frequently occurring duplicate single log entry in the target log are stored in the duplicate single log list; Remove the duplicate single logs that have been stored in the duplicate single log list from each of the duplicate single logs, and return to the step of determining the duplicate single log that appears most frequently in the target log based on each of the duplicate single logs, until the duplicate single log list contains a preset number of duplicate single logs; The duplicate single log entries in the duplicate single log list are sorted from highest to lowest frequency of occurrence. Correspondingly, extracting the target duplicate single log entry that appears multiple times in the target log includes: When the target duplicate single log is extracted for the first time, the first duplicate single log in the list of duplicate single logs is extracted.

3. The log compression method according to claim 2, characterized in that, Before obtaining the number of duplicate lines appearing in the target log of the first line of the duplicate log segment, and after confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment, the method further includes: Obtain the minimum threshold for the number of duplicate log segments; wherein, the minimum threshold for the number of duplicate log segments represents the minimum number of times the duplicate log segment appears in the target log; Determine the number of times the duplicate log segment is repeated in the target log; Determine whether the number of repetitions is not less than the minimum threshold for the number of repetitive log segments; If so, proceed to the step of obtaining the number of duplicate lines in the target log where the first line of the duplicate log segment appears; If not, then extract the next duplicate single log in the duplicate single log list as the new target duplicate single log, and return to the step of determining whether there are multiple identical single logs in each single log after the target duplicate single log based on the position of the target duplicate single log in the target log.

4. The log compression method according to claim 1 or 3, characterized in that, After confirming that the target duplicate single log and multiple identical single logs constitute a duplicate log segment, the method further includes: Determine whether the single log entry exists in the list of duplicate single log entries among the identical single log entries; If it is confirmed that the same log entry exists in the duplicate log entry list, then the log entry in the duplicate log entry list will be deleted.

5. The log compression method according to claim 3, characterized in that, After obtaining a preset number of duplicate single log entries based on the target log, the method further includes: Determine whether the number of occurrences of the duplicate single log entry is less than the minimum threshold for the number of duplicate log segments; If such duplicate log entries exist, those that appear less than the minimum threshold for the number of duplicate log segments will be removed.

6. A log compression device, characterized in that, include: The first acquisition module is used to acquire the target log and extract the target duplicate single log that appears multiple times in the target log; The judgment module is used to determine whether there are multiple identical single logs in each subsequent single log based on the position of the target duplicate single log in the target log; if multiple identical single logs exist, the confirmation module is triggered. The confirmation module is used to confirm that the target duplicate single log and multiple identical single logs constitute a duplicate log segment; The second acquisition module is used to acquire the number of duplicate lines in the target log where the first line of the duplicate log segment appears. A marking module is used to mark and replace the duplicate log segments in the target log according to the number of duplicate lines, so as to obtain the compressed target log; The judgment module includes: The second acquisition submodule is used to acquire the maximum number of lines threshold of a duplicate log segment; wherein, the maximum number of lines threshold of a duplicate log segment represents the maximum number of lines of a single log entry in the duplicate log segment; The first judgment submodule is used to determine, based on the position of the target duplicate single log in the target log, whether there are multiple identical single logs in each of the single logs of the maximum number of lines in the duplicate log segment after the target duplicate single log; The confirmation module includes: The third acquisition submodule is used to acquire the minimum number of lines threshold for a duplicate log segment; wherein, the minimum number of lines threshold for a duplicate log segment represents the minimum number of lines in a single log entry in the duplicate log segment; the minimum number of lines threshold for a duplicate log segment is less than the maximum number of lines threshold for a duplicate log segment; The second determining submodule is used to determine the number of identical single log entries; The second judgment submodule is used to determine whether the number of identical single log entries is not less than the minimum number of lines in the duplicate log segment threshold; if it is confirmed that the number of identical single log entries is less than the minimum number of lines in the duplicate log segment threshold, the second extraction submodule is triggered; if it is confirmed that the number of identical single log entries is not less than the minimum number of lines in the duplicate log segment threshold, the first confirmation submodule is triggered. The second extraction submodule is used to extract the next duplicate single log in the duplicate single log list as the new target duplicate single log, and trigger the judgment module; The first confirmation submodule is used to confirm that the target duplicate single log and each of the same single log constitutes a duplicate log segment.

7. A log compression device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the log compression method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the log compression method as described in any one of claims 1 to 5.

Images