Log compression method and device based on structured marking and hybrid coding

Abstract

The invention discloses a log compression method and device based on structured marking and hybrid coding, and solves the technical problem that the compression rate is obviously low due to an existing log compression method. The method comprises the following steps: acquiring an original log message, and preprocessing the original log message by adopting a predefined separator set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence; classifying the dynamic label sequence, and outputting a plurality of structured labels and a plurality of unstructured labels; based on the plurality of structured markers, generating a refined skeleton and a simplified sub-marker matrix; generating a dictionary file and a binary coded data file according to the static mark sequence, the plurality of unstructured marks, the refined skeleton and the simplified sub-mark matrix; and compressing the dictionary file and the binary coded data file based on a compression algorithm to generate a complete compressed log file.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and in particular to a log compression method and apparatus based on structured tagging and hybrid coding. Background Technology

[0002] The volume of log data has grown exponentially, which has not only directly increased the cost of purchasing storage hardware, but also brought a series of derivative expenses such as data transmission bandwidth consumption, long-term backup energy consumption, and storage cluster operation and maintenance management. This has brought heavy comprehensive cost pressure to the IT systems of enterprises and institutions, and log storage optimization has become a core pain point in the operation and maintenance of large-scale software systems.

[0003] System logs are not merely redundant data that can be arbitrarily discarded. As a crucial carrier recording software operating status, error details, and user operation trajectories, they are the core basis for rapid fault location and diagnosis, system performance modeling and optimization, and security incident tracing and auditing, directly impacting system operation and maintenance efficiency and risk control capabilities. Furthermore, according to industry regulatory compliance requirements such as network security level protection and internal operation and maintenance management standards, log data needs to be archived and retained long-term, ensuring lossless restoration and searchability during the archiving period. This necessitates that log storage solutions address both the cost pressure of massive amounts of data and data availability. Therefore, efficient log compression technology that does not diminish log value becomes the key support for resolving this contradiction.

[0004] Existing log compression methods are typically designed based on the character repetition characteristics of general text. They reduce data volume by recording repeated strings using dictionary encoding and assigning short codes to high-frequency characters using Huffman coding. These methods are suitable for various general text or byte streams without specific formats, and do not need to distinguish the data source or inherent characteristics. They only compress based on the surface character distribution patterns. However, machine-generated log data has highly redundant characteristics and contains rich fixed template structures and numerical patterns. These algorithms completely ignore this essential characteristic of logs and still treat them as unstructured byte streams or text streams. They can only capture surface character repetitions and cannot use template structures and numerical patterns to eliminate deep redundancy. Therefore, when processing highly redundant machine-generated logs, the compression rate is significantly low. Summary of the Invention

[0005] This invention provides a log compression method and apparatus based on structured tagging and hybrid coding, which solves the technical problem that existing log compression methods result in significantly low compression ratios.

[0006] The first aspect of this invention provides a log compression method based on structured tagging and hybrid coding, comprising:

[0007] The original log message is obtained and preprocessed using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence.

[0008] The dynamic tag sequence is classified to output multiple structured tags and multiple unstructured tags;

[0009] Based on the multiple structured tags, a refined skeleton and a simplified sub-tag matrix are generated;

[0010] Based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix, a dictionary file and a binary encoded data file are generated;

[0011] The dictionary file and the binary encoded data file are compressed using a compression algorithm to generate a complete compressed log file.

[0012] Optionally, the preprocessing of the original log message using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence includes:

[0013] The original log message is segmented into an ordered sequence of markers using the predefined set of delimiters;

[0014] Based on the two types of regular expressions, dynamic and static tag recognition are performed on the ordered tag sequence to obtain dynamic and static tag sequences.

[0015] Optionally, classifying the dynamic marker sequence to output multiple structured markers and multiple unstructured markers includes:

[0016] Perform a character-by-character scan on the dynamic marker sequence to identify whether it contains special delimiters;

[0017] Dynamic tags containing the special delimiter in the dynamic tag sequence are determined to be structured tags, while dynamic tags not containing the special delimiter are determined to be unstructured tags.

[0018] Optionally, generating a refined skeleton and a simplified sub-tag matrix based on multiple structured tags includes:

[0019] Extract the delimiter skeleton from each of the structured markers;

[0020] The structured tags with the same delimiter skeleton are grouped and decomposed sequentially to generate a sub-tag matrix;

[0021] Pattern mining and constant extraction are performed on the sub-label matrix to obtain a refined skeleton and a simplified sub-label matrix.

[0022] Optionally, generating a dictionary file and a binary encoded data file based on the static tag sequence, the plurality of unstructured tags, the refined skeleton, and the simplified sub-tag matrix includes:

[0023] The static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix are subjected to data type identification and classification to generate numerical tags, text tags, and hybrid tags.

[0024] The numerical tags, text tags, and hybrid tags are adapted and encoded respectively to generate dictionary files and binary encoded data files.

[0025] Optionally, the compression of the dictionary file and the binary encoded data file based on the compression algorithm to generate a complete compressed log file includes:

[0026] The dictionary file and the binary encoded data file are collected and integrated into a single archive package using a standard archiving format;

[0027] The single archive package is subjected to secondary compression using the compression algorithm to generate a complete compressed log file.

[0028] A second aspect of the present invention provides a log compression apparatus based on structured tagging and hybrid coding, comprising:

[0029] The acquisition module is used to acquire raw log messages and preprocess the raw log messages using a predefined delimiter set and two types of regular expressions to generate dynamic and static tag sequences.

[0030] The identification module is used to classify the dynamic tag sequence and output multiple structured tags and multiple unstructured tags;

[0031] A generation module is used to generate a refined skeleton and a simplified sub-tag matrix based on multiple structured tags;

[0032] The integration module is used to generate a dictionary file and a binary encoded data file based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix;

[0033] The compression module is used to compress the dictionary file and the binary encoded data file based on a compression algorithm to generate a complete compressed log file.

[0034] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the log compression method based on structured tagging and hybrid coding as described above.

[0035] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the log compression method based on structured tagging and hybrid coding as described above.

[0036] The fifth aspect of the present invention provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the log compression method based on structured tagging and hybrid coding as described above.

[0037] As can be seen from the above technical solutions, the present invention has the following advantages:

[0038] The above-mentioned technical solution of the present invention provides a log compression method based on structured tagging and hybrid encoding. The method obtains the original log message and preprocesses it using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence. The dynamic tag sequence is classified, outputting multiple structured tags and multiple unstructured tags. Based on the multiple structured tags, a refined skeleton and a simplified sub-tag matrix are generated. According to the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix, a dictionary file and a binary encoded data file are generated. The compression algorithm is then applied to the compressed data. The dictionary file and binary encoded data file are compressed to generate a complete compressed log file. Based on the above scheme, this invention accurately distinguishes the dynamic and static characteristics of the log through the preprocessing stage, avoiding the crude processing of the log as unstructured data. Then, through the subdivision and refinement of the skeleton of the structured tag and the simplification of the generation of the sub-tag matrix, the hidden structured rules and redundant information inside the log are deeply mined, rather than just staying at the surface character level. Subsequently, the encoded file is generated and compressed based on the multi-class processed data, realizing the accurate adaptation of log data characteristics and the effective elimination of deep redundancy, and ultimately significantly improving the log compression rate. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 The following is a flowchart of the steps of a log compression method based on structured tagging and hybrid coding provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram illustrating an example of a structured tag and sub-tag matrix provided in Embodiment 1 of the present invention;

[0042] Figure 3 This is an overall framework diagram of a log compression method based on structured tagging and hybrid coding provided in Embodiment 1 of the present invention;

[0043] Figure 4 This is a structural block diagram of a log compression device based on structured tagging and hybrid coding, provided in Embodiment 2 of the present invention. Detailed Implementation

[0044] This invention provides a log compression method and apparatus based on structured tagging and hybrid coding, which solves the technical problem that existing log compression methods result in significantly low compression ratios.

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that in the optional embodiments of the present invention, the object information and other related data involved require the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. That is to say, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0046] Terminology Explanation:

[0047] Token: The smallest unit of log parsing, such as a word, number, or sequence of symbols.

[0048] Structured tokens: Complex tokens containing specific delimiters (such as hyphens, colons, square brackets, etc.), which are composed of a delimiter skeleton and multiple sub-tokens, and have a decomposable internal structure. For example, the date "2025-12-18" or the process name "ftpd

[4305] " with an ID.

[0049] Sub-token: A finer-grained basic unit extracted from structured markup, typically a word without punctuation, including words consisting solely of numbers, letters, or a mixture of letters and numbers. For example, "2025", "12", and "18" are extracted from "2025-12-18".

[0050] Delimiter Skeleton: A structural pattern in complex markup consisting of delimiters (such as punctuation marks). For example, the skeleton of "2025-12-18" is "<>-<>-<>".

[0051] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a log compression method based on structured tagging and hybrid coding, as provided in Embodiment 1 of the present invention.

[0052] This invention provides a log compression method based on structured tagging and hybrid coding, comprising:

[0053] Step 101: Obtain the original log message and preprocess it using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence.

[0054] It should be noted that the original log message is divided into an ordered sequence of tags by a predefined set of delimiters. Then, two types of regular expressions are used to filter and identify the tags. The tags that match successfully form a dynamic tag sequence, while the tags that do not match are combined to form a static tag sequence.

[0055] Further, step 101 may include the following sub-steps:

[0056] S11. Divide the raw log message into an ordered sequence of markers using a predefined set of delimiters;

[0057] S12. Based on two types of regular expressions, perform dynamic and static tag recognition on the ordered tag sequence to obtain dynamic tag sequence and static tag sequence.

[0058] Raw log messages refer to unstructured text data that records system operating status, operation trajectory, error information, etc., and are the initial input data for log compression processing.

[0059] An ordered tag sequence refers to a set of tags that retains the original order of tags after splitting the original log message using a predefined set of delimiters. It is the direct processing object for subsequent tag identification.

[0060] There are two types of regular expressions, which are specific rule expressions used for dynamic tag recognition. One type is used to match tags containing numbers, and the other type is used to match tags containing paths. Together, they constitute the criteria for distinguishing between dynamic and static tags.

[0061] It should be noted that the core task of this step is to preprocess the raw log stream (i.e., the raw log messages), and the process includes two stages: tag parsing and type classification.

[0062] First, the raw log messages are segmented into ordered tokenization sequences using a predefined set of delimiters (such as spaces, commas, tabs, etc.), strictly preserving their original order. Then, the system categorizes each token:

[0063] 1) Identification of Dynamic Tokens: The system first uses predefined regular expressions to match tokens containing numbers, and specialized path matching expressions to actively identify variable parts in log messages and mark them as dynamic tokens. Specifically, the regular expression "^\S*\d\S*$" is used to identify tokens containing numbers, and the regular expression "( / [ˆ / ]*)+|([a-zA-Z]:\\(?:[ˆ\\]*\\)" is used to identify tokens containing paths.

[0064] 2) Determination of Static Tokens: The remaining parts of the log message that are not matched by the above regular expressions are automatically regarded as fixed template content and combined to form a static token sequence.

[0065] In this embodiment, the original log message is segmented by segment using a predefined set of delimiters (including spaces, commas, tabs, etc.), strictly preserving the order of each tag in the original log to form an ordered tag sequence. Each tag in this ordered tag sequence is matched and validated using two types of regular expressions. One type of regular expression is used to identify tags containing numbers, and the other type is used to identify tags containing paths. Tags that are successfully matched by the two types of regular expressions are integrated into a dynamic tag sequence in their original order. Tags that are not matched by either regular expression are determined to be fixed template content and are combined in their original order to form a static tag sequence. This step achieves accurate separation of the dynamic variable part and the static template part in the log, providing a foundation for subsequent mining of the internal structured patterns of the log and elimination of deep redundancy.

[0066] Step 102: Classify the dynamic marker sequence and output multiple structured markers and multiple unstructured markers.

[0067] It should be noted that each token in the dynamic token sequence is scanned character by character to identify whether it contains a special delimiter that is not a letter or number. The tokens containing the special delimiter are determined to be structured tokens, and the tokens that do not contain the special delimiter are determined to be unstructured tokens. In this way, multiple structured tokens and multiple unstructured tokens are output.

[0068] Furthermore, step 102 may include the following sub-steps:

[0069] S21. Perform a character-by-character scan on the dynamically marked sequence to identify whether it contains special delimiters;

[0070] S22. Dynamic tags containing special delimiters in the dynamic tag sequence are determined to be structured tags, and dynamic tags not containing special delimiters are determined to be unstructured tags.

[0071] It should be noted that the dynamic tokens are further subdivided into structured and unstructured tokens: For the dynamic tokens identified in the above steps, the system further performs a character-by-character scan. If the scan finds that the token contains special characters that are not letters or numbers (i.e., separators, such as hyphens "-", colons ":", periods ".", square brackets "[]", etc.), it is determined to be a structured token and will be the focus of subsequent in-depth analysis; conversely, if the token consists only of letters or numbers and does not contain special separators, it is determined to be an unstructured token. Specifically, this invention uses the regular expression "(([\p{L}\p{N}]+)|([ˆ\p{L}\p{N}]+))" to perform a character-by-character scan and identify special characters that are not letters or numbers as separators.

[0072] This refined classification mechanism ensures that subsequent steps can apply the most effective compression strategies for different data characteristics (such as dictionary encoding for static tags and skeleton decomposition for structured tags).

[0073] In this embodiment, each dynamic token in the dynamic token sequence is scanned character by character to accurately identify whether the token contains special non-alphanumeric separators (such as hyphens, colons, square brackets, etc.). Based on the scanning results, dynamic tokens containing such special separators are clearly identified as structured tokens, while dynamic tokens that do not contain such special separators and consist only of letters or numbers are identified as unstructured tokens. This step achieves refined classification of dynamic tokens, providing a key foundation for further in-depth exploration of the redundancy patterns within structured tokens.

[0074] Step 103: Based on multiple structured tags, generate a refined skeleton and a simplified sub-tag matrix.

[0075] It should be noted that the delimiter skeleton of each structured tag is extracted, structured tags with the same skeleton are grouped and decomposed into sub-tag sequences to form a sub-tag matrix, and then constants are extracted through pattern mining, finally generating a refined skeleton and a simplified sub-tag matrix.

[0076] Furthermore, step 103 may include the following sub-steps:

[0077] S31. Extract the delimiter skeleton from each structured marker;

[0078] S32. Group and decompose the structured tags with the same delimiter skeleton in sequence to generate a sub-tag matrix;

[0079] S33. Perform pattern mining and constant extraction on the sub-label matrix to obtain the refined skeleton and simplified sub-label matrix.

[0080] The delimiter skeleton refers to the pattern composed of special delimiters extracted from structured markup. It is an abstract representation of the internal format of structured markup and is used as the basis for grouping structured markup.

[0081] A sub-label matrix refers to a set of column-homogeneous labels formed by decomposing structured labels with the same delimiter skeleton into sub-label sequences and arranging them column-wise. It is the direct processing carrier for pattern mining.

[0082] Pattern mining refers to a series of processing operations performed on the sub-label matrix, including determining key positions, splitting and recombining the matrix, and extracting constants, in order to discover fixed patterns and redundant information in the matrix.

[0083] Constant extraction refers to the process of identifying the common constant columns among all records in a sub-label matrix using a frequent itemset mining algorithm and extracting the corresponding values. It is the core operation for simplifying matrices and optimizing skeletons.

[0084] The refined skeleton refers to the optimized skeleton form that removes redundancy and contains fixed constants after embedding the values ​​obtained from the constants into the placeholders of the original separator skeleton.

[0085] The simplified sub-label matrix refers to the sub-label matrix after pattern mining and constant extraction, in which constant redundancy is removed, and only column-oriented homogeneous sets with variable sub-labels are retained.

[0086] It should be noted that, regarding the "structured tags" identified in the above steps, this step aims to uncover their deep redundancy and achieve a breakthrough in compression performance. The system first extracts the delimiter patterns in the tags as "skeletons" by scanning character by character (for example, extracting "<>-<>-<>" from "2024-01-01", and extracting "<>[<>]" from "ftpd

[4305] "). Tags with the same skeleton are grouped together, directly eliminating the redundancy of the delimiter skeleton. Specifically, this invention uses the regular expression "(([\p{L}\p{N}]+)|([ˆ\p{L}\p{N}]+))" to perform character-by-character scanning and identify special characters that are not letters or numbers as delimiters. Next, based on the skeleton, the complex marker is decomposed into an ordered sequence of sub-markers, forming a sub-marker matrix (e.g., "2024-01-01" is decomposed into ["2024", "01", "01"], and multiple sets of sub-markers with the same skeleton marker form a column-homogeneous matrix). For a specific example, please refer to Figure (2).

[0087] Based on this, the system further optimizes itself through a three-step model, with each step executed according to a clearly defined process:

[0088] The first step is to locate the key positions in the sub-label matrix.

[0089] First, perform a "numerical count" on each column of the sub-label matrix—for example, if a column has three values, "07", "08", and "09", appearing 4 times, 1 time, and 1 time respectively, plot these counts using a histogram. Next, calculate the "dynamic threshold": divide the total number of rows in this column by the number of distinct values ​​(e.g., 6 rows ÷ 3 values ​​= 2). Values ​​appearing ≥ this threshold are the "representative values" (e.g., "07" appearing 4 times ≥ 2 is a representative value). If a column has only one unique value, it is directly embedded as a constant into the skeleton.

[0090] Next, the columns with representative values ​​are sorted: first, the "number of distinct values" is compared (fewer values ​​take precedence); if the numbers are the same, the "dominance ratio" (number of times the representative value appears ÷ total number of rows, higher ratio takes precedence); if they are still the same, the "Shannon entropy" is compared (the more concentrated the numerical distribution, the lower the entropy value, and the higher the priority). Finally, verification is performed: only when the number of representative values ​​in a column is ≤40 (default threshold), or the dominance ratio is ≥0.6 (default threshold), is the column confirmed as a "critical position".

[0091] The second step is to split and recombine the sub-label matrix (i.e., the simplified sub-label matrix) according to the rules.

[0092] First, divide the values ​​in the key positions into two categories: the previously found "representative values" (e.g., "07") and other "minor values" (e.g., "08" and "09"). Then, check the number of minor values: if there are ≤3 (the default threshold), perform a "full restructuring"—create a separate submatrix for each value (including representative and minor values), and refine the skeleton (e.g., the original "2024-<>-<>" is refined into "2024-07-<>", "2024-08-<>", and "2024-09-<>" because the key column is the month); if there are more than 3 minor values, perform a "partial restructuring"—only create a submatrix with a refined skeleton for the representative values, and all minor values ​​remain under the original skeleton.

[0093] The third step is to extract constants to form the final framework.

[0094] Specifically, the FP-Growth algorithm for frequent itemset mining scans each reorganized submatrix: each row in the matrix is ​​treated as a "record," and each sub-tag plus its column position (e.g., "01" + the third column) is treated as an "entry." The algorithm only finds "common entries" that are present in all records (i.e., the minimum support is set to 100%). The columns corresponding to these common entries are "constant columns" (e.g., the third column of a certain submatrix is ​​all "01"). Finally, the values ​​of these constant columns are embedded into the corresponding skeleton placeholders. For example, replacing the placeholder in "2024-08-<>" with "01" yields the final refined skeleton "2024-08-01".

[0095] In this embodiment, each structured tag is scanned character by character to identify special delimiters and extract the corresponding delimiter skeleton. Structured tags with the same delimiter skeleton are then grouped together, and each structured tag within a group is decomposed into an ordered sequence of sub-tags. These sub-tag sequences are arranged column-wise to form a sub-tag matrix. Next, pattern mining is performed on this sub-tag matrix—first, the key positions of the matrix are determined, then the matrix is ​​split and recombined according to rules, and finally, constant columns are extracted through frequent itemset mining. These constants are embedded into skeleton placeholders while removing redundant constants from the matrix, ultimately yielding a refined skeleton and a simplified sub-tag matrix. This step deeply mines the fixed patterns and redundant information within the structured tags, avoiding redundant residues caused by neglecting such structural features in existing log compression methods, effectively improving the compression potential of log data.

[0096] Step 104: Generate a dictionary file and a binary encoded data file based on the static tag sequence, multiple unstructured tags, refined skeleton, and simplified sub-tag matrix.

[0097] It should be noted that the first step is to identify the corresponding data type and match it with a specific encoding scheme. Adaptive encoding is then performed on different data types, and duplicate content is mapped to a unique identifier and stored in a dictionary file. The encoded content is then integrated to generate a binary encoded data file.

[0098] Furthermore, step 104 may include the following sub-steps:

[0099] S41. Perform data type identification and classification on the static tag sequence, multiple unstructured tags, refined skeleton and simplified sub-tag matrix to generate numerical tags, text tags and mixed tags;

[0100] S42. Adapt and encode the numeric tags, text tags, and mixed tags respectively to generate dictionary files and binary encoded data files.

[0101] Data type identification and classification refers to the process of analyzing the features of each data item in a static tag sequence, multiple unstructured tags, a refined skeleton, and a simplified sub-tag matrix, and classifying and integrating them into numerical, textual, and mixed tags based on the data composition characteristics.

[0102] Numerical tags refer to a set of tags formed by integrating data content that only has numerical characteristics after data type identification and classification. They are one of the processing objects of adaptive encoding.

[0103] Text-based tags refer to a set of tags formed by integrating data content that only has textual characteristics after data type identification and classification. They are one of the processing objects for adaptive encoding.

[0104] Hybrid tags refer to a set of tags formed by integrating data content that has both numerical and textual characteristics after data type identification and classification. It is one of the processing objects of adaptive encoding.

[0105] Adaptive encoding refers to matching specific encoding rules and performing targeted encoding processing operations for different data characteristics of numeric tags, text tags, and mixed tags.

[0106] A dictionary file is a file that stores the mapping relationship between recurring content in the data and its corresponding unique identifier during the adaptation encoding process. It is one of the output files after encoding processing.

[0107] A binary encoded data file is a file that integrates and converts the results of numerical tags, text tags, and mixed tags after their respective adaptation encodings, and stores the encoded content in binary form. It is one of the output files after encoding processing.

[0108] It should be noted that this step will match a specific encoding scheme for each data type to ensure that each type of data can be compressed efficiently. The specific operation is as follows:

[0109] 1) Numerical data: Targeted optimization of coding:

[0110] For purely numeric sub-tokens (such as "2024" or "01" in timestamps, or the standalone numeric token "4305"), the processing is divided into three steps:

[0111] The first step is to group the numbers by length: separate the numbers by the number of digits (e.g., 1 digit, 2 digits, 3 or more digits), because numbers with similar numbers of digits usually have a more concentrated range of values. Each group will be labeled with a unique letter (e.g., a label for 1-digit numbers). 2 posts The original numbers will be replaced with labels thereafter.

[0112] The second step is to determine whether to use "dynamic Delta encoding": randomly select the first 10 numbers and calculate the difference between each number and the previous number (for example, the difference between the sequence [100,101,102] is [1,1]). If the average difference is smaller than the original number, use the difference to replace the original number for storage; if the numbers are irregular (such as random port numbers), store the original number directly to avoid doing useless work.

[0113] The third step, for composite values ​​(such as the timestamp "17:41:53" split into [17,41,53]): concatenate the numbers in the same row into a larger number (such as 174153), then calculate and store the difference. If the concatenated difference is more compact, use this method; otherwise, process them separately by column.

[0114] Finally, all values ​​are further compressed using "flexible encoding" to store the numbers in a more space-efficient byte format (for example, 35 originally occupies 4 bytes, but after compression it only occupies 1 byte).

[0115] 2) Text data: Dictionary index replacement:

[0116] For plain text sub-tags (such as "ftpd" and "QuorumPeerConfig") and the previously extracted skeleton patterns (such as "<>-<>-<>"), a "dictionary-based accounting" approach is used:

[0117] First, create two dedicated dictionaries: one to store text sub-tags and skeleton patterns (such as "ftpd" "<>-<>-<>"), and the other to store fixed text sequences in the log (such as "main:@").

[0118] Assign a unique number to each item stored in the dictionary (e.g., "ftpd"=1, "<>-<>-<>"=2). When encountering duplicate text or skeletons, use the number to replace the original text for storage.

[0119] To differentiate between different types of alternative content, unique tags are added to the numbers: the numbers for structured skeletons are labeled as |tag| (e.g., |2|), and the numbers for plain text are labeled as <*> to avoid confusion.

[0120] 3) Mixed data: Unified encoding after conversion

[0121] For mixed sub-tags containing both text and numbers (such as "ftpd

[4305] " "QuorumPeerConfig@334"), the encoding is completed in three steps:

[0122] The first step is to assign a unique numeric ID to the text part of the mixed tag (e.g., "ftpd" "QuorumPeer Config") by column (the text IDs in the same column are consecutive, e.g., the first column "ftpd"=1, "Quorum Peer Config"=2), and store the "text-ID" correspondence in a dictionary.

[0123] The second step is to assign odd / even labels to the numbers and IDs: In order to distinguish between the original numbers and the IDs converted from text, a simple conversion is performed - the original numbers are multiplied by 2 (to become even numbers, such as 4305→8610, 334→668), and the text IDs are multiplied by 2 and added by 1 (to become odd numbers, such as 1→3, 2→5). This will allow for accurate reconstruction later.

[0124] The third step is unified compression: After conversion, the mixed tags are all turned into numbers (e.g., "ftpd

[4305] " → [3,8610], "QuorumPeerConfig@334" → [5,668]), and then processed using the "dynamic Delta encoding + flexible encoding" mentioned earlier to achieve efficient compression.

[0125] In this embodiment, data feature analysis is performed on each item in the static tag sequence, multiple unstructured tags, refined skeleton, and simplified sub-tag matrix to complete accurate data type identification and classification. Based on the composition characteristics of the data, they are integrated into numerical tags, text tags, and mixed tags. For the unique data characteristics of each type of tag, the corresponding exclusive adaptation encoding scheme is matched and encoding processing is performed. The content that appears repeatedly during the encoding process is mapped to a unique identifier for retention. At the same time, the results of the various tags after adaptation encoding are integrated and transformed to finally generate a dictionary file and a binary encoded data file. This step implements differentiated and accurate encoding for the characteristics of various processed log data, avoids the defect of indiscriminate encoding of general compression algorithms, maximizes the elimination of redundant information in the encoding process, and further improves the compression efficiency of log data.

[0126] Step 105: Compress the dictionary file and binary encoded data file based on the compression algorithm to generate a complete compressed log file.

[0127] It should be noted that the dictionary file and binary encoded data file are first collected and integrated, and then the integrated file is compressed based on the preset compression algorithm to further remove redundant data, and finally generate a complete compressed log file.

[0128] Furthermore, step 105 may include the following sub-steps:

[0129] S51, collect dictionary files and binary encoded data files, and integrate them into a single archive package using a standard archive format;

[0130] S52. Perform secondary compression on a single archive package using a compression algorithm to generate a complete compressed log file.

[0131] Standard archive format refers to a pre-defined general file integration and packaging format, which serves as the basis for ensuring the integrity, relevance, and subsequent parsing of the collected dictionary files and binary encoded data files.

[0132] A single archive package refers to a unified file package formed by encapsulating and integrating the collected dictionary files and binary encoded data files according to a standard archive format. It is the direct object of secondary compression.

[0133] A complete compressed log file refers to the final file generated after performing secondary compression on a single archive using a compression algorithm. This file retains the complete and valid information of the log, and its storage size is greatly reduced.

[0134] It should be noted that, as the final step in the entire compression process, this step is responsible for integrating and compressing the structured intermediate data output by the preceding components, ensuring both data integrity and storage compactness. First, all processed core intermediate data needs to be collected, strictly corresponding to the output results of the preceding components. This mainly includes two types of key files: First, dictionary files, containing dedicated token and template dictionaries. The token dictionary stores unstructured strings, delimiter skeleton patterns of structured tags, and the mapping relationship between "string-numeric IDs" in mixed types. The template dictionary stores fixed sequence templates composed of static tags, various labels, and placeholders. Both dictionaries provide the necessary mapping basis for subsequent lossless decompression. Second, binary encoded data files, covering the flexible encoding and dynamic Delta encoding results of numerical data, ID replacement sequences for string types, column-oriented encoding products of sub-tag matrices, and ID-encoded binary streams of static tag sequences. These files have all undergone redundancy removal in the preceding steps and possess a high compression foundation.

[0135] Then, the tar archiving tool is used to integrate all the collected intermediate files into a single archive package according to the POSIX standard tar format. During the archiving process, the relative path indexes and original file structures of each file are fully preserved without modifying the core content of the files. This direct aggregation method avoids the loss or disorder of scattered files and provides a unified data source for subsequent overall compression, helping the compression algorithm to efficiently utilize the redundant features across files.

[0136] Finally, a second compression is performed on the tar archive, employing a "general compression tool + standardized configuration" approach to balance compression performance and compatibility. The default compression scheme is the LZMA algorithm (Lempel-Ziv-Markov chain Algorithm), with the compression level set to Level 6 to balance compression ratio and efficiency. It is also compatible with mainstream general compression tools such as Gzip and Bzip2, allowing users to switch flexibly according to their specific needs. The compression process only affects the entire archive, without damaging the original structure and data content of the internal files. The compressed file generates a single-suffix log archive. The file header includes the compression algorithm identifier and original data size information, providing a basis for decompression. The final output supports lossless decompression verification, ensuring that the original log data can be completely restored after decompression without any information loss.

[0137] In this embodiment, the generated dictionary file and binary encoded data file are first uniformly collected to ensure the integrity and data correlation of the two types of files. Then, the collected two types of files are packaged and integrated according to a preset standard archiving format to form a single archive package. Subsequently, a preset compression algorithm is called to perform a targeted secondary compression operation on the single archive package to further remove surface redundancy in the data within the package, and finally generate a complete compressed log file. This process follows the previous structured mining, accurate classification and adaptation encoding of log data, realizes unified data management through archiving and integration, and completes the layered deep redundancy elimination of log data through secondary compression. Compared with the existing general compression algorithm that only performs surface compression on unstructured logs, this method realizes fine-grained compression processing of log data throughout the entire process, and significantly improves the log compression rate.

[0138] For comparison of technical effectiveness, existing technologies can be used as a reference. System logs are an important resource for fault diagnosis, performance modeling, and security auditing. Due to compliance and operational needs, logs typically require long-term archiving and storage. Therefore, efficient log compression technology is crucial for reducing storage overhead.

[0139] Current log compression technologies are mainly divided into two categories:

[0140] 1) General compression algorithms (such as Gzip, Bzip2): Treat logs as ordinary text streams and process them using general compression algorithms.

[0141] 2) Log-specific compression tool:

[0142] Parsing-based methods (such as LogZip, LogReducer, and LogShrink) utilize the "template-parameter" structure of logs to extract and store fixed templates and variable parameters separately. These methods typically treat the identified parameters as a whole and store them using dictionary encoding; some methods (such as LogReducer) apply numerical encoding optimization to the identified pure numeric sequences.

[0143] Numerical methods (such as Denum) focus on optimizing numeric tokens in logs. These methods primarily compress numbers by grouping them according to their length and first digit characteristics, and then combining dictionary encoding and numerical encoding techniques.

[0144] Based on the above, the shortcomings of existing compression technologies can be divided into three parts:

[0145] 1. General compression algorithm: Treats logs as unstructured byte streams or text streams, completely ignoring the rich template structure and numerical patterns contained in the logs. Therefore, when processing highly redundant machine-generated logs, the compression rate is much lower than that of dedicated algorithms.

[0146] 2. Insufficient Granularity of Structure Recognition (Regarding Existing Dedicated Methods): Although existing dedicated technologies can identify log templates, they usually treat parameters as indivisible minimum processing units. For parameters with complex formats and delimiters (such as "2015-07-09" or "ftpd

[4305] "), existing tools cannot recognize their fixed internal "skeleton" structure and treat them as ordinary strings, resulting in a large amount of structured redundancy not being eliminated.

[0147] 3. Limited and inefficient encoding strategies (for mixed-type data): Existing technologies lack sophisticated type awareness. Numerical methods are only effective for pure numbers; parsing-based methods typically employ only general dictionary encoding for structured tokens that are not purely numeric. This approach ignores the data characteristics of different parts within the structured token (such as split numbers and text), failing to match optimal encoding strategies and thus limiting compression ratios.

[0148] As mentioned above, with the development of large-scale software systems, the amount of log data (files recording system operation) is growing exponentially, bringing enormous storage cost pressure. This invention aims to solve the problem of low compression efficiency in existing technologies when processing heterogeneous log data with complex formats and mixed character types (such as identifiers with delimiters, dates, etc.) due to the inability to recognize their internal fine structure. It provides a log compression method based on structured tagging and hybrid encoding. By deeply mining the "skeleton" structure within the parameters and introducing a fine-grained type-aware mechanism, appropriate encoding algorithms are used for pure numeric, text, and mixed-type data, thereby significantly improving the compression ratio.

[0149] Specifically, please refer to Figure 3 The overall framework of this invention is mainly divided into four components. The tag analyzer is responsible for preprocessing the raw log data, dividing the data into three types: structured tags, unstructured tags, and static tag sequences. Subsequently, the structured tag sequences are further processed using a structured tag processor. Based on the symbol skeleton of the structured tags, the data is grouped, and then template mining is used to refine and simplify the sub-tag matrix. Finally, the refined skeleton and sub-tag matrix obtained by the structured tag processor, along with the unstructured tags and static tag sequences, are mixed and encoded to generate a dictionary file and a binary file. Finally, a packer is used to compress these files, ultimately forming a complete compressed log file.

[0150] In summary, the fine-grained decomposition technique based on delimiter skeleton proposed in this invention uses punctuation marks as a skeleton to deeply decompose complex log tags into sub-tag sequences, breaking through the limitation of traditional methods that only stop at the word level. Simultaneously, through a type-aware adaptive hybrid encoding strategy, a multi-level classification mechanism (static / structured / unstructured) is established, and the optimal encoding algorithm is matched for each of the decomposed fine-grained data types to maximize the compression ratio.

[0151] Experiments show that on multiple public datasets, this invention achieves an average compression rate improvement of approximately 11.11% compared to state-of-the-art log compression tools, significantly reducing storage costs for large-scale systems. Furthermore, it effectively solves the problem of traditional methods struggling to handle heterogeneous data such as mixed IDs and complex timestamps, fully extracting redundant information hidden within the tags. While improving compression rate, it maintains a high processing throughput (approximately 10MB / s), making it suitable for log archiving needs in real-world production environments.

[0152] In this embodiment of the invention, a log compression method based on structured tagging and hybrid encoding is provided. The method acquires the original log message and preprocesses it using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence. The dynamic tag sequence is classified, outputting multiple structured tags and multiple unstructured tags. Based on the multiple structured tags, a refined skeleton and a simplified sub-tag matrix are generated. A dictionary file and a binary encoded data file are generated based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix. A compression algorithm is then applied. The dictionary file and binary encoded data file are compressed to generate a complete compressed log file. Based on the above scheme, this invention accurately distinguishes the dynamic and static characteristics of the log through the preprocessing stage, avoiding the crude processing of the log as unstructured data. Then, through the subdivision and refinement of the skeleton of structured tags and the simplification of the generation of sub-tag matrices, the hidden structured rules and redundant information inside the log are deeply mined, rather than just staying at the surface character level. Subsequently, based on the multi-class processed data, encoded files are generated and compressed, realizing accurate adaptation to log data characteristics and effective elimination of deep redundancy, ultimately significantly improving the log compression rate.

[0153] Please see Figure 4 , Figure 4 This is a structural block diagram of a log compression device based on structured tagging and hybrid coding, provided in Embodiment 2 of the present invention.

[0154] This invention provides a log compression device based on structured tagging and hybrid coding, comprising:

[0155] The acquisition module 401 is used to acquire the raw log message and preprocess the raw log message using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence.

[0156] The identification module 402 is used to classify the dynamic marker sequence and output multiple structured markers and multiple unstructured markers;

[0157] The generation module 403 is used to generate a refined skeleton and a simplified sub-tag matrix based on multiple structured tags;

[0158] Integration module 404 is used to generate dictionary files and binary encoded data files based on static tag sequences, multiple unstructured tags, refined skeletons, and simplified sub-tag matrices;

[0159] Compression module 405 is used to compress dictionary files and binary encoded data files based on compression algorithms to generate complete compressed log files.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the log compression method based on structured tagging and hybrid coding as described in the above embodiments.

[0162] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the log compression method based on structured tagging and hybrid coding as described in the above embodiments.

[0163] This invention also provides a computer program product, including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the log compression method based on structured tagging and hybrid coding as described in the above embodiments.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log compression method based on structured tagging and hybrid coding, characterized in that, include: The original log message is obtained and preprocessed using a predefined delimiter set and two types of regular expressions to generate a dynamic tag sequence and a static tag sequence. The dynamic tag sequence is classified to output multiple structured tags and multiple unstructured tags; Based on the multiple structured tags, a refined skeleton and a simplified sub-tag matrix are generated; Based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix, a dictionary file and a binary encoded data file are generated; The dictionary file and the binary encoded data file are compressed using a compression algorithm to generate a complete compressed log file.

2. The log compression method based on structured tagging and hybrid coding according to claim 1, characterized in that, The preprocessing of the original log message using a predefined delimiter set and two types of regular expressions to generate dynamic and static tag sequences includes: The original log message is segmented into an ordered sequence of markers using the predefined set of delimiters; Based on the two types of regular expressions, dynamic and static tag recognition are performed on the ordered tag sequence to obtain dynamic and static tag sequences.

3. The log compression method based on structured tagging and hybrid coding according to claim 1, characterized in that, The classification of the dynamic marker sequence outputs multiple structured markers and multiple unstructured markers, including: Perform a character-by-character scan on the dynamic marker sequence to identify whether it contains special delimiters; Dynamic tags containing the special delimiter in the dynamic tag sequence are determined to be structured tags, while dynamic tags not containing the special delimiter are determined to be unstructured tags.

4. The log compression method based on structured tagging and hybrid coding according to claim 1, characterized in that, The process of generating a refined skeleton and a simplified sub-tag matrix based on multiple structured tags includes: Extract the delimiter skeleton from each of the structured markers; The structured tags with the same delimiter skeleton are grouped and decomposed sequentially to generate a sub-tag matrix; Pattern mining and constant extraction are performed on the sub-label matrix to obtain a refined skeleton and a simplified sub-label matrix.

5. The log compression method based on structured tagging and hybrid coding according to claim 1, characterized in that, The step of generating a dictionary file and a binary encoded data file based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix includes: The static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix are subjected to data type identification and classification to generate numerical tags, text tags, and hybrid tags. The numerical tags, text tags, and hybrid tags are adapted and encoded respectively to generate dictionary files and binary encoded data files.

6. The log compression method based on structured tagging and hybrid coding according to claim 1, characterized in that, The process of compressing the dictionary file and the binary encoded data file using a compression algorithm to generate a complete compressed log file includes: The dictionary file and the binary encoded data file are collected and integrated into a single archive package using a standard archiving format; The single archive package is subjected to secondary compression using the compression algorithm to generate a complete compressed log file.

7. A log compression device based on structured tagging and hybrid coding, characterized in that, include: The acquisition module is used to acquire raw log messages and preprocess the raw log messages using a predefined delimiter set and two types of regular expressions to generate dynamic and static tag sequences. The identification module is used to classify the dynamic tag sequence and output multiple structured tags and multiple unstructured tags; A generation module is used to generate a refined skeleton and a simplified sub-tag matrix based on multiple structured tags; The integration module is used to generate a dictionary file and a binary encoded data file based on the static tag sequence, multiple unstructured tags, the refined skeleton, and the simplified sub-tag matrix; The compression module is used to compress the dictionary file and the binary encoded data file based on a compression algorithm to generate a complete compressed log file.

8. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the log compression method based on structured tagging and hybrid coding as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the log compression method based on structured tagging and hybrid encoding as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the log compression method based on structured tagging and hybrid coding as described in any one of claims 1-6.

Images