Log compression method, apparatus, device, and medium
By obtaining semantic vectors through log compression methods and using a global template library to match and compress log lines, the problems of low compression rate and low query efficiency in existing technologies are solved, achieving efficient log compression and fast query analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 湖南三一智慧新能源设计有限公司
- Filing Date
- 2026-03-02
- Publication Date
- 2026-07-21
AI Technical Summary
Existing log compression methods have limited compression rates, and the compressed data cannot be directly used for querying and analysis. Furthermore, they cannot adapt to log format changes brought about by system updates or new failure modes, resulting in low efficiency.
By obtaining the first semantic vector of the log line to be compressed, matching log templates using a global template library, compressing based on the target log template and variable parameter values, constructing a compression database, and supporting fast querying and statistical analysis of template identifiers.
It achieves extremely high compression ratio and query efficiency, enabling rapid retrieval and statistical analysis without complete decompression, and adapts to system updates and new failure modes.
Smart Images
Figure CN122432129A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of log processing technology, and in particular to a log compression method, apparatus, device, and medium. Background Technology
[0002] As distributed systems continue to expand in scale and increase in complexity, the log data generated during system operation is growing explosively. Log data is an important basis for system monitoring, fault diagnosis, and performance analysis, but its massive nature also brings huge challenges to storage, transmission, and analysis.
[0003] Currently, general compression algorithms have limited compression ratios, and the compressed data cannot be directly used for querying and analysis; it must be decompressed first, which is inefficient. Compression methods based on fixed keywords or regular expressions require prior knowledge of key information, have poor flexibility, and cannot adapt to log format changes brought about by system updates or new failure modes. Therefore, using existing log compression methods is inefficient for log compression. Summary of the Invention
[0004] This invention provides a log compression method, apparatus, device, and medium to address the limitations of existing general compression algorithms, such as limited compression ratios and the inefficiency of decompressing compressed data before it can be directly used for querying and analysis. Furthermore, compression methods based on fixed keywords or regular expressions require prior knowledge of key information, lack flexibility, and cannot adapt to log format changes brought about by system updates or new failure modes. Therefore, this invention overcomes the shortcomings of existing log compression methods in terms of low efficiency. It achieves this by automatically extracting log templates representing the system's operating status from a massive global template library using the first semantic vector of the log to be compressed, and transforming specific log instances into a compact format of template identifiers and variable parameter values (i.e., key parameters), thereby achieving a very high compression ratio and improving log compression efficiency.
[0005] This invention provides a log compression method, comprising the following steps: Obtain the first semantic vectors corresponding to multiple log lines to be compressed; Each of the first semantic vectors is matched with each log template vector in the global template library to obtain the target log template that matches each of the first semantic vectors; Based on the target log templates and the variable parameter values in each log line to be compressed, each log line to be compressed is compressed to obtain a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
[0006] According to a log compression method provided by the present invention, the step of compressing each log line to be compressed based on each target log template and each log line to be compressed to obtain a compressed database containing multiple compressed records includes: extracting the variable parameter values from each log line to be compressed based on each target log template; and compressing each log line to be compressed based on each variable parameter value and a template identifier corresponding to each target log template to obtain a compressed database containing multiple compressed records.
[0007] According to a log compression method provided by the present invention, obtaining the first semantic vector corresponding to a plurality of log lines to be compressed includes: performing masking processing on the variable parameter values in each of the log lines to be compressed to obtain the masked log lines to be compressed corresponding to each of the log lines to be compressed; and performing vectorization processing on each of the masked log lines to be compressed to obtain the first semantic vector corresponding to each of the log lines to be compressed.
[0008] According to a log compression method provided by the present invention, determining the global template library includes: obtaining second semantic vectors corresponding to multiple original log lines; performing clustering processing on each second semantic vector to obtain multiple first log line clusters; calculating the information entropy corresponding to each word based on the frequency of occurrence of each word in the first log line clusters; determining the log template corresponding to each first log line cluster based on the information entropy; and constructing the global template library based on all the log templates.
[0009] According to a log compression method provided by the present invention, after obtaining a compressed database containing multiple compressed records, the method further includes: querying the compressed record corresponding to the query condition from the compressed database based on the user's query condition; determining the log template corresponding to the template identifier from the global template library based on the template identifier in the compressed record; and filling the variable parameter bits in the log template with the parameter list in the compressed record to obtain the decompressed log line.
[0010] According to a log compression method provided by the present invention, the method further includes: determining the occurrence count of the variable parameter value when compressing the log line to be compressed; and determining aggregation information based on the variable parameter value in the compression database and the occurrence count of the variable parameter value when a user initiates an aggregation request.
[0011] According to a log compression method provided by the present invention, the method further includes: during the matching process of the first semantic vector and the log template vector, statistically analyzing the feature information of the log lines to be compressed corresponding to the first semantic vectors that failed to match; if the feature information does not meet the information threshold, clustering the log lines to be compressed that failed to match to obtain multiple second log line clusters; and updating the global template library based on each second log line cluster.
[0012] The present invention also provides a log compression device, comprising the following modules: The first acquisition module is used to acquire the first semantic vectors corresponding to multiple log lines to be compressed; The matching module is used to match each of the first semantic vectors with each log template vector in the global template library to obtain the target log template that matches each of the first semantic vectors. A compression module is used to compress each of the target log templates and the variable parameter values in each of the log lines to be compressed, thereby obtaining a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with a log line to be compressed. The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements any of the log compression methods described above.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log compression method as described above.
[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the log compression methods described above.
[0015] The log compression method, apparatus, device, and medium provided by this invention obtain first semantic vectors corresponding to multiple log lines to be compressed; match each first semantic vector with each log template vector in a global template library to obtain target log templates that match each first semantic vector; and compress each log line to be compressed based on each target log template and the variable parameter values in each log line to be compressed, resulting in a compressed database containing multiple compressed records, with each compressed record corresponding one-to-one with a log line to be compressed. In this way, by using the first semantic vectors of the logs to be compressed, log templates representing the system's operating status are automatically extracted from the massive logs in the global template library, and specific log instances are transformed into a compact format of template identifiers and variable parameter values (i.e., key parameters), thereby achieving an extremely high compression ratio and improving log compression efficiency. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating the log compression method provided by the present invention.
[0018] Figure 2 This is a schematic diagram of the log compression device provided by the present invention.
[0019] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0021] Traditional log compression methods, such as general compression algorithms (like GZIP), can reduce data size, but their compression ratio is limited, and the compressed data cannot be used directly for querying and analysis; it must be decompressed first, resulting in low efficiency in log compression.
[0022] In addition, log compression methods based on fixed keywords or regular expressions require prior knowledge of key information, lack flexibility, and cannot adapt to log format changes brought about by system updates or new failure modes.
[0023] To address the aforementioned problems, this invention provides a log compression method. This method involves obtaining first semantic vectors corresponding to multiple log lines to be compressed; matching each first semantic vector with a log template vector in a global template library to obtain target log templates that match each first semantic vector; and compressing each log line to be compressed based on the target log templates and the variable parameter values in each log line to be compressed, resulting in a compressed database containing multiple compressed records, with each compressed record corresponding one-to-one with a log line to be compressed. In this way, by using the first semantic vectors of the logs to be compressed, log templates representing the system's operating status are automatically extracted from the massive logs in the global template library, and specific log instances are transformed into a compact format using template identifiers and variable parameter values, thereby achieving a very high compression ratio and improving log compression efficiency.
[0024] The following is combined with Figure 1 The present invention describes a log compression method that is adaptable to any distributed system. The execution subject of this method can be an electronic device or a log compression device installed in the electronic device. The log compression device can be implemented by software, hardware, or a combination of both.
[0025] Figure 1 This is a flowchart illustrating the log compression method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain the first semantic vector corresponding to multiple log lines to be compressed.
[0026] Here, the first semantic vector is used to represent the high-dimensional semantic information of the log line to be compressed.
[0027] Optionally, the log lines to be compressed can be the initial log lines collected directly, or log lines that have been preprocessed from the initial log lines. Preprocessing may include format specification, deduplication, etc.
[0028] Here, the first semantic vector can be determined by a pre-trained language model, which may include bidirectional encoder representations from transformers (BERT) and sentence-level bidirectional encoder representations from transformers (Sentence-BERT).
[0029] Here, the log lines to be compressed can be raw log streams collected in real time or near real time from various nodes of the distributed system (such as application servers, databases, middleware).
[0030] Furthermore, obtaining the first semantic vector corresponding to the multiple log lines to be compressed includes: masking the variable parameter values in each log line to be compressed to obtain the masked log lines to be compressed corresponding to each log line to be compressed; and vectorizing each masked log line to be compressed to obtain the first semantic vector corresponding to each log line to be compressed.
[0031] Here, variable parameter values include, but are not limited to, user identifiers (such as 123, 456), IP addresses (such as 192.168.1.1, 10.0.0.1), file paths (such as / home / user / file.txt), timestamps (such as 2025-12-19 10:00:00), etc. It should be noted that masking refers to obscuring the variable parameter values.
[0032] For example, the original log is: User 123 logged in from 192.168.1.1, and after masking, it becomes: User [MASK] logged in from [MASK].
[0033] For example, User 123 logged in from 192.168.1.1 and User 456 logged in from 10.0.0.1 are two log entries. The unchanging parts (core semantics) are User and logged in from, which are fixed templates and parameter names that determine the core semantics of the log. The variable parts are 123, 456 (user ID), 192.168.1.1, 10.0.0.1 (IP address), as well as file paths (such as / home / user / file.txt), timestamps (such as 2025-12-19 10:00:00), etc., which are all specific parameter values of this type.
[0034] Here, vectorization refers to converting unstructured / semi-structured log text (in string form) into a numerical vector (a set of ordered numbers) that can be computed by a computer.
[0035] In this embodiment of the invention, by performing masking and vectorization processing on the log lines to be compressed, the semantic information of the logs is effectively captured, going beyond simple keyword matching and improving the compression ratio and compression efficiency.
[0036] Step 102: Match each of the first semantic vectors with each log template vector in the global template library to obtain the target log template that matches each of the first semantic vectors.
[0037] Here, the global template library can be obtained through offline learning based on deep semantic representation.
[0038] Here, the global template library can adopt version management to support template compatibility during system rolling upgrades.
[0039] Here, the matching method can be based on the similarity between the first semantic vector and the log target vector.
[0040] Optional methods for calculating similarity include, but are not limited to, Euclidean distance, Manhattan distance, and cosine distance.
[0041] Here, the target log template refers to a log template similar to the first semantic vector. The log template includes a static part and variable parameter bits. It should be noted that the variable parameter bits in the log target do not contain specific values.
[0042] For example, User [] logged in from IP []at [*] is the log template.
[0043] Further, determining the global template library includes: obtaining second semantic vectors corresponding to multiple original log lines; performing clustering processing on each second semantic vector to obtain multiple first log line clusters; calculating the information entropy corresponding to each word based on the frequency of occurrence of each word in the first log line cluster; determining the log template corresponding to each first log line cluster based on the information entropy; and constructing the global template library based on all the log templates.
[0044] Here, the raw log lines can be preprocessed log lines. The raw log lines come from multiple nodes in the distributed system.
[0045] For example, methods for obtaining multiple raw log lines include: (1) Multi-source log collection: Collect raw log streams from various nodes of the distributed system (such as application servers, databases, middleware) in real time or near real time. Supports collection of logs in various formats, such as plain text logs, JSON format logs, Syslog, etc.
[0046] (2) Log line normalization: Clean the collected raw log lines, including removing meaningless spaces, standardizing the timestamp format, and identifying and separating log level (such as INFO, ERROR, WARN) and other metadata.
[0047] (3) Temporary storage: The normalized log data is stored in a temporary buffer to provide a data source, i.e., the log lines to be compressed, for subsequent offline learning or online compression.
[0048] Here, clustering methods can include density-based spatial clustering of applications with noise (DBSCAN) or hierarchical density-based spatial clustering of applications with noise (HDBSCAN) and other clustering algorithms. It should be noted that DBSCAN can automatically discover clusters of any shape and effectively identify noise points, i.e., rare or novel log patterns.
[0049] It should be noted that the first log line cluster is a non-noise cluster. After identifying noise points, the DBSCAN clustering algorithm retains the noise points but does not extract log templates from them.
[0050] Here, information entropy is used to characterize the similarity of words. If the information entropy is 0, it means that the words at the same position in each log line in the first log line cluster are the same; otherwise, they are different. The larger the information entropy, the more diverse the words at the same position in each log line are.
[0051] For example, if the information entropy of words at the same position in the first log line cluster is 0, it means that all words at that position are the same and can be retained as a static fixed part; if the information entropy of words at the same position in the first log line cluster is greater than 0 and less than or equal to 1, it means that the word diversity at that position is low and can be selected as a static fixed part; if the information entropy of words at the same position in the first log line cluster is greater than 1, it means that the word diversity is high and that position is determined as a strongly variable parameter part.
[0052] For example, taking three log entries within the cluster: User [NUM] logged in from [IP], User [NUM] logged in from [IP], and User [NUM] failed to connect to [IP], the alignment results of the three log entries after word segmentation are shown in Table 1: Table 1. Alignment results after word segmentation for the three log entries. Position Index Log 1 word Log 2 word Log 3 word 0 User User User 1 [NUM] [NUM] [NUM] 2 logged logged failed 3 in in in 4 from from connect 5 [IP] [IP] [IP] Among them, for the first in the cluster For each position, the information entropy is calculated using the following formula (1): (1) in, Indicates the first in the cluster Information entropy of a word at a given position Indicates the number of logs. Greater than or equal to 0, Indicates the first The first log entry Frequency of words at each position It represents the logarithm with base 2.
[0053] Taking the above three log entries at position 2 as an example: the total number of log lines is 3, the word distribution at position 2 is: logged (2 times), failed (1 time), the probability calculation is: p1=2 / 3, p2=1 / 3, and the information entropy of position 2 is calculated as follows (2): (2) At position 0 in the three log entries, all entries are for the user, and the H value is 0, indicating a static part.
[0054] For example, the information entropy at each word position in all log lines within the first log line cluster is calculated, and low-entropy (high-frequency) words are retained as the static part of the template, while the positions of high-entropy (low-frequency) words are marked as variable parameter bits to obtain the log template.
[0055] Optionally, a unique template identifier can be assigned to each log template, and the mapping relationship between each log template and the template identifier can be stored to obtain a global template library.
[0056] In this invention, multiple log lines are divided into different log line clusters through clustering, and the log template of each log line cluster is extracted, which reduces the amount of data. Furthermore, a readable template library is constructed based on the log templates, providing a foundation for subsequent retrieval and analysis.
[0057] Step 103: Based on the target log templates and the variable parameter values in the log lines to be compressed, compress each log line to be compressed to obtain a compressed database containing multiple compressed records.
[0058] The compressed record corresponds one-to-one with the log line to be compressed.
[0059] Here, a compressed record is a compressed block after the log line to be compressed has been compressed.
[0060] It should be noted that the compression and determination of the global template library can adopt a distributed deployment architecture, which can be deployed as an edge-cloud collaborative architecture. Online compression is performed at the edge nodes, which significantly reduces the amount of data transmitted over the network; offline template learning is performed at the central cloud node to determine the global target library and global management.
[0061] Furthermore, the choice of the model for vectorization, i.e., the pre-trained language model, needs to strike a balance between semantic representation accuracy and computational overhead. For example, a lightweight model can be used on resource-constrained edge nodes.
[0062] Here, the cloud-edge system can output key performance indicators, including but not limited to compression ratio and size of the compressed template library.
[0063] Furthermore, the step of compressing each log line to be compressed based on each target log template and the variable parameter values in each log line to be compressed to obtain a compressed database containing multiple compressed records includes: extracting the variable parameter values from each log line to be compressed based on each target log template; and compressing each log line to be compressed based on each variable parameter value and the template identifier corresponding to each target log template to obtain a compressed database containing multiple compressed records.
[0064] Here, the template identifier can be a template identity document (ID), such as 123, 456, etc.
[0065] Specifically, after matching the target log template, the variable parameter values are extracted from the log lines to be compressed based on the target log template. The variable parameter values and template identifier are preserved to obtain the compressed block. In other words, even when the log lines to be compressed are compressed into compressed blocks, the variable parameter values and template identifier can still be seen.
[0066] For example, the original log "User Alice logged in from IP 192.168.1.1 at 2023-10-27 10:00:01" is compressed to [20231027100001] [TID_005][Alice, 192.168.1.1].
[0067] In this invention, a framework combining offline learning and online compression is used to automatically extract log templates representing the system's operating status from massive logs, and to convert specific log instances into a compact format of template identifiers and key parameters, thereby achieving an extremely high compression ratio.
[0068] Furthermore, after obtaining the compressed database containing multiple compressed records, the method further includes: querying the compressed record corresponding to the query condition from the compressed database based on the user's query condition; determining the log template corresponding to the template identifier from the global template library based on the template identifier in the compressed record; and filling the variable parameter bits in the log template with the parameter list in the compressed record to obtain the decompressed log line.
[0069] Here, the decompressed log lines are also called reconstructed log lines.
[0070] Here, user query conditions include, but are not limited to, template content, template ID, and time range. Template content refers to the content within the log template. For example, the log template could be: User [variable parameter] logged in from [variable parameter], where User can be the template content.
[0071] To facilitate quick use by non-professional users, the system will preset some commonly used query condition templates, map log blocks into structured tables, and finally assemble them into SQL query condition statements, which users can directly select without manual input.
[0072] Optionally, the corresponding compressed record can be found directly based on the user's query conditions at once; alternatively, multiple candidate compressed records can be queried based on some of the user's query conditions, and the final compressed record can be found from the candidate compressed records based on the remaining conditions.
[0073] For example, the user query conditions include the template of login error, 2025-12-19 10:00-12:00, and user ID is 150. The specific query process includes: (1) Template matching: find template ID=123 (content contains login error); (2) Compressed block location: find 10h and 11h of template_123; (3) Block metadata filtering: 10h block: user ID range 100-200, contains 150 → valid; 11h block: user ID range 200-300, does not contain 150 → filtered; (4) Decompress as needed: decompress 10h block, reconstruct the original log line of user ID=150, and return it to the user.
[0074] Key performance indicators may also include decompression throughput and query response time.
[0075] In this invention, a readable log template library is constructed, enabling rapid template-based retrieval and statistical analysis of compressed logs without complete decompression, thus greatly improving query efficiency.
[0076] Furthermore, it also includes: when compressing the log line to be compressed, determining the occurrence count of the variable parameter value; when a user initiates an aggregation request, determining aggregation information based on the variable parameter value in the compression database and the occurrence count of the variable parameter value.
[0077] Here, aggregation requests are used to perform statistical analysis on log lines, such as counting the number of occurrences of each error code in the past hour or counting the access frequency of user 123. The aggregation information is the statistical result corresponding to the aggregation request.
[0078] Here, during the compression stage, the values of all variable parameters and the frequency of their occurrence are extracted and stored. When a user initiates an aggregation request, it is only necessary to read the parameters in the compressed package for statistics and directly perform calculations such as summation, technical mean, and average.
[0079] For example, the aggregation request is to count the occurrence of each error code in the past hour. The calculation logic is to filter compressed blocks that match the time range, merge the error code frequency dictionaries of all blocks, and then sum them according to the error codes.
[0080] For example, the aggregation request is to count the access frequency of user 123. The calculation logic is to filter the compressed blocks with user ID 123 and accumulate the values of user ID 123 in all blocks.
[0081] In this invention, statistical aggregation analysis of specific parameters (such as the number of errors and the frequency of user access) is supported. The calculation is performed directly based on the parameter list in the compressed record, without the need to query from the original log or reconstruct the complete log.
[0082] Furthermore, it also includes: during the matching process between the first semantic vector and the log template vector, statistically analyzing the feature information of the log lines to be compressed corresponding to the first semantic vectors that failed to match; if the feature information does not meet the information threshold, clustering the log lines to be compressed that failed to match to obtain multiple second log line clusters; and updating the global template library based on each second log line cluster.
[0083] Here, the first semantic vector of a failed match means that no target log template corresponding to the log line to be compressed was found in the global template library, indicating that the log line to be compressed corresponds to a new log template.
[0084] Here, the feature information may include the number of log lines to be compressed or the storage time, and the threshold information may include a quantity threshold or a time threshold. When the number of log lines to be compressed and / or the storage time exceed the corresponding information threshold, the log lines to be compressed that fail to match are clustered to obtain multiple second log line clusters.
[0085] Here, the clustering method can be either DBSCAN or HDBSCAN.
[0086] Specifically, after obtaining multiple second log line clusters, the information entropy of words at the same position in all log lines within the second log line cluster is calculated. New templates are extracted based on the information entropy, and the global template library is updated based on the new templates.
[0087] Here, during the incremental learning process of updating the global template library, a resource limit can be set to avoid impacting the online compression service and protect resources.
[0088] For example, firstly, a global template library and a new pattern buffer are defined. The global template library is an existing template library that contains information such as template ID, template content, template vector, creation time, and number of logs. The new pattern buffer stores logs that do not match existing templates.
[0089] Algorithm for determining whether log lines to be compressed are added to the template library: (1) Generate semantic vectors using a pre-trained language model, calculate the cosine similarity between the log semantic vector and the template vector, and if the similarity is greater than or equal to the threshold (e.g., 0.9), it is determined that there is already a global template; otherwise, it is a new pattern buffer. (2) If the conditions are met, the incremental learning process is triggered, i.e., the global template library is updated: the buffer log records more than the set value, such as 1000; or the buffer log storage time reaches the specified value, such as 24 hours; (3) Incremental clustering generates new templates, deduplicatizes logs, and retains logs with unique masks; use DBSCAN for clustering (same as offline process) to generate new clusters; extract new log templates for each new cluster (using the location information entropy method). (4) The new template is merged into the global template library; (5) Update the template library and index.
[0090] The merged template library is persisted (e.g., to a database / file), and the template vector index is updated to improve subsequent matching speed.
[0091] It should be noted that in the log template lifecycle management, a recent access timestamp and access frequency are maintained for each log template, and old templates that have not been used for a long time are cleaned up regularly to control the size of the template library.
[0092] In this embodiment of the invention, by updating the global template library in real time, the accuracy and reliability of the global template library are improved, and the efficiency of compression and decompression is increased.
[0093] The following are application scenarios of the log compression method provided by this invention.
[0094] Taking the distributed system log processing of a certain e-commerce platform as an example, the specific implementation process is as follows: (1) Data collection and preprocessing: Collect approximately 1TB of raw log data from 50 application nodes over the past week; standardize the logs, for example, unify various time formats to “YYYY-MM-DD HH:MM:SS i.e. year-month-day hour-minute-second”.
[0095] (2) Offline template learning: The log lines were converted into 384-dimensional vectors using the language model (Sentence-BERT); DBSCAN clustering was performed with eps=0.5 and min_samples=10, resulting in 320 effective clusters and a small number of noise points; 320 log templates were extracted, such as template TID_042: "Order [] payment failed with error code []"; a global template library containing 320 entries was constructed.
[0096] (3) Online compression: After the real-time log stream is preprocessed, it is quickly matched with the template library. For example, a new log “Order 202310270001 payment failed with error code 5001” is matched with TID_042.
[0097] (4) Extract parameters [202310270001, 5001].
[0098] (5) Generate compressed record: [20231027143005] [TID_042][202310270001, 5001]. The original log is about 120 bytes, and the compressed record is about 40 bytes, with a compression ratio of 3:1.
[0099] (6) Query and analysis: Operation and maintenance personnel need to count the number of "payment failed" errors in the past hour. The system directly scans the records with template ID TID_042 in the compressed log and counts the number. The response time is in the millisecond range, and there is no need to decompress TB-level data.
[0100] (7) Dynamic update: After the system upgrade, a new error log "Transaction timeout after [*]ms" appears. After accumulating a certain number, incremental learning is triggered, a new template TID_321 is generated and added to the template library.
[0101] The log compression apparatus provided by the present invention is described below. The log compression apparatus described below and the log compression method described above can be referred to in correspondence.
[0102] Figure 2 This is a schematic diagram of the log compression device provided by the present invention, as shown below. Figure 2 As shown, the log compression device 200 includes the following: The first acquisition module 210 is used to acquire the first semantic vectors corresponding to multiple log lines to be compressed; The matching module 220 is used to match each of the first semantic vectors with each log template vector in the global template library to obtain a target log template that matches each of the first semantic vectors. Compression module 230 is used to compress each log line to be compressed based on each target log template and the variable parameter values in each log line to be compressed, to obtain a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
[0103] In this embodiment of the invention, the compression module 230 is specifically used to: extract the variable parameter value from each of the log lines to be compressed based on each of the target log templates; and compress each of the log lines to be compressed based on each of the variable parameter values and the template identifier corresponding to each of the target log templates to obtain a compressed database containing multiple compressed records.
[0104] In this embodiment of the invention, the first acquisition module 210 is specifically used to: perform masking processing on the variable parameter values in each of the log lines to be compressed to obtain the masked log lines to be compressed corresponding to each of the log lines to be compressed; and perform vectorization processing on each of the masked log lines to be compressed to obtain the first semantic vector corresponding to each of the log lines to be compressed.
[0105] In this embodiment of the invention, the log compression device 200 further includes a global template library determination module, specifically used for: obtaining second semantic vectors corresponding to multiple original log lines; performing clustering processing on each of the second semantic vectors to obtain multiple first log line clusters; calculating the information entropy corresponding to each word based on the frequency of occurrence of each word in the first log line cluster; determining the log template corresponding to each of the first log line clusters based on the information entropy; and constructing the global template library based on all the log templates.
[0106] In this embodiment of the invention, after obtaining the compressed database containing multiple compressed records, the log compression device 200 further includes a query module, specifically used for: querying the compressed record corresponding to the query condition from the compressed database based on the user's query condition; determining the log template corresponding to the template identifier from the global template library based on the template identifier in the compressed record; and filling the variable parameter bits in the log template with the parameter list in the compressed record to obtain the decompressed log line.
[0107] In this embodiment of the invention, the log compression device 200 further includes an aggregation module, specifically used for: determining the occurrence count of the variable parameter value when compressing the log line to be compressed; and determining aggregation information based on the variable parameter value in the compression database and the occurrence count of the variable parameter value when a user initiates an aggregation request.
[0108] In this embodiment of the invention, the log compression device 200 further includes an update module, specifically configured to: during the matching process between the first semantic vector and the log template vector, statistically analyze the feature information of the log lines to be compressed corresponding to the first semantic vectors that failed to match; if the feature information does not meet the information threshold, cluster the log lines to be compressed that failed to match to obtain multiple second log line clusters; and update the global template library based on each second log line cluster.
[0109] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 3 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a log compression method. This method includes: obtaining first semantic vectors corresponding to multiple log lines to be compressed; matching each first semantic vector with each log template vector in a global template library to obtain target log templates that match each first semantic vector; and compressing each log line to be compressed based on each target log template and the variable parameter values in each log line to be compressed to obtain a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with each log line to be compressed.
[0110] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log compression method provided by the above methods. The method includes: obtaining a first semantic vector corresponding to a plurality of log lines to be compressed; matching each of the first semantic vectors with each log template vector in a global template library to obtain a target log template that matches each of the first semantic vectors; and compressing each of the log lines to be compressed based on each of the target log templates and the variable parameter values in each of the log lines to be compressed to obtain a compressed database containing a plurality of compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the log compression method provided by the above methods. The method includes: obtaining first semantic vectors corresponding to a plurality of log lines to be compressed; matching each of the first semantic vectors with each log template vector in a global template library to obtain a target log template matching each of the first semantic vectors; and compressing each of the log lines to be compressed based on each of the target log templates and the variable parameter values in each of the log lines to be compressed to obtain a compressed database containing a plurality of compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A log compression method, characterized in that, include: Obtain the first semantic vectors corresponding to multiple log lines to be compressed; Each of the first semantic vectors is matched with each log template vector in the global template library to obtain the target log template that matches each of the first semantic vectors; Based on the target log templates and the variable parameter values in each log line to be compressed, each log line to be compressed is compressed to obtain a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
2. The log compression method according to claim 1, characterized in that, The step involves compressing each log line to be compressed based on the target log templates and the variable parameter values in each log line to be compressed, resulting in a compressed database containing multiple compressed records, including: Based on each of the target log templates, extract the variable parameter values from each of the log lines to be compressed; Based on the variable parameter values and the template identifiers corresponding to the target log templates, each log line to be compressed is compressed to obtain a compressed database containing multiple compressed records.
3. The log compression method according to claim 1 or 2, characterized in that, The step of obtaining the first semantic vector corresponding to multiple log lines to be compressed includes: The variable parameter values in each of the log lines to be compressed are masked to obtain the masked log lines to be compressed corresponding to each of the log lines to be compressed. The log lines to be compressed under each mask are vectorized to obtain the first semantic vector corresponding to each log line to be compressed.
4. The log compression method according to claim 1 or 2, characterized in that, Determining the global template library includes: Obtain the second semantic vectors corresponding to multiple original log lines; Clustering is performed on each of the second semantic vectors to obtain multiple first log line clusters; Based on the frequency of each word in the first log line cluster, calculate the information entropy corresponding to each word; Based on the information entropy, determine the log template corresponding to each of the first log line clusters; Based on all the aforementioned log templates, construct the global template library.
5. The log compression method according to claim 1 or 2, characterized in that, After obtaining the compressed database containing multiple compressed records, the method further includes: Based on the user's query criteria, retrieve the compressed record corresponding to the query criteria from the compressed database; Based on the template identifier in the compressed record, determine the log template corresponding to the template identifier from the global template library; The parameter list in the compressed record is filled into the variable parameter bits in the log template to obtain the decompressed log line.
6. The log compression method according to claim 1 or 2, characterized in that, Also includes: When compressing the log lines to be compressed, determine the number of times the variable parameter value appears; When a user initiates an aggregation request, aggregation information is determined based on the variable parameter values in the compressed database and the frequency of occurrence of those variable parameter values.
7. The log compression method according to claim 1 or 2, characterized in that, Also includes: During the matching process between the first semantic vector and the log template vector, the feature information of the log lines to be compressed corresponding to the first semantic vectors that failed to match is statistically analyzed. If the feature information does not meet the information threshold, the log lines that fail to match are clustered to obtain multiple second log line clusters; The global template library is updated based on each of the second log line clusters.
8. A log compression device, characterized in that, include: The first acquisition module is used to acquire the first semantic vectors corresponding to multiple log lines to be compressed; The matching module is used to match each of the first semantic vectors with each log template vector in the global template library to obtain the target log template that matches each of the first semantic vectors. The compression module is used to compress each of the log lines to be compressed based on the target log templates and the variable parameter values in each of the log lines to be compressed, to obtain a compressed database containing multiple compressed records, wherein each compressed record corresponds one-to-one with the log line to be compressed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the log compression method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the log compression method as described in any one of claims 1 to 7.