Log data compression method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a log data compression method and device, equipment, a storage medium and a program product, and is applied to the technical field of big data. The method comprises the following steps: acquiring an original log file, and preprocessing the original log file to obtain an intermediate log file; performing clustering processing on the intermediate log file to obtain a plurality of log data clusters corresponding to the intermediate log file; for each log data cluster, extracting a variable set of the log data cluster; determining a target compression mode corresponding to the log data cluster according to a variable type corresponding to the variable set; and performing compression processing on the log data cluster according to the target compression mode to obtain log compression data. Therefore, the electronic equipment adopts different compression modes for different variable types, so that the compression efficiency can be improved, and meanwhile, the compression effect can be optimized.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a log data compression method, device, equipment, storage medium and program product. Background Art

[0002] With the continuous development of Internet technology, the digital transformation of finance and other fields has gradually deepened. In each data center, various types of systems and applications will generate corresponding log files. Data centers usually need to store log files and mine and analyze logs to predict risks in advance or locate the cause of failures in a timely manner.

[0003] However, due to the complexity and diversity of equipment and software in the data center, there are a large number of log files, so the log files need to be compressed.

[0004] In the related art, the compression method of log files is usually to directly compress the log data based on a general compression algorithm. This compression method is not efficient and has a poor compression effect. Summary of the invention

[0005] The embodiments of the present application provide a log data compression method, apparatus, device, storage medium, and program product, which can improve the compression efficiency of log data and optimize the compression effect of log data.

[0006] In a first aspect, an embodiment of the present application provides a log data compression method, comprising:

[0007] Obtaining an original log file, and preprocessing the original log file to obtain an intermediate log file;

[0008] Performing clustering processing on the intermediate log files to obtain multiple log data clusters corresponding to the intermediate log files;

[0009] For each log data cluster, extract a variable set of the log data cluster;

[0010] Determining a target compression method corresponding to the log data cluster according to a variable type corresponding to the variable set;

[0011] The log data cluster is compressed according to the target compression method to obtain log compressed data.

[0012] In a possible implementation manner, preprocessing the original log file to obtain an intermediate log file includes:

[0013] The original time data in the original log file is format-converted to generate the intermediate log file; the intermediate log file includes the target time data in the target format.

[0014] In a possible implementation, clustering the intermediate log files to obtain a plurality of log data clusters corresponding to the intermediate log files includes:

[0015] Extracting text vectors corresponding to each of the intermediate log files, and determining text signatures corresponding to the intermediate log files according to the text vectors;

[0016] For any two intermediate log files, if the similarity between the two text signatures corresponding to the two intermediate log files meets the target condition, the two intermediate log files are added to the same log data cluster.

[0017] In a possible implementation, the method further includes:

[0018] Determine the log mode and index information corresponding to the log data cluster; the index information is used to associate the intermediate log files in the log data cluster;

[0019] The target condition corresponding to the log data cluster is adjusted according to the log mode.

[0020] In a possible implementation, determining the target compression mode corresponding to the log data cluster according to the variable type corresponding to the variable set includes:

[0021] If the variable type corresponding to the variable set is a digital type, determining that the target compression method is a digital compression method;

[0022] If the variable type corresponding to the variable set is a string type, determining the target compression method according to the length of the string;

[0023] If the variable type corresponding to the variable set is a special character type, the target compression method is determined to be a dictionary mapping compression method.

[0024] In a possible implementation, the method further includes:

[0025] Pre-decompressing the compressed log data to obtain pre-decompressed log data and index information;

[0026] The pre-decompressed log data is decompressed according to the index information and the target decompression method corresponding to the target compression method to restore the original log file.

[0027] In a second aspect, an embodiment of the present application provides a log data compression device, including:

[0028] An acquisition module is used to acquire an original log file and pre-process the original log file to obtain an intermediate log file;

[0029] A clustering module, used to perform clustering processing on the intermediate log files to obtain multiple log data clusters corresponding to the intermediate log files;

[0030] An extraction module, used for extracting a variable set of each log data cluster;

[0031] A determination module, used to determine a target compression method corresponding to the log data cluster according to a variable type corresponding to the variable set;

[0032] The compression module is used to compress the log data cluster according to the target compression method to obtain log compressed data.

[0033] In a possible implementation manner, the acquisition module is specifically configured to:

[0034] The original time data in the original log file is format-converted to generate the intermediate log file; the intermediate log file includes the target time data in the target format.

[0035] In a possible implementation, the clustering module is specifically used to:

[0036] Extracting text vectors corresponding to each of the intermediate log files, and determining text signatures corresponding to the intermediate log files according to the text vectors;

[0037] For any two intermediate log files, if the similarity between two text signatures corresponding to the two intermediate log files meets the target condition, the two intermediate log files are added to the same log data cluster.

[0038] In a possible implementation manner, the device is further used for:

[0039] Determine the log mode and index information corresponding to the log data cluster; the index information is used to associate the intermediate log files in the log data cluster;

[0040] The target condition corresponding to the log data cluster is adjusted according to the log mode.

[0041] In a possible implementation manner, the determining module is specifically configured to:

[0042] If the variable type corresponding to the variable set is a digital type, determining that the target compression method is a digital compression method;

[0043] If the variable type corresponding to the variable set is a string type, determining the target compression method according to the length of the string;

[0044] If the variable type corresponding to the variable set is a special character type, the target compression method is determined to be a dictionary mapping compression method.

[0045] In a possible implementation manner, the device is further used for:

[0046] Pre-decompressing the compressed log data to obtain pre-decompressed log data and index information;

[0047] The pre-decompressed log data is decompressed according to the index information and the target decompression method corresponding to the target compression method to restore the original log file.

[0048] In a third aspect, an embodiment of the present application provides a log data compression device, including: a memory, a processor;

[0049] The memory stores computer-executable instructions;

[0050] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the log data compression method as described in any one of the first aspects above.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the log data compression method as described in any one of the first aspects above.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the log data compression method as described in any one of the first aspects above.

[0053] The log data compression method, device, equipment, storage medium and program product provided in the embodiments of the present application obtain the original log file, and pre-process the original log file to obtain an intermediate log file; cluster the intermediate log file to obtain multiple log data clusters corresponding to the intermediate log file; for each log data cluster, extract the variable set of the log data cluster; determine the target compression method corresponding to the log data cluster according to the variable type corresponding to the variable set; compress the log data cluster according to the target compression method to obtain log compressed data. In the present application, the electronic device obtains the original log file, and after pre-processing and clustering, it can obtain multiple log data clusters, and then determines the corresponding target compression method according to the variable type of the variable set in the log data cluster, and compresses based on the target compression method to obtain log compressed data. In this way, the electronic device can improve the compression efficiency and optimize the compression effect by adopting different compression methods for different variable types. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0055] Figure 1 A flow chart of a log data compression method provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of another log data compression method provided in an embodiment of the present application;

[0057] Figure 3 A structural diagram of a log data compression device provided in an embodiment of the present application;

[0058] Figure 4 A structural schematic diagram of a log data compression device provided in an embodiment of the present application.

[0059] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present application, and are not intended to limit the present application.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation portals for users to choose to authorize or refuse. In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0062] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0063] It should be noted that the log data compression method, device, equipment, storage medium and program product of the present application can be used in the field of big data technology, and can also be used in any field outside the field of big data technology. The present application does not limit the specific application field of the log data compression method, device, equipment, storage medium and program product.

[0064] In data centers in the financial and other fields, almost all systems and applications generate log files. The log files can include time-stamped footprints, recorded behaviors, conditions, and events. By mining and analyzing logs, data centers can predict potential risks in their own operations in advance, and can also quickly locate and determine the cause of failures after failures occur. Timely discovery of abnormal logs can reduce the occurrence of failures and ensure smooth business operations.

[0065] However, there are tens of thousands of devices in the data center, and the installed software and applications are complex and diverse. Each device and each application has its own log print, which leads to the rapid expansion of log data in the data center, and the large-scale log data needs to be saved for a certain period of time. In order to reduce storage costs and improve storage efficiency, data centers usually need to compress log files.

[0066] In order to restore the original log information for log analysis and anomaly detection, data centers usually need to perform lossless compression on log files. In related technologies, data centers usually directly use general compression algorithms to compress log files. This log data compression method has low compression efficiency and poor compression effect. There is still room for improvement in compression efficiency and compression rate.

[0067] In order to solve the above problems, the log data compression method, device, equipment, storage medium and program product in the present application, the electronic device obtains the original log file, first pre-processes the original log file to obtain the intermediate log file corresponding to the original log file; then clusters the intermediate log file to obtain multiple log data clusters; then extracts the variable set in each log data cluster, and then determines the corresponding target compression method based on the type of the variable set, compresses the log data cluster based on the target compression method, and obtains the target compressed data. In this way, the electronic device can improve the compression efficiency by adopting different compression methods for different variable types on the basis of log clustering, and can also optimize the compression effect, which can save disk capacity and network bandwidth during transmission to a certain extent.

[0068] The general compression algorithm does not take into account the specific structure and application scenarios of the log. Using more efficient algorithm processing and compression storage design can optimize the compression rate and time, which helps save disk capacity and network bandwidth during transmission. General compression algorithms generally use the general characteristics of English characters or the characteristics of text structure. Although the log data can be compressed directly using a general compression program, there is still room for improvement in efficiency.

[0069] The scheme shown in the present application is described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.

[0070] Figure 1 A flow chart of a log data compression method provided in an embodiment of the present application. Figure 1 , the log data compression method may include:

[0071] S101: Obtain an original log file, and pre-process the original log file to obtain an intermediate log file.

[0072] The execution subject of the embodiment of the present application can be an electronic device, such as a mobile phone, a computer, a server, etc., and can be a log data compression system installed in the electronic device, or a log data compression device arranged in the electronic device. The log data compression device can be implemented by software, or by a combination of software and hardware. For ease of understanding, hereinafter, the execution subject is an electronic device as an example for explanation.

[0073] In the embodiment of the present application, the original log file may refer to the original log data that needs to be compressed. Preprocessing may refer to the preliminary processing of the original log file, which may specifically include extraction processing and format conversion of time data. The intermediate log file may refer to the intermediate log data obtained after preprocessing.

[0074] In this step, the electronic device obtains the original log file that needs to be compressed, and the electronic device can first pre-process the original log file to obtain an intermediate log file. Specifically, since the variable corresponding to the time part of the original log file is meaningless, the electronic device can extract the time data in the original log file and convert the format, and finally obtain a unified target format timestamp in the intermediate log file. The electronic device can then perform clustering processing on other log data in the intermediate log file except the timestamp, thereby improving the efficiency and rationality of the clustering processing.

[0075] Of course, the electronic device may also perform other types of preprocessing on the original log file, which may be specifically configured based on actual needs, and the embodiments of the present application are not limited to this.

[0076] S102: performing clustering processing on the intermediate log files to obtain a plurality of log data clusters corresponding to the intermediate log files.

[0077] In an embodiment of the present application, a log data cluster may refer to a collection of multiple intermediate log files with high similarity. Specifically, after obtaining the intermediate log files, the electronic device may perform clustering processing on the intermediate log files. Specifically, based on a log clustering algorithm (such as LogReduce), the intermediate log files with high similarity (for example, the similarity meets target conditions such as a threshold) may be clustered to ultimately obtain multiple log data clusters. Different log data clusters may correspond to different log modes, which may specifically refer to the structural form of the log, etc. In this way, in an embodiment of the present application, the electronic device obtains multiple log data clusters by performing clustering processing on the intermediate log files, which can achieve rapid classification processing of the log data and facilitate the subsequent use of different compression methods.

[0078] S103 . For each log data cluster, extract a variable set of the log data cluster.

[0079] In an embodiment of the present application, a variable set may refer to a set of variables included in a log data cluster. Specifically, after the electronic device performs clustering processing on the intermediate log file, for each log data cluster, the electronic device may extract each variable in the log data cluster by column through methods such as regular expressions or delimiter parsing, and obtain a variable set corresponding to the log data cluster. The variable set may specifically be a variable list, which may include variables such as a user ID (Identity document, ID), an operation type, a status code, and a resource name.

[0080] S104: Determine a target compression method corresponding to the log data cluster according to the variable type corresponding to the variable set.

[0081] In the embodiments of the present application, the variable type may refer to the specific type of the variable in the variable set, such as a numeric type, a string type, or a special character type. The numeric type may refer to a variable type including pure numbers; the string type may refer to a character variable type that can be converted into numbers; and the special character type may refer to a character variable type that cannot be converted into numbers. The target compression method may refer to a log data compression method corresponding to the variable type.

[0082] In this step, after the electronic device obtains the variable set of the log data cluster, it can determine the corresponding target compression method according to the different variable types of the variable set. Exemplarily, for variables of digital string type, the electronic device can determine the target compression method as a separate digital compression method, such as a digital compression algorithm, etc.; for character string type, the electronic device can use the corresponding target compression method according to the length of the variable to optimize the compression effect; and for special character types that cannot be directly converted to numbers, the electronic device can process them by creating a dictionary mapping method, and the corresponding dictionary compression method (such as a dictionary compression algorithm, etc.) can be used as the target compression method. In this way, the electronic device uses different compression algorithms for different variable types, which can make the compression process of the log data correspond to the characteristics of the variable, and can improve the compression efficiency and compression effect of the log data.

[0083] S105 . Compress the log data cluster according to the target compression method to obtain log compressed data.

[0084] In the embodiment of the present application, the log compression data may refer to the compressed data finally obtained after the original log file is compressed. After determining the target compression method, the electronic device may compress the log data cluster according to the target compression method to obtain the final log compression data. It should be noted that, since there may be multiple variable types in the variable set of a log data cluster, the log data cluster may correspond to multiple target compression methods. The electronic device may compress the variable type corresponding to the log data cluster according to the target compression method to obtain log compression data, thereby realizing fast and efficient compression processing of the log data.

[0085] The log data compression method provided by the embodiment of the present application is that the electronic device obtains the original log file, and pre-processes the original log file to obtain an intermediate log file; clusters the intermediate log file to obtain multiple log data clusters corresponding to the intermediate log file; extracts the variable set of the log data cluster for each log data cluster; determines the target compression method corresponding to the log data cluster according to the variable type corresponding to the variable set; compresses the log data cluster according to the target compression method to obtain log compressed data. In the present application, the electronic device obtains the original log file, and after pre-processing and clustering, it can obtain multiple log data clusters, and then determines the corresponding target compression method according to the variable type of the variable set in the log data cluster, and compresses based on the target compression method to obtain log compressed data. In this way, the electronic device can improve the compression efficiency and optimize the compression effect by adopting different compression methods for different variable types.

[0086] Based on the above embodiments, Figure 2 A flowchart of another log data compression method provided in an embodiment of the present application. Figure 2 , the log data compression method may include:

[0087] S201, obtaining an original log file; performing format conversion on original time data in the original log file to generate an intermediate log file; the intermediate log file includes target time data in a target format.

[0088] In the embodiment of the present application, the original time data may refer to the time part in the original log file. The target format may refer to a unified time data format. The target time data may refer to the time data after the format is unified.

[0089] Specifically, since the original log files include log files generated by different devices and different applications, and the format of the original time data is not unified, the electronic device can extract the original time data in the original log files, and perform format conversion on the original time data, converting the original time data into target time data in a target format, which can specifically be a timestamp in the target format, and finally obtain an intermediate log file including the target time data. In this way, the electronic device can improve the accuracy of the compression processing by converting and unifying the format of the original time data in the original log files.

[0090] S202: extracting text vectors corresponding to each intermediate log file, and determining text signatures corresponding to the intermediate log files according to the text vectors.

[0091] In the embodiment of the present application, the text vector may refer to the text feature vector corresponding to each intermediate log file. The text signature may refer to the unique text identifier corresponding to the intermediate log file, and the text signature may be used for similarity calculation in the subsequent clustering process.

[0092] Specifically, after obtaining the intermediate log file, the electronic device can first extract character items from the intermediate log file. The character items can specifically refer to characters or character combinations, such as words, phrases, etc.; then the electronic device can determine the weights corresponding to each character item based on the frequency, position and other statistical characteristics of the character item in the text. Afterwards, the electronic device can use a hash function to calculate the hash value corresponding to each character item based on the weight of the character item. The selection of the hash function should ensure that different character items are mapped to different hash values ​​as much as possible, while reducing hash conflicts. Then the electronic device can convert the hash values ​​corresponding to each character item into vectors, and aggregate the vectors corresponding to each character item to obtain the text vector corresponding to the intermediate log file.

[0093] Afterwards, the electronic device can perform binarization processing on the text vector to obtain the text signature corresponding to the intermediate log file. Specifically, for each element in the text vector, if the element is greater than 0, the element is converted to 1; if it is not greater than 0, the element is converted to 1, so that the text vector can be converted into a binary vector containing only 0 and 1, and the binary vector is the text signature of the intermediate log file. Due to the binarization process, the text signature is more compact and can achieve rapid calculation of similarity.

[0094] S203: For any two intermediate log files, if the similarity between the two text signatures corresponding to the two intermediate log files meets the target condition, the two intermediate log files are added to the same log data cluster.

[0095] In the embodiment of the present application, similarity may refer to the degree of similarity between two text signatures, which may be specifically represented by the Hamming distance. The Hamming distance refers to the number of different characters in the corresponding positions between the two text signatures. The smaller the Hamming distance, the more similar the two text signatures are; the larger the Hamming distance, the less similar the two text signatures are. The target condition may refer to the judgment condition that two intermediate log files belong to the same log data cluster, which may specifically refer to a threshold condition, for example, the Hamming distance between the two text signatures is less than the target threshold, etc.

[0096] In this step, after the electronic device obtains the text signature corresponding to each intermediate log file, for any two intermediate log files, the electronic device can determine the similarity of the two text signatures corresponding to the two intermediate log files. If the similarity meets the target condition, the electronic device can add the two intermediate log files to the same log data cluster. After the electronic device has completed the pairwise comparison of all intermediate log files, it can obtain multiple log data clusters to implement clustering processing for the intermediate log files.

[0097] In the embodiment of the present application, in the intermediate log file, the variable belongs to the dynamically changing part. In addition, there is a fixed log mode in the intermediate log file, and the log mode reflects the specific execution event. In a typical log file, the log recorded in each line has a fixed structure. The log file consists of lines, and the lines are composed of tags. Each line is separated by a line end mark (Token), and the end mark is separated by a space. There is a certain similarity between the logs of each line, such as the date, time, and network protocol address (Internet Protocol, IP) of adjacent line logs. And the log format of the same device or the log format of the same application, etc. On this basis, regardless of the timestamp, the similarity between the logs is mainly reflected in the log mode. The electronic device can extract the log mode to cluster the intermediate log file.

[0098] Specifically, the electronic device can compare the text similarity of the rest of the intermediate log file except the target time data, cluster the intermediate log files with the same log mode (i.e., the similarity of the text signature meets the target condition), and obtain multiple log data clusters, each log data cluster corresponds to a log mode, and each log mode only occupies one storage space, so as to avoid the situation where redundant information occupies storage space. At the same time, the variables in each intermediate log file can be associated through indexes and log modes.

[0099] In a possible implementation, during the clustering process of the intermediate log files, the log compression method may further include the following steps:

[0100] Determine the log mode and index information corresponding to the log data cluster; the index information is used to associate the intermediate log files in the log data cluster; and adjust the target condition corresponding to the log data cluster according to the log mode.

[0101] In the embodiment of the present application, the index information may refer to a data index corresponding to a log data cluster, which may be associated with an intermediate log file including a specific variable.

[0102] In the embodiment of the present application, since different intermediate log files have different characteristics, and the clustering processing algorithm also has uncertainty. If the electronic device directly uses the same target condition for clustering during the clustering process, the clustering result may be too rough or too detailed. On this basis, the electronic device can control the clustering accuracy to achieve better clustering results. The accuracy of the clustering process is related to the log mode. The electronic device can dynamically adjust the target condition according to the log mode of the log data cluster. Specifically, the target threshold of the Hamming distance in the log data cluster can be adjusted to achieve changes in the cluster size, optimize the clustering effect, and achieve fast and accurate clustering for large-scale intermediate log files.

[0103] In this step, during the clustering process of the intermediate log files, the electronic device can adjust the accuracy of the clustering process by dynamically adjusting the target conditions. Specifically, the electronic device can determine the log mode and index information corresponding to the log data cluster, and adjust the target conditions corresponding to the log data cluster according to the log mode.

[0104] When determining the log pattern of the log data cluster, the electronic device can use a pattern analysis algorithm such as a greedy algorithm to determine it. Specifically, the electronic device can read all the intermediate log files in the log data cluster to obtain all the terms (or words, phrases) that appear in the log data cluster; then the electronic device can traverse each intermediate log file to determine a set of candidate words; then the term with the highest frequency of all intermediate log files is used as the first word of the log pattern. If there are multiple terms with the same frequency, the term with the highest log coverage (i.e., the number of intermediate log files containing the term) is selected as the first word of the log pattern; then starting from the first term, the electronic device can add other candidate words in subsequent positions, and at the same time, based on the support of the log pattern in the intermediate log file as a judgment indicator, a longer log pattern is constructed, and finally the log pattern corresponding to the log data cluster is output.

[0105] Of course, the electronic device may also use other algorithms to implement the calculation of the log mode, which can be flexibly selected based on actual needs, and the embodiments of the present application are not limited to this.

[0106] S204: For each log data cluster, extract a variable set of the log data cluster.

[0107] S205 . Determine a target compression method corresponding to the log data cluster according to the variable type corresponding to the variable set.

[0108] In the embodiment of the present application, after clustering the intermediate log files, the electronic device can extract the variables in each log data cluster to obtain the variable set of the log data cluster. Then, the electronic device can determine the target compression method to be adopted when compressing the log data cluster based on the variable type of the variable set.

[0109] In a possible implementation, the target compression mode in step S205 may be determined by the following steps (1) to (3):

[0110] (1) If the variable type corresponding to the variable set is a digital type, the target compression method is determined to be a digital compression method.

[0111] In an embodiment of the present application, if the variable type in the variable set is a digital type, the electronic device may adopt a digital compression method, specifically a digital compression algorithm, etc., as a target compression method.

[0112] Taking the digital compression algorithm as an example, the digital compression algorithm adopted by the electronic device can be formed based on the combination of run-length coding, bit packing and delta coding, wherein run-length coding is used to process continuously repeated values; bit packing is used to reduce the number of bits required to store each value; delta coding is used to process the difference between consecutive values. In the process of run-length coding, different values ​​are stored and compressed by Delta coding and bit packing, and the matching count and the mismatching count are stored and compressed by bit packing. For different digital types, such as integer type (int) or long integer type (long), and whether the digital type is monotonous, the electronic device can adopt different digital compression algorithms, which can be specifically named according to the order of digital, encoding or decoding, and the order of digital type. For example, non-monotonic-encoding-integer type can be represented as unorder_encode_32, and non-monotonic-encoding-long integer type can be represented as unorder_encode_64. The target patterns of the log data cluster can themselves be connected to each other and compressed by the deflate algorithm (Deflate), and the length of each log pattern can be specifically compressed by the unorder_encode_32 algorithm. And, the log index corresponding to the log mode may be first sorted in ascending order and then compressed by a monotonic_encode_32 algorithm.

[0113] Of course, the electronic device may also use other types of digital compression algorithms to compress digital variables, and the embodiments of the present application are not limited to this.

[0114] (2) If the variable type corresponding to the variable set is a string type, the target compression method is determined according to the length of the string.

[0115] In the embodiment of the present application, since the conversion from a string to a number will result in the loss of variables in some cases (such as when the string is too long, etc.), for example, when the electronic device converts "0076" to an integer, it only obtains "76" and loses the first two characters. Therefore, if the variable type in the variable set is a string type that can be converted to a number, the electronic device can first determine the length of the string. When the length of the string variable is greater than a preset length threshold, the electronic device needs to store the position and length of the string variable, and compress the string variable in combination with a digital compression method, so that it can be accurately restored during decompression; when the length of the string variable is not greater than the preset length threshold, the electronic device can directly use the digital compression method as the target compression method.

[0116] (3) If the variable type corresponding to the variable set is a special character type, the target compression method is determined to be a dictionary mapping compression method.

[0117] In an embodiment of the present application, if the variable type in the variable set is a special character type that cannot be converted into a number, the electronic device can create a dictionary to map the special characters to numbers, and then use the dictionary mapping compression method as the target compression method.

[0118] It should be noted that for special character type variables that cannot be converted into numbers, the electronic device can create a dictionary to map the special character variables to digital variables. However, when using the dictionary to map the string, it is necessary to monitor the ratio of the number of special character type variables to the total number of variables in the variable set. If the ratio is greater than a preset ratio threshold (e.g., 20%), the dictionary creation efficiency is low, and compared with direct compression, the compression rate is not good. At this time, other compression algorithms can be used. For example, when the proportion of special character type variables exceeds the preset ratio threshold, a general compression algorithm can be directly used as the target compression method to compress the special character type variables. Of course, other compression algorithms can also be used as the target compression method, and the embodiments of the present application do not limit this. When the proportion of special character type variables is not greater than the preset ratio threshold, the electronic device can use a dictionary mapping compression algorithm as the target compression method.

[0119] In an embodiment of the present application, the electronic device, based on log clustering, adopts different compression algorithms as target compression methods according to the variable types of the variable set in the log data cluster, which can improve the efficiency of log data compression, shorten the compression time of log data, and also optimize the compression effect of log data.

[0120] S206 , compressing the log data cluster according to the target compression method to obtain log compressed data.

[0121] In the embodiment of the present application, after determining the target compression method, the electronic device can compress the log data cluster according to the target compression method to obtain final log compression data.

[0122] S207: Pre-decompress the compressed log data to obtain pre-decompressed log data and index information.

[0123] S208: Decompress the pre-decompressed log data according to the index information and the target decompression method corresponding to the target compression method, and restore the original log file.

[0124] In the embodiment of the present application, after the electronic device obtains the log compression data, during the log recovery process, the electronic device can determine the target decompression method corresponding to the target compression method, decompress the log compression data, and restore the original log file. In this step, during the log recovery process, the electronic device can first perform a file pre-decompression process on the log compression data according to the target compression method to obtain pre-decompressed log data and index information; then decompress the pre-decompression result according to the target decompression method combined with the index information to obtain the original log file.

[0125] Since log information such as log patterns, variable sets, and index information are compressed by different algorithms and then stored in the same file as log compressed data, the electronic device needs to determine the position of each part during the decompression process. The pre-decompression process is used to obtain the relative position of each part, pre-decompress the file, and determine the relevant log information for decompression for subsequent log recovery. Specifically, the electronic device can decompress the target storage location (such as the last 4 bytes) according to the storage format of the compression structure in the log compressed data to obtain the encoding size of each data block, such as the log pattern, index information, and other related information. For example, the electronic device can determine the various log patterns in the log compressed data according to the encoding size, and decompress the index information corresponding to each log pattern according to the encoding size of the index information. After that, the electronic device can restore the pre-decompressed log data according to the target decompression method based on the log pattern and index information.

[0126] During the restoration process, the electronic device can decompress the tag information, which records whether the variable type is a string type, decompress the position of the first part of the numerical variable that is zero and the tag information of the variable length, determine the position and length of the string variable, and subsequently use it to restore the string type variable; then the electronic device can decompress the list of the number of variables in the storage mode and the tag information whether the variable is a digital type, which can subsequently be used to restore the digital type variable; decompress the dictionary tag information that encodes the special character type variable, which can subsequently be used to restore the special character type variable.

[0127] Based on the above process, the electronic device can construct an unordered mapping table, whose key is the index information and the value is the corresponding log file. In the initial state, the electronic device needs to determine the number of variables corresponding to each log mode. If the number of variables is zero, that is, there are no variables in the log at this time, the log mode and the log content are the same, and the log mode can be filled into the unordered mapping table according to the index information corresponding to the log mode (i.e., the index number).

[0128] Then, for the case where the number of variables is not zero, the electronic device needs to traverse the log mode. Since there is no difference between the log mode and the log wildcard, the actual variables can be added to each log corresponding to the log mode. Exemplarily, if part is a variable name (or variable flag), the electronic device needs to append the variable part to the log corresponding to the log mode. That is, the electronic device needs to repeatedly add the common components and actual variables (names) of the log mode until the entire log mode is traversed.

[0129] For the variable part of the log (the variable flag part, i.e., the actual variable value), the electronic device first determines the variable type at the corresponding position according to the tag information obtained by decompression for each log to be reconstructed. For digital types, if the type is an integer type, the target decompression method non-monotonic-encoding-integer type (unorder_decode_32) corresponding to the target compression method non-monotonic-encoding-integer type (unorder_encode_32) is used to decompress the integer type variable; similarly, if the specific type of the digital type is a long integer type, the electronic device can use the target decompression method non-monotonic-encoding-long integer type (unorder_decode_64) algorithm corresponding to the target compression method non-monotonic-encoding-integer type (unorder_encode_64) to decompress the long integer type.

[0130] For special character types, electronic devices can compress using the dictionary mapping decoding algorithm corresponding to the dictionary mapping compression method. Specifically, they can first use unorder_decode_32 to decompress, and then use the dictionary to map to special characters; for string types, electronic devices can decompress according to the Deflate algorithm. Finally, electronic devices can separate log files according to variable lengths and perform variable checks to ensure the accuracy of restoration. During the log recovery process, the space complexity of the entire log recovery process is O(n), and the time complexity is also O(n).

[0131] It should be noted that, by default, the electronic device can use the Deflate algorithm as a dictionary compression mapping method, but other general compression algorithms can also be used. In addition, the electronic device can use the PPMd compression algorithm (Prediction by Partial Matching, PPMd) for string compression. In this way, in the embodiment of the present application, the electronic device adopts different target compression methods based on log clustering and the variable type of the variable set in the log data cluster. Compared with the method of directly compressing log data based on a general compression algorithm in the related art, the compression efficiency can be improved and the compression efficiency can be optimized.

[0132] Figure 3 This is a schematic diagram of the structure of a log data compression device provided in an embodiment of the present application. Figure 3 , the log data compression device 30 may include:

[0133] The acquisition module 31 is used to acquire the original log file and pre-process the original log file to obtain an intermediate log file;

[0134] A clustering module 32, used to perform clustering processing on the intermediate log files to obtain multiple log data clusters corresponding to the intermediate log files;

[0135] An extraction module 33, for extracting a variable set of the log data cluster for each log data cluster;

[0136] A determination module 34, configured to determine a target compression method corresponding to the log data cluster according to a variable type corresponding to the variable set;

[0137] The compression module 35 is used to compress the log data cluster according to the target compression method to obtain log compressed data.

[0138] In a possible implementation, the acquisition module 31 is specifically configured to:

[0139] The format of the original time data in the original log file is converted to generate an intermediate log file; the intermediate log file includes the target time data in the target format.

[0140] In a possible implementation, the clustering module 32 is specifically configured to:

[0141] Extracting text vectors corresponding to each intermediate log file, and determining text signatures corresponding to the intermediate log files according to the text vectors;

[0142] For any two intermediate log files, if the similarity of the two text signatures corresponding to the two intermediate log files meets the target condition, the two intermediate log files are added to the same log data cluster.

[0143] In a possible implementation manner, the device 30 is further used for:

[0144] Determine the log mode and index information corresponding to the log data cluster; the index information is used to associate the intermediate log files in the log data cluster;

[0145] Adjust the target conditions corresponding to the log data cluster according to the log pattern.

[0146] In a possible implementation, the determination module 34 is specifically configured to:

[0147] If the variable type corresponding to the variable set is a digital type, the target compression method is determined to be a digital compression method;

[0148] If the variable type corresponding to the variable set is a string type, the target compression method is determined according to the length of the string;

[0149] If the variable type corresponding to the variable set is a special character type, the target compression method is determined to be a dictionary mapping compression method.

[0150] In a possible implementation manner, the device 30 is further used for:

[0151] Pre-decompressing the compressed log data to obtain pre-decompressed log data and index information;

[0152] According to the index information and the target decompression method corresponding to the target compression method, the pre-decompressed log data is decompressed to restore the original log file.

[0153] The log data compression device 30 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0154] Figure 4 This is a structural diagram of a log data compression device provided in an embodiment of the present application. Figure 4The log data compression device 40 may include: a memory 41 and a processor 42. Exemplarily, the memory 41 and the processor 42 are connected to each other via a bus 43.

[0155] The memory 41 is used to store program instructions;

[0156] The processor 42 is used to execute the program instructions stored in the memory to implement the log data compression method shown in the above embodiment.

[0157] Figure 4 The log data compression device 40 shown can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0158] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned log data compression method.

[0159] The embodiment of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, may implement the above-mentioned log data compression method.

[0160] It should be noted that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0161] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DR RAM). It should be noted that when the processor is a general purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor. It should be noted that the memory described herein is intended to include but is not limited to these and any other suitable types of memory.

[0162] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0163] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] Regarding the various modules / units contained in the various devices and products described in the above embodiments, they may be software modules / units, or hardware modules / units, or they may be partially software modules / units and partially hardware modules / units. Each device and product may be applied to or integrated in a chip, a chip module or a terminal device. Exemplarily, for each device and product applied to or integrated in a chip, each module / chip contained therein may be implemented in the form of hardware such as circuits, or at least some modules / units may be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining modules / units may be implemented in the form of hardware such as circuits.

[0167] In the present application, the term "include" and its variations may refer to non-restrictive inclusion; the term "or" and its variations may refer to "and / or". The terms "first", "second", etc. in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In the present application, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0168] The above are only some embodiments of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A log data compression method, characterized in that: include: Obtaining an original log file, and preprocessing the original log file to obtain an intermediate log file; Performing clustering processing on the intermediate log files to obtain multiple log data clusters corresponding to the intermediate log files; For each log data cluster, extract a variable set of the log data cluster; Determining a target compression method corresponding to the log data cluster according to a variable type corresponding to the variable set; The log data cluster is compressed according to the target compression method to obtain log compressed data.

2. The method according to claim 1, characterized in that The preprocessing of the original log file to obtain an intermediate log file includes: The original time data in the original log file is format-converted to generate the intermediate log file; the intermediate log file includes the target time data in the target format.

3. The method according to claim 1, characterized in that The clustering process of the intermediate log files to obtain a plurality of log data clusters corresponding to the intermediate log files includes: Extracting text vectors corresponding to each of the intermediate log files, and determining text signatures corresponding to the intermediate log files according to the text vectors; For any two intermediate log files, if the similarity between the two text signatures corresponding to the two intermediate log files meets the target condition, the two intermediate log files are added to the same log data cluster.

4. The method according to claim 3, characterized in that: The method further comprises: Determine the log mode and index information corresponding to the log data cluster; the index information is used to associate the intermediate log files in the log data cluster; The target condition corresponding to the log data cluster is adjusted according to the log mode.

5. The method according to claim 1, characterized in that The determining, according to the variable type corresponding to the variable set, a target compression method corresponding to the log data cluster includes: If the variable type corresponding to the variable set is a digital type, determining that the target compression method is a digital compression method; If the variable type corresponding to the variable set is a string type, determining the target compression method according to the length of the string; If the variable type corresponding to the variable set is a special character type, the target compression method is determined to be a dictionary mapping compression method.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Pre-decompressing the compressed log data to obtain pre-decompressed log data and index information; The pre-decompressed log data is decompressed according to the index information and the target decompression method corresponding to the target compression method to restore the original log file.

7. A log data compression device, characterized in that: include: An acquisition module is used to acquire an original log file and pre-process the original log file to obtain an intermediate log file; A clustering module, used to perform clustering processing on the intermediate log files to obtain multiple log data clusters corresponding to the intermediate log files; An extraction module, used for extracting a variable set of each log data cluster; A determination module, used to determine a target compression method corresponding to the log data cluster according to a variable type corresponding to the variable set; The compression module is used to compress the log data cluster according to the target compression method to obtain log compressed data.

8. A log data compression device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the log data compression method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the log data compression method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the log data compression method according to any one of claims 1 to 6 is implemented.

Images