Log compression, log restoration method and device, computer equipment and storage medium

Abstract

This application relates to a log compression and log restoration method, apparatus, computer device, storage medium, and computer program product. Embodiments of this invention can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving. The method includes: using candidate log templates that successfully match the string arrangement information of template strings with the string arrangement information of the target log as initial log templates; from the log strings of the target log, using log strings that match the variable class template strings in the initial log template as first initial strings, and log strings that match the constant class template strings in the initial log template as second initial strings; generating log compression data based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string, and the first positional relationship between the first and second initial strings in the target log, thereby improving the log compression rate.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log compression and log restoration method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, logs have emerged. Logs are files used to record system or application operation events and play an important role in processing historical data, locating problems, and understanding system activities.

[0003] Traditional technologies typically store or transmit raw logs directly. However, the volume of logs is usually quite large, and current log processing methods have low compression rates, increasing storage and transmission pressure. Summary of the Invention

[0004] Therefore, it is necessary to provide a log compression and log restoration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the log compression rate in response to the above-mentioned technical problems.

[0005] This application provides a log compression method. The method includes:

[0006] Obtain the target log;

[0007] The log string arrangement information corresponding to the target log is matched with the template string arrangement information corresponding to the candidate log template, and the successfully matched candidate log template is used as the initial log template.

[0008] From the log strings of the target log, the log string that matches the variable class template string in the initial log template is taken as the first initial string, and the log string that matches the constant class template string in the initial log template is taken as the second initial string;

[0009] Obtain the first encoded data corresponding to each initial string;

[0010] Based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log, log compressed data corresponding to the target log is generated.

[0011] This application also provides a log compression apparatus. The apparatus includes:

[0012] The log acquisition module is used to acquire target logs;

[0013] The log template determination module is used to match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and use the successfully matched candidate log template as the initial log template.

[0014] The string classification module is used to select the log string that matches the variable class template string in the initial log template as the first initial string and the log string that matches the constant class template string in the initial log template as the second initial string from the log strings of the target log.

[0015] The encoded data acquisition module is used to acquire the first encoded data corresponding to each first initial string;

[0016] The log compression data generation module is used to generate log compression data corresponding to the target log based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the log compression method above.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the log compression method above.

[0019] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the log compression method above.

[0020] The aforementioned log compression method, apparatus, computer equipment, storage medium, and computer program product acquire a target log; match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to candidate log templates, and use the successfully matched candidate log templates as initial log templates; from the various log strings in the target log, select the log string that matches the variable class template string in the initial log template as the first initial string, and the log string that matches the constant class template string in the initial log template as the second initial string; acquire the first encoded data corresponding to each first initial string; and generate log compressed data corresponding to the target log based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first and second initial strings in the target log. In this way, by representing the second initial string in the target log with the template identifier corresponding to the initial log template determined based on the string arrangement information, log data can be effectively compressed, improving the log compression rate. Converting a target log composed of a large number of strings into log compressed data composed of template identifiers, first encoded data, and first positional relationships can effectively reduce the amount of log data and improve the log compression rate.

[0021] This application provides a log restoration method. The method includes:

[0022] Obtain the compressed log data corresponding to the target log; the compressed log data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable class template string in the initial log template, and the second initial string is the log string in the target log that matches the constant class template string in the initial log template.

[0023] Obtain the initial log template corresponding to the template identifier, and obtain the log string corresponding to the first encoded data;

[0024] Based on the first positional relationship, the variable class template string in the initial log template is replaced with the corresponding log string to obtain the target log.

[0025] This application also provides a log restoration apparatus. The apparatus includes:

[0026] The log compression data acquisition module is used to acquire the log compression data corresponding to the target log. The log compression data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable class template string in the initial log template, and the second initial string is the log string in the target log that matches the constant class template string in the initial log template.

[0027] The log composition data acquisition module is used to acquire the initial log template corresponding to the template identifier and to acquire the log string corresponding to the first encoded data.

[0028] The log restoration module is used to replace the variable class template string in the initial log template with the corresponding log string based on the first positional relationship, so as to obtain the target log.

[0029] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the log restoration method above.

[0030] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the log restoration method above.

[0031] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the log restoration method above.

[0032] The aforementioned log restoration method, apparatus, computer equipment, storage medium, and computer program product obtain compressed log data corresponding to the target log. This compressed log data is generated based on a template identifier corresponding to an initial log template, first encoded data corresponding to each first initial string in the target log, and a first positional relationship between the first and second initial strings in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to candidate log templates, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable-type template string in the initial log template, and the second initial string is the log string in the target log that matches the constant-type template string in the initial log template. The process involves obtaining the initial log template corresponding to the template identifier, obtaining the log string corresponding to the first encoded data, and replacing the variable-type template string in the initial log template with the corresponding log string based on the first positional relationship to obtain the target log. In this way, the compressed log data is generated based on the template identifier corresponding to the initial log template, the target encoded data corresponding to each log string belonging to the variable type in the target log, and the positional relationship between the variable type strings and constant type strings in the target log. Representing the constant strings in the target log using the template identifier corresponding to the initial log template determined by the arrangement information of the constant type strings effectively compresses log data and improves the log compression ratio. Converting the target log, composed of a large number of strings, into compressed log data composed of template identifiers, target encoded data, and positional relationships effectively reduces the amount of log data and improves the log compression ratio. During log restoration, it is only necessary to obtain the corresponding log template based on the template identifier in the compressed log data, obtain the corresponding log string based on the target encoded data in the compressed log data, and quickly and accurately restore the original log based on the log template corresponding to the template identifier, the log string corresponding to the target encoded data, and the positional relationship in the compressed log data. Attached Figure Description

[0033] Figure 1 This is an application environment diagram of the log compression method and log restoration method in one embodiment;

[0034] Figure 2 This is a flowchart illustrating a log compression method in one embodiment;

[0035] Figure 3 This is a flowchart illustrating the log compression method in another embodiment;

[0036] Figure 4 This is a flowchart illustrating the log compression method in yet another embodiment;

[0037] Figure 5 This is a schematic diagram of a data block storage method in one embodiment;

[0038] Figure 6 This is a flowchart illustrating a log restoration method in one embodiment;

[0039] Figure 7 This is a schematic diagram of the architecture of a log compression method in one embodiment;

[0040] Figure 8 This is a structural block diagram of a log compression device in one embodiment;

[0041] Figure 9 This is a structural block diagram of a log restoration device in one embodiment;

[0042] Figure 10 This is an internal structural diagram of a computer device in one embodiment;

[0043] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0046] The solutions provided in this application involve technologies such as natural language processing in artificial intelligence, and are specifically illustrated through the following embodiments:

[0047] The log compression and log restoration methods provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0048] Both the terminal and the server can be used independently to execute the log compression method and log restoration method provided in the embodiments of this application.

[0049] For example, the server retrieves the target log locally, matches the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log templates, and uses the successfully matched candidate log templates as the initial log templates. From the various log strings in the target log, the server selects the log strings that match the variable-type template strings in the initial log template as the first initial string, and the log strings that match the constant-type template strings in the initial log template as the second initial string. The server retrieves the first encoded data corresponding to each first initial string, and based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first and second initial strings in the target log, generates the compressed log data corresponding to the target log. The server can store the compressed log data locally or transmit it to other devices.

[0050] The server retrieves compressed log data corresponding to the target log locally. This compressed data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first and second initial strings in the target log. The server retrieves the initial log template corresponding to the template identifier, obtains the log strings corresponding to the first encoded data, and, based on the first positional relationship, replaces the variable-type strings in the initial log template with the corresponding log strings to obtain the target log. The server can perform data analysis on the restored target log or transmit it to other devices.

[0051] The terminal and server can also work together to execute the log compression method and log restoration method provided in the embodiments of this application.

[0052] For example, the server retrieves the target log from the terminal. The server matches the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log templates, and uses the successfully matching candidate log templates as the initial log templates. From the various log strings in the target log, the server selects the log string that matches the variable class template string in the initial log template as the first initial string, and the log string that matches the constant class template string in the initial log template as the second initial string. The server obtains the first encoded data corresponding to each first initial string, and based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first and second initial strings in the target log, generates compressed log data corresponding to the target log. The server can store the compressed log data locally or transmit it to other devices.

[0053] The terminal sends a log retrieval request to the server, carrying the log identifier corresponding to the target log. Upon receiving the request, the server retrieves the compressed log data corresponding to the target log. This compressed data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first and second initial strings in the target log. The server retrieves the initial log template corresponding to the template identifier, obtains the log strings corresponding to the first encoded data, and, based on the first positional relationship, replaces the variable-type strings in the initial log template with the corresponding log strings to obtain the target log. The server can then send the restored target log to the terminal for viewing or data analysis.

[0054] In one embodiment, such as Figure 2 As shown, a log compression method is provided. Taking the application of this method to a computer device as an example, the computer device can be the one described above. Figure 1 Terminal 102 or server 104 in the reference. Figure 2 The log compression method includes the following steps:

[0055] Step S202: Obtain the target log.

[0056] Step S204: Match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and use the successfully matched candidate log template as the initial log template.

[0057] In this context, "target log" refers to any log file. Logs are files that store records of messages generated by the operating system or applications. Logs record detailed information about the operation of various services within the operating system or applications, playing a crucial role in processing historical data, locating problems, and understanding the activities of the operating system or applications. Logs include, but are not limited to, application logs, security logs, and system logs.

[0058] Log string arrangement information refers to the arrangement of strings corresponding to the log, used to indicate the order of each log string. Template string arrangement information refers to the arrangement of strings corresponding to the log template, used to indicate the order of each template string within the log template.

[0059] A string is a sequence of characters consisting of at least one character. Characters in a string include at least one type of character such as a number, letter, or symbol. A string can also be called a token. A log string refers to the string within a log entry; a log entry can be segmented to obtain its individual log strings. Various methods can be used for log segmentation. A template string refers to the string within a log template, which is determined when the log template is generated.

[0060] A log template is a template used to represent the general format of a type of log. A log template includes a constant section and a variable section. The constant section is represented by a constant-class template string, and the variable section is represented by a variable-class template string. The constant section represents the common parts of the logs corresponding to the log template; that is, the constant-class template string can represent the common strings in the logs corresponding to the log template. The variable section represents the different parts of the logs corresponding to the log template; that is, the variable-class template string can represent the different strings in the logs corresponding to the log template.

[0061] Log templates can be generated based on logs. When generating a log template, the constant portion can be derived from high-frequency strings in the log, and the variable portion from low-frequency strings. High-frequency strings can be used as the constant portion of the log template, while low-frequency strings between adjacent high-frequency strings can be represented by preset strings, which are then used as the variable portion. All variable portions of the log template can be represented by the same preset string or different preset strings. The preset strings can be any string.

[0062] In one embodiment, all low-frequency strings between adjacent high-frequency strings can be represented by a preset string, that is, a preset string can refer to at least one log string. For example, a log template can be represented as "token1+*+token4+*+token6+token10", where token1, token4, token6, and token10 represent constant-type template strings, and "*" represents variable-type template strings. The "*" between token1 and token4 indicates that within a certain type of log corresponding to this log template, there is a log string between token1 and token4. It can be understood that the number of log strings between token1 and token4 can be the same or different in the various logs corresponding to this log template. For example, in log A corresponding to this log template, there is one log string between token1 and token4; in log B corresponding to this log template, there are three log strings between token1 and token4.

[0063] It is understandable that all low-frequency strings between adjacent high-frequency strings can also be represented by at least two preset strings. For example, if the number of low-frequency strings between adjacent high-frequency strings is odd, then one preset string is used to represent all low-frequency strings between adjacent high-frequency strings; if the number of low-frequency strings between adjacent high-frequency strings is even, then two preset strings are used to represent all low-frequency strings between adjacent high-frequency strings.

[0064] Of course, all low-frequency strings before the starting high-frequency string (i.e., the first high-frequency string) can also be represented by a preset string, and all low-frequency strings after the ending high-frequency string (i.e., the last high-frequency string) can also be represented by a preset string.

[0065] Candidate log templates refer to log templates that are proposed for matching. Candidate log templates are existing log templates. Initial log templates refer to candidate log templates that have been determined to match the target log based on string arrangement information.

[0066] Specifically, the computer device can obtain the target log locally or from other devices, perform word segmentation on the target log to obtain the corresponding log strings and log string arrangement information. The computer device can obtain at least one candidate log template locally or from other devices, and the candidate log template has corresponding template string arrangement information. The computer device can match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template. If the constant part of the candidate log template appears sequentially in the target log, and there is a log string in the target log that matches the variable part of the candidate log template, then it is determined that the target log and the candidate log template are successfully matched, and the successfully matched candidate log template is used as the initial log template.

[0067] In one embodiment, the computer device can obtain a preset delimiter and perform word segmentation on the target log based on the preset delimiter, dividing the target log into multiple log strings. The preset delimiter refers to a pre-set, pre-specified delimiter used for word segmentation of the log. For example, commas, semicolons, and spaces can be used as preset delimiters.

[0068] In one embodiment, a preset delimiter can be used as a preset string in the log template. Preset delimiters are typically meaningless strings; using a preset delimiter as a preset string avoids confusion between constant-type template strings and variable-type template strings.

[0069] In one embodiment, if there are at least two successfully matched candidate log templates, one of the successfully matched candidate log templates can be randomly selected for log compression, or the candidate log template with the largest number of template strings can be selected from all the successfully matched candidate log templates for log compression.

[0070] Step S206: From the log strings of the target log, the log string that matches the variable class template string in the initial log template is taken as the first initial string, and the log string that matches the constant class template string in the initial log template is taken as the second initial string.

[0071] Specifically, after determining the initial log template, the computer device can use the log strings in the target log that match the variable class template string in the initial log template as the first initial string, and the log strings in the target log that match the constant class template string in the initial log template as the second initial string. For example, the log strings in the target log that match the variable class template string can be used as the second initial string, and the log strings between adjacent second initial strings can be used as the first initial string.

[0072] Step S208: Obtain the first encoded data corresponding to each first initial string.

[0073] Here, the encoded data corresponding to the string refers to the data obtained after encoding the string. Different strings correspond to different encoded data. The first encoded data refers to the encoded data corresponding to the first initial string in the target log.

[0074] Specifically, after determining the first initial string, the computer device can obtain the first encoded data corresponding to each first initial string in the target log. The first encoded data is used to replace the corresponding first initial string in the target log when compressing the target log.

[0075] In one embodiment, the encoded data corresponding to each string can be pre-set, so that the computer device can obtain the first encoded data corresponding to the first initial string from the pre-set encoded data. Of course, the computer device can also encode the first initial string in the target log in real time to obtain the corresponding first encoded data.

[0076] In one embodiment, the string can be encrypted to obtain the corresponding encoded data, or a custom algorithm or formula can be used to encode the string to obtain the corresponding encoded data.

[0077] In one embodiment, the amount of data in the first encoded data corresponding to the first initial string is less than the amount of data in the first initial string. For example, the first initial string consists of six characters, and the target encoded data corresponding to the first initial string consists of three strings. Thus, when compressing the target log, replacing the original log string with the smaller first encoded data can effectively reduce the amount of log data and improve the log compression rate. In one embodiment, if the amount of data in the first initial string is less than or equal to a preset value, then the first initial string is used as the encoded data corresponding to the first initial string; if the amount of data in the first initial string is greater than the preset value, then the first initial string is encoded to obtain the encoded data corresponding to the first initial string.

[0078] Step S210: Based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log, generate the log compressed data corresponding to the target log.

[0079] The template identifier is an identifier used to uniquely identify a log template, and can specifically include a string of at least one character such as letters, numbers, or symbols.

[0080] The first positional relationship refers to the positional relationship between the first initial string and the second initial string in the target log. It is used to indicate whether there is a second initial string between adjacent first initial strings and whether there is a first initial string between adjacent second initial strings.

[0081] Specifically, after determining the initial log template, the computer device can generate compressed log data corresponding to the target log based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first and second initial strings in the target log. For example, the template identifier, each first encoded data, and the first positional relationship can be combined to form compressed log data. In this way, log compression of the target log is achieved through a series of data processing steps, and the original target log can be compressed into compressed log data. It can be understood that representing all the second initial strings in the target log with a single template identifier makes the size of the compressed log data smaller than the size of the target log. Furthermore, representing all the second initial strings in the target log with a single template identifier and representing the first initial strings in the target log with target encoded data also helps protect the security of the log strings in the target log; even if a suspicious device obtains the compressed log data, it will be unable to reconstruct the target log.

[0082] In the above log compression method, the following steps are taken: First, the target log is obtained. Then, the log string arrangement information corresponding to the target log is matched with the template string arrangement information corresponding to the candidate log templates. The successfully matched candidate log templates are used as the initial log templates. From the various log strings in the target log, log strings matching the variable class template strings in the initial log template are used as the first initial strings, and log strings matching the constant class template strings in the initial log template are used as the second initial strings. First encoded data corresponding to each first initial string is obtained. Based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first and second initial strings in the target log, compressed log data corresponding to the target log is generated. In this way, by representing the second initial string in the target log using the template identifier corresponding to the initial log template determined based on the string arrangement information, log data can be effectively compressed, improving the log compression rate. Converting the target log, which consists of a large number of strings, into compressed log data composed of template identifiers, first encoded data, and first positional relationships can effectively reduce the amount of log data and improve the log compression rate.

[0083] In one embodiment, step S204 includes:

[0084] When the target log contains all the constant class template strings in the candidate log template, and the constant class template strings appear in the same order in the target log and the candidate log template, and there is a log string in the target log that matches the variable class template string, the target log and the candidate log template are considered to be successfully matched.

[0085] Specifically, when performing template matching based on string arrangement information, if the target log contains each constant class template string in the candidate log template, and the constant class template strings appear in the same order in the target log and the candidate log template, and there is a log string in the target log that matches the variable class template string, it indicates that the target log has the log format indicated by the candidate log template, and then it is determined that the target log and the candidate log template match successfully.

[0086] For example, the target log consists of token1, token2, token3, token4, token5, and token6, and is represented as "token1+token2+token3+token4+token5+token6". The candidate log template is represented as "token1+*+token4+*+token6", where token1, token4, and token6 represent constant-type template strings, and "*" represents variable-type template strings. Similar to the candidate log template, the target log also appears in the order of token1, then token4, then token6, with other log strings appearing between token1 and token4, and between token4 and token6. Token2 and token3 can be considered to match the "*" between token1 and token4 in the candidate log template, and token5 can be considered to match the "*" between token4 and token6 in the candidate log template.

[0087] In the above embodiments, when the target log contains each constant class template string in the candidate log template, and the constant class template strings appear in the same order in the target log and the candidate log template, and there is a log string in the target log that matches the variable class template string, it can be quickly determined that the target log and the candidate log template are successfully matched.

[0088] In one embodiment, step S208 includes:

[0089] Obtain the dynamic dictionary; the dynamic dictionary includes candidate strings and their corresponding candidate encoding data. The target word frequency information of the candidate string is greater than the preset word frequency information, and the data volume of the candidate encoding data is less than the data volume of the corresponding candidate string. The number of candidate strings in the dynamic dictionary is dynamically updated. In the dynamic dictionary, query the encoding data corresponding to each first initial string. Based on the candidate encoding data corresponding to the candidate strings that match the first initial string in the dynamic dictionary, obtain the first encoding data corresponding to each first initial string that was successfully queried, and use the first initial strings that failed to be queried as the first encoding data corresponding to each first initial string that failed to be queried.

[0090] The dynamic dictionary is a collection of candidate strings. It records information about these candidate strings. A candidate string is a string whose target word frequency is greater than a preset word frequency; in other words, the dynamic dictionary only records strings whose target word frequency is greater than a preset word frequency.

[0091] Target term frequency information refers to the latest term frequency information of a string, used to characterize the frequency or number of times the string is used in a large amount of known and retrieved logs. Preset term frequency information is used to determine whether a string can be added to the dynamic dictionary. Preset term frequency information can be set according to actual needs.

[0092] The dynamic dictionary records candidate strings and their corresponding encoded data. Candidate encoded data refers to the encoded data corresponding to a candidate string. The amount of candidate encoded data for a candidate string is less than the amount of candidate string data. Therefore, when compressing logs, replacing the original log string with encoded data can effectively reduce the data volume.

[0093] It's understandable that the number of candidate strings in the dynamic dictionary is updated dynamically. Once a new string's target word frequency exceeds the preset word frequency, that string can be added to the dynamic dictionary.

[0094] Specifically, the computer device can determine the first encoded data corresponding to each first initial string in the target log based on a dynamic dictionary. The computer device can obtain the dynamic dictionary and query the encoded data corresponding to each first initial string. If a candidate string matching the first initial string can be found in the dynamic dictionary, the candidate encoded data corresponding to that candidate string is used as the first encoded data corresponding to the first initial string. In other words, the computer device obtains the first encoded data corresponding to each successfully queried first initial string based on the candidate encoded data corresponding to the candidate strings matching the first initial string in the dynamic dictionary. If no candidate string matching the first initial string can be found in the dynamic dictionary, the first initial string is directly used as its corresponding first encoded data. That is, the computer device can directly use each first initial string that failed to be queried as its corresponding first encoded data.

[0095] For example, if the dynamic dictionary includes tokenA, the encoded data corresponding to tokenA in the dynamic dictionary is 1. If the first initial string corresponding to the target log is also tokenA, then the first encoded data corresponding to that first initial string is 1. If the first initial string of the target log is tokenF, and tokenF is not recorded in the dynamic dictionary, then the first encoded data corresponding to that first initial string is tokenF.

[0096] In one embodiment, a dynamic dictionary specific to the application is created based on logs generated by the same application. Different dynamic dictionaries can be created for different applications, improving the adaptability of the dynamic dictionary to different applications. A dynamic dictionary contains only strings relevant to one application, effectively controlling the amount of data in the dynamic dictionary and thus improving query speed. In this way, when retrieving target encoded data based on the dynamic dictionary, the corresponding target encoded data can be quickly found in the dynamic dictionary corresponding to the application to which the target log belongs.

[0097] In one embodiment, candidate encoded data is generated based on the sorting information of candidate strings in a dynamic dictionary. The sorting information indicates the candidate string's position within the dynamic dictionary. For example, if the dynamic dictionary contains 100 candidate strings, and the sorting is determined by the time the candidate string was added to the dictionary, then if candidate string A is the 11th string in the dynamic dictionary, its corresponding candidate encoded data is 11. Thus, for each new candidate string added to the dynamic dictionary, no complex encoding processing is required; candidate encoded data can be quickly generated based on the sorting information of the candidate strings.

[0098] Furthermore, the candidate encoded data can consist of a dictionary identifier corresponding to the dynamic dictionary and the sorting information of the candidate strings within the dynamic dictionary. A dictionary identifier is an identifier used to uniquely identify a dynamic dictionary; specifically, it can include a string of at least one character, such as letters, numbers, or symbols. Different dynamic dictionaries correspond to different dictionary identifiers.

[0099] In one embodiment, the dynamic dictionary may also record the target word frequency information corresponding to the candidate strings. The target word frequency information in the dynamic dictionary can also be dynamically updated. When processing new logs, the word frequency information of the corresponding candidate strings in the dynamic dictionary is updated promptly. If a new log contains candidate strings recorded in the dynamic dictionary, the word frequency information corresponding to that candidate string in the dynamic dictionary is updated.

[0100] In one embodiment, to ensure the timeliness of the dynamic dictionary, candidate strings whose word frequency information remains unchanged within a preset time period can be filtered out and deleted. If the word frequency information of a candidate string in the dynamic dictionary remains unchanged within the preset time period, it indicates that the candidate string is no longer frequently used and can be filtered out from the dynamic dictionary to reduce its data volume. The computer device can periodically filter out candidate strings that need to be deleted from the dynamic dictionary and reorganize the dynamic dictionary at regular intervals.

[0101] In one embodiment, the amount of candidate strings in the dynamic dictionary is greater than a preset value. For example, the dynamic dictionary only stores strings with a length greater than five. This avoids blindly encoding strings, and strings with a data size less than or equal to the preset value do not need to be encoded.

[0102] In the above embodiments, a dynamic dictionary is obtained. The dynamic dictionary includes candidate strings and corresponding candidate encoding data. The target word frequency information corresponding to the candidate string is greater than the preset word frequency information, and the data volume of the candidate encoding data is less than the data volume of the corresponding candidate string. The number of candidate strings in the dynamic dictionary is dynamically updated. The encoding data corresponding to each first initial string is queried in the dynamic dictionary. Based on the candidate encoding data corresponding to the candidate strings that match the first initial string in the dynamic dictionary, the first encoding data corresponding to each successfully queried first initial string is obtained, and the first encoding data corresponding to each unqueried first initial string is used as the first encoding data corresponding to each unqueried first initial string. In this way, the first encoding corresponding to the first initial string can be quickly determined based on the dynamic dictionary. Furthermore, the data volume of the encoding data in the dynamic dictionary is less than the data volume of the corresponding string, which can effectively improve the log compression rate. If the corresponding encoding data cannot be found in the action dictionary, the original string is directly used as the corresponding encoding data, which can quickly determine the encoding data.

[0103] In one embodiment, the log compression method further includes:

[0104] Each time a log is retrieved, the current strings corresponding to the current log are determined. If the current string already exists in the dynamic dictionary, the word frequency information of the current string in the dynamic dictionary is updated based on the current log. If the current string does not exist in the dynamic dictionary, the historical word frequency information corresponding to the current string is updated based on the current log to obtain the target word frequency information corresponding to the current string. If the target word frequency information corresponding to the current string is greater than the preset word frequency information, the encoded data corresponding to the current string is generated, and the current string, the corresponding encoded data, and the target word frequency information are added to the dynamic dictionary.

[0105] Historical word frequency information refers to the word frequency information obtained before processing the current log. Target word frequency information refers to the latest word frequency information. For example, if the current string obtained by segmenting the current log is "test", the historical word frequency information for "test" is 10. After updating, the target word frequency information for "test" is 11.

[0106] Specifically, the dynamic dictionary records the word frequency information corresponding to candidate strings. Each time logs are acquired and compressed, the computer device can determine the current strings corresponding to the current log. If the current string already exists in the dynamic dictionary, its word frequency information is updated based on the current log. If the current string does not exist in the dynamic dictionary, its historical word frequency information is updated based on the current log to obtain the target word frequency information. As the number of logs increases, the word frequency information of the strings also increases. If the target word frequency information corresponding to the current string increases to a level greater than the preset word frequency information, the computer device can generate target encoding data corresponding to the current string and add the current string, its target encoding data, and the target word frequency information to the dynamic dictionary.

[0107] In one embodiment, the computer device can first use a word frequency statistics table to record the word frequency information corresponding to each string. If the word frequency information of a string is greater than the preset word frequency information, then the string is moved from the word frequency statistics table to the dynamic dictionary.

[0108] In the above embodiments, each time a log is retrieved, the current strings corresponding to the current log are determined. If the current string already exists in the dynamic dictionary, the word frequency information of the current string in the dynamic dictionary is updated based on the current log. If the current string does not exist in the dynamic dictionary, the historical word frequency information corresponding to the current string is updated based on the current log to obtain the target word frequency information corresponding to the current string. When the target word frequency information corresponding to the current string is greater than a preset word frequency information, the encoded data corresponding to the current string is generated, and the current string, the corresponding encoded data, and the target word frequency information are added to the dynamic dictionary. In this way, the data in the dynamic dictionary is updated dynamically in real time, which can ensure the validity and accuracy of the data.

[0109] In one embodiment, such as Figure 3 As shown, log compression methods also include:

[0110] Step S302: Match the number of log strings corresponding to the target log with the number of template strings corresponding to each initial log template, and use the successfully matched initial log template as the intermediate log template.

[0111] The log string count is a statistical measure of the number of log strings in the target log. For example, if the target log consists of token1, token2, token3, token4, token5, and token6, then the target log has a count of 6 log strings.

[0112] The template string count is a statistical measure of the number of strings in the log template. For example, if the log template consists of token1, *, token4, *, and token6, then the corresponding template string count is 5.

[0113] Intermediate log templates refer to the initial log templates that successfully match the target log based on the number of strings.

[0114] Specifically, after determining the initial log template, the computer device can further match the target log and the initial log template based on the number of log strings corresponding to the target log and the number of template strings corresponding to the initial log template. The successfully matched initial log template is used as an intermediate log template, and log compression is ultimately performed using the intermediate log template. For example, the initial log template corresponding to the number of template strings whose data difference from the number of log strings is less than a preset difference can be used as the intermediate log template. The template identifier corresponding to such an intermediate log template can refer to more constant-type strings in the target log, resulting in a higher log compression rate.

[0115] It's understandable that if there's only one initial log template, the target log is matched against that initial log template to determine whether log compression should ultimately be based on the initial log template. If there are at least two initial log templates, the target log is matched against each of the at least two initial log templates, and log compression is performed based on the successfully matched initial log template from among the at least two initial log templates.

[0116] Step S304: From the log strings of the target log, the log string that matches the variable class template string in the intermediate log template is taken as the first intermediate string, and the log string that matches the constant class template string in the intermediate log template is taken as the second intermediate string.

[0117] Step S306: Obtain the second encoded data corresponding to each of the first intermediate strings.

[0118] It is understandable that the determination of the first intermediate string and the second intermediate string can refer to the determination of the first initial string and the second initial string mentioned above, and the acquisition of the second encoded data can refer to the acquisition of the first encoded data mentioned above, which will not be repeated here.

[0119] Step S308: Based on the template identifier corresponding to the intermediate log template, each second encoded data, and the second positional relationship between the first intermediate string and the second intermediate string in the target log, generate the log compression data corresponding to the target log.

[0120] The second positional relationship refers to the positional relationship between the first intermediate string and the second intermediate string in the target log, which is used to indicate whether there is a second intermediate string between adjacent first intermediate strings and whether there is a first intermediate string between adjacent second intermediate strings.

[0121] It is understandable that if the intermediate log template and the initial log template are the same log template, then the first initial string and the first intermediate string are the same string, the second initial string and the second intermediate string are the same string, the first encoded data and the second encoded data are the same encoded data, and the first positional relationship and the second positional relationship are the same positional relationship, so there is no need to repeatedly determine or obtain them.

[0122] Specifically, after determining the target log template, the computer device can generate compressed log data corresponding to the target log based on the template identifier corresponding to the intermediate log template, the second encoded data corresponding to each first intermediate string in the target log, and the second positional relationship between the first and second intermediate strings in the target log. For example, the template identifier, each second encoded data, and the second positional relationship can be combined to form compressed log data.

[0123] In one embodiment, if there are at least two successfully matched initial log templates, one of the successfully matched initial log templates can be randomly selected for log compression, or the initial log template with the largest number of template strings can be selected from all the successfully matched initial log templates for log compression.

[0124] In the above embodiments, the number of log strings corresponding to the target log is matched with the number of template strings corresponding to each initial log template, and the successfully matched initial log templates are used as intermediate log templates. From the log strings of the target log, the log strings that match the variable class template strings in the intermediate log templates are used as first intermediate strings, and the log strings that match the constant class template strings in the intermediate log templates are used as second intermediate strings. Second encoded data corresponding to each first intermediate string is obtained. Based on the template identifier corresponding to the intermediate log template, each second encoded data, and the second positional relationship between the first and second intermediate strings in the target log, compressed log data corresponding to the target log is generated. In this way, by further matching the target log and the initial log templates based on the number of strings, and performing log compression based on the successfully matched initial log templates, the compression efficiency can be effectively guaranteed, and the log compression rate can be significantly improved.

[0125] In one embodiment, step S302 includes:

[0126] The number of log strings and the template matching threshold are merged to obtain the first fused data corresponding to the target log. When the number of template strings corresponding to the initial log template is greater than or equal to the first fused data, it is determined that the target log and the initial log template are successfully matched, and the successfully matched initial log template is used as the intermediate log template. When the number of template strings corresponding to the initial log template is less than the first fused data, it is determined that the target log and the initial log template are not matched.

[0127] The template matching threshold refers to the threshold used for template matching. The template matching threshold can be set according to actual needs.

[0128] Specifically, when matching the target log with the initial log template, the computer device can fuse the number of log strings corresponding to the target log and the template matching threshold to obtain the first fused data corresponding to the target log. For example, the first fused data can be obtained by multiplying the number of log strings by the template matching threshold; or by using the difference between the number of log strings and the template matching threshold as the first fused data; and so on. When the number of template strings corresponding to the initial log template is greater than or equal to the first fused data, the computer device can determine that the target log and the initial log template have successfully matched, and use the successfully matched initial log template as an intermediate log template. When the number of template strings corresponding to the initial log template is less than the first fused data, the computer device can determine that the target log and the initial log template have failed to match.

[0129] It's understandable that if the number of template strings corresponding to the initial log template is greater than or equal to the number of strings in the first fused data, it indicates that the initial log template has a sufficient number of strings compared to the target log. Using such an initial log template for log compression can achieve high compression efficiency and a high compression ratio. Conversely, if the number of template strings corresponding to the initial log template is less than the number of strings in the first fused data, it indicates that the initial log template does not have a sufficient number of strings compared to the target log. Using such an initial log template for log compression will not yield high compression efficiency.

[0130] In the above embodiments, the number of log strings and the template matching threshold are fused to obtain the first fused data corresponding to the target log. When the number of template strings corresponding to the initial log template is greater than or equal to the first fused data, it is determined that the target log and the initial log template have successfully matched, and the successfully matched initial log template is used as the intermediate log template. When the number of template strings corresponding to the initial log template is less than the first fused data, it is determined that the target log and the initial log template have failed to match. In this way, log compression based on the initial log template with a template string count greater than or equal to the first fused data can effectively ensure compression efficiency and significantly improve the log compression rate.

[0131] In a specific embodiment, assuming the number of tokens in the target log is L(log), the template matching success threshold (i.e., the template matching threshold) is T(match), and the value of T(match) is between 0 and 1, the sum of the number of constants and the number of variables in the initial log template (there can be at most one variable between two constants) is S(template). If S(template)≥T(match)*L(log), then it is determined that the target log and the initial log template are successfully matched, and the initial log template is used as the intermediate log template.

[0132] In one embodiment, such as Figure 4 As shown, log compression methods also include:

[0133] Step S402: When the target log and candidate log template fail to match or the target log and initial log template fail to match, the log strings in the target log are divided into variable log strings and constant log strings. Based on the variable log strings and constant log strings, a target log template corresponding to the target log is generated. The target word frequency information corresponding to the variable log strings is less than the target word frequency information corresponding to the constant log strings.

[0134] Based on the target term frequency information of log strings, log strings can be divided into variable strings and constant strings. The target term frequency information corresponding to variable strings is less than that corresponding to constant strings. It can be considered that, for a given log entry, variable strings are low-frequency strings, and constant strings are high-frequency strings. High-frequency strings are used frequently in a large number of log entries and can be considered the constant part of the log. Low-frequency strings are used less frequently in a large number of log entries and can be considered the variable part of the log.

[0135] The target log template is a template generated based on the target log.

[0136] Specifically, if the target log and the candidate log template fail to match, or the target log and the initial log template fail to match, the computer device can create a new log template corresponding to the target log, generate a target log template corresponding to the target log, and perform log compression based on the target log template.

[0137] Computer equipment can categorize log strings in a target log into variable strings and constant strings based on target term frequency information. Log strings with target term frequencies greater than a preset threshold are designated as constant strings, while those with target term frequencies less than or equal to the preset threshold are designated as variable strings. Furthermore, the computer equipment can generate a target log template corresponding to the target log based on these variable and constant log strings. For example, constant log strings can be used as constant template strings in the target log template, and variable log strings can be used as variable template strings in the target log template.

[0138] In one embodiment, after determining the log strings corresponding to the target log, the computer device can obtain the historical word frequency information corresponding to each log string, update the historical word frequency information based on the target log, and thus obtain the target word frequency information corresponding to each log string. For example, if a log string obtained by segmenting the target log is "test", and the historical word frequency information corresponding to "test" is 10, after updating, the target word frequency information corresponding to "test" is 11.

[0139] Step S404: Obtain the third-encoded data corresponding to each log string belonging to the variable class log string.

[0140] Step S406: Based on the template identifier corresponding to the target log template, each third-level encoded data, and the third positional relationship between variable-type log strings and constant-type log strings in the target log, generate the log compression data corresponding to the target log.

[0141] The third encoded data refers to the encoded data corresponding to the log string that belongs to the variable class log string in the target log.

[0142] It is understandable that the method for obtaining the third encoded data can refer to the method for obtaining the first encoded data mentioned above, and will not be repeated here.

[0143] The third positional relationship refers to the positional relationship between variable-type log strings and constant-type log strings in the target log. It is used to indicate whether there are constant-type log strings between adjacent variable-type log strings, and whether there are variable-type log strings between adjacent constant-type log strings.

[0144] Specifically, after generating the target log template, the computer device can generate compressed log data corresponding to the target log based on the template identifier corresponding to the target log template, the third-encoded data corresponding to each log string belonging to the variable class in the target log, and the third positional relationship between the variable class log strings and the constant class log strings in the target log. For example, the template identifier, each third-encoded data, and the third positional relationship are combined to form compressed log data.

[0145] In the above embodiments, when the target log and candidate log templates fail to match, or when the target log and the initial log template fail to match, the log strings in the target log are divided into variable-type log strings and constant-type log strings. Based on the variable-type log strings and constant-type log strings, a target log template corresponding to the target log is generated. The target word frequency information corresponding to the variable-type log strings is less than that corresponding to the constant-type log strings. The third-level encoding data corresponding to each log string belonging to the variable-type log strings is obtained. Based on the template identifier corresponding to the target log template, each third-level encoding data, and the third positional relationship between the variable-type log strings and constant-type log strings in the target log, log compression data corresponding to the target log is generated. In this way, when the target log and existing log templates fail to match, a target log template of its own is generated. Log compression based on the target log template can effectively improve the log compression rate. Subsequently, the target log template can also be used for log compression of other logs.

[0146] In one embodiment, a target log template corresponding to the target log is generated based on variable-type log strings and constant-type log strings, including:

[0147] The number of log strings and the template generation threshold are merged to obtain the second fused data corresponding to the target log. The number of variable-type log strings and constant-type log strings in the target log is counted to obtain the number of reference strings corresponding to the target log. Specifically, when there is a variable-type log string between adjacent constant-type log strings in the target log, the count of the variable-type log strings between adjacent constant-type log strings is a preset number. When there is a variable-type log string before the starting constant-type log string or after the ending constant-type log string in the target log, the count of the variable-type log strings before the starting constant-type log string or after the ending constant-type log string is a preset number. When the number of reference strings is greater than or equal to the second fused data, a target log template is generated based on the variable-type log strings and constant-type log strings. The number of template strings corresponding to the target log template is the number of reference strings. When the number of reference strings is less than the second fused data, the log compressed data corresponding to the target log is obtained based on the encoded data corresponding to each log string in the target log.

[0148] The template generation threshold refers to the threshold used for template creation. The template generation threshold can be set according to actual needs. The preset quantity can also be set according to actual needs.

[0149] The starting constant class log string refers to the first constant class log string in the target log. The ending constant class log string refers to the last constant class log string in the target log.

[0150] Specifically, before creating a new target log template, the computer device can further determine whether it is necessary to create a new log template to avoid the newly created log template failing to achieve high compression efficiency.

[0151] The computer device can fuse the number of log strings corresponding to the target log with a template generation threshold to obtain second fused data for the target log. For example, it can multiply the number of log strings by the template generation threshold to obtain the second fused data; or use the difference between the number of log strings and the template generation threshold as the second fused data; and so on. The computer device can also count the number of variable-type log strings and constant-type log strings in the target log to obtain the number of reference strings corresponding to the target log. When performing the count, if there is a variable-type log string between adjacent constant-type log strings in the target log, the count of the variable-type log strings between adjacent constant-type log strings is set to a preset number. If there is a variable-type log string before the starting constant-type log string in the target log, the count of the variable-type log strings before the starting constant-type log string is set to a preset number. If there is a variable-type log string after the ending constant-type log string, the count of the variable-type log strings after the ending constant-type log string is set to a preset number. For example, if there are variable log strings between adjacent constant log strings in the target log, regardless of how many variable log strings exist between adjacent constant log strings in the target log, the number of variable log strings between adjacent constant log strings will be counted as one.

[0152] If the number of reference strings is greater than or equal to the second fused data, it indicates that the generated target log template contains a sufficient number of template strings, or more constant-type template strings. Such a target log template can achieve high compression efficiency. In this case, the computer device can generate the target log template based on the variable-type and constant-type log strings in the target log. The number of template strings corresponding to the generated target log template is equal to the number of reference strings. If the number of reference strings is less than the second fused data, it indicates that the generated target log template contains insufficient template strings, or more constant-type template strings. This cannot achieve high compression efficiency. In this case, the computer device can obtain the encoded data corresponding to each log string in the target log and generate compressed log data corresponding to the target log based on the obtained encoded data.

[0153] In the above embodiments, the number of log strings and the template generation threshold are fused to obtain the second fused data corresponding to the target log; the number of variable-type log strings and constant-type log strings in the target log is counted to obtain the number of reference strings corresponding to the target log; when the number of reference strings is greater than or equal to the second fused data, a target log template is generated based on the variable-type log strings and constant-type log strings; the number of template strings corresponding to the target log template is the number of reference strings; when the number of reference strings is less than the second fused data, the log compressed data corresponding to the target log is obtained based on the encoded data corresponding to each log string in the target log. Thus, if the number of reference strings is greater than or equal to the second fused data, a target log template is generated, ensuring that log compression based on the target log template has good compression efficiency and effectively improves the log compression rate.

[0154] In a specific embodiment, assuming the number of tokens in the target log is L(log), the successful template generation threshold (i.e., the template generation threshold) is T(create), where T(create) is in the range of 0 to 1, and the sum of the number of constants and variables in the target log (there can be at most one variable between two constants) is S(template), if S(template)≥T(create)*L(log), then the target log template corresponding to the target log is generated.

[0155] In one embodiment, a target log template is generated based on variable-type log strings and constant-type log strings, including:

[0156] The constant log string is used as the target template string. Based on the variable log string, a preset string is added between each target template string to obtain the target log template.

[0157] The target template string is used as the constant class template string in the target log template, and the preset string is used as the variable class template string in the target log template. When there is a variable class log string between adjacent constant class log strings in the target log, a preset number of preset strings are added between the corresponding adjacent target template strings. When there is a variable class log string before the starting constant class log string, a preset number of preset strings are added before the corresponding target template string. When there is a variable class log string after the ending constant class log string, a preset number of preset strings are added after the corresponding target template string.

[0158] Specifically, when directly generating a target log template or determining to generate a target log template after judgment, the computer device can use constant-type log strings as target template strings. Based on variable-type log strings, preset strings are added between each target template string to obtain the target log template. The target template strings are used as constant-type template strings in the target log template, and the preset strings are used as variable-type template strings in the target log template. When adding preset strings, if there are variable-type log strings between adjacent constant-type log strings in the target log, the computer device adds a preset number of preset strings between the corresponding adjacent target template strings; if there is a variable-type log string before the starting constant-type log string, the computer device adds a preset number of preset strings before the corresponding target template string; if there is a variable-type log string after the ending constant-type log string, the computer device adds a preset number of preset strings after the corresponding target template string.

[0159] For example, a target log might be represented as "token1+token2+token3+token4+token5+token6+token7+token8", where strings token1, token4, and token6 are constant log strings, and token2, token3, token5, token7, and token8 are variable log strings. The corresponding target log template is represented as "token1+*+token4+*+token6+*". This target log template can compress multiple logs with the format "token1+*+token4+*+token6+*".

[0160] In the above embodiments, constant-type log strings are used as target template strings. Based on variable-type log strings, preset strings are added between each target template string to obtain the target log template. The target template strings are used as constant-type template strings in the target log template, and the preset strings are used as variable-type template strings in the target log template. In this way, target log templates can be quickly generated based on constant-type and variable-type log strings.

[0161] In one embodiment, the current log template is an initial log template, an intermediate log template, or a target log template. When the current log template is an initial log template, the current variable class string is a first initial string, the current constant class string is a second initial string, the current encoded data is first encoded data, and the current positional relationship is a first positional relationship. When the current log template is an intermediate log template, the current variable class string is a first intermediate string, the current constant class string is a second intermediate string, the current encoded data is second encoded data, and the current positional relationship is a second positional relationship. When the current log template is a target log template, the current variable class string is a variable class log string, the current constant class string is a constant class log string, the current encoded data is third encoded data, and the current positional relationship is a third positional relationship.

[0162] The process of generating compressed log data includes the following steps:

[0163] Arrange the current encoded data according to the order of appearance of each current variable class string in the target log to obtain the initial encoding sequence; based on the position of the current constant class string in the target log among the current variable class strings, insert position identifiers at the corresponding positions in the initial encoding sequence to obtain the target encoding sequence; and obtain the log compressed data based on the template identifier corresponding to the current log template and the target encoding sequence.

[0164] Specifically, when generating target compressed data based on template identifiers, encoded data, and positional relationships, the computer device can arrange the current encoded data in an ordered manner according to the appearance order of each current variable class string in the target log to obtain an initial encoding sequence. Then, based on the appearance position of the current constant class string in the target log among the current variable class strings, position identifiers are inserted at corresponding positions in the initial encoding sequence to obtain the target encoding sequence. Finally, based on the template identifier corresponding to the current log template and the target encoding sequence, the log compressed data is obtained. The position identifier is used to identify whether adjacent current variable class strings in the target log are directly adjacent or indirectly adjacent, and whether there are other strings between adjacent current variable class strings in the target log.

[0165] For example, the target log is represented by "token1+token2+token3+token4+token5+token6+token7+token8", and the initial log template is represented by "token1+*+token4+*+token6+*", with the template identifier A corresponding to the initial log template. Here, token2, token3, token5, token7, and token8 are the first initial strings, and the first encoded data corresponding to token2, token3, token5, token7, and token8 are 2, 3, 5, 7, and 8 respectively. Strings token1, token4, and token6 are the second initial strings. The target compressed data corresponding to the target log can be represented by "A-2-3-&-5-&-7-8". "&" represents the position identifier.

[0166] When restoring the target log based on the target compressed data, the initial log template "token1+*+token4+*+token6+*" corresponding to A is obtained based on A in the target compressed data. Based on the initial log template, it can be seen that the target log contains token1, token4, and token6, and token1, token4, and token6 appear in sequence. There are other strings between token1 and token4, between token4 and token6, and after token6 there are other strings.

[0167] Based on the numbers 2, 3, 5, 7, and 8 in the target compressed data, obtain tokens 2, 3, 5, 7, and 8, respectively. Based on the '&' in the target compressed data, determine that tokens 2 and 3 are located between tokens 1 and 4, token 5 is located between tokens 4 and 6, and tokens 7 and 8 are located after token 6.

[0168] The final target log obtained by restoring the target compressed data is "token1+token2+token3+token4+token5+token6+token7+token8".

[0169] In the above embodiments, the current encoded data are arranged according to the order of appearance of each current variable class string in the target log to obtain an initial encoding sequence. Based on the position of the current constant class string in the target log among the current variable class strings, position identifiers are inserted at the corresponding positions in the initial encoding sequence to obtain the target encoding sequence. Based on the template identifier corresponding to the current log template and the target encoding sequence, the log compressed data is obtained. In this way, the target encoding sequence not only records the encoded data and appearance order of the variable class strings, but also records whether adjacent variable class strings are separated by constant class strings. Based on the template identifier, the log template that records the constant class strings in the log can be found. The log compressed data obtained based on the template identifier and the target encoding sequence can effectively restore the original target log while maintaining an excellent log compression ratio.

[0170] In one embodiment, log compressed data is obtained based on the template identifier and target encoding sequence corresponding to the current log template, including:

[0171] Based on the template identifier and target encoding sequence corresponding to the current log template, generate log encoding data corresponding to the target log; perform general compression on the log encoding data to obtain compressed log data.

[0172] General-purpose compression refers to universal and basic compression methods. For example, compression is performed using general-purpose compression tools or algorithms. General-purpose compression tools or algorithms include, but are not limited to, ZSTD (Zstandard, a new open-source lossless compression algorithm) and Deflate (a lossless data compression algorithm that uses both the LZ77 algorithm and Huffman coding).

[0173] Specifically, in order to further improve the compression rate, the computer equipment can first generate the log encoding data corresponding to the target log based on the template identifier and target encoding sequence corresponding to the current log template, and then perform general compression on the log encoding data to further reduce the amount of data, thereby obtaining compressed log data.

[0174] In one embodiment, log-coded data can be compressed in blocks. If the amount of log-coded data in a log entry is greater than or equal to a preset data size, then that log-coded data is compressed as a single data block. If the amount of log-coded data in a log entry is less than the preset data size, then the log-coded data for the next log entry is retrieved. If the sum of the log-coded data from two log entries is greater than or equal to the preset data size, then the log-coded data from the two log entries is compressed as a single data block. If the sum of the log-coded data is still less than the preset data size, then the log-coded data for the next log entry is retrieved, and so on, until the sum of the log-coded data is greater than or equal to the preset data size, forming a single data block for general compression.

[0175] In one embodiment, reference Figure 5 This refers to the storage method for data blocks. When storing data blocks, it is necessary to record the block size and the log encoding data corresponding to each log in the data block. The log encoding data corresponding to a log includes the log size, log ID, log template ID, and the encoded data (also known as parameters) corresponding to each variable string.

[0176] In the above embodiments, log encoded data corresponding to the target log is generated based on the template identifier and target encoding sequence corresponding to the current log template; the log encoded data is then subjected to general compression to obtain compressed log data. In this way, generating log encoded data first and then performing general compression to obtain compressed log data can further improve the log compression rate.

[0177] In one embodiment, such as Figure 6 As shown, a log restoration method is provided. Taking the application of this method to a computer device as an example, the computer device can be the one described above. Figure 1 Terminal 102 or server 104 in the reference. Figure 6 The log restoration method includes the following steps:

[0178] Step S602: Obtain the log compression data corresponding to the target log; the log compression data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log.

[0179] The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template. The candidate log template that matches successfully is used as the initial log template. The first initial string is the log string that matches the variable class template string in the initial log template among the various log strings of the target log. The second initial string is the log string that matches the constant class template string in the initial log template among the various log strings of the target log.

[0180] It is understood that the process of generating compressed log data can be referred to the contents of the various embodiments of the aforementioned log compression method, and will not be repeated here.

[0181] Specifically, when it is necessary to view or analyze the target log, the computer device can obtain the compressed log data corresponding to the target log and restore the original target log based on the compressed log data.

[0182] In one embodiment, if the log compressed data is obtained by general compression of log encoded data, then the log compressed data is first decompressed in a general manner, and then the original target log is restored based on the log encoded data.

[0183] In one embodiment, the terminal may send a log retrieval request carrying a log identifier of the target log to the server. The server retrieves the compressed log data corresponding to the target log based on the log identifier, restores the original target log based on the compressed log data, and sends the original target log to the terminal.

[0184] Step S604: Obtain the initial log template corresponding to the template identifier, and obtain the log string corresponding to the first encoded data.

[0185] Step S606: Based on the first positional relationship, replace the variable class template string in the initial log template with the corresponding log string to obtain the target log.

[0186] Specifically, when performing log restoration, the computer device can obtain the log string corresponding to the first encoded data based on the template identifier in the log compressed data, and replace the variable class template string in the initial log template with the corresponding log string based on the first encoded data in the log compressed data, so as to obtain the target log.

[0187] It is understandable that if the compressed log data is generated based on an intermediate log template or a target log template, a similar method can be used to restore the original target log.

[0188] In one embodiment, a computer device can retrieve the log string corresponding to the first encoded data from a dynamic dictionary based on the first encoded data.

[0189] The above log restoration method obtains the compressed log data corresponding to the target log. The compressed log data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first and second initial strings in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string that matches the variable class template string in the initial log template among the various log strings in the target log, and the second initial string is the log string that matches the constant class template string in the initial log template among the various log strings in the target log. The method obtains the initial log template corresponding to the template identifier and the log string corresponding to the first encoded data. Based on the first positional relationship, the variable class template string in the initial log template is replaced with the corresponding log string to obtain the target log. In this way, the compressed log data is generated based on the template identifier corresponding to the initial log template, the target encoded data corresponding to each log string belonging to the variable type in the target log, and the positional relationship between the variable type strings and constant type strings in the target log. Representing the constant strings in the target log using the template identifier corresponding to the initial log template determined by the arrangement information of the constant type strings effectively compresses log data and improves the log compression ratio. Converting the target log, composed of a large number of strings, into compressed log data composed of template identifiers, target encoded data, and positional relationships effectively reduces the amount of log data and improves the log compression ratio. During log restoration, it is only necessary to obtain the corresponding log template based on the template identifier in the compressed log data, obtain the corresponding log string based on the target encoded data in the compressed log data, and quickly and accurately restore the original log based on the log template corresponding to the template identifier, the log string corresponding to the target encoded data, and the positional relationship in the compressed log data.

[0190] In one specific embodiment, reference Figure 7 The log compression method includes the following steps:

[0191] 1. Log parsing

[0192] The system performs word segmentation and term frequency statistics on the log data stream written by users. Word segmentation divides a log entry into multiple strings called tokens (or words). Log entries can be segmented using specified delimiters (such as commas, semicolons, etc.), and the frequency of the resulting tokens can be counted.

[0193] 2. Generate dynamic dictionaries online.

[0194] If the word frequency of a token reaches the dictionary generation threshold T(create_dict), then the token is added to the dynamic dictionary; otherwise, the word frequency statistics are continuously updated.

[0195] Once a token is added to the dynamic dictionary, the dictionary stores the token, its corresponding encoded data, and word frequency. When processing new logs, the word frequency in the dynamic dictionary is updated promptly to facilitate the generation of log templates later.

[0196] The token's encoded data (code) in the dynamic dictionary uses a format of: a 1-byte identifier + a variable-length code of less than or equal to 4 bytes. This encoding method allows the size of the token's encoded data to range from 2 to 5 bytes. The 1-byte identifier is the dictionary identifier for the dynamic dictionary, and the variable-length code is an integer generated auto-incrementing according to the dictionary size. To ensure that encoding is effective, the original token data size must be larger than its corresponding encoded data. Therefore, the dynamic dictionary only processes tokens longer than 5 bytes; other tokens are processed through a short word dictionary. The short word dictionary is only used to cache frequently used tokens with a length less than 5 bytes and is not encoded for them.

[0197] 3. Online matching and generation of dynamic templates.

[0198] For each log entry, an online matching and dynamic template generation process is performed. The constant portion of the log template stores high-frequency words, while the variable portion uses a wildcard "<*>" to represent low-frequency words in the log.

[0199] 3-1 Template Matching

[0200] The current log is matched with the existing log template. If the match is successful, the log is encoded based on the existing log template.

[0201] Assuming the current log has L(log) tokens, the template matching success threshold T(match) (ranging from 0 to 1), and the sum of the number of constants and variables in the existing log template (there can be at most one variable between two constants) is S(template), if S(template)≥T(match)*L(log), and the constant tokens in the existing log template appear in the log in sequence with consistent relative order, and the remaining tokens in the log can also be matched by wildcards in the existing log template, then the current log is determined to be a successful match with the existing log template.

[0202] 3-2 Template Generation

[0203] If the current log fails to match an existing log template, a new log template can be generated for the current log, and the log can be encoded based on the new log template.

[0204] Based on the word frequency statistics in the dynamic dictionary, tokens whose word frequency reaches the threshold T (const) are used as the constant part of the new log template, and the remaining tokens are used as the variable part of the new log template. The variable part is represented by the wildcard "<*>". There can be at most one variable token between two constant tokens.

[0205] Assuming the current number of tokens in the log is L(log), the successful template generation threshold is T(create) (ranging from 0 to 1), and the sum of the number of constants and variables in the new log template (e.g., there can be at most one variable between two constants) is S(template), if S(template)≥T(create)*L(log), then the log template is successfully created.

[0206] 4. Encoding

[0207] Logs that successfully match a template or successfully create a template are divided into two parts: constants and variables. The constant part is encoded using a dynamic template, while the variable part is encoded using a dynamic dictionary. Logs that fail to create a template are directly encoded using a dynamic dictionary.

[0208] 5. Compress using a general compression tool.

[0209] The encoded log data is compressed in blocks using general compression tools (such as ZSTD and Deflate) to further process redundancy within the blocks.

[0210] The process of decoding log data includes the following steps:

[0211] 1. Identify the constant portion of the log.

[0212] Read the template encoding sequence number in the log encoding data, find the log template corresponding to the template encoding sequence number, and determine the constant part in the log based on the found log template.

[0213] 2. Determine the variable section in the log.

[0214] If a dictionary identifier is read from the log encoded data, then the dynamic dictionary corresponding to that dictionary identifier is searched, and the token corresponding to the encoded data is searched in the dynamic dictionary to obtain the variable part of the log.

[0215] 3. Combining the constant part and the variable part

[0216] The original log is obtained by restoring the constant and variable parts of the log to their original order based on the log template.

[0217] For example, the current log entry is {Timestamp:“[2022-5-1 12:00:00]”,URL:“ / is / a / test”,Message:“a / test”,Server_ip:127.0.0.1}. The encoding process for this log entry is as follows:

[0218] The tokens corresponding to the current log are shown in Table 1.

[0219] Table 1

[0220]

[0221] Assuming there are currently no existing log templates that can be successfully matched, let T(const) be 11, T(create) be 0.5, and T(match) be 0.5. Then the newly created log templates are shown in Table 2.

[0222] Table 2

[0223]

[0224] If S(template) is 11, the conditions for successful template creation are met. If the encoding sequence number (i.e., template identifier) ​​Template ID of the new log template is 1, then the data after the current log is replaced by the new log template is shown in Table 3.

[0225] Table 3

[0226]

[0227] In this process, the encoding sequence number of the new log template replaces the constant part in the current log, and the following 5 tokens belong to the variable part of the template and are encoded using a dynamic dictionary.

[0228] The final encoded data of the current log is shown in Table 4.

[0229] Table 4

[0230]

[0231] Here, 1 represents a one-byte dictionary identifier with the value of ASCII code 1. The following 6 or 5 represents the encoded data in the dynamic dictionary. It can be understood that time information does not need to be encoded.

[0232] The current log decoding process is as follows:

[0233] Parse constants: Read the log template with Template ID 1: {Timestamp:“<*>”, URL:“<*>a / <*>Message:“a / <*>Server_ip:<*>}

[0234] Parsing variables: Read the raw data "[2022-5-1 12:00:00]" and " / is / ", and write them directly in order. Read the byte with ASCII value 1, indicating that the dynamic dictionary needs to be queried. Based on the dictionary codes 6 and 5, retrieve the raw data "test", and "127.0.0.1" from the action dictionary.

[0235] The template constants and parsed variables are restored in order to obtain: Timestamp: "[2022-5-1 12:00:00]", URL: " / is / a / test", Message: "a / test", Server_ip: 127.0.0.1.

[0236] In the above embodiments, the log compression method of this application can simultaneously address both local and global redundancy. The use of dynamic dictionaries and dynamic templates reduces global redundancy, while block-based log storage effectively improves the compression rate by reducing local redundancy. Furthermore, both dynamic templates and dynamic dictionaries are globally shared, generated online, and continuously updated, facilitating maintenance. Block-based log storage supports random read operations. Logs only need to be scanned once for encoding and compression, effectively reducing the number of scans during the encoding and compression process and improving encoding and compression efficiency. The log compression method of this application can achieve streaming processing, batch reading logs into memory, batch encoding and compressing log data, and persistent storage. This log compression method reduces the storage space consumption of log data, saves costs, and ensures high performance for random read and write operations.

[0237] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0238] Based on the same inventive concept, embodiments of this application also provide a log compression apparatus for implementing the log compression method described above, and a log restoration apparatus for implementing the log restoration method described above. The solutions provided by the above apparatuses are similar to the solutions described in the above methods. Therefore, the specific limitations in one or more log compression apparatus embodiments provided below can be found in the limitations of the log compression method above, and the specific limitations in one or more log restoration apparatus embodiments provided below can be found in the limitations of the log restoration method above, and will not be repeated here.

[0239] In one embodiment, such as Figure 8 As shown, a log compression device is provided, including: a log acquisition module 802, a log template determination module 804, a string classification module 806, an encoded data acquisition module 808, and a log compression data generation module 810, wherein:

[0240] The log acquisition module 802 is used to acquire target logs.

[0241] The log template determination module 804 is used to match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and use the successfully matched candidate log template as the initial log template.

[0242] The string classification module 806 is used to select the log string that matches the variable class template string in the initial log template as the first initial string and the log string that matches the constant class template string in the initial log template as the second initial string from the log strings of the target log.

[0243] The encoded data acquisition module 808 is used to acquire the first encoded data corresponding to each first initial string.

[0244] The log compression data generation module 810 is used to generate log compression data corresponding to the target log based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log.

[0245] The aforementioned log compression device represents the second initial string in the target log using a template identifier corresponding to the initial log template determined based on string arrangement information. This effectively compresses log data and improves the log compression rate. It converts the target log, which consists of a large number of strings, into compressed log data composed of a template identifier, first encoded data, and a first positional relationship. This effectively reduces the amount of log data and improves the log compression rate.

[0246] In one embodiment, the log template determination module is further configured to determine that the target log and the candidate log template are successfully matched when the target log contains each constant class template string in the candidate log template, the constant class template strings appear in the same order in the target log and the candidate log template, and there is a log string in the target log that matches the variable class template string.

[0247] In one embodiment, the encoding data acquisition module is further configured to acquire a dynamic dictionary; the dynamic dictionary includes candidate strings and candidate encoding data corresponding to the candidate strings, wherein the target word frequency information corresponding to the candidate string is greater than the preset word frequency information, the data volume of the candidate encoding data is less than the data volume of the corresponding candidate string, and the number of candidate strings in the dynamic dictionary is dynamically updated; in the dynamic dictionary, the encoding data corresponding to each first initial string is queried; based on the candidate encoding data corresponding to the candidate strings that match the first initial strings in the dynamic dictionary, the first encoding data corresponding to each first initial string that was successfully queried is obtained, and the first initial strings that failed to be queried are used as the first encoding data corresponding to each first initial string that failed to be queried.

[0248] In one embodiment, the log compression device is further used for:

[0249] Each time a log is retrieved, the corresponding strings for that current log are determined.

[0250] If the current string already exists in the dynamic dictionary, update the word frequency information of the current string in the dynamic dictionary based on the current log;

[0251] If the current string does not exist in the dynamic dictionary, update the historical word frequency information corresponding to the current string based on the current log to obtain the target word frequency information corresponding to the current string. If the target word frequency information corresponding to the current string is greater than the preset word frequency information, generate the encoded data corresponding to the current string, and add the current string, the corresponding encoded data, and the target word frequency information to the dynamic dictionary.

[0252] In one embodiment, the log template determination module is further configured to match the number of log strings corresponding to the target log with the number of template strings corresponding to each initial log template, and use the successfully matched initial log template as an intermediate log template.

[0253] The string classification module is also used to select the log string that matches the variable class template string in the intermediate log template as the first intermediate string and the log string that matches the constant class template string in the intermediate log template as the second intermediate string from the various log strings in the target log.

[0254] The encoded data acquisition module is also used to acquire the second encoded data corresponding to each of the first intermediate strings.

[0255] The log compression data generation module is also used to generate log compression data corresponding to the target log based on the template identifier corresponding to the intermediate log template, each second encoded data, and the second positional relationship between the first intermediate string and the second intermediate string in the target log.

[0256] In one embodiment, the log template determination module is further configured to fuse the number of log strings and the template matching threshold to obtain the first fused data corresponding to the target log; when the number of template strings corresponding to the initial log template is greater than or equal to the first fused data, it is determined that the target log and the initial log template are successfully matched, and the successfully matched initial log template is used as the intermediate log template; when the number of template strings corresponding to the initial log template is less than the first fused data, it is determined that the target log and the initial log template are not matched.

[0257] In one embodiment, the log template determination module is further configured to, when the target log and candidate log templates fail to match or the target log and initial log template fail to match, divide each log string in the target log into variable log strings and constant log strings, and generate a target log template corresponding to the target log based on the variable log strings and constant log strings; the target word frequency information corresponding to the variable log strings is less than the target word frequency information corresponding to the constant log strings.

[0258] The encoded data acquisition module is also used to acquire the third encoded data corresponding to each log string belonging to the variable class log string.

[0259] The log compression data generation module is also used to generate log compression data corresponding to the target log based on the template identifier corresponding to the target log template, each third-level encoded data, and the third positional relationship between variable-type log strings and constant-type log strings in the target log.

[0260] In one embodiment, the log template determination module is further configured to fuse the number of log strings and the template generation threshold to obtain second fused data corresponding to the target log; count the number of variable-type log strings and constant-type log strings in the target log to obtain the number of reference strings corresponding to the target log; wherein, when there are variable-type log strings between adjacent constant-type log strings in the target log, the count of variable-type log strings between adjacent constant-type log strings is a preset number; when there are variable-type log strings before the starting constant-type log string or after the ending constant-type log string in the target log, the count of variable-type log strings before the starting constant-type log string or after the ending constant-type log string is a preset number; when the number of reference strings is greater than or equal to the second fused data, a target log template is generated based on the variable-type log strings and constant-type log strings; the number of template strings corresponding to the target log template is the number of reference strings; when the number of reference strings is less than the second fused data, log compressed data corresponding to the target log is obtained based on the encoded data corresponding to each log string in the target log.

[0261] In one embodiment, the log template determination module is further configured to use constant-type log strings as target template strings, and based on variable-type log strings, add preset strings between each target template string to obtain a target log template; wherein, the target template strings are used as constant-type template strings in the target log template, and the preset strings are used as variable-type template strings in the target log template; when there are variable-type log strings between adjacent constant-type log strings in the target log, a preset number of preset strings are added between the corresponding adjacent target template strings; when there is a variable-type log string before the starting constant-type log string, a preset number of preset strings are added before the corresponding target template string; when there is a variable-type log string after the ending constant-type log string, a preset number of preset strings are added after the corresponding target template string.

[0262] In one embodiment, the current log template is an initial log template, an intermediate log template, or a target log template. When the current log template is an initial log template, the current variable class string is a first initial string, the current constant class string is a second initial string, the current encoded data is first encoded data, and the current positional relationship is a first positional relationship. When the current log template is an intermediate log template, the current variable class string is a first intermediate string, the current constant class string is a second intermediate string, the current encoded data is second encoded data, and the current positional relationship is a second positional relationship. When the current log template is a target log template, the current variable class string is a variable class log string, the current constant class string is a constant class log string, the current encoded data is third encoded data, and the current positional relationship is a third positional relationship.

[0263] The log compression data generation module is also used to arrange the current encoded data according to the order of appearance of each current variable class string in the target log to obtain the initial encoding sequence; based on the appearance position of the current constant class string in the target log among each current variable class string, it inserts position identifiers at the corresponding positions in the initial encoding sequence to obtain the target encoding sequence; and based on the template identifier corresponding to the current log template and the target encoding sequence, it obtains the log compressed data.

[0264] In one embodiment, the log compression data generation module is further configured to generate log encoding data corresponding to the target log based on the template identifier and target encoding sequence corresponding to the current log template; and perform general compression on the log encoding data to obtain log compression data.

[0265] In one embodiment, such as Figure 9 As shown, a log restoration device is provided, including: an M module, an N module, and an L module, wherein:

[0266] The log compression data acquisition module 902 is used to acquire the log compression data corresponding to the target log. The log compression data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable class template string in the initial log template, and the second initial string is the log string in the target log that matches the constant class template string in the initial log template.

[0267] The log composition data acquisition module 904 is used to obtain the initial log template corresponding to the template identifier and the log string corresponding to the first encoded data.

[0268] The log restoration module 906 is used to replace the variable class string in the initial log template with the corresponding log string based on the first position relationship to obtain the target log.

[0269] The aforementioned log restoration device generates compressed log data based on the template identifier corresponding to the initial log template, the target encoded data corresponding to each log string belonging to the variable type in the target log, and the positional relationship between variable type strings and constant type strings in the target log. Constant strings in the target log are represented by the template identifier corresponding to the initial log template, determined based on the arrangement information of constant type strings. This effectively compresses log data and improves the log compression rate. Converting a target log composed of a large number of strings into compressed log data composed of template identifiers, target encoded data, and positional relationships effectively reduces the amount of log data and improves the log compression rate. During log restoration, it is only necessary to obtain the corresponding log template based on the template identifier in the compressed log data, obtain the corresponding log string based on the target encoded data in the compressed log data, and quickly and accurately restore the original log based on the log template corresponding to the template identifier, the log string corresponding to the target encoded data, and the positional relationship in the compressed log data.

[0270] The modules in the aforementioned log compression and log restoration devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0271] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as dynamic dictionaries and log templates. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a log compression method or a log restoration method.

[0272] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a log compression method or a log restoration method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0273] Those skilled in the art will understand that Figure 10 , 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0274] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0275] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0276] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0277] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0278] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0279] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0280] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A log compression method, characterized in that, The method includes: Obtain the target log; The log string arrangement information corresponding to the target log is matched with the template string arrangement information corresponding to the candidate log template, and the successfully matched candidate log template is used as the initial log template. From the log strings of the target log, the log string that matches the variable class template string in the initial log template is taken as the first initial string, and the log string that matches the constant class template string in the initial log template is taken as the second initial string; Obtain the first encoded data corresponding to each initial string; Based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log, log compressed data corresponding to the target log is generated.

2. The method according to claim 1, characterized in that, The step of matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template, includes: When the target log contains each constant class template string in the candidate log template, and the constant class template strings appear in the same order in the target log and the candidate log template, and there is a log string in the target log that matches the variable class template string, it is determined that the target log and the candidate log template are successfully matched.

3. The method according to claim 1, characterized in that, The step of obtaining the first encoded data corresponding to each first initial string includes: Obtain a dynamic dictionary; the dynamic dictionary includes candidate strings and candidate encoding data corresponding to the candidate strings, the target word frequency information corresponding to the candidate strings is greater than the preset word frequency information, the data volume of the candidate encoding data is less than the data volume of the corresponding candidate strings, and the number of candidate strings in the dynamic dictionary is dynamically updated; In the dynamic dictionary, query the encoded data corresponding to each of the first initial strings; Based on the candidate encoding data corresponding to the candidate strings that match the first initial string in the dynamic dictionary, the first encoding data corresponding to each first initial string that was successfully queried is obtained, and the first initial strings that failed to be queried are used as the first encoding data corresponding to each first initial string that failed to be queried.

4. The method according to claim 3, characterized in that, The method further includes: Each time a log is retrieved, the corresponding strings for that current log are determined. If the current string already exists in the dynamic dictionary, update the word frequency information of the current string in the dynamic dictionary based on the current log; If the current string does not exist in the dynamic dictionary, the historical word frequency information corresponding to the current string is updated based on the current log to obtain the target word frequency information corresponding to the current string. When the target word frequency information corresponding to the current string is greater than the preset word frequency information, the encoded data corresponding to the current string is generated, and the current string, the corresponding encoded data, and the target word frequency information are added to the dynamic dictionary.

5. The method according to claim 1, characterized in that, The method further includes: The number of log strings corresponding to the target log is matched with the number of template strings corresponding to each initial log template, and the initial log template that matches successfully is used as the intermediate log template. From the log strings of the target log, the log string that matches the variable class template string in the intermediate log template is taken as the first intermediate string, and the log string that matches the constant class template string in the intermediate log template is taken as the second intermediate string; Obtain the second encoded data corresponding to each of the first intermediate strings; Based on the template identifier corresponding to the intermediate log template, each second encoded data, and the second positional relationship between the first intermediate string and the second intermediate string in the target log, log compressed data corresponding to the target log is generated.

6. The method according to claim 5, characterized in that, The step of matching the number of log strings corresponding to the target log with the number of template strings corresponding to each initial log template, and using the successfully matched initial log template as the intermediate log template, includes: The number of log strings and the template matching threshold are combined to obtain the first fused data corresponding to the target log; When the number of template strings corresponding to the initial log template is greater than or equal to the first fused data, it is determined that the target log and the initial log template are successfully matched, and the successfully matched initial log template is used as the intermediate log template. If the number of template strings corresponding to the initial log template is less than the first fused data, it is determined that the target log and the initial log template have failed to match.

7. The method according to claim 5, characterized in that, The method further includes: When the target log and the candidate log template fail to match, or the target log and the initial log template fail to match, each log string in the target log is divided into variable log strings and constant log strings. Based on the variable log strings and the constant log strings, a target log template corresponding to the target log is generated. The target word frequency information corresponding to the variable log strings is less than the target word frequency information corresponding to the constant log strings. Retrieve the third-encoded data corresponding to each log string belonging to the variable class log string; Based on the template identifier corresponding to the target log template, each third-encoded data, and the third positional relationship between the variable-type log string and the constant-type log string in the target log, the log compressed data corresponding to the target log is generated.

8. The method according to claim 7, characterized in that, The step of generating the target log template corresponding to the target log based on the variable class log string and the constant class log string includes: The number of log strings and the template generation threshold are combined to obtain the second fused data corresponding to the target log; The number of variable-type log strings and constant-type log strings in the target log is counted to obtain the number of reference strings corresponding to the target log; wherein, when there is a variable-type log string between adjacent constant-type log strings in the target log, the number of variable-type log strings between adjacent constant-type log strings is a preset number; when there is a variable-type log string before the starting constant-type log string or after the ending constant-type log string in the target log, the number of variable-type log strings before the starting constant-type log string or after the ending constant-type log string is a preset number. When the number of reference strings is greater than or equal to the second fused data, the target log template is generated based on the variable log strings and the constant log strings; the number of template strings corresponding to the target log template is the number of reference strings. When the number of reference strings is less than the second fused data, the log compressed data corresponding to the target log is obtained based on the encoded data corresponding to each log string in the target log.

9. The method according to claim 8, characterized in that, The step of generating the target log template based on the variable class log string and the constant class log string includes: Using the constant log string as the target template string, and based on the variable log string, a preset string is added between each target template string to obtain the target log template; Wherein, the target template string is used as the constant class template string in the target log template, and the preset string is used as the variable class template string in the target log template; when there is a variable class log string between adjacent constant class log strings in the target log, a preset number of preset strings are added between the corresponding adjacent target template strings; when there is a variable class log string before the starting constant class log string, a preset number of preset strings are added before the corresponding target template string; when there is a variable class log string after the ending constant class log string, a preset number of preset strings are added after the corresponding target template string.

10. The method according to any one of claims 1 to 9, characterized in that, The current log template is either an initial log template, an intermediate log template, or a target log template. When the current log template is an initial log template, the current variable class string is a first initial string, the current constant class string is a second initial string, the current encoded data is first encoded data, and the current position relationship is a first position relationship. When the current log template is an intermediate log template, the current variable class string is a first intermediate string, the current constant class string is a second intermediate string, the current encoded data is second encoded data, and the current position relationship is a second position relationship. When the current log template is a target log template, the current variable class string is a variable class log string, the current constant class string is a constant class log string, the current encoded data is third encoded data, and the current position relationship is a third position relationship. The process of generating the compressed log data includes the following steps: Arrange the current encoded data according to the order of appearance of each current variable string in the target log to obtain the initial encoding sequence; Based on the position of the current constant string in the target log among the various current variable strings, a position identifier is inserted at the corresponding position in the initial encoding sequence to obtain the target encoding sequence; The compressed log data is obtained based on the template identifier corresponding to the current log template and the target encoding sequence.

11. The method according to claim 10, characterized in that, The step of obtaining the compressed log data based on the template identifier corresponding to the current log template and the target encoding sequence includes: Based on the template identifier corresponding to the current log template and the target encoding sequence, generate log encoding data corresponding to the target log; The log encoded data is subjected to general compression to obtain the log compressed data.

12. A log restoration method, characterized in that, The method includes: Obtain the compressed log data corresponding to the target log; the compressed log data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable class template string in the initial log template, and the second initial string is the log string in the target log that matches the constant class template string in the initial log template. Obtain the initial log template corresponding to the template identifier, and obtain the log string corresponding to the first encoded data; Based on the first positional relationship, the variable class string in the initial log template is replaced with the corresponding log string to obtain the target log.

13. A log compression device, characterized in that, The device includes: The log acquisition module is used to acquire target logs; The log template determination module is used to match the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and use the successfully matched candidate log template as the initial log template. The string classification module is used to select the log string that matches the variable class template string in the initial log template as the first initial string and the log string that matches the constant class template string in the initial log template as the second initial string from the log strings of the target log. The encoded data acquisition module is used to acquire the first encoded data corresponding to each first initial string; The log compression data generation module is used to generate log compression data corresponding to the target log based on the template identifier corresponding to the initial log template, each first encoded data, and the first positional relationship between the first initial string and the second initial string in the target log.

14. A log restoration device, characterized in that, The device includes: The log compression data acquisition module is used to acquire the log compression data corresponding to the target log. The log compression data is generated based on the template identifier corresponding to the initial log template, the first encoded data corresponding to each first initial string in the target log, and the first positional relationship between the first initial string and the second initial string in the target log. The initial log template is obtained by matching the log string arrangement information corresponding to the target log with the template string arrangement information corresponding to the candidate log template, and using the successfully matched candidate log template as the initial log template. The first initial string is the log string in the target log that matches the variable class template string in the initial log template, and the second initial string is the log string in the target log that matches the constant class template string in the initial log template. The log composition data acquisition module is used to acquire the initial log template corresponding to the template identifier and to acquire the log string corresponding to the first encoded data. The log restoration module is used to replace the variable class string in the initial log template with the corresponding log string based on the first positional relationship, so as to obtain the target log.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11 or 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11 or 12.

17. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 11 or 12.

Images