Log compressed file generation method, log compressed file decompression method and device

Abstract

The invention discloses a log compressed file generation method and device and a log compressed file decompression method and device, and relates to the technical field of data processing. On the basis of obtaining a log source file and a dictionary tree and generating an initial log compressed file, a first command line interface log is determined from a plurality of command line interface logs included in the log source file; determining a log timestamp in the first command line interface log as a reference timestamp, and writing the reference timestamp into a file header of the initial log compressed file; further, according to the dictionary tree and the reference timestamp, compressing each command line interface log, and generating a corresponding log mark value; further, each log mark value is written into a data area of the initial log compressed file, and the dictionary tree is written into a dictionary tree area of the initial log compressed file, so that a final log compressed file is generated; therefore, the problem of storage space waste is solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for generating log compressed files, a method for decompressing log compressed files, and an apparatus. Background Technology

[0002] The operation, maintenance and management of electronic equipment typically relies on the collection, storage and analysis of command-line interface logs.

[0003] In related technologies, command-line interface logs are typically stored in plain text format; however, this approach leads to wasted storage space. Summary of the Invention

[0004] This application provides a method for generating compressed log files, a method for decompressing compressed log files, and an apparatus to at least solve the problem of wasted storage space caused by related technologies.

[0005] This application provides a method for generating compressed log files, including: obtaining a log source file and a trie to generate an initial compressed log file; determining the first command-line interface log from multiple command-line interface logs included in the log source file; determining the log timestamp in the first command-line interface log as a base timestamp; writing the base timestamp into the header of the initial compressed log file; compressing each command-line interface log according to the trie and the base timestamp to generate corresponding log tag values; writing each log tag value into the data area of ​​the initial compressed log file, and writing the trie into the trie area of ​​the initial compressed log file to generate the final compressed log file.

[0006] This application also provides a method for decompressing a log compressed file, comprising: obtaining a log compressed file; reading the file header of the log compressed file to obtain a base timestamp and the root node offset of the trie; determining the trie area from the log compressed file based on the root node offset, and reading the trie data in the trie area to generate a trie; reading the data area of ​​the log compressed file to obtain a log tag value; and generating a command-line interface log based on the log tag value, the trie, and the base timestamp.

[0007] This application also provides a log compressed file generation apparatus, comprising: a first acquisition module, configured to acquire a log source file and a trie, and generate an initial log compressed file; a first processing module, configured to determine a first command-line interface log from multiple command-line interface logs included in the log source file; a first generation module, configured to determine the log timestamp in the first command-line interface log as a base timestamp; the first processing module is further configured to write the base timestamp into the file header of the initial log compressed file; compress each command-line interface log according to the trie and the base timestamp to generate corresponding log tag values; the first generation module is further configured to write each log tag value into the data area of ​​the initial log compressed file, and write the trie into the trie area of ​​the initial log compressed file to generate a final log compressed file.

[0008] This application also provides a log compressed file decompression device, comprising: a second acquisition module for acquiring a log compressed file; a second processing module for reading the file header of the log compressed file to obtain a reference timestamp and the root node offset of the trie; determining the trie area from the log compressed file based on the root node offset, and reading the trie data in the trie area to generate a trie; reading the data area of ​​the log compressed file to obtain a log tag value; and a second generation module for generating a command-line interface log based on the log tag value, the trie, and the reference timestamp.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described log compressed file generation methods or the steps of any of the above-described log compressed file decompression methods.

[0010] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described log compressed file generation methods or the steps of any of the above-described log compressed file decompression methods.

[0011] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described log compressed file generation methods or the steps of any of the above-described log compressed file decompression methods.

[0012] The log compressed file generation method, log compressed file decompression method, and device provided in this application, based on obtaining the log source file and the trie and generating an initial log compressed file, determine the first command-line interface log from multiple command-line interface logs included in the log source file; then determine the log timestamp in the first command-line interface log as the base timestamp and write the base timestamp into the file header of the initial log compressed file; then, compress each command-line interface log according to the trie and the base timestamp to generate corresponding log tag values; further, write each log tag value into the data area of ​​the initial log compressed file, and write the trie into the trie area of ​​the initial log compressed file to generate the final log compressed file; that is, use the log tag values ​​determined based on the trie and the base timestamp to replace the plain text format command-line interface logs for storage, reducing the storage space occupied by storing command-line interface logs, thus solving the problem of wasted storage space. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of the log compression file generation method provided in the embodiments of this application;

[0015] Figure 2 A flowchart illustrating the log compression file generation method provided in this application embodiment;

[0016] Figure 3 A flowchart illustrating the log compression file decompression method provided in this application embodiment;

[0017] Figure 4 This is a schematic diagram of the log compression file generation device provided in the embodiments of this application;

[0018] Figure 5 A schematic diagram of the structure of the log compression file decompression device provided in the embodiments of this application;

[0019] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] The operation, maintenance, and management of electronic devices typically rely on the collection, storage, and analysis of command-line interface (COM) logs. Currently, these logs are usually stored in plain text format; however, this approach leads to wasted storage space.

[0023] To address the aforementioned technical problems, this application proposes the following technical concept: Based on obtaining the log source file and trie, and generating an initial compressed log file, the first command-line interface log is determined from multiple command-line interface logs included in the log source file; then, the log timestamp in the first command-line interface log is determined as the base timestamp, and written into the header of the initial compressed log file; furthermore, each command-line interface log is compressed according to the trie and the base timestamp to generate corresponding log tag values; further still, each log tag value is written into the data area of ​​the initial compressed log file, and the trie is written into the trie area of ​​the initial compressed log file to generate the final compressed log file; that is, the log tag values ​​determined based on the trie and the base timestamp are used to replace the plain text format command-line interface logs for storage, reducing the storage space occupancy rate of the command-line interface logs and thus solving the problem of wasted storage space.

[0024] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The specific application environment architecture or specific hardware architecture on which the log compression file generation method depends is described here.

[0026] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario for the log compression file generation method provided in this application embodiment. For example... Figure 1 As shown, this scenario includes a storage device 101 and a terminal device 102. The storage device 101 is used to obtain the log source file and a trie, generating an initial compressed log file; from multiple command-line interface logs included in the log source file, it determines the first command-line interface log; it determines the log timestamp in the first command-line interface log as the base timestamp; it writes the base timestamp into the header of the initial compressed log file; based on the trie and the base timestamp, it compresses each command-line interface log, generating corresponding log tag values; it writes each log tag value into the data area of ​​the initial compressed log file, and writes the trie into the trie area of ​​the initial compressed log file to generate the final compressed log file. After generating the final compressed log file, the storage device 101 sends the final compressed log file to the terminal device 102, so that the terminal device 102 can store the final compressed log file or read and decompress it.

[0027] Figure 2 A flowchart illustrating the log compression file generation method provided in this application embodiment is shown below. Figure 2 As shown, embodiments of this application provide a method for generating compressed log files. The method is described in detail below:

[0028] Step S201: Obtain the log source file and trie, and generate the initial log compressed file.

[0029] For example, the log source file includes multiple command-line interface logs. Specifically, after the command-line interface logs are generated, they are persisted to the log source file. The trie is pre-stored in the storage device 101. In one possible implementation, the method provided in this application embodiment is executed based on the storage device 101. The storage device 101 can also update the trie and store the updated trie so that when the method provided in this application embodiment is executed subsequently, step S201 and subsequent steps are executed based on the updated trie.

[0030] For example, generating an initial log compressed file includes: generating a file format marker value according to a preset file format; and generating the initial log compressed file according to the file format marker value. Specifically, for example, generating a file format marker value "0x43 0x4C 0x49 0x5A" according to the preset file format "CLIZ" corresponding to the log compressed file; creating an empty file corresponding to the log compressed file; initializing the file header, data area, and dictionary area of ​​the empty file corresponding to the log compressed file; writing the file format marker value "0x43 0x4C 0x49 0x5A" into the file header of the empty file corresponding to the log compressed file; and generating the initial log compressed file according to the initialized data area and the initialized dictionary area.

[0031] Step S202: Determine the first command-line interface log from the multiple command-line interface logs included in the log source file.

[0032] For example, the command-line interface log includes a log sequence number and a log timestamp; wherein, the log sequence number is used to indicate the sequential number of the command-line interface log based on time order, and the log timestamp is used to indicate the time when the command-line interface log was generated, such as 2025-05-01-12:31:45.123, and the corresponding data format is year-month-day-hour:minute:second.millisecond.

[0033] Furthermore, based on the log sequence numbers corresponding to the multiple command-line interface logs included in the log source file, the first command-line interface log can be determined.

[0034] Step S203: Determine the log timestamp in the first command-line interface log as the base timestamp.

[0035] Step S204: Write the base timestamp into the header of the initial log compressed file.

[0036] For example, if the log timestamp 2025-05-01-12:31:45.123 is determined as the base timestamp, then the log timestamp 2025-05-01-12:31:45.123 is formatted, and the formatted log timestamp 2025-05-01-12:31:45.123 is written into the header of the initial log compressed file. For example, the formatting of the log timestamp 2025-05-01-12:31:45.123 yields a Unix millisecond timestamp 1746165105123, which is then written into the header of the initial log compressed file.

[0037] Step S205: Compress the logs of each command-line interface according to the trie and the base timestamp, and generate the corresponding log tag values.

[0038] Specifically, for each command-line interface log, the specific implementation steps of step S205 include:

[0039] Step A: Segment the command-line interface log to determine numeric fields, log timestamps, text fields, and command-line interface commands.

[0040] For example, the command-line interface log includes a log sequence number, a log timestamp, a user identifier, an operation source network address, a log result, and a command-line interface command; wherein the log sequence number and the log result are defined as numeric fields, and the user identifier and the operation source network address are defined as text fields.

[0041] Step B: Determine the text tag value based on the trie and the text class field.

[0042] For example, the text tag value corresponding to the text class field is determined based on the matching relationship between the string in the trie and the text class field. Specifically, the text class field includes a user identifier and an operation source network address; the text tag value corresponding to the user identifier is determined based on the matching relationship between the string in the trie and the user identifier; the text tag value corresponding to the operation source network address is determined based on the matching relationship between the string in the trie and the operation source network address.

[0043] Specifically, the implementation steps of step B include:

[0044] Step B1: Determine whether the string in the trie matches the text field.

[0045] Step B2: If the string in the trie matches the text field, then the tag value corresponding to the string that matches the text field is determined as the text tag value.

[0046] For example, taking the operation source network address in a text field as an example, if the operation source network address is 192.168.1.101, and the string "192.168.1.101" in the trie is consistent with the operation source network address "192.168.1.101", then the tag value corresponding to the string "192.168.1.101" in the trie is determined as the text tag value.

[0047] Step B3: If the string in the trie is inconsistent with the text field, then create a new character node in the trie according to the text field to obtain the new string corresponding to the text field.

[0048] For example, taking the source network address of a text field as an example, if the source network address is 192.168.1.102, and the string in the trie is inconsistent with the source network address "192.168.1.102", then a new character node is created in the trie based on the source network address "192.168.1.102" to obtain a new string corresponding to the text field. Since the character nodes in the trie constitute the string "192.168.1.101", a new character node "2" only needs to be created after the string "192.168.1.10" to obtain the new string "192.168.1.102". The strings "192.168.1.101" and "192.168.1.102" share the string "192.168.1.10", thus saving storage space.

[0049] Step B4: Generate a new tag value based on the newly created string.

[0050] Step B5: Define the newly created tag value as a text tag value.

[0051] For example, after obtaining the newly created string "192.168.1.102" corresponding to the operation source network address "192.168.1.102", a new tag value corresponding to the newly created string can be generated, and then the new tag value can be determined as the text tag value of the operation source network address "192.168.1.102".

[0052] Furthermore, the method provided in this application embodiment also includes: updating the trie of newly created character nodes according to the newly created tag value to obtain the updated trie; wherein, the updated trie is used to compress the command-line interface log after the current command-line interface log, avoiding the need to create new character nodes again to obtain the new string and generate the corresponding newly created tag value when the operation source network address of the subsequent command-line interface log is "192.168.1.102" during the compression process of the subsequent command-line interface log, thereby improving data processing efficiency.

[0053] Step C: Determine the command-line tag value based on the trie and command-line interface commands.

[0054] Specifically, the specific implementation steps of step C include:

[0055] Step C1 involves semantic segmentation of the command-line interface commands to obtain the command type field, command body field, and command parameter field.

[0056] For example, if the command-line interface command is "mcsop mkuser -usergrp 0 -name test -disabled", then the command-line interface command is semantically segmented to obtain the command type field "mcsop", the command body field "mkuser", and the command parameter field "-usergrp 0 -name test -disabled".

[0057] Step C2: Determine the command type tag value based on the trie and the command type field.

[0058] In this embodiment, the specific implementation process of step C2 is similar in principle and technical effect to that of step B, and will not be repeated here. For example, based on the command type field "mcsop", the corresponding command type tag value "0x0001" is determined from the trie.

[0059] Step C3: Determine the command body tag value based on the trie and the command body field.

[0060] In this embodiment, the specific implementation process of step C3 is similar in principle and technical effect to that of step B, and will not be repeated here. For example, based on the command body field "mkuser", the corresponding command body tag value "0x000A" is determined from the trie.

[0061] Step C4: Determine the command parameter tag value based on the trie and the command parameter field.

[0062] In one possible implementation, the specific implementation process of step C4 is similar in principle and technical effect to the specific implementation process of step B, and will not be repeated here.

[0063] In another possible implementation, the specific implementation process of step C4 includes: classifying the command parameter fields according to the parameter type with value and the parameter type without value to obtain parameter fields with value and parameter fields without value; determining the first command parameter tag value according to the trie and the parameter fields with value; determining the second command parameter tag value according to the trie and the parameter fields without value; and concatenating the first command parameter tag value and the second command parameter tag value to obtain the command parameter tag value.

[0064] Specifically, for example, the command parameter field "-usergrp 0 -name test -disabled" can be categorized into the parameter field with value "-usergrp 0", the parameter field with value "-name test", and the parameter field without value "-disabled". Furthermore, for example, based on the value-bearing parameter field "-usergrp 0", the corresponding first command parameter flag value is determined from the trie to be "0x0003 0x0105", where "-usergrp" corresponds to 0x0003 and "0" corresponds to 0x0105; based on the value-bearing parameter field "-name test", the corresponding first command parameter flag value is determined from the trie to be "0x0005 0x020C"; based on the valueless parameter field "-disabled" being a valueless parameter, the highest bit of the second command parameter flag value is set to "1" as an identifier for a valueless parameter, so the corresponding second command parameter flag value is determined from the trie to be "0x8008", where the highest bit "0x8000" in "0x8008" is a valueless identifier. Then, by concatenating the first command parameter marker value "0x00030x0105", the first command parameter marker value "0x0005 0x020C", and the second command parameter marker value "0x8008", the command parameter marker value corresponding to the command parameter field "-usergrp 0 -name test -disabled" is obtained as "0x0003 0x0105 0x0005 0x020C 0x8008", which means that the command parameter field is stored through chained storage.

[0065] In this embodiment, by classifying the command parameter fields according to parameter types with values ​​and parameter types without values, the storage redundancy rate of the final command parameter tag values ​​is reduced, and the parsing efficiency of command parameter tag values ​​is improved when decompressing the log compressed file.

[0066] Step C5: Obtain the first separator tag value and the second separator tag value from the trie; wherein, the first separator tag value is used to separate the command type tag value and the command body tag value, and the second separator tag value is used to separate the command body tag value and the command parameter tag value.

[0067] In this embodiment, the first dividing marker value is "0xA0" and the second dividing marker value is "0xA1".

[0068] Step C6: Concatenate the command type marker value, the first separator marker value, the command body marker value, the second separator marker value, and the command parameter marker value to obtain the command line marker value.

[0069] For example, the command type marker value is "0x0001", the command body marker value is "0x000A", the command parameter marker value is "0x0003 0x0105 0x0005 0x020C 0x8008", the first separator marker value is "0xA0", and the second separator marker value is "0xA1". The command type marker value, the first separator marker value, the command body marker value, the second separator marker value, and the command parameter marker value are concatenated in the order to obtain the corresponding command line marker value "0x0001 0xA0 0x000A 0xA10x0003 0x0105 0x0005 0x020C 0x8008".

[0070] In this embodiment, the trie implements a cross-field sharing and reuse mechanism. For example, if the user identifier in the text field and the command body field in the command line interface command are the same in the first N characters, then the string composed of the character nodes corresponding to the first N characters in the trie is shared. The above method improves the file compression rate of the log compressed file.

[0071] Step D: Calculate the difference between the base timestamp and the log timestamp to generate the log time offset.

[0072] For example, the base timestamp is the log timestamp in the first command-line interface log, and then the log timestamps in each command-line interface log are converted into the number of milliseconds that differ from the base timestamp, that is, the log time offset is generated.

[0073] Step E involves concatenating the numeric field, text tag value, command line tag value, and log time offset to obtain the log tag value.

[0074] For example, the log tag value can be obtained by concatenating the numeric field, text tag value, command line tag value, and log time offset based on the order of log sequence number, log timestamp, user identifier, operation source network address, log result, and command line interface command.

[0075] Step S206: Write each log tag value into the data area of ​​the initial log compressed file, and write the trie into the trie area of ​​the initial log compressed file to generate the final log compressed file.

[0076] In one possible implementation, after writing each log tag value to the data area of ​​the initial log compressed file, the trie is written to the trie area of ​​the initial log compressed file, thereby generating the final log compressed file.

[0077] Furthermore, the specific implementation process of writing the trie into the trie area of ​​the initial log compression file includes:

[0078] Step S206a: Determine the root node offset of the trie based on the log tag values ​​corresponding to each command-line interface log written to the data area of ​​the initial log compressed file.

[0079] For example, based on the log tag values ​​corresponding to each command-line interface log written to the data area of ​​the initial log compressed file, the amount of data stored in the data area of ​​the initial log compressed file is determined, and then the position of the root node of the storage trie is determined according to the amount of data stored in the data area, that is, the offset of the root node of the trie is determined.

[0080] Step S206b: Write the root node offset into the header of the initial log compressed file.

[0081] For example, the root node offset is written into the header of the initial log compressed file so that when the log compressed file is decompressed, the position of the root node of the trie can be determined from the log compressed file by reading the root node offset of the trie in the header.

[0082] Step S206c: Based on the root node offset, write the trie into the trie area of ​​the initial log compressed file.

[0083] For example, the position of the trie region in the initial log compressed file is determined based on the root node offset, and then the position of the root node of the trie in the trie region is determined. Then, based on the root node of the trie, the trie is written into the trie region of the initial log compressed file, thereby storing the trie in the form of trie data in the initial log compressed file.

[0084] Furthermore, if the trie is updated during the process of determining each log tag value, the updated trie is written into the trie area of ​​the log compressed file. The implementation process of writing the updated trie into the trie area of ​​the log compressed file is similar in principle and technical effect to the specific implementation process of steps S206a-S206c, and will not be elaborated further here.

[0085] In this embodiment, based on obtaining the log source file and the trie, and generating an initial compressed log file, the first command-line interface log is determined from the multiple command-line interface logs included in the log source file. Then, the log timestamp in the first command-line interface log is determined as the base timestamp and written into the header of the initial compressed log file. Next, each command-line interface log is compressed according to the trie and the base timestamp to generate corresponding log tag values. Further, each log tag value is written into the data area of ​​the initial compressed log file, and the trie is written into the trie area of ​​the initial compressed log file to generate the final compressed log file. That is, the log tag values ​​determined based on the trie and the base timestamp replace the plain text format command-line interface logs for storage, reducing the storage space occupied by the command-line interface logs and solving the problem of wasted storage space.

[0086] Figure 3 This is a flowchart illustrating the log compression file decompression method provided in the embodiments of this application, as shown below. Figure 3 As shown, the method includes:

[0087] Step S301: Obtain the log compressed file.

[0088] Step S302: Read the header of the log compressed file to obtain the base timestamp and the root node offset of the trie.

[0089] In this embodiment, the file header of the log compressed file is read to obtain the file format marker value. Based on the file format marker value, it is determined whether the file format of the log compressed file meets the decompression conditions. If the decompression conditions are met, the file header of the log compressed file is further read to obtain the base timestamp and the root node offset of the trie. If the decompression conditions are not met, a file format error message is generated to inform the user that the format of the log compressed file does not meet the decompression conditions.

[0090] Step S303: Based on the root node offset, determine the trie area from the log compressed file and read the trie data in the trie area to generate the trie.

[0091] Step S304: Read the data area of ​​the log compressed file to obtain the log tag value.

[0092] Step S305: Generate command-line interface logs based on log tag values, a trie, and a baseline timestamp.

[0093] Specifically, the specific implementation steps of step S305 include:

[0094] Step S305a: Determine the numeric field, text field, command line field, and log time offset based on the log flag value.

[0095] Step S305b: Calculate the sum of the log time offset and the base timestamp to obtain the log timestamp.

[0096] In this embodiment, the log time offset is stored in milliseconds. The data format of the base timestamp and the log timestamp is "year-month-day-hour:minute:second.millisecond". Therefore, the data format of the log time offset is first converted to "year-month-day-hour:minute:second.millisecond", and then combined with the base timestamp, the log timestamp can be obtained.

[0097] Step S305c: Based on the text tag value, perform a string lookup in the trie to obtain the text class field.

[0098] Step S305d: Based on the command line marker value, perform a string lookup in the trie to obtain the command line interface command.

[0099] For example, if the command line flag value is "0x0001 0xA0 0x000A 0xA1 0x0003 0x0105 0x0005 0x020C 0x8008", then a string search in the trie will yield the command type field "mcsop" corresponding to "0x0001", the command body field "mkuser" corresponding to "0x000A", and the command parameter field "-usergrp0 -name test -disabled" corresponding to "0x0003 0x0105 0x0005 0x020C 0x8008". This leads to the command line command "mcsop mkuser -usergrp 0 -nametest -disabled".

[0100] Step S305e involves concatenating numeric fields, log timestamps, text fields, and command-line interface commands to obtain the command-line interface log.

[0101] For example, based on the order of log sequence number, log timestamp, user identifier, operation source network address, log result, and command-line interface command, the numeric fields, log timestamp, text fields, and command-line interface command are concatenated to obtain the command-line interface log.

[0102] Furthermore, in one possible implementation, after step S305, the method provided in this application embodiment further includes: deleting the trie.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0104] Figure 4 This is a schematic diagram of the log compression file generation device provided in an embodiment of this application. Figure 4 As shown, embodiments of this application also provide a log compression file generation device 40, including a first acquisition module 401, a first processing module 402, and a first generation module 403.

[0105] The first acquisition module 401 is used to acquire the log source file and the trie to generate the initial log compressed file.

[0106] The first processing module 402 is used to determine the first command-line interface log from the multiple command-line interface logs included in the log source file.

[0107] The first generation module 403 is used to determine the log timestamp in the first command-line interface log as the base timestamp.

[0108] The first processing module 402 is also used to write the base timestamp into the header of the initial log compression file; and to compress the logs of each command-line interface according to the trie and the base timestamp to generate the corresponding log tag value.

[0109] The first generation module 403 is also used to write each log tag value into the data area of ​​the initial log compressed file and write the trie into the trie area of ​​the initial log compressed file to generate the final log compressed file.

[0110] In one possible implementation, when the first processing module 402 compresses each command-line interface log based on the trie and the base timestamp to generate the corresponding log tag value, it specifically performs the following steps for each command-line interface log: splitting the command-line interface log to determine numeric fields, log timestamps, text fields, and command-line interface commands; determining text tag values ​​based on the trie and text fields; determining command-line tag values ​​based on the trie and command-line interface commands; calculating the difference between the base timestamp and the log timestamp to generate a log time offset; and concatenating the numeric fields, text tag values, command-line tag values, and log time offset to obtain the log tag value.

[0111] In one possible implementation, when the first processing module 402 determines the command line tag value based on the trie and the command line interface command, it specifically performs the following: semantic segmentation of the command line interface command to obtain a command type field, a command body field, and a command parameter field; determines the command type tag value based on the trie and the command type field; determines the command body tag value based on the trie and the command body field; determines the command parameter tag value based on the trie and the command parameter field; obtains a first separator tag value and a second separator tag value from the trie; wherein the first separator tag value is used to separate the command type tag value and the command body tag value, and the second separator tag value is used to separate the command body tag value and the command parameter tag value; and concatenates the command type tag value, the first separator tag value, the command body tag value, the second separator tag value, and the command parameter tag value to obtain the command line tag value.

[0112] In one possible implementation, when determining the text tag value based on the trie and the text class field, the first processing module 402 is specifically used to: determine whether the string in the trie is consistent with the text class field; if the string in the trie is consistent with the text class field, then determine the tag value corresponding to the string corresponding to the text class field as the text tag value; if the string in the trie is inconsistent with the text class field, then create a new character node in the trie based on the text class field to obtain a new string corresponding to the text class field; generate a new tag value based on the new string; and determine the new tag value as the text tag value; the first processing module 402 is also used to: update the trie that creates the new character node based on the new tag value to obtain an updated trie.

[0113] In one possible implementation, when the first generation module 403 writes the trie into the trie area of ​​the initial log compressed file, it is specifically used to: write the updated trie into the trie area of ​​the log compressed file.

[0114] In one possible implementation, when the first generation module 403 writes the trie into the trie area of ​​the initial log compressed file, it specifically performs the following steps: determining the root node offset of the trie based on the log tag values ​​corresponding to each command-line interface log written to the data area of ​​the initial log compressed file; writing the root node offset into the file header of the initial log compressed file; and writing the trie into the trie area of ​​the initial log compressed file based on the root node offset.

[0115] In one possible implementation, when generating the initial log compressed file, the first acquisition module 401 is specifically used to: generate a file format marker value according to a preset file format; and generate the initial log compressed file according to the file format marker value.

[0116] For a description of the features in the embodiment corresponding to the log compression file generation device 40, please refer to the relevant description in the embodiment corresponding to the log compression file generation method, which will not be repeated here.

[0117] Figure 5 This is a schematic diagram of the log compression file decompression device provided in an embodiment of this application. Figure 5 As shown, embodiments of this application also provide a log compression file decompression device 50, including a second acquisition module 501, a second processing module 502, and a second generation module 503.

[0118] The second acquisition module 501 is used to acquire the log compressed file.

[0119] The second processing module 502 is used to read the file header of the log compressed file to obtain the base timestamp and the root node offset of the trie; determine the trie area from the log compressed file based on the root node offset, and read the trie data in the trie area to generate the trie; and read the data area of ​​the log compressed file to obtain the log tag value.

[0120] The second generation module 503 is used to generate command-line interface logs based on log tag values, a trie, and a base timestamp.

[0121] In one possible implementation, when the second generation module 503 generates the command-line interface log based on the log tag value, the trie, and the base timestamp, it specifically performs the following steps: determining the numeric field, text tag value, command-line tag value, and log time offset based on the log tag value; calculating the sum of the log time offset and the base timestamp to obtain the log timestamp; performing a string query in the trie based on the text tag value to obtain the text field; performing a string query in the trie based on the command-line tag value to obtain the command-line interface command; and concatenating the numeric field, log timestamp, text field, and command-line interface command to obtain the command-line interface log.

[0122] For a description of the features of the log compression file decompression device 50 in the corresponding embodiment, please refer to the relevant description of the log compression file decompression method in the corresponding embodiment, which will not be repeated here.

[0123] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0124] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-described log compressed file generation method embodiment or log compressed file decompression method embodiment.

[0125] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0126] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0127] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0128] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0129] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the log compressed file generation method or the steps in the log compressed file decompression method.

[0130] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0131] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the log compressed file generation method or the steps in the log compressed file decompression method.

[0132] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in any of the above embodiments of the log compressed file generation method or the steps in the log compressed file decompression method.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] The above provides a detailed description of a log compressed file generation method, a log compressed file decompression method, and an apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for generating compressed log files, characterized in that, include: Obtain the log source file and trie, and generate the initial compressed log file; From the multiple command-line interface logs included in the log source file, determine the first command-line interface log; The log timestamp in the first command-line interface log is determined as the base timestamp; Write the base timestamp into the header of the initial log compressed file; Based on the trie and the base timestamp, the logs of each command-line interface are compressed to generate corresponding log tag values; Each of the log tag values ​​is written to the data area of ​​the initial log compressed file, and the trie is written to the trie area of ​​the initial log compressed file to generate the final log compressed file.

2. The log compressed file generation method according to claim 1, characterized in that, The step of compressing the command-line interface logs based on the trie and the base timestamp to generate corresponding log tag values ​​includes: For each command-line interface log: The command-line interface logs are broken down to identify numeric fields, log timestamps, text fields, and command-line interface commands. The text tag value is determined based on the trie and the text class field; Determine the command-line flag value based on the trie and the command-line interface commands; Calculate the difference between the base timestamp and the log timestamp to generate the log time offset; The log tag value is obtained by concatenating the numeric field, the text tag value, the command line tag value, and the log time offset.

3. The log compressed file generation method according to claim 2, characterized in that, The step of determining the command-line flag value based on the trie and the command-line interface commands includes: The command-line interface commands are semantically segmented to obtain command type field, command body field, and command parameter field; Determine the command type tag value based on the trie and the command type field; Determine the command body tag value based on the trie and the command body field; Determine the command parameter tag value based on the trie and the command parameter field; Obtain a first separator tag value and a second separator tag value from the trie; wherein the first separator tag value is used to separate the command type tag value and the command body tag value, and the second separator tag value is used to separate the command body tag value and the command parameter tag value; The command type marker value, the first separator marker value, the command body marker value, the second separator marker value, and the command parameter marker value are concatenated to obtain the command line marker value.

4. The log compressed file generation method according to claim 2, characterized in that, The step of determining the text tag value based on the trie and the text class field includes: Determine whether the string in the trie matches the text class field; If the string in the trie matches the text class field, then the tag value corresponding to the string corresponding to the text class field is determined as the text tag value; If the string in the trie is inconsistent with the text class field, a new character node is created in the trie according to the text class field to obtain a new string corresponding to the text class field; Generate a new tag value based on the newly created string; The newly created tag value is determined as the text tag value; The method further includes: Based on the newly created tag value, the trie that creates the new character node is updated to obtain the updated trie.

5. The log compressed file generation method according to claim 4, characterized in that, The step of writing the trie into the trie area of ​​the initial log compressed file includes: The updated trie is written into the trie area of ​​the log compressed file.

6. The log compressed file generation method according to claim 1, characterized in that, The step of writing the trie into the trie area of ​​the initial log compressed file includes: The root node offset of the trie is determined based on the log tag values ​​corresponding to each command-line interface log written to the data area of ​​the initial log compressed file. Write the root node offset into the header of the initial log compressed file; Based on the root node offset, the trie is written into the trie area of ​​the initial log compressed file.

7. The log compressed file generation method according to any one of claims 1-6, characterized in that, The process of generating the initial compressed log file includes: Generate file format tag values ​​based on the preset file format; The initial log compressed file is generated based on the file format tag value.

8. A method for decompressing a log compressed file, characterized in that, include: Obtain the compressed log file; Read the header of the compressed log file to obtain the base timestamp and the root node offset of the trie; Based on the root node offset, the trie region is determined from the log compressed file, and the trie data in the trie region is read to generate the trie; Read the data area of ​​the compressed log file to obtain the log tag value; The command-line interface log is generated based on the log tag value, the trie, and the base timestamp.

9. The log compression file decompression method according to claim 8, characterized in that, The step of generating command-line interface logs based on the log tag value, the trie, and the base timestamp includes: Based on the log tag values, determine the numeric fields, text tag values, command line tag values, and log time offsets; The log timestamp is obtained by summing the log time offset and the base timestamp. Based on the text tag value, a string search is performed in the trie to obtain the text class field; Based on the command line tag value, a string search is performed in the trie to obtain the command line interface command; The command-line interface log is obtained by concatenating the numerical fields, the log timestamp, the text fields, and the command-line interface commands.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the log compressed file generation method as described in any one of claims 1 to 7, or to implement the steps of the log compressed file decompression method as described in any one of claims 8 to 9.

Images