Log compressed file generation method, log compressed file decompression method and equipment

Abstract

This application discloses a method for generating and decompressing compressed log files, and an apparatus for doing so, relating to the field of data processing technology. Based on obtaining a source log file and a trie, and generating an initial compressed log file, the method identifies the first command-line interface log from multiple command-line interface logs included in the source log file. Then, the log timestamp in the first command-line interface log is determined as the base timestamp and written into the header of the initial compressed log file. Next, each command-line interface log is compressed according to the trie and the base timestamp to generate corresponding log tag values. Finally, each log tag value is written into the data area of ​​the initial compressed log file, and the trie is written into the trie area of ​​the initial compressed log file to generate the final compressed log file. This solves the problem of wasted storage space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a log compressed file generation method, a log compressed file decompression method and equipment. BACKGROUND

[0002] For the operation maintenance management of electronic equipment, usually relies on the collection, storage and analysis of command line interface logs.

[0003] In the related art, for the collected command line interface logs, the command line interface logs are usually stored in the form of pure text format; however, the related art solution causes the problem of storage space waste. SUMMARY

[0004] The present application provides a log compressed file generation method, a log compressed file decompression method and equipment to at least solve the problem of storage space waste caused by the related art.

[0005] The present application provides a log compressed file generation method, comprising: obtaining a log source file and a dictionary tree, generating an initial log compressed file; determining a first command line interface log from a plurality of command line interface logs included in the log source file; determining a log timestamp in the first command line interface log as a reference timestamp; writing the reference timestamp into a file header of the initial log compressed file; compressing each command line interface log according to the dictionary tree and the reference timestamp to generate a corresponding log mark value; writing each log mark value into a data area of the initial log compressed file, and writing the dictionary tree into a dictionary tree area of the initial log compressed file to generate a final log compressed file.

[0006] The present application also provides a log compressed file decompression method, comprising: obtaining a log compressed file; reading a file header of the log compressed file to obtain a reference timestamp and a root node offset of a dictionary tree; determining a dictionary tree area from the log compressed file according to the root node offset, and reading dictionary tree data in the dictionary tree area to generate a dictionary tree; reading a data area of the log compressed file to obtain a log mark value; and generating a command line interface log according to the log mark value, the dictionary tree and the reference timestamp.

[0007] The application further provides a log compression file generation device, comprising: a first acquisition module, configured to acquire a log source file and a dictionary tree, and generate an initial log compression file; a first processing module, configured to determine a first command line interface log from a plurality of command line interface logs included in the log source file; a first generation module, configured to determine a log timestamp in the first command line interface log as a reference timestamp; the first processing module is further configured to write the reference timestamp into a file header of the initial log compression file; compress each command line interface log according to the dictionary tree and the reference timestamp to generate a corresponding log mark value; the first generation module is further configured to write each log mark value into a data area of the initial log compression file, and write the dictionary tree into a dictionary tree area of the initial log compression file, so as to generate a final log compression file.

[0008] The application further provides a log compression file decompression device, comprising: a second acquisition module, configured to acquire a log compression file; a second processing module, configured to read a file header of the log compression file to obtain a reference timestamp and a root node offset of a dictionary tree; determine a dictionary tree area from the log compression file according to the root node offset, and read dictionary tree data in the dictionary tree area to generate the dictionary tree; read a data area of the log compression file to obtain a log mark value; and a second generation module, configured to generate a command line interface log according to the log mark value, the dictionary tree and the reference timestamp.

[0009] The application further provides an electronic device, comprising: a memory, configured to store a computer program; and a processor, configured to execute the computer program to implement steps of any one of the log compression file generation methods or steps of any one of the log compression file decompression methods.

[0010] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement steps of any one of the log compression file generation methods or steps of any one of the log compression file decompression methods.

[0011] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement steps of any one of the log compression file generation methods or steps of any one of the log compression file decompression methods.

[0012] The log compression file generation method, the log compression file decompression method and the device provided by the application, on the basis of obtaining a log source file and a dictionary tree, generating an initial log compression file, determining a first command line interface log from a plurality of command line interface logs included in the log source file; then determining a log timestamp in the first command line interface log as a reference timestamp, and writing the reference timestamp into a file header of the initial log compression file; then, according to the dictionary tree and the reference timestamp, compressing each command line interface log to generate a corresponding log mark value; further, writing each log mark value into a data area of the initial log compression file, and writing the dictionary tree into a dictionary tree area of the initial log compression file, to generate a final log compression file; that is, replacing the command line interface log in pure text format with the log mark value determined based on the dictionary tree and the reference timestamp for storage, reducing the storage space occupancy rate of the command line interface log, that is, solving the problem of storage space waste. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0014] Figure 1 The application scenario diagram of the log compression file generation method provided by the embodiments of the application is shown.

[0015] Figure 2 The flowchart of the log compression file generation method provided by the embodiments of the application is shown.

[0016] Figure 3 The flowchart of the log compression file decompression method provided by the embodiments of the application is shown.

[0017] Figure 4 The structure diagram of the log compression file generation device provided by the embodiments of the application is shown.

[0018] Figure 5 The structure diagram of the log compression file decompression device provided by the embodiments of the application is shown.

[0019] Figure 6 The structure diagram of the electronic device provided by the application is shown. DETAILED DESCRIPTION

[0020] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application. Based on the embodiments of the present application, any person skilled in the art can obtain all other embodiments without creative work.

[0021] It should be noted that in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0022] For the operation maintenance management of electronic equipment, it is usually dependent on the collection, storage and analysis of command line interface logs. In the related art, for the collected command line interface logs, the command line interface logs are usually stored in the form of pure text format. However, the scheme of the related art causes the problem of storage space waste.

[0023] In order to solve the above technical problems, the embodiments of the present application propose the following technical concept: on the basis of obtaining a log source file and a dictionary tree and generating an initial log compressed file, a first command line interface log is determined from a plurality of command line interface logs included in the log source file; then the log timestamp in the first command line interface log is determined as a reference timestamp, and the reference timestamp is written into the file header of the initial log compressed file; then, according to the dictionary tree and the reference timestamp, each command line interface log is compressed to generate a corresponding log marker value; further, each log marker value is written into the data area of the initial log compressed file, and the dictionary tree is written into the dictionary tree area of the initial log compressed file, to generate a final log compressed file; that is, the log marker value determined based on the dictionary tree and the reference timestamp is used to replace the command line interface log in the pure text format for storage, which reduces the storage space occupancy rate of the command line interface log, i.e. solves the problem of storage space waste.

[0024] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0025] In combination with the specific application environment architecture or specific hardware architecture on which the log compressed file generation method is dependent, the specific application environment architecture or specific hardware architecture is described here.

[0026] ReferenceFigure 1 , Figure 1 This is a schematic diagram illustrating an application scenario for the log compression file generation method provided in this application embodiment. For example... Figure 1 As shown, this scenario includes a storage device 101 and a terminal device 102. The storage device 101 is used to obtain the log source file and a trie, generating an initial compressed log file; from multiple command-line interface logs included in the log source file, it determines the first command-line interface log; it determines the log timestamp in the first command-line interface log as the base timestamp; it writes the base timestamp into the header of the initial compressed log file; based on the trie and the base timestamp, it compresses each command-line interface log, generating corresponding log tag values; it writes each log tag value into the data area of ​​the initial compressed log file, and writes the trie into the trie area of ​​the initial compressed log file to generate the final compressed log file. After generating the final compressed log file, the storage device 101 sends the final compressed log file to the terminal device 102, so that the terminal device 102 can store the final compressed log file or read and decompress it.

[0027] Figure 2 A flowchart illustrating the log compression file generation method provided in this application embodiment is shown below. Figure 2 As shown, embodiments of this application provide a method for generating compressed log files. The method is described in detail below:

[0028] Step S201: Obtain the log source file and trie, and generate the initial log compressed file.

[0029] For example, the log source file includes multiple command-line interface logs. Specifically, after the command-line interface logs are generated, they are persisted to the log source file. The trie is pre-stored in the storage device 101. In one possible implementation, the method provided in this application embodiment is executed based on the storage device 101. The storage device 101 can also update the trie and store the updated trie so that when the method provided in this application embodiment is executed subsequently, step S201 and subsequent steps are executed based on the updated trie.

[0030] Exemplarily, the initial log compression file is generated, including: generating a file format marker value according to a preset file format; and generating the initial log compression file according to the file format marker value. Specifically, for example, the file format marker value "0x43 0x4C 0x49 0x5A" is generated according to the preset file format "CLIZ" corresponding to the log compression file; an empty file corresponding to the log compression file is established; file header initialization, data area initialization, and dictionary tree area initialization are performed on the empty file corresponding to the log compression file; the file format marker value "0x43 0x4C 0x49 0x5A" is written into the file header of the empty file corresponding to the log compression file, and the initial log compression file is generated according to the initialized data area and the initialized dictionary tree area.

[0031] Step S202: determining a first command line interface log from a plurality of command line interface logs included in the log source file.

[0032] Exemplarily, the command line interface log includes a log sequence number and a log timestamp; the log sequence number is used to indicate a sequential number of the command line interface log based on time order, and the log timestamp is used to indicate a generation moment of the command line interface log; the log timestamp is, for example, 2025-05-01-12:31:45.123, and the corresponding data format is year-month-day-hour:minute:second.millisecond.

[0033] Further, the first command line interface log can be determined based on the log sequence numbers corresponding to the plurality of command line interface logs included in the log source file.

[0034] Step S203: determining the log timestamp in the first command line interface log as a reference timestamp.

[0035] Step S204: writing the reference timestamp into the file header of the initial log compression file.

[0036] Exemplarily, for example, the log timestamp 2025-05-01-12:31:45.123 is determined as the reference timestamp, then the log timestamp 2025-05-01-12:31:45.123 is format-converted, and the format-converted log timestamp 2025-05-01-12:31:45.123 is written into the file header of the initial log compression file. For example, the log timestamp 2025-05-01-12:31:45.123 is format-converted to obtain a Unix millisecond timestamp 1746165105123, and the Unix millisecond timestamp 1746165105123 is written into the file header of the initial log compression file.

[0037] Step S205: compressing each command line interface log according to the dictionary tree and the reference timestamp to generate a corresponding log mark value.

[0038] Specifically, for each command line interface log, the specific implementation steps of step S205 include:

[0039] Step A, splitting the command line interface log to determine the numerical field, log timestamp, text field, and command line interface command.

[0040] For example, the command line interface log includes log sequence number, log timestamp, user identification, operation source network address, log result, and command line interface command; wherein the log sequence number and log result are determined as numerical fields, and the user identification and operation source network address are determined as text fields.

[0041] Step B, determining the text mark value according to the dictionary tree and the text field.

[0042] For example, according to the matching relationship between the string in the dictionary tree and the text field, the text mark value corresponding to the text field is determined. Specifically, the text field includes user identification and operation source network address; according to the matching relationship between the string in the dictionary tree and the user identification, the text mark value corresponding to the user identification is determined; and according to the matching relationship between the string in the dictionary tree and the operation source network address, the text mark value corresponding to the operation source network address is determined.

[0043] Specifically, the specific implementation steps of step B include:

[0044] Step B1, judging whether the string in the dictionary tree is consistent with the text field.

[0045] Step B2, if the string in the dictionary tree is consistent with the text field, the mark value corresponding to the string corresponding to the text field is determined as the text mark value.

[0046] For example, taking the operation source network address in the text field as an example, the operation source network address is 192.168.1.101, the string "192.168.1.101" in the dictionary tree is consistent with the operation source network address "192.168.1.101", and the mark value corresponding to the string "192.168.1.101" in the dictionary tree is determined as the text mark value.

[0047] Step B3, if the string in the dictionary tree is not consistent with the text field, a new character node is created in the dictionary tree according to the text field to obtain a new string corresponding to the text field.

[0048] Exemplarily, taking the operation source network address in the text type field as an example, the operation source network address is 192.168.1.102, and the string in the dictionary tree is inconsistent with the operation source network address “192.168.1.102”. Therefore, a new character node is created in the dictionary tree according to the operation source network address “192.168.1.102” to obtain a newly created string corresponding to the text type field. Since the character nodes in the dictionary tree form the string “192.168.1.101”, only a new character node “2” needs to be created after the string “192.168.1.10” formed by the character nodes to obtain the newly created string “192.168.1.102”. Since the string “192.168.1.101” and the string “192.168.1.102” share the string “192.168.1.10”, storage space is saved.

[0049] Step B4, generating a newly created mark value according to the newly created string.

[0050] Step B5, determining the newly created mark value as a text mark value.

[0051] Exemplarily, after obtaining the newly created string “192.168.1.102” corresponding to the operation source network address “192.168.1.102”, a newly created mark value corresponding to the newly created string can be generated, and then the newly created mark value is determined as the text mark value of the operation source network address “192.168.1.102”.

[0052] Further, the method provided by the embodiment of the application further includes: updating the dictionary tree in which the new character node is created according to the newly created mark value to obtain an updated dictionary tree; wherein the updated dictionary tree is used to compress the command line interface log after the current command line interface log, thereby avoiding the need to create a new character node again to obtain a newly created string and generate a corresponding newly created mark value when the operation source network address “192.168.1.102” of the subsequent command line interface log during the compression of the subsequent command line interface log, thereby improving the data processing efficiency.

[0053] Step C, determining a command line mark value according to the dictionary tree and the command line interface command.

[0054] Specifically, the specific implementation steps of step C include:

[0055] Step C1, performing semantic segmentation on the command line interface command to obtain a command type field, a command body field and a command parameter field.

[0056] For example, the command line interface command is "mcsop mkuser -usergrp 0 -name test -disabled", the command line interface command is semantically segmented to obtain the command type field "mcsop", the command body field "mkuser", and the command parameter field "-usergrp 0 -name test -disabled".

[0057] Step C2, determining the command type tag value according to the dictionary tree and the command type field.

[0058] In the embodiment, the implementation process of step C2 is similar to the implementation principle and technical effect of the implementation process of step B, and will not be repeated here. For example, based on the command type field "mcsop", the corresponding command type tag value "0x0001" is determined from the dictionary tree.

[0059] Step C3, determining the command body tag value according to the dictionary tree and the command body field.

[0060] In the embodiment, the implementation process of step C3 is similar to the implementation principle and technical effect of the implementation process of step B, and will not be repeated here. For example, based on the command body field "mkuser", the corresponding command body tag value "0x000A" is determined from the dictionary tree.

[0061] Step C4, determining the command parameter tag value according to the dictionary tree and the command parameter field.

[0062] In one possible implementation, the implementation process of step C4 is similar to the implementation principle and technical effect of the implementation process of step B, and will not be repeated here.

[0063] In another possible implementation, the implementation process of step C4 includes: classifying the command parameter field according to the value parameter type and the valueless parameter type to obtain the value parameter field and the valueless parameter field; determining the first command parameter tag value according to the dictionary tree and the value parameter field; determining the second command parameter tag value according to the dictionary tree and the valueless parameter field; and splicing the first command parameter tag value and the second command parameter tag value to obtain the command parameter tag value.

[0064] Specifically, for example, the command parameter field "-usergrp 0 -name test -disabled" is classified to obtain the value parameter field "-usergrp 0", the value parameter field "-name test", and the valueless parameter field "-disabled". Further, for example, based on the value parameter field "-usergrp 0", the corresponding first command parameter marker value is determined from the dictionary tree as "0x0003 0x0105", wherein "-usergrp" corresponds to 0x0003 and "0" corresponds to 0x0105; based on the value parameter field "-name test", the corresponding first command parameter marker value is determined from the dictionary tree as "0x0005 0x020C"; and based on the valueless parameter field "-disabled" being a valueless parameter, the highest bit of the second command parameter marker value is set as "1" as an identifier of the valueless parameter, and the corresponding second command parameter marker value is determined from the dictionary tree as "0x8008", wherein the highest bit "0x8000" in "0x8008" is the valueless identifier. Further, the first command parameter marker value "0x0003 0x0105", the first command parameter marker value "0x0005 0x020C", and the second command parameter marker value "0x8008" are spliced to obtain the command parameter marker value corresponding to the command parameter field "-usergrp 0 -name test -disabled" as "0x0003 0x0105 0x0005 0x020C 0x8008", that is, the command parameter field is stored in the form of chain storage.

[0065] In the step of the embodiment, the command parameter field is classified according to the value parameter type and the valueless parameter type, so that the storage redundancy rate of the finally obtained command parameter marker value is reduced, and the parsing efficiency of the command parameter marker value is improved when the log compressed file is decompressed.

[0066] In step C5, the first separation marker value and the second separation marker value are obtained from the dictionary tree; wherein the first separation marker value is used to separate the command type marker value and the command body marker value, and the second separation marker value is used to separate the command body marker value and the command parameter marker value.

[0067] In the embodiment, the first separation marker value is "0xA0", and the second separation marker value is "0xA1".

[0068] In step C6, the command type marker value, the first separation marker value, the command body marker value, the second separation marker value, and the command parameter marker value are spliced to obtain the command line marker value.

[0069] Exemplarily, the command type tag value is "0x0001", the command body tag value is "0x000A", the command parameter tag value is "0x0003 0x0105 0x0005 0x020C 0x8008", the first split tag value is "0xA0", the second split tag value is "0xA1", the command line tag value corresponding to the order of the command type tag value, the first split tag value, the command body tag value, the second split tag value and the command parameter tag value is "0x0001 0xA0 0x000A 0xA1 0x0003 0x0105 0x0005 0x020C 0x8008".

[0070] In the step of the embodiment, the dictionary tree realizes the sharing and multiplexing mechanism across fields, for example, the user identifier in the text field and the command body field in the command line interface command are the same in the first N characters, and then the dictionary tree is shared by the string composed of the character nodes corresponding to the first N characters. The above method improves the file compression rate of the log compression file.

[0071] Step D, calculating the difference between the reference timestamp and the log timestamp to generate a log time offset.

[0072] Exemplarily, the reference timestamp is the log timestamp in the first command line interface log, and then the log timestamp in each command line interface log is converted into the difference in milliseconds from the reference timestamp, that is, the log time offset is generated.

[0073] Step E, concatenating the numerical value field, the text tag value, the command line tag value and the log time offset to obtain the log tag value.

[0074] Exemplarily, based on the order of the log sequence number, the log timestamp, the user identifier, the operation source network address, the log result and the command line interface command, the numerical value field, the text tag value, the command line tag value and the log time offset are concatenated, and the log tag value is obtained.

[0075] Step S206: writing each log tag value into the data area of the initial log compression file, and writing the dictionary tree into the dictionary tree area of the initial log compression file to generate the final log compression file.

[0076] In a possible implementation, after writing each log tag value into the data area of the initial log compression file, the dictionary tree is written into the dictionary tree area of the initial log compression file, and then the final log compression file is generated.

[0077] Further, the specific implementation process of writing the dictionary tree into the dictionary tree area of the initial log compression file includes:

[0078] Step S206a, according to the log mark value corresponding to each command line interface log written to the data area of the initial log compressed file, the root node offset of the dictionary tree is determined.

[0079] Illustratively, based on the log mark value corresponding to each command line interface log written to the data area of the initial log compressed file, the data amount stored in the data area of the initial log compressed file is determined, and then according to the data amount stored in the data area, the position of the root node of the dictionary tree is determined, that is, the root node offset of the dictionary tree is determined.

[0080] Step S206b, the root node offset is written to the file header of the initial log compressed file.

[0081] Illustratively, the root node offset is written to the file header of the initial log compressed file, so as to determine the position of the root node of the dictionary tree from the log compressed file by reading the root node offset of the dictionary tree in the file header when the log compressed file is decompressed.

[0082] Step S206c, according to the root node offset, the dictionary tree is written to the dictionary tree area of the initial log compressed file.

[0083] Illustratively, according to the root node offset, the position of the dictionary tree area in the initial log compressed file is determined, and then the position of the root node of the dictionary tree in the dictionary tree area is determined, and then based on the root node of the dictionary tree, the dictionary tree is written to the dictionary tree area of the initial log compressed file, and then the dictionary tree is stored in the initial log compressed file in the form of dictionary tree data.

[0084] Further, if the dictionary tree is updated during the determination of each log mark value, the updated dictionary tree is written to the dictionary tree area of the log compressed file. Wherein, the implementation process of writing the updated dictionary tree to the dictionary tree area of the log compressed file is similar to the implementation principle and technical effect of the specific implementation process of steps S206a-S206c, which will not be repeated here.

[0085] In the embodiment, on the basis of obtaining the log source file and the dictionary tree and generating the initial log compression file, a first command line interface log is determined from the plurality of command line interface logs included in the log source file; then, a log timestamp in the first command line interface log is determined as a reference timestamp, and the reference timestamp is written into a file header of the initial log compression file; then, according to the dictionary tree and the reference timestamp, each command line interface log is compressed to generate a corresponding log marker value; further, each log marker value is written into a data area of the initial log compression file, and the dictionary tree is written into a dictionary tree area of the initial log compression file to generate a final log compression file; that is, the log marker value determined based on the dictionary tree and the reference timestamp is used to replace the command line interface log in pure text format for storage, thereby reducing the storage space occupancy rate of the stored command line interface log, that is, solving the problem of storage space waste.

[0086] Figure 3 A flowchart of a log compression file decompression method provided by the embodiment is shown in FIG. 8, and the method includes the following steps. Figure 3

[0087] Step S301: Obtain a log compression file.

[0088] Step S302: Read a file header of the log compression file to obtain a reference timestamp and a root node offset of a dictionary tree.

[0089] In the embodiment, the file header of the log compression file is read to obtain a file format marker value; whether the file format of the log compression file meets a decompression condition is judged according to the file format marker value; if the decompression condition is met, the file header of the log compression file is further read to obtain the reference timestamp and the root node offset of the dictionary tree; if the decompression condition is not met, a prompt information of file format error is generated to prompt the user that the format of the log compression file does not meet the decompression condition.

[0090] Step S303: Determine a dictionary tree area from the log compression file according to the root node offset, and read dictionary tree data in the dictionary tree area to generate the dictionary tree.

[0091] Step S304: Read a data area of the log compression file to obtain log marker values.

[0092] Step S305: Generate a command line interface log according to the log marker values, the dictionary tree and the reference timestamp.

[0093] Specifically, the specific implementation steps of step S305 include the following steps.

[0094] Step S305a: Determine a numerical field, a text marker value, a command line marker value and a log time offset according to the log marker value. ​

[0095] Step S305b, the sum of the log time offset and the reference timestamp is calculated to obtain the log timestamp.

[0096] In this embodiment, the log time offset is stored in the form of milliseconds, and the data format of the reference timestamp and the log timestamp is "year-month-day-hour: minute: second.millisecond", so the data format of the log time offset is first converted into the form of "year-month-day-hour: minute: second.millisecond", and then the log timestamp is obtained by combining the reference timestamp.

[0097] Step S305c, according to the text mark value, the string query is performed in the dictionary tree to obtain the text type field.

[0098] Step S305d, according to the command line mark value, the string query is performed in the dictionary tree to obtain the command line interface command.

[0099] Exemplarily, for example, the command line mark value is "0x0001 0xA0 0x000A 0xA1 0x0003 0x0105 0x0005 0x020C 0x8008", then the string query is performed in the dictionary tree, and the command type field "mcsop" corresponding to the command line interface command, the command body field "mkuser" corresponding to the command line interface command, and the command parameter field "-usergrp 0 -name test -disabled" corresponding to the command line interface command of "0x0003 0x0105 0x0005 0x020C 0x8008" are obtained; and then the command line interface command "mcsop mkuser -usergrp 0 -name test -disabled" is obtained.

[0100] Step S305e, the numerical value type field, the log timestamp, the text type field and the command line interface command are spliced to obtain the command line interface log.

[0101] Exemplarily, the numerical value type field, the log timestamp, the text type field and the command line interface command are spliced to obtain the command line interface log based on the sequence of the log serial number, the log timestamp, the user identifier, the operation source network address, the log result and the command line interface command.

[0102] Further, in a possible implementation, after step S305, the method provided by the embodiment of the application further includes: deleting the dictionary tree.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment.

[0104] Figure 4 A structural schematic diagram of a log compression file generation apparatus provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the embodiments of the present application further provide a log compression file generation apparatus 40, which comprises a first acquisition module 401, a first processing module 402 and a first generation module 403. Figure 4

[0105] The first acquisition module 401 is configured to acquire a log source file and a dictionary tree, and generate an initial log compression file.

[0106] The first processing module 402 is configured to determine a first command line interface log from a plurality of command line interface logs included in the log source file.

[0107] The first generation module 403 is configured to determine a log timestamp in the first command line interface log as a reference timestamp.

[0108] The first processing module 402 is further configured to write the reference timestamp into a file header of the initial log compression file, and compress each command line interface log according to the dictionary tree and the reference timestamp to generate a corresponding log mark value.

[0109] The first generation module 403 is further configured to write each log mark value into a data area of the initial log compression file, and write the dictionary tree into a dictionary tree area of the initial log compression file, so as to generate a final log compression file.

[0110] In a possible implementation, when the first processing module 402 compresses each command line interface log according to the dictionary tree and the reference timestamp to generate a corresponding log mark value, the first processing module 402 is specifically configured to: for each command line interface log, split the command line interface log to determine a numerical value field, a log timestamp, a text field and a command line interface command; determine a text mark value according to the dictionary tree and the text field; determine a command line mark value according to the dictionary tree and the command line interface command; calculate a difference value between the reference timestamp and the log timestamp to generate a log time offset; and splice the numerical value field, the text mark value, the command line mark value and the log time offset to obtain the log mark value.

[0111] ​In a possible implementation, the first processing module 402, when determining the command line mark value according to the dictionary tree and the command line interface command, is specifically configured to: perform semantic segmentation on the command line interface command to obtain a command type field, a command body field, and a command parameter field; determine a command type mark value according to the dictionary tree and the command type field; determine a command body mark value according to the dictionary tree and the command body field; determine a command parameter mark value according to the dictionary tree and the command parameter field; obtain a first separation mark value and a second separation mark value from the dictionary tree; the first separation mark value is used to separate the command type mark value and the command body mark value, and the second separation mark value is used to separate the command body mark value and the command parameter mark value; and concatenate the command type mark value, the first separation mark value, the command body mark value, the second separation mark value, and the command parameter mark value to obtain the command line mark value.

[0112] In a possible implementation, the first processing module 402, when determining the text mark value according to the dictionary tree and the text type field, is specifically configured to: determine whether a string in the dictionary tree is consistent with the text type field; if the string in the dictionary tree is consistent with the text type field, determine a mark value corresponding to a string corresponding to the text type field as the text mark value; if the string in the dictionary tree is not consistent with the text type field, create a new character node in the dictionary tree according to the text type field to obtain a newly created string corresponding to the text type field; generate a newly created mark value according to the newly created string; and determine the newly created mark value as the text mark value. The first processing module 402 is further configured to: update the dictionary tree in which the new character node is created according to the newly created mark value to obtain an updated dictionary tree.

[0113] In a possible implementation, the first generating module 403, when writing the dictionary tree into the dictionary tree area of the initial log compression file, is specifically configured to: write the updated dictionary tree into the dictionary tree area of the log compression file.

[0114] In a possible implementation, the first generating module 403, when writing the dictionary tree into the dictionary tree area of the initial log compression file, is specifically configured to: determine a root node offset of the dictionary tree according to log mark values corresponding to the command line interface logs written into the data area of the initial log compression file; write the root node offset into a file header of the initial log compression file; and write the dictionary tree into the dictionary tree area of the initial log compression file according to the root node offset.

[0115] In a possible implementation, the first generating module 403, when writing the dictionary tree into the dictionary tree area of the initial log compression file, is specifically configured to: determine a root node offset of the dictionary tree according to log mark values corresponding to the command line interface logs written into the data area of the initial log compression file; write the root node offset into a file header of the initial log compression file; and write the dictionary tree into the dictionary tree area of the initial log compression file according to the root node offset.

[0116] The features of the embodiment of the log compressed file generation apparatus 40 can be referred to the related description of the embodiment of the log compressed file generation method, which will not be repeated here.

[0117] Figure 5 The structural schematic diagram of the log compressed file decompression apparatus provided by the embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, the embodiment of the present application further provides a log compressed file decompression apparatus 50, which comprises a second acquisition module 501, a second processing module 502 and a second generation module 503. Figure 5

[0118] The second acquisition module 501 is configured to acquire the log compressed file.

[0119] The second processing module 502 is configured to read the file header of the log compressed file to obtain the reference timestamp and the root node offset of the dictionary tree; determine the dictionary tree area from the log compressed file according to the root node offset, and read the dictionary tree data in the dictionary tree area to generate the dictionary tree; and read the data area of the log compressed file to obtain the log marker value.

[0120] The second generation module 503 is configured to generate the command line interface log according to the log marker value, the dictionary tree and the reference timestamp.

[0121] In a possible implementation, when the second generation module 503 generates the command line interface log according to the log marker value, the dictionary tree and the reference timestamp, the second generation module 503 is specifically configured to: determine the numerical value field, the text marker value, the command line marker value and the log time offset according to the log marker value; calculate the sum of the log time offset and the reference timestamp to obtain the log timestamp; perform string query in the dictionary tree according to the text marker value to obtain the text field; perform string query in the dictionary tree according to the command line marker value to obtain the command line interface command; and splice the numerical value field, the log timestamp, the text field and the command line interface command to obtain the command line interface log.

[0122] The features of the embodiment of the log compressed file decompression apparatus 50 can be referred to the related description of the embodiment of the log compressed file decompression method, which will not be repeated here.

[0123] Figure 6 The structural schematic diagram of the electronic device provided by the present application is shown in FIG. 7. As shown in FIG. 7, the electronic device 60 provided by the embodiment of the present application comprises at least one processor 601 and a memory 602. Optionally, the electronic device 60 further comprises a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected through a bus. Figure 6

[0124] ​​In the implementation process, the at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 performs the log compressed file generation method embodiments or the log compressed file decompression method embodiments described above.

[0125] The specific implementation process of the processor 601 can refer to the method embodiments described above, which have similar implementation principles and technical effects, and details are not described here again in the embodiment.

[0126] In the embodiments described above, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0127] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0128] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the application does not limit to only one bus or one type of bus.

[0129] The embodiments of the application also provide a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the steps in any one of the log compressed file generation method embodiments or the steps in the log compressed file decompression method embodiments when running.

[0130] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0131] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the steps in any of the log compression file generation method embodiments or the steps in the log compression file decompression method embodiments.

[0132] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in any of the log compression file generation method embodiments or the steps in the log compression file decompression method embodiments.

[0133] The skilled in the art can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] The above describes in detail a log compression file generation method, a log compression file decompression method and equipment provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper, and the above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be pointed out that for the ordinary skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A log compression file generation method characterized by, The method comprises the following steps: obtaining a log source file and a dictionary tree; generating an initial log compressed file; the log compressed file is an empty file, and the log compressed file comprises an initialized file header, a data area and a dictionary tree area; determining a first command line interface log from a plurality of command line interface logs comprised in the log source file; determining a log timestamp in the first command line interface log as a reference timestamp; writing the reference timestamp into the file header of the initial log compressed file; for each command line interface log: splitting the command line interface log to determine a numerical field, a log timestamp, a text field and a command line interface command; determining a text token value according to the dictionary tree and the text field; determining a command line token value according to the dictionary tree and the command line interface command; calculating a difference value between the reference timestamp and the log timestamp to generate a log time offset; splicing the numerical field, the text token value, the command line token value and the log time offset to obtain a log token value; the dictionary tree is constructed to share character nodes across fields for strings of the text field and the command line interface command; writing each log token value into the data area of the initial log compressed file, and writing the dictionary tree into the dictionary tree area of the initial log compressed file to generate a final log compressed file.

2. The log compression file generation method according to claim 1, characterized by, The method further comprises: determining a command type token value according to the dictionary tree and the command type field; determining a command body token value according to the dictionary tree and the command body field; determining a command parameter token value according to the dictionary tree and the command parameter field; obtaining a first separation token value and a second separation token value from the dictionary tree; wherein the first separation token value is used to separate the command type token value and the command body token value, and the second separation token value is used to separate the command body token value and the command parameter token value; splicing the command type token value, the first separation token value, the command body token value, the second separation token value and the command parameter token value to obtain the command line token value. The method further comprises:

3. The log compression file generation method according to claim 1, characterized by, determining whether the string in the dictionary tree is consistent with the text field; if the string in the dictionary tree is consistent with the text field, determining a token value corresponding to the string corresponding to the text field as the text token value; if the string in the dictionary tree is not consistent with the text field, creating a new character node in the dictionary tree according to the text field to obtain a newly created string corresponding to the text field; generating a newly created token value according to the newly created string; determining the newly created token value as the text token value; The method further comprises: ​ According to the new mark value, the dictionary tree of creating a new character node is updated to obtain an updated dictionary tree.

4. The log compression file generation method according to claim 3, characterized by, The writing of the dictionary tree into the dictionary tree area of the initial log compression file comprises: The writing of the updated dictionary tree into the dictionary tree area of the log compression file.

5. The log compression file generation method according to claim 1, characterized by, The writing of the dictionary tree into the dictionary tree area of the initial log compression file comprises: According to the log mark value corresponding to each command line interface log written into the data area of the initial log compression file, the root node offset of the dictionary tree is determined; The root node offset is written into the file header of the initial log compression file; According to the root node offset, the dictionary tree is written into the dictionary tree area of the initial log compression file.

6. The log compression file generation method according to any one of claims 1 to 5, characterized by, The generation of the initial log compression file comprises: According to a preset file format, a file format mark value is generated; According to the file format mark value, the initial log compression file is generated.

7. A log compressed file decompression method, characterized by, It comprises: Obtaining a log compression file; Reading the file header of the log compression file to obtain a reference timestamp and a root node offset of a dictionary tree; According to the root node offset, a dictionary tree area is determined from the log compression file, and dictionary tree data in the dictionary tree area is read to generate a dictionary tree; Reading the data area of the log compression file to obtain a log mark value; According to the log mark value, the dictionary tree and the reference timestamp, a command line interface log is generated; The log compression file is generated according to the log compression file generation method of any one of claims 1-6.

8. The log compressed file decompression method according to claim 7, wherein, The generation of the command line interface log according to the log mark value, the dictionary tree and the reference timestamp comprises: According to the log mark value, a numerical field, a text mark value, a command line mark value and a log time offset are determined; The sum of the log time offset and the reference timestamp is calculated to obtain a log timestamp; According to the text mark value, a string query is performed in the dictionary tree to obtain a text field; According to the command line mark value, a string query is performed in the dictionary tree to obtain a command line interface command; The numerical field, the log timestamp, the text field and the command line interface command are spliced to obtain the command line interface log.

9. An electronic device, comprising: It comprises: A memory for storing a computer program; A processor for executing the computer program to realize the steps of the log compression file generation method of any one of claims 1-6, or to realize the steps of the log compression file decompression method of claim 7 or 8.

Images