A server log data compression method, system, terminal and storage medium

Abstract

The application relates to the field of log compression, and particularly discloses a server log data compression method, a server log data compression system, a terminal and a storage medium, which collect log files; read text content in the log files, split the text content in the log files by taking spaces as identifiers to obtain multiple segments; cyclically compare the segments, and record segments with the same content as an element; replace each element in the text content by using a marker to obtain a new log file; wherein the marker comprises a two-bit length marker and a three-bit length marker; one element corresponds to one marker; and the new log file is compressed. The application divides segments by taking spaces as identifiers to find repeated elements, performs replacement, and then performs compression, so that the number of characters in the log file is reduced, and the compression effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of log compression, in particular to a server log data compression method, system, terminal and storage medium. BACKGROUND

[0002] A server log is one or more log files created and maintained automatically by a server, which contains a list of its executed activities. For example, the log of a web server, which contains the history of page requests. Currently, log files usually append content to the end of the file. The added information is about web page requests, including client IP address, request date / time, requested web page, HTTP code, number of bytes served, user agent, referrer, etc. These data may be written in one file or separated into different logs, such as access log, error log, referrer log, etc.

[0003] Multiple projects are usually running on a server, and each project outputs a large amount of logs every day. The server storage space cannot accommodate the log increment, so it is necessary to compress the log files regularly and store them after compression to reduce space occupation. The common log file compression method mainly has two steps: log file collection and log file compression. The common algorithm for text data compression is ZIP using LZ77 and Huffman coding for compression, which reduces data volume and storage space, and improves the storage efficiency of the server. However, the traditional log file compression only compresses the data volume of the log file itself to reduce its space occupation, and does not process the content in the log file. However, the log file actually has a large number of repetitive phrases, such as words, sentences, descriptions, dates, IP addresses, ports, etc. For this type of log file, the traditional compression method does not have a targeted processing, and cannot achieve a better compression effect. SUMMARY

[0004] To solve the above problems, the present application provides a server log data compression method, system, terminal and storage medium, which realizes segmenting by space to find repeated elements, replacing and then compressing to reduce the number of characters in the log file and improve the compression effect.

[0005] In a first aspect, the technical solution of the present application provides a server log data compression method, which includes the following steps:

[0006] Collecting log files;

[0007] Reading the text content in the log file, and dividing the text content in the log file by space as an identifier to obtain multiple segments;

[0008] Cycling and comparing each segment, and recording the segments with the same content as an element;

[0009] The new log file is obtained by replacing each element in the text content with a marker, wherein the marker includes a two-digit marker and a three-digit marker; one element corresponds to one marker;

[0010] The new log file is compressed.

[0011] In an optional embodiment, the replacement of each element in the text content with a marker specifically includes:

[0012] All markers are sorted; the two-digit markers are arranged in front of the three-digit markers, so as to replace the two-digit markers first, and if there are still elements to be replaced after the two-digit markers are used up, the three-digit markers are used to replace the remaining elements;

[0013] The number of occurrences of each element is counted.

[0014] The element is taken as a key value, and the number of occurrences of the element is taken as a value value, so that the element and the number of occurrences thereof are stored in the form of a key-value pair, and all elements and occurrence number key-value pairs constitute an element array;

[0015] The key-value pairs in the element array are sorted in descending order of the number of occurrences of the elements;

[0016] The first marker in the marker sequence replaces the value of the first key-value pair in the element array, the second marker in the marker sequence replaces the value of the second key-value pair in the element array, and so on, until all values are replaced, to form an element marker mapping relationship array;

[0017] Based on the mapping relationship in the element marker mapping relationship array, each element in the log file is replaced.

[0018] In an optional embodiment, the method further includes the following steps:

[0019] A mapping relationship file is created;

[0020] The element marker mapping relationship array is written into the created mapping relationship file;

[0021] When the new log file is compressed, the new log file and the mapping relationship file are compressed together.

[0022] In an optional embodiment, the log file is collected, specifically including:

[0023] All log files of the target program are collected; the number of log files is not less than 2;

[0024] Correspondingly, the method specifically includes the following steps:

[0025] reading the text content in the first log file, splitting the text content in the first log file by spaces to obtain a plurality of segments;

[0026] reading the text content in the second log file, splitting the text content in the second log file by spaces to obtain a plurality of segments;

[0027] and so on until the text content of all log files is divided into segments;

[0028] for all segments, comparing each segment, and recording segments with the same content as an element;

[0029] replacing each element in each log file with a marker to obtain a plurality of new log files;

[0030] compressing all new log files together.

[0031] In an optional embodiment, replacing each element in each log file with a marker specifically includes:

[0032] sorting all markers; wherein two-digit length markers are arranged in front of three-digit length markers to preferentially replace with two-digit length markers, and if there are still elements to be replaced after the two-digit length markers are used up, then three-digit length markers are used to replace the remaining elements;

[0033] counting and storing the number of occurrences of each element in the first log file;

[0034] counting and storing the number of occurrences of each element in the second log file;

[0035] and so on until the number of occurrences of each element in all log files is obtained;

[0036] counting the total number of occurrences of each element in all log files;

[0037] storing the element and its total number of occurrences in the form of a key-value pair, with the element as the key value and the total number of occurrences as the value, and all elements and total number of occurrences key-value pairs constitute an element array;

[0038] sorting the key-value pairs in the element array in descending order of the total number of occurrences of the elements;

[0039] The first marker in the marker sequence replaces the value of the first key-value pair in the element array, the second marker in the marker sequence replaces the value of the second key-value pair in the element array, and so on until all values are replaced, thereby forming an element marker mapping relationship array;

[0040] Based on the mapping relationship in the element marker mapping relationship array, each element in each log file is replaced.

[0041] In an optional embodiment, the method further comprises the following steps:

[0042] Creating a mapping relationship file;

[0043] Writing the element marker mapping relationship array into the created mapping relationship file;

[0044] When compressing all new log files together, all new log files and the mapping relationship file are compressed together.

[0045] In an optional embodiment, the two-bit length marker includes a one-bit separator and a one-bit marker value, and the one-bit marker value is any one of the numbers 0 to 9, 26 lowercase letters, and 26 uppercase letters;

[0046] The three-bit length marker includes a one-bit separator and a two-bit marker value, and the two-bit marker value is any combination of two of the numbers 0 to 9, 26 lowercase letters, and 26 uppercase letters;

[0047] When sorting all markers, the markers are sorted in the following marker value order: 0, 1, …, 9, a, b, …, z, A, B, …, Z, 00, 01, …, 99, 0a, 0b, …, 9z, a0, a1, …, z9, 0A, 0B, …, 9Z, A0, A1, …, Z9, aa, ab, …, zz, aA, aB, …, aZ, Aa, Ab, …, Zz, AA, AB, …, ZZ.

[0048] In a second aspect, the technical solution of the present application provides a server log data compression system, comprising,

[0049] Log collection module: collecting log files;

[0050] Fragment segmentation module: reading the text content in the log file, and segmenting the text content in the log file with spaces as markers to obtain a plurality of fragments;

[0051] Element analysis module: cyclically comparing each fragment, and recording fragments with the same content as an element;

[0052] Element replacement module: replace each element in the text content with a marker to obtain a new log file; wherein the marker includes a two-bit length marker and a three-bit length marker; one element corresponds to one marker;

[0053] Log compression module: compress the new log file.

[0054] In a third aspect, the technical solution of the present application provides a terminal, comprising:

[0055] A memory for storing a server log data compression program;

[0056] A processor for executing the server log data compression program to realize the steps of the server log data compression method according to any one of the above.

[0057] In a fourth aspect, the technical solution of the present application provides a computer readable storage medium, wherein the readable storage medium stores a server log data compression program, and the server log data compression program is executed by a processor to realize the steps of the server log data compression method according to any one of the above.

[0058] The server log data compression method, system, terminal and storage medium provided by the present application have the following beneficial effects compared with the prior art: by using the characteristics of the log file, the text content is divided into multiple segments using spaces as markers, and then the same elements are filtered out and replaced with shorter markers, and finally compressed, which greatly reduces the number of characters in the log file. On this basis, compression is performed again to achieve the purpose of reducing the size of the compressed log file and reducing the hard disk space occupation. The present application divides segments by spaces to find repeated elements, replaces them, and then compresses them to reduce the number of characters in the log file and improve the compression effect. At the same time, two or three bit length markers are used for replacement, the marker bit length is short, space is saved, and the number of element occurrences is sorted, the marker replaced by the element with more occurrences occupies less space, further saving file space occupation, and optimizing compression effect. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0060] Figure 1 is a server log data compression method flowchart of the first embodiment of the present application.

[0061] Figure 2 is a flowchart of a specific embodiment of a server log data compression method of the first embodiment of the present application.

[0062] Figure 3 is a flowchart of a server log data compression method of the second embodiment of the present application.

[0063] Figure 4 is a flowchart of a specific embodiment of a server log data compression method of the second embodiment of the present application.

[0064] Figure 5 is a schematic block diagram of a server log data compression system structure provided by the embodiments of the present application.

[0065] Figure 6 is a schematic diagram of a terminal structure provided by the embodiments of the present application. DETAILED DESCRIPTION

[0066] In order to make the personnel in the technical field better understand the present application scheme, the present application is further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without making creative efforts belong to the scope of protection of the present application.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0068] The present application is directed to the characteristics of high-frequency occurrence of the same words, IP, port and date in the service end log content, and provides a server log data compression method. Before the log file is compressed, the log file is preprocessed, and each segment in the log file is replaced with a marker. Since there are a large number of repeated phrases in the log file, fewer markers can be used to achieve replacement. After replacement, the number of characters of the log file is reduced, and then compression is performed, so as to achieve the purpose of reducing the file size after log compression and reducing the hard disk space occupation.

[0069] Figure 1 is a flowchart of a server log data compression method of the first embodiment of the present application. Among them, Figure 1The executing entity can be a server log data compression system. The server log data compression method is performed by computer equipment; correspondingly, the server log data compression method system runs on the computer equipment. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0070] like Figure 1 As shown, the method includes the following steps.

[0071] S1, collect log files.

[0072] S2 reads the text content from the log file, splits the text content in the log file using spaces as identifiers, and obtains multiple segments.

[0073] Log files typically use spaces to separate phrases; therefore, this embodiment uses spaces as identifiers to segment the text content into several segments.

[0074] S3: Iterate through and compare each segment, and record segments with the same content as one element.

[0075] Fragments with identical content are grouped together as one element, and a log file will contain multiple elements.

[0076] S4 uses identifiers to replace each element in the text content to obtain a new log file.

[0077] The identifiers include two-digit and three-digit identifiers; one element corresponds to one identifier.

[0078] Replace each element with a shorter identifier to reduce the number of characters in the log file.

[0079] S5 compresses the new log file.

[0080] It should be noted that although the LA77 compression algorithm replaces repetitive statements, it uses a moving window approach to find repetitive statements, which is inefficient. In contrast, this embodiment, based on the characteristics of log files, directly uses spaces as identifiers to divide segments and find repetitive statements, which is more efficient than the LA77 algorithm.

[0081] The server log data compression method provided in this invention utilizes the characteristics of log files. It uses spaces as identifiers to divide the text content into multiple segments, then filters out identical elements and replaces them with shorter identifiers. Finally, compression is performed. After element replacement, the number of characters in the log file is significantly reduced. Further compression on this basis achieves the goal of reducing the size of the compressed log file and thus reducing disk space usage. This invention uses spaces to divide segments, find duplicate elements, replace them, and then compress, improving the compression effect.

[0082] To further understand the present invention, a specific embodiment is provided below to provide a more detailed description of the invention.

[0083] Figure 2 This is a flowchart illustrating the specific implementation method, as shown below. Figure 2 As shown, the method includes the following steps.

[0084] S101, collect log files.

[0085] S102: Read the text content in the log file, and split the text content in the log file using spaces as identifiers to obtain multiple segments.

[0086] S103, compare each segment in a loop, and record segments with the same content as one element.

[0087] S104, sort all identifiers.

[0088] The identifiers include two-digit and three-digit length identifiers. During sorting, the two-digit length identifiers are placed before the three-digit length identifiers, so that the two-digit length identifiers are used first for replacement. After the two-digit length identifiers are used up, if there are still elements that have not been replaced, the three-digit length identifiers are used to replace the remaining elements.

[0089] One element corresponds to one identifier.

[0090] In one alternative implementation, the two-digit length identifier includes a separator and a flag value, wherein the flag value is any one of the numbers from 0 to 9, 26 lowercase letters, and 26 uppercase letters; the three-digit length identifier includes a separator and two flag values, wherein the two flag values ​​are a combination of any two of the numbers from 0 to 9, 26 lowercase letters, and 26 uppercase letters.

[0091] When sorting all identifiers, they are sorted in the following order of identifier values: 0, 1, ..., 9, a, b, ..., z, A, B, ..., Z, 00, 01, ..., 99, 0a, 0b, ..., 9z, a0, a1, ..., z9, 0A, 0B, ..., 9Z, A0, A1, ..., Z9, aa, ab, ..., zz, aA, aB, ..., aZ, Aa, Ab, ..., Zz, AA, AB, ..., ZZ.

[0092] S105, count the number of times each element appears.

[0093] S106, use the element as the key value and the number of times the element appears as the value value, so that the element and its number of appearances are stored in the form of key-value pairs, and all key-value pairs of elements and their number of appearances constitute an element array.

[0094] S107 Sort the key-value pairs in the element array in descending order of the frequency of element occurrence.

[0095] S108, replace the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replace the value of the second key-value pair in the element array with the second identifier in the identifier sequence, and so on, until all values ​​are replaced, thus forming an element identifier mapping relationship array.

[0096] S109, based on the mapping relationship in the element identifier mapping relationship array, replace each element in the log file.

[0097] This specific implementation stores elements and their occurrence counts in key-value pairs for easy sorting and element replacement. The occurrence counts in the key-value pairs are replaced with identifiers to obtain the mapping relationship between elements and identifiers. Then, each segment in the log file is replaced according to this mapping relationship, realizing the preprocessing of replacing the original content in the log file with shorter identifiers.

[0098] S110, Create a mapping file.

[0099] S111, write the element identifier mapping array to the created mapping file.

[0100] S112, compress the new log file and mapping relationship file together.

[0101] After establishing the mapping relationship, store the mapping relationship in a mapping relationship file. Compress the mapping relationship file together with the new log file so that when reading the log file later, the mapping relationship can be parsed.

[0102] In one specific implementation, the data file is split using spaces as delimiters, and the frequency of each word, IP address, date, etc., is counted repeatedly. Finally, the data is sorted in descending order: the most frequent occurrences are listed first, and the least frequent occurrences are listed last. For example, the following two log samples (total 293 characters):

[0103] 2023-04-14 07:32:54,681 WARN [Caesium-1-3] [atlassian.pats.user.DeletedUserPruningService]onUserDeleted Can not delete tokens for user lvbinbin

[0104] 2023-04-14 07:32:54,683 WARN [Caesium-1-3] [atlassian.pats.user.DeletedUserPruningService]onUserDeleted Can not delete tokens for userliuxiongbin

[0105] After repeated data analysis, the following K:V array was obtained: K represents the key name, and V represents the number of times it appears.

[0106] 2023-04-14:2

[0107] WARN:2

[0108] [Caesium-1-3]:2

[0109] [atlassian.pats.user.DeletedUserPruningService]:2

[0110] onUserDeleted:2

[0111] Can:2

[0112] not:2

[0113] delete:2

[0114] tokens:2

[0115] for:2

[0116] user:2

[0117] lvbinbin:1

[0118] liuxiongbin:1

[0119] 07:32:54, 681:1

[0120] 07:32:54,683:1}

[0121] Replacement flags range from &1 to &ZZ, such as &0, &1, &2…&a, &b…&Z, etc., with a minimum length of 2 characters and a maximum length of 3 characters, containing 10 digits, 26 lowercase letters, and 26 uppercase letters, for a total of 3844 flags. This is sufficient for most log files.

[0122] The following replacement marker array is obtained after the replacement:

[0123] 2023-04-14:&0

[0124] WARN:&1

[0125] [Caesium-1-3]:&2

[0126] [atlassian.pats.user.DeletedUserPruningService]:&3

[0127] onUserDeleted:&4

[0128] Can:&5

[0129] not:&6

[0130] delete:&7

[0131] tokens:&8

[0132] for:&9

[0133] user:&a

[0134] lvbinbin:&b

[0135] liuxiongbin:&c

[0136] 07:32:54,681:&d}

[0137] Replace the log file with the replacement mark. The result after replacement in Example 1 is as follows: (55 characters in total) is 19% of the original content.

[0138] &0&d&1&2&3&4&5&6&7&8&9&a&b

[0139] &0&e&1&2&3&4&5&6&7&8&9&a&b&c

[0140] Figure 3 This is a schematic flowchart of the server log data compression method according to the second embodiment of the present invention. Figure 3 The executing entity can be a server log data compression system. The server log data compression method is performed by computer equipment; correspondingly, the server log data compression method system runs on the computer equipment. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0141] This embodiment is implemented for multiple log files, collecting logs on a program-by-program basis. Each program generates several log files, which are then processed and compressed uniformly. For example... Figure 3 As shown, the method includes the following steps.

[0142] SS1 collects all log files of the target program.

[0143] The number of log files must be no less than two.

[0144] SS2 reads the text content from the first log file, splits the text content in the first log file into multiple segments using spaces as identifiers; it then reads the text content from the second log file, splits the text content in the second log file into multiple segments using spaces as identifiers, and so on, until the text content of all log files has been segmented.

[0145] SS3 compares all fragments in a loop and records fragments with the same content as one element.

[0146] The text content of all log files is analyzed uniformly. Fragments of the same content appearing in different log files are considered to be the same element. For example, if the fragment 2023-04-14 appears in all log files, this fragment is replaced with the same identifier in all log files.

[0147] SS4 uses identifiers to replace elements in various log files, resulting in multiple new log files.

[0148] It should be noted that the identifiers include two-digit and three-digit identifiers; one element corresponds to one identifier.

[0149] SS5 compresses all new log files together.

[0150] The server log data compression method provided in this invention utilizes the characteristics of log files. It uses spaces as identifiers to divide the text content into multiple segments, then filters out identical elements and replaces them with shorter identifiers. Finally, compression is performed. After element replacement, the number of characters in the log file is significantly reduced. Further compression on this basis achieves the goal of reducing the size of the compressed log file and thus reducing disk space usage. This invention uses spaces to divide segments, find duplicate elements, replace them, and then compress, improving the compression effect.

[0151] To further understand the present invention, a specific embodiment is provided below to provide a more detailed description of this embodiment.

[0152] Figure 4 This is a schematic diagram of the specific implementation method, such as... Figure 4 As shown, this specific implementation includes the following steps.

[0153] SS101 collects all log files of the target program.

[0154] The number of log files must be no less than two.

[0155] SS102 reads the text content from the first log file, splits the text content in the first log file into multiple segments using spaces as identifiers; reads the text content from the second log file, splits the text content in the second log file into multiple segments using spaces as identifiers; and so on, until the text content of all log files has been segmented.

[0156] SS103 compares each segment in a loop for all segments and records segments with the same content as one element.

[0157] SS104 sorts all identifiers.

[0158] Two-length markers are placed before three-length markers, with two-length markers being used first for replacement. After two-length markers are used up, if there are still elements that have not been replaced, three-length markers are used to replace the remaining elements.

[0159] In one alternative implementation, the two-digit length identifier includes a separator and a flag value, wherein the flag value is any one of the numbers from 0 to 9, 26 lowercase letters, and 26 uppercase letters; the three-digit length identifier includes a separator and two flag values, wherein the two flag values ​​are a combination of any two of the numbers from 0 to 9, 26 lowercase letters, and 26 uppercase letters.

[0160] When sorting all identifiers, they are sorted in the following order of identifier values: 0, 1, ..., 9, a, b, ..., z, A, B, ..., Z, 00, 01, ..., 99, 0a, 0b, ..., 9z, a0, a1, ..., z9, 0A, 0B, ..., 9Z, A0, A1, ..., Z9, aa, ab, ..., zz, aA, aB, ..., aZ, Aa, Ab, ..., Zz, AA, AB, ..., ZZ.

[0161] SS105 counts the occurrences of each element in the first log file and stores the counts; counts the occurrences of each element in the second log file and stores the counts; and so on, until the occurrence counts of each element in all log files are obtained.

[0162] SS106 counts the total number of times each element appears in all log files.

[0163] SS107 uses elements as keys and the total number of times each element appears as values, storing elements and their total occurrences as key-value pairs. All key-value pairs of elements and their total occurrences form an element array.

[0164] SS108 sorts the key-value pairs in an array of elements in descending order of the total number of times each element appears.

[0165] SS109 replaces the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replaces the value of the second key-value pair in the element array with the second identifier in the identifier sequence, and so on, until all values ​​have been replaced, thus forming an element identifier mapping array.

[0166] SS110 replaces each element in each log file based on the mapping relationship in the element identifier mapping relationship array.

[0167] This specific implementation stores elements and their total occurrences in key-value pairs for easy sorting and element replacement. The total occurrences in the key-value pairs are replaced with identifiers to obtain a mapping relationship between elements and identifiers. Then, each segment in the log file is replaced according to this mapping relationship, realizing the preprocessing of replacing the original content in the log file with shorter identifiers.

[0168] SS111, Create a mapping file.

[0169] SS112 writes the element identifier mapping array to the created mapping file.

[0170] SS113 compresses all new log files and mapping files together.

[0171] After establishing the mapping relationship, store the mapping relationship in a mapping relationship file. Compress the mapping relationship file together with the new log file so that when reading the log file later, the mapping relationship can be parsed.

[0172] The foregoing has described in detail an embodiment of a server log data compression method. Based on the server log data compression method described in the above embodiment, this invention also provides a server log data compression system corresponding to the method.

[0173] Figure 5 This is a schematic block diagram of a server log data compression system 500 provided in an embodiment of the present invention. The server log data compression system 500 can be divided into multiple functional modules according to its functions, such as... Figure 5 As shown, the functional modules may include: a log collection module 510, a fragment segmentation module 520, an element analysis module 530, an element replacement module 540, a log compression module 550, and a mapping file storage module 560. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory.

[0174] Log collection module 510: Collects log files.

[0175] Fragment segmentation module 520: Reads the text content in the log file, segments the text content in the log file using spaces as identifiers, and obtains multiple fragments.

[0176] Element Analysis Module 530: Iterate through and compare each segment, and record segments with the same content as one element.

[0177] Element replacement module 540: Uses identifiers to replace each element in the text content to obtain a new log file; the identifiers include two-digit and three-digit length identifiers; one identifier corresponds to one element.

[0178] Log compression module 550: Compresses new log files.

[0179] In an optional implementation, the element replacement module 540 uses identifiers to replace each element in the text content. Specifically, this includes: sorting all identifiers; counting the occurrences of each element; storing elements as key-value pairs and their occurrences as values, with all key-value pairs forming an element array; sorting the key-value pairs in the element array in descending order of element occurrences; replacing the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replacing the value of the second key-value pair with the second identifier in the identifier sequence, and so on, until all values ​​are replaced, forming an element identifier mapping array; and replacing each element in the log file based on the mapping relationships in the element identifier mapping array. Two-digit length identifiers precede three-digit length identifiers, prioritizing the use of two-digit length identifiers. After using two-digit length identifiers, if there are still elements left to replace, three-digit length identifiers are then used to replace the remaining elements.

[0180] In an optional implementation, system 500 further includes a mapping file storage module 560 configured to create mapping relationship files and write element identifier mapping relationship arrays to the created mapping relationship files. When the log compression module 550 compresses a new log file, it compresses both the new log file and the mapping relationship file together.

[0181] In an optional implementation, the log collection module 510 collects all log files of the target program; the number of log files is not less than two.

[0182] Accordingly, the segmentation module 520 is specifically configured to: read the text content in the first log file, segment the text content in the first log file using spaces as identifiers, and obtain multiple segments; read the text content in the second log file, segment the text content in the second log file using spaces as identifiers, and obtain multiple segments; and so on, until the text content of all log files has been segmented.

[0183] Meanwhile, the element analysis module 530 is specifically configured to iteratively compare all segments, recording segments with identical content as a single element. The element replacement module 540 is specifically configured to replace each element in each log file using identifiers, resulting in multiple new log files. The log compression module 550 is specifically configured to perform compression processing on all the new log files together.

[0184] In an optional implementation, the element replacement module 540 uses identifiers to replace elements in each log file, specifically including: sorting all identifiers; counting and storing the occurrences of each element in the first log file; counting and storing the occurrences of each element in the second log file; and so on, until the occurrence counts of each element in all log files are obtained; counting the total occurrence counts of each element in all log files; storing elements as key values ​​and the total occurrence counts of elements as value values, forming an element array; sorting the key-value pairs in the element array in descending order of the total occurrence counts; replacing the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replacing the value of the second key-value pair in the element array with the second identifier in the identifier sequence, and so on, until all values ​​are replaced, forming an element identifier mapping relationship array; and replacing each element in each log file based on the mapping relationship in the element identifier mapping relationship array. Two-length markers are placed before three-length markers, with two-length markers being used first for replacement. After two-length markers are used up, if there are still elements that have not been replaced, three-length markers are used to replace the remaining elements.

[0185] In an optional implementation, the log compression module 550 compresses all new log files and mapping relationship files together.

[0186] In one optional implementation, a two-digit length identifier includes a separator and a flag value, wherein the flag value is any one of the following: a number from 0 to 9, 26 lowercase letters, or 26 uppercase letters; a three-digit length identifier includes a separator and two flag values, wherein the two flag values ​​are any combination of any two of the following: a number from 0 to 9, 26 lowercase letters, or 26 uppercase letters; when sorting all identifiers, they are sorted according to the following flag value order: 0, 1, ..., 9, a, b, ..., z, A, B, ..., Z, 00, 01, ..., 99, 0a, 0b, ..., 9z, a0, a1, ..., z9, 0A, 0B, ..., 9Z, A0, A1, ..., Z9, aa, ab, ..., zz, aA, aB, ..., aZ, Aa, Ab, ..., Zz, AA, AB, ..., ZZ.

[0187] Since the server log data compression system in this embodiment is used to implement the aforementioned server log data compression method, its function corresponds to the function of the above method, and will not be described again here.

[0188] Figure 6 A schematic diagram of a terminal 600 provided in an embodiment of the present invention includes: a processor 610, a memory 620, and a communication unit 630. The processor 610 is used to implement the following steps when executing the server log data compression program stored in the memory 620:

[0189] Collect log files;

[0190] Read the text content from the log file, and split the text content in the log file using spaces as the identifier to obtain multiple segments;

[0191] Loop through and compare each segment, and record segments with the same content as one element;

[0192] The text content is replaced with identifiers to obtain a new log file; the identifiers include two-digit and three-digit length identifiers; one identifier corresponds to one element.

[0193] Compress the new log file.

[0194] This invention utilizes the characteristics of log files, using spaces as identifiers to divide the text content into multiple segments, then filters out identical elements and replaces them with shorter identifiers, and finally compresses them. After element replacement, the number of characters in the log file is greatly reduced. On this basis, further compression is performed to reduce the size of the compressed log file and reduce hard disk space usage.

[0195] The terminal 600 includes a processor 610, a memory 620, and a communication unit 630. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figures does not constitute a limitation of the present invention. It can be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0196] The memory 620 can be used to store the execution instructions of the processor 610. The memory 620 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 620 are executed by the processor 610, the terminal 600 is able to perform some or all of the steps in the above method embodiments.

[0197] The processor 610 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 620, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 610 may only include a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0198] The communication unit 630 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It can receive user data sent by other terminals or send user data to other terminals.

[0199] The present invention also provides a computer storage medium, which may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0200] The computer storage medium stores a server log data compression program, which, when executed by the processor, performs the following steps:

[0201] Collect log files;

[0202] Read the text content from the log file, and split the text content in the log file using spaces as the identifier to obtain multiple segments;

[0203] Loop through and compare each segment, and record segments with the same content as one element;

[0204] The text content is replaced with identifiers to obtain a new log file; the identifiers include two-digit and three-digit length identifiers; one identifier corresponds to one element.

[0205] Compress the new log file.

[0206] This invention utilizes the characteristics of log files, using spaces as identifiers to divide the text content into multiple segments, then filters out identical elements and replaces them with shorter identifiers, and finally compresses them. After element replacement, the number of characters in the log file is greatly reduced. On this basis, further compression is performed to reduce the size of the compressed log file and reduce hard disk space usage.

[0207] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0208] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0209] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0210] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0211] The above-disclosed embodiments are merely preferred embodiments of the present invention, but the present invention is not limited thereto. Any non-creative variations that can be conceived by those skilled in the art, as well as any improvements and modifications made without departing from the principles of the present invention, should fall within the protection scope of the present invention.

Claims

1. A method for compressing server log data, characterized in that, Includes the following steps: Collect log files; Read the text content from the log file, and split the text content in the log file using spaces as the identifier to obtain multiple segments; Compare each segment in a loop, and record segments with the same content as one element; Use identifiers to replace each element in the text content to obtain a new log file; The identifiers include two-digit and three-digit length identifiers; one element corresponds to one identifier for compressing the new log file. Use identifiers to replace individual elements in the text content, specifically including: Sort all identifiers; two-digit identifiers are placed before three-digit identifiers, so that two-digit identifiers are used first for replacement. After two-digit identifiers are used up, if there are still elements that have not been replaced, then three-digit identifiers are used to replace the remaining elements. Count the number of times each element appears; The elements are used as keys and the number of times each element appears is used as values, so that the elements and their occurrences are stored as key-value pairs. All key-value pairs of elements and their occurrences constitute an element array. Sort the key-value pairs in the element array in descending order of the frequency of their occurrences; Replace the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replace the value of the second key-value pair in the element array with the second identifier in the identifier sequence, and so on, until all values ​​have been replaced, thus forming an element identifier mapping array; Based on the mapping relationships in the element identifier mapping relationship array, the elements in the log file are replaced.

2. The server log data compression method according to claim 1, characterized in that, The method also includes the following steps: Create a mapping file; Write the element identifier mapping array to the created mapping file; When compressing a new log file, both the new log file and the mapping relationship file are compressed together.

3. The server log data compression method according to claim 1, characterized in that, The collection log files specifically include: Collect all log files of the target program; the number of log files should be no less than 2. The process of reading the text content from the log file, splitting the text content in the log file using spaces as identifiers to obtain multiple segments, specifically includes: Read the text content from the first log file, and split the text content in the first log file into multiple segments using spaces as identifiers; Read the text content from the second log file, and split the text content in the second log file using spaces as identifiers to obtain multiple segments; This process continues until the text content of all log files has been divided into segments; The process of replacing each element in the text content with identifiers to obtain a new log file specifically includes: Use identifiers to replace each element in each log file to obtain multiple new log files; The compression process for the new log files specifically includes: compressing all the new log files together.

4. The server log data compression method according to claim 3, characterized in that, Use identifiers to replace elements within each log file, specifically including: Sort all identifiers in each log file; two-digit identifiers are placed before three-digit identifiers, so that two-digit identifiers are used first for replacement. After two-digit identifiers are used up, if there are still elements that have not been replaced, then three-digit identifiers are used to replace the remaining elements. Count the number of occurrences of each element in the first log file and store the results. Count the number of occurrences of each element in the second log file and store the results. This process continues until the occurrence count of each element in all log files is obtained; Count the total number of times each element appears in all log files; Use the element as the key and the total number of times the element appears as the value, so that the element and its total number of appearances are stored in the form of key-value pairs. All key-value pairs of elements and total number of appearances constitute an element array. Sort the key-value pairs in the element array in descending order of the total number of times each element appears; Replace the value of the first key-value pair in the element array with the first identifier in the identifier sequence, replace the value of the second key-value pair in the element array with the second identifier in the identifier sequence, and so on, until all values ​​have been replaced, thus forming an element identifier mapping array; Based on the mapping relationships in the element identifier mapping relationship array, each element in each log file is replaced.

5. The server log data compression method according to claim 4, characterized in that, The method also includes the following steps: Create a mapping file; Write the element identifier mapping array to the created mapping file; When compressing all new log files together, all new log files and mapping relationship files are compressed together.

6. The server log data compression method according to claim 1 or 4, characterized in that, The two-digit length identifier consists of a separator and a flag value, where the flag value is any one of the following: a number from 0 to 9, one of the 26 lowercase letters, or one of the 26 uppercase letters. The three-digit length identifier consists of a 1-digit separator and a 2-digit flag value. The 2-digit flag value is a combination of any two of the following: numbers from 0 to 9, 26 lowercase letters, and 26 uppercase letters. When sorting all identifiers, they are sorted in the following order of identifier values: 0, 1, ..., 9, a, b, ..., z, A, B, ..., Z, 00, 01, ..., 99, 0a, 0b, ..., 9z, a0, a1, ..., z9, 0A, 0B, ..., 9Z, A0, A1, ..., Z9, aa, ab, ..., zz, aA, aB, ..., aZ, Aa, Ab, ..., Zz, AA, AB, ..., ZZ.

7. A server log data compression system, the system being used to implement the server log data compression method as described in any one of claims 1 to 6, characterized in that, include, Log collection module: collects log files; Fragment segmentation module: Reads the text content in the log file, segments the text content in the log file using spaces as identifiers, and obtains multiple fragments; Element Analysis Module: Iterates through and compares each segment, and records segments with the same content as one element; Element replacement module: Uses identifiers to replace each element in the text content to obtain a new log file; the identifiers include two-digit and three-digit length identifiers; one identifier corresponds to one element; Log compression module: Compresses new log files.

8. A terminal, characterized in that, include: The storage device is used to store the server log data compression program; A processor, configured to implement the steps of the server log data compression method as described in any one of claims 1-6 when executing the server log data compression program.

9. A computer-readable storage medium, characterized in that, The readable storage medium stores a server log data compression program, which, when executed by a processor, implements the steps of the server log data compression method as described in any one of claims 1-6.

Images