Computer data intelligent management method
Through the method based on tree structure and text analysis, redundant removal and compression of log data is solved, and the problem of poor effectiveness of traditional compression algorithms is achieved, and efficient storage and management of log data is realized.
Patent Information
- Application Number
- CN202411487512.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional algorithms for compressing log data fail to effectively reduce the redundancy of log data, resulting in poor final compression of log data and affecting the work efficiency of the computer.
Using a tree structure and text analysis method, the log data is sorted, redundant, and the primary and advanced compressed coding tables are constructed, and the storage tree is built to improve the compression rate of the log data.
It significantly reduces the amount of stored data and improves the efficient storage, management and analysis capabilities of log data.
Smart Images

Figure CN119003471A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data compression, and in particular to a computer data intelligent management method. Background Art
[0002] In a computer system, logs are an important means of recording system activities, events and errors. However, as the scale and complexity of the system increase, the amount of log data is also growing rapidly. The log data is becoming more and more massive, and the storage space occupied is increasing. This leads to certain restrictions in log retrieval and log analysis, thereby reducing the work efficiency of the computer. The traditional algorithm for compressing log data does not take into account the large amount of redundant components in the log data, resulting in poor final compression effect of the log data. Therefore, this embodiment proposes a log storage method based on tree structure and text analysis, which reduces the redundancy of log files while utilizing the retrieval characteristics of the tree structure to improve the retrieval of the log, thereby achieving efficient storage, management and analysis of log data. Summary of the invention
[0003] The present invention provides a computer data intelligent management method to solve the existing problem that the traditional algorithm for compressing log data has a poor compression effect on the log data.
[0004] A computer data intelligent management method of the present invention adopts the following technical solution: The following steps are involved: Collect log data to obtain data to be compressed; The data to be compressed are sorted to obtain a sequence of data to be compressed, and a summary short code of the data to be compressed and a difference file of the data to be compressed are obtained according to adjacent data to be compressed in the sequence of data to be compressed; Remove redundancy in the difference file of the data to be compressed to obtain the target file of the data to be compressed; convert the target file of the data to be compressed into a primary compression code table of the data to be compressed; obtain a row number threshold of the primary compression code table of the data to be compressed according to the number of data in each column in the primary compression code table of the data to be compressed; obtain an advanced compression code table of the data to be compressed according to the row number threshold of the primary compression code table of the data to be compressed and the primary compression code table of the data to be compressed; obtain a final compression code table of the data to be compressed according to the value of each column of data in the advanced compression code table of the data to be compressed; compress the final compression code table of the data to be compressed to obtain a compression result of the difference file in the data to be compressed; According to the compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed, the compression log is obtained; and a storage tree is constructed to store the compressed log in the storage tree.
[0005] Preferably, the collecting log data to obtain the data to be compressed includes the following specific methods: All log data are retrieved from the log repository, and standard normalization processing is performed on each log data to obtain the standard normalization processing result of each log data, which is recorded as each data to be compressed.
[0006] Preferably, the method of sorting the data to be compressed to obtain a data sequence to be compressed, and obtaining a difference file between the summary short code of the data to be compressed and the data to be compressed according to adjacent data to be compressed in the data sequence to be compressed includes the following specific methods: For The data to be compressed and The data to be compressed is The first of the data to be compressed characters, and Whether the first character of the data to be compressed is exactly the same, if not, the first character The first character in the data to be compressed is recorded as the difference character. If they are exactly the same, the first character is recorded as the difference character. The first character in the data to be compressed is recorded as the same character, and the The classification of the first character in the data to be compressed; For Classify all characters in the data to be compressed and get the All the different characters and the same characters in the data to be compressed, and so on, according to each data to be compressed and the adjacent previous data to be compressed, obtain the different characters and the same characters in each data to be compressed except the first data to be compressed; Record the data consisting of the difference characters in each piece of data to be compressed except the first piece of data to be compressed as a difference file; Get the The summary of all the same characters in the data to be compressed is recorded as The summary short code of the data to be compressed.
[0007] Preferably, the removing redundancy in the difference file of the data to be compressed and obtaining the target file of the data to be compressed comprises the following specific methods: For The data to be compressed is obtained All blank text in the difference file of the data to be compressed is replaced by Replace all blank text in the difference file of the data to be compressed with special characters , and get the first The difference file of the data to be compressed is recorded as The characteristic file of the data to be compressed; Replace all uppercase English letters in the feature file of the data to be compressed with lowercase English letters to obtain the The target file for the data to be compressed.
[0008] Preferably, the specific method of converting the target file of the data to be compressed into the primary compression coding table of the data to be compressed includes: For The target file of the data to be compressed, statistics The number of character types in the target file of the data to be compressed is recorded as , preset an empty column number equal to Each character is placed in the first row of the compression code table, and the index position of any character in the target file of the data to be compressed is filled in line by line under the column where the character is located in the first row of the compression code table in order to obtain the first The primary compression code table of the data to be compressed.
[0009] Preferably, the method of obtaining the row number threshold of the primary compression coding table of the data to be compressed according to the number of data in each column in the primary compression coding table of the data to be compressed includes the following specific methods: For The primary compression code table of the data to be compressed is used to obtain the row number threshold of the primary compression code table according to the number of columns in the primary compression code table and the amount of data in each column. The specific calculation formula is: In the formula, Indicates The row number threshold of the primary compression code table of the data to be compressed; Indicates The number of character types in the target file of the data to be compressed; Indicates The target file of the data to be compressed is The number of characters; Indicates The character in The column number order value in the primary compression code table of the data to be compressed; Indicates a round-up operation.
[0010] Preferably, the step of obtaining the advanced compression coding table of the data to be compressed according to the row number threshold of the primary compression coding table of the data to be compressed and the primary compression coding table of the data to be compressed comprises the following specific methods: The first The number of rows in the primary compression code table of the data to be compressed is less than The columns are recorded as short columns, and the number of rows is equal to The column is recorded as the base column, and the number of rows is greater than The columns are recorded as long columns; A long column, The first in a long line The data after the row is recorded as Long columns of data to be transferred; According to The distribution of the base column, short column and long column in the primary compression code table of the data to be compressed, as well as the data to be transferred in the long column, is obtained by filling 0 and transferring the data to be transferred from the long column to the short column until all the long columns are converted into base columns. An intermediate compression code table of the data to be compressed; The first In the intermediate compression code table of the data to be compressed, add several zeros to the end of each column of data until the number of data in each column is , get the An advanced compression code table for the data to be compressed.
[0011] Preferably, the method of obtaining the final compression coding table of the data to be compressed according to the value of each column of data in the advanced compression coding table of the data to be compressed includes the following specific methods: For The first entry in the advanced compression code table of the data to be compressed Column data, get the The first entry in the advanced compression code table of the data to be compressed The rounded-off result of the mean of all data in the column is recorded as the reference value. The first entry in the advanced compression code table of the data to be compressed Replace all 0s in the column data with the reference value, and then The first entry in the advanced compression code table of the data to be compressed Divide all data in the column by the reference value to get the The first entry in the advanced compression code table of the data to be compressed The remainder and multiple of each data in the column data; For The first entry in the advanced compression code table of the data to be compressed Column data data, the first The first entry in the advanced compression code table of the data to be compressed Column data The sum of the remainder and multiple of the data is taken as the The first entry in the advanced compression code table of the data to be compressed Column data The characterization value of the data is obtained The representation value of each data in the advanced compression coding table of the data to be compressed, and replace the All the data in the advanced compression coding table of the data to be compressed are obtained The final compression code table of the data to be compressed.
[0012] Preferably, the method of compressing the final compression coding table of the data to be compressed to obtain the compression result of the difference file in the data to be compressed includes the following specific methods: For The final compression code table of the data to be compressed is scanned column by column in an S shape to obtain the The sequence of the final compression code table of the data to be compressed is recorded as the target sequence, and the target sequence is compressed using the LZ77 compression algorithm to obtain the compression result of the target sequence, and the compression result of the target sequence is recorded as the compression result of the difference file in the data to be compressed.
[0013] Preferably, the method of obtaining a compression log according to the compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed; and constructing a storage tree to store the compressed log in the storage tree includes the following specific methods: For The data to be compressed will be The compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed is recorded as Get all compressed logs and take the first piece of data to be compressed as a compressed log. A storage tree is constructed in a time-oriented manner. The node where the newly stored log file is located is used as the query root node. After each new node is inserted, the height difference between the left subtree and the right subtree is made 1. Finally, all compressed logs are stored in the storage tree.
[0014] The beneficial effects of the technical solution of the present invention are as follows: the present application collects log data to obtain data to be compressed; sorts the data to be compressed to obtain a sequence of data to be compressed, and obtains a summary short code of the data to be compressed and a difference file of the data to be compressed according to adjacent data to be compressed in the sequence of data to be compressed. Since there are a large number of repeated characters in the log data, the present application greatly reduces the data required to be stored by only analyzing and processing the difference file; obtains a final compression coding table of the data to be compressed according to the difference file of the data to be compressed, and achieves the ultimate compression of the difference file by analyzing the difference file; compresses the final compression coding table of the data to be compressed to obtain a compression result of the difference file in the data to be compressed; obtains a compressed log according to the summary short code of the data to be compressed and the compression result of the difference file in the data to be compressed; constructs a storage tree to store the compressed log in the storage tree, and finally improves the efficient storage, management and analysis of the log data. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 A flowchart of the steps of a computer data intelligent management method of the present invention; Figure 2 Convert the target file of the present invention into a primary compression code representation; Figure 3 It is the intermediate compression coding of the present invention; Figure 4 This is the advanced compression coding of the present invention. DETAILED DESCRIPTION
[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a computer data intelligent management method proposed by the present invention, its specific implementation method, structure, features and effects, in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0019] The specific scheme of a computer data intelligent management method provided by the present invention is described in detail below with reference to the accompanying drawings.
[0020] See also Figure 1 , which shows a flowchart of a method for intelligent management of computer data provided by an embodiment of the present invention, the method comprising the following steps: Step S001: Collect log data to obtain data to be compressed.
[0021] It should be noted that in a computer system, logs are an important means of recording system activities, events and errors. They record various operations, state changes and error information that occur when the system is running, but with the increase in system scale and complexity, the amount of log data is also growing rapidly, and the log data is becoming more and more huge, occupying more and more storage space, which leads to certain restrictions when performing log retrieval and log analysis, thereby reducing the work efficiency of the computer; therefore, this embodiment proposes a log storage method based on tree structure and text analysis, which reduces the redundancy of log files while using the retrieval characteristics of the tree structure to improve the retrieval of the log, and improves the efficient storage, management and analysis of log data. To this end, it is first necessary to obtain the log data of the computer.
[0022] Specifically, all log data are retrieved from the log storage repository, and standard normalization processing is performed on each piece of log data to obtain a standard normalization processing result for each piece of log data, which is recorded as each piece of data to be compressed.
[0023] It should be noted that each log data is log data collected at the same timestamp, and in order to be able to analyze the log data more conveniently, it is also necessary to perform standard normalization processing on the log data to obtain the data to be compressed, and all the data to be compressed; since the specific method of performing standard normalization processing on the log data is a well-known prior art, it will not be repeated in this embodiment.
[0024] At this point, all the data to be compressed are obtained.
[0025] Step S002: sorting the data to be compressed to obtain a sequence of data to be compressed, and obtaining a summary short code of the data to be compressed and a difference file of the data to be compressed according to adjacent data to be compressed in the sequence of data to be compressed.
[0026] It should be noted that, as a computer data intelligent management method, the specific application scenario of this embodiment is to compress and store log data, so as to make the storage space occupied by the compression results of all the data to be compressed as small as possible. Therefore, this embodiment obtains the same parts and different parts between different data to be compressed, and only analyzes the different parts, so as to achieve the purpose of making the storage space occupied by the compression results of all the data to be compressed smaller.
[0027] Specifically, obtain the timestamp of each log data corresponding to each piece of data to be compressed, record it as the timestamp of each piece of data to be compressed, sort each piece of data to be compressed according to the timestamp of the data to be compressed, and obtain a sequence of data to be compressed; For The data to be compressed and The data to be compressed is The first characters, and Whether the first character of the data to be compressed is exactly the same, if not, the first character The first character in the data to be compressed is recorded as the difference character. If they are exactly the same, the first character is recorded as the difference character. The first character in the data to be compressed is recorded as the same character, and the The classification of the first character in the data to be compressed; For Classify all characters in the data to be compressed and get the All the different characters and the same characters in the data to be compressed, and so on, according to each data to be compressed and the adjacent previous data to be compressed, obtain the different characters and the same characters in each data to be compressed except the first data to be compressed; The data consisting of the difference characters in each piece of data to be compressed except the first piece of data to be compressed is recorded as a difference file.
[0028] Furthermore, for the The data to be compressed is obtained by using the MD5 information digest algorithm. The summary of all the same characters in the data to be compressed is recorded as The summary short code of the data to be compressed is not described in detail in this embodiment because the MD5 information summary algorithm is a well-known prior art.
[0029] It should be noted that The method for classifying all characters in the data to be compressed is the same as the method for classifying the characters in the first The method for classifying the first character in the data to be compressed is the same, so this embodiment will not be repeated. The same characters of the data to be compressed are The data to be compressed and The characters in each character position of the data to be compressed are the same, so the first The data to be compressed is obtained The same characters in the data to be compressed.
[0030] So far, the summary short code and difference file of the data to be compressed.
[0031] Step S003: remove redundancy in the difference file of the data to be compressed, and obtain the target file of the data to be compressed; convert the target file of the data to be compressed into a primary compression coding table of the data to be compressed; obtain the row number threshold of the primary compression coding table of the data to be compressed according to the number of data in each column in the primary compression coding table of the data to be compressed; obtain the advanced compression coding table of the data to be compressed according to the row number threshold of the primary compression coding table of the data to be compressed and the primary compression coding table of the data to be compressed; obtain the final compression coding table of the data to be compressed according to the value of each column of data in the advanced compression coding table of the data to be compressed; compress the final compression coding table of the data to be compressed to obtain the compression result of the difference file in the data to be compressed.
[0032] It should be noted that the summary short code and difference file of the data to be compressed are obtained through step S002. Since the summary short codes of all data to be compressed except the first data to be compressed can be obtained through the previous data to be compressed, only the difference file of the data to be compressed needs to be analyzed.
[0033] It should be further explained that since the difference file of each data to be compressed is composed of the remaining characters after removing the same characters in the data to be compressed, there is a large amount of blank text in the difference file of the data to be compressed, that is, there is a large amount of redundancy in the difference file of the data to be compressed. Therefore, in order to better compress the difference file of the data to be compressed, it is necessary to remove the redundancy of the difference file of the data to be compressed.
[0034] Specifically, for The data to be compressed is obtained All blank text in the difference file of the data to be compressed is replaced by Replace all blank text in the difference file of the data to be compressed with special characters , and get the first The difference file of the data to be compressed is recorded as Feature file of the data to be compressed.
[0035] It should be noted that after obtaining the characteristic files of each piece of data to be compressed except the first piece of data to be compressed, the characteristic files of the data to be compressed can be compressed; and the data specifically processed in this embodiment is computer log data. Since the data to be compressed is log data that has been processed by standard normalization, only uppercase and lowercase letters and Arabic numerals exist in the data to be compressed; however, in this embodiment, in the process of removing the redundancy of the difference files of the data to be compressed, special characters are added. , that is, there are no special characters in the data to be compressed , so you can use special characters Indicates blank text in the data to be compressed, ensuring that the log data is the same as the original data when the log data is restored later. After obtaining the feature file of the data to be compressed, the feature file of the data to be compressed can be further analyzed to achieve compression of the feature file of the data to be compressed.
[0036] It should be further explained that, since the case of English letters in the log data does not affect the readability and analyzability of the log data, the compression effect of the data to be compressed can be further improved by the case of English letters in the data to be compressed.
[0037] Specifically, all uppercase English letters in the feature file of each piece of data to be compressed are replaced with lowercase English letters to obtain a target file of each piece of data to be compressed.
[0038] It should be noted that, since the first data to be compressed has no feature file, the target file of each data to be compressed is the target file of each data to be compressed except the first data to be compressed; and after replacing all uppercase English letters in each data to be compressed with lowercase English letters, each data to be compressed can be analyzed; since each data to be compressed is a one-dimensional data sequence, and converting the one-dimensional sequence into a two-dimensional matrix can better analyze the data structure in the one-dimensional sequence, so this embodiment converts the feature file of each data to be compressed into a two-dimensional matrix for analysis.
[0039] Specifically, for The target file of the data to be compressed, statistics The number of character types in the target file of the data to be compressed is recorded as , preset an empty column number equal to Each character is placed in the first row of the compression code table, and the index position of any character in the target file of the data to be compressed is filled in line by line under the column where the character is located in the first row of the compression code table in order to obtain the first The primary compression code table of the data to be compressed is as follows: Figure 2 shown.
[0040] It should be noted that, in the primary compression coding table of the data to be compressed, the amount of data in each column is not exactly the same, which will result in a large waste of storage space. In order to further improve the compression rate, this embodiment records the columns with a large amount of data in the primary compression coding table as long columns, and the columns with a small amount of data as short columns. By transferring part of the data in the long column to the short column, the compression rate is further improved.
[0041] Specifically, for The primary compression code table of the data to be compressed is used to obtain the row number threshold of the primary compression code table according to the number of columns in the primary compression code table and the amount of data in each column. The specific calculation formula is: In the formula, Indicates The row number threshold of the primary compression code table of the data to be compressed; Indicates The number of character types in the target file of the data to be compressed; Indicates The target file of the data to be compressed is The number of characters; Indicates The character in The column number order value in the primary compression code table of the data to be compressed; Indicates a round-up operation.
[0042] The first The number of rows in the primary compression code table of the data to be compressed is less than The columns are recorded as short columns, and the number of rows is equal to The column is recorded as the base column, and the number of rows is greater than The columns are recorded as long columns; A long column, The first in a long line The data after the row is recorded as Long columns of data to be transferred; Starting from the first short column, The difference between the number of data in the first short column is recorded as , record the long column closest to the first short column as the first transfer long column of the first short column. If there are two long columns closest to the first short column, record the long column on the left of the first short column as the first transfer long column of the first short column. Get the distance between the first short column and the first transfer long column of the first short column and record it as ,like Greater than or equal to Then the first short column is recorded as the base column; like Less than , then add at the end of the first short column 0, get the data to be transferred in the first transfer long column of the first short column, if the number of data to be transferred in the first transfer long column of the first short column is greater than , then transfer the first part of the first short column to the last part of the long column The data are transferred to the first short column to obtain the first transfer result of the first short column, and the first transfer result of the first short column is recorded as the reference column; If the number of data to be transferred in the first transfer long column of the first short column is equal to , then all the data to be transferred in the first transfer long column of the first short column are transferred to the first short column to obtain the first transfer result of the first short column, and the first transfer result of the first short column and the first transfer long column of the first short column are recorded as the reference column; If the number of data to be transferred in the first transfer long column of the first short column is less than , then all the data to be transferred in the first transfer long column of the first short column are transferred to the first short column, and the first transfer long column of the first short column is recorded as the reference column, and the first transfer result of the first short column is obtained; Will The difference between the number of data in the first transfer result of the first short column is recorded as , record the long column that is the second closest to the first short column as the second transfer long column of the first short column. If there are two long columns that are the second closest to the first short column, record the long column on the left of the first short column as the second transfer long column of the first short column. Get the distance between the first short column and the second transfer long column of the first short column and record it as ,like Greater than or equal to Then the first transfer result of the first short column is recorded as the base column; like Less than , then add at the end of the first short column 0, get the data to be transferred in the second transfer long column of the first short column, if the number of data to be transferred in the second transfer long column of the first short column is greater than , then transfer the second of the first short column to the last of the long column The data are transferred to the first short column to obtain the second transfer result of the first short column, and the first transfer result of the first short column is recorded as the reference column; If the number of data to be transferred in the second transfer long column of the first short column is equal to , then transfer the second of the first short column to the last of the long column The data are transferred to the first short column to obtain the second transfer result of the first short column, and the first transfer result of the first short column and the second transfer long column of the first short column are recorded as the reference column; If the number of data to be transferred in the second transfer long column of the first short column is less than , then all the data to be transferred in the second transfer long column of the first short column are transferred to the first transfer result of the first short column, and the second transfer long column of the first short column is recorded as the reference column to obtain the second transfer result of the first short column; And so on, until the first short column is obtained. The first short column is The result of the first transfer is the base column; the transfer of the first short column is completed; then the second short column is transferred, and the third short column is transferred, until all long columns are converted into base columns, and the first An intermediate compression code table of the data to be compressed; The first In the intermediate compression code table of the data to be compressed, add several zeros to the end of each column of data until the number of data in each column is , get the The advanced compression coding table of the data to be compressed; Figure 3 , Figure 4 shown.
[0043] It should be noted that, when transferring short columns, in order to determine which long columns the transferred data in the transferred short columns are transferred from, it is necessary to add 0 at the end of the short column before transferring the data, resulting in the number of data in each column after the transfer being greater than the average number of each character in the target file of the data to be compressed. Therefore, it is necessary to add the index value of each character when calculating the row number threshold of the primary compression coding table of the data to be compressed, so that the row number threshold of the primary compression coding table of the data to be compressed is greater than the average number of each character in the target file of the data to be compressed, so as to ensure that the maximum number of rows in the primary compression coding table is shortened as much as possible to improve the compression ratio.
[0044] It should be further explained that since each data in the primary coding table represents the position of each character in the target file, there is no 0 in the primary coding table. Therefore, 0 can be added to indicate the source of the data after 0, and at the same time, data preparation is made for the subsequent compression of the data through advanced compression coding.
[0045] Specifically, for The first entry in the advanced compression code table of the data to be compressed Column data, get the The first entry in the advanced compression code table of the data to be compressed The rounded-off result of the mean of all data in the column is recorded as the reference value. The first entry in the advanced compression code table of the data to be compressed Replace all 0s in the column data with the reference value, and then The first entry in the advanced compression code table of the data to be compressed Divide all data in the column by the reference value to get the The first entry in the advanced compression code table of the data to be compressed The remainder and multiple of each data in the column data; For The first entry in the advanced compression code table of the data to be compressed Column data data, the first The first entry in the advanced compression code table of the data to be compressed Column data The sum of the remainder and multiple of the data is taken as the The first entry in the advanced compression code table of the data to be compressed Column data The characterization value of the data is obtained The representation value of each data in the advanced compression coding table of the data to be compressed, and replace the All the data in the advanced compression coding table of the data to be compressed are obtained The final compression code table of the data to be compressed.
[0046] It should be noted that since the advanced compression coding table is a table consisting of 0 and the index position of each character, the data values in the advanced compression coding are generally large. In order to further improve the compression rate, this embodiment calculates the baseline value of each column of data to obtain the characterization value of each data. Since the characterization value of each data is smaller than its original value, the final compression coding table is obtained. Therefore, the purpose of improving the compression rate can be achieved by compressing the final compression coding table.
[0047] Specifically, for The final compression code table of the data to be compressed is scanned column by column in an S shape to obtain the The sequence of the final compression coding table of the data to be compressed is recorded as the target sequence. Since the S-shaped scanning is a well-known prior art, it is not described in detail in this embodiment. The target sequence is compressed using the LZ77 compression algorithm to obtain the compression result of the target sequence, and the compression result of the target sequence is recorded as the compression result of the difference file in the data to be compressed. Since the LZ77 compression algorithm is a well-known prior art, it is not described in detail in this embodiment.
[0048] It should be noted that this step compresses the difference files in the data to be compressed by analyzing the difference files in the data to be compressed, and the compression result of the tag sequence obtained in this step is the compression result of the difference files in the data to be compressed.
[0049] Step S004: Obtain compression logs according to the summary short code of the data to be compressed and the compression results of the difference files in the data to be compressed; construct a storage tree to store the compressed logs in the storage tree.
[0050] It should be noted that the summary short code of the data to be compressed and the compression results of the difference files in the data to be compressed obtained through steps S002 and S003 are respectively the compression results of the data composed of the same characters in the data to be compressed and the compression results of the data composed of different characters. By storing the summary short code of the data to be compressed and the compression results of the difference files in the data to be compressed, the space occupied by storing log data can be saved.
[0051] Specifically, for The data to be compressed will be The compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed is recorded as Get all compressed logs and take the first piece of data to be compressed as a compressed log. A storage tree is constructed in a time-oriented manner. The node where the newly stored log file is located is used as the query root node. After each new node is inserted, the height difference between the left subtree and the right subtree is made 1. Finally, all compressed logs are stored in the storage tree.
[0052] It should be noted that since the specific process of constructing the storage tree is a well-known prior art, it will not be repeated in this embodiment. After all compressed logs are stored in the storage tree, the storage of the log data is completed. When it is needed, it is only necessary to find the required log according to the tree index of the storage tree and restore the data log to obtain the corresponding computer log.
[0053] At this point, this embodiment is completed.
[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A computer data intelligent management method, characterized in that: The method comprises the following steps: Collect log data to obtain data to be compressed; The data to be compressed are sorted to obtain a sequence of data to be compressed, and a summary short code of the data to be compressed and a difference file of the data to be compressed are obtained according to adjacent data to be compressed in the sequence of data to be compressed; Remove redundancy in the difference file of the data to be compressed to obtain the target file of the data to be compressed; convert the target file of the data to be compressed into a primary compression code table of the data to be compressed; obtain a row number threshold of the primary compression code table of the data to be compressed according to the number of data in each column in the primary compression code table of the data to be compressed; obtain an advanced compression code table of the data to be compressed according to the row number threshold of the primary compression code table of the data to be compressed and the primary compression code table of the data to be compressed; obtain a final compression code table of the data to be compressed according to the value of each column of data in the advanced compression code table of the data to be compressed; compress the final compression code table of the data to be compressed to obtain a compression result of the difference file in the data to be compressed; According to the compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed, the compression log is obtained; and a storage tree is constructed to store the compressed log in the storage tree.
2. A computer data intelligent management method according to claim 1, characterized in that: The method of collecting log data to obtain the data to be compressed includes: All log data are retrieved from the log repository, and standard normalization processing is performed on each log data to obtain the standard normalization processing result of each log data, which is recorded as each data to be compressed.
3. A computer data intelligent management method according to claim 1, characterized in that: The method of sorting the data to be compressed to obtain a sequence of data to be compressed, and obtaining a difference file of the summary short code of the data to be compressed and the data to be compressed according to adjacent data to be compressed in the sequence of data to be compressed includes the following specific methods: For The data to be compressed and The data to be compressed is The first of the data to be compressed characters, and Whether the first character of the data to be compressed is exactly the same, if not, the first character The first character in the data to be compressed is recorded as the difference character. If they are exactly the same, the first character is recorded as the difference character. The first character in the data to be compressed is recorded as the same character, and the The classification of the first character in the data to be compressed; For Classify all characters in the data to be compressed and get the All the different characters and the same characters in the data to be compressed, and so on, according to each data to be compressed and the adjacent previous data to be compressed, obtain the different characters and the same characters in each data to be compressed except the first data to be compressed; Record the data consisting of the difference characters in each piece of data to be compressed except the first piece of data to be compressed as a difference file; Get the The summary of all the same characters in the data to be compressed is recorded as The summary short code of the data to be compressed.
4. A computer data intelligent management method according to claim 1, characterized in that: The method of removing redundancy in the difference file of the data to be compressed and obtaining the target file of the data to be compressed includes: For The data to be compressed is obtained All blank text in the difference file of the data to be compressed is replaced by Replace all blank text in the difference file of the data to be compressed with special characters , and get the first The difference file of the data to be compressed is recorded as A feature file of the data to be compressed; The first Replace all uppercase English letters in the feature file of the data to be compressed with lowercase English letters to obtain the The target file for the data to be compressed.
5. A computer data intelligent management method according to claim 1, characterized in that: The specific method of converting the target file of the data to be compressed into the primary compression coding table of the data to be compressed is as follows: For The target file of the data to be compressed, statistics The number of character types in the target file of the data to be compressed is recorded as , preset an empty column number equal to Each character is placed in the first row of the compression code table, and the index position of any character in the target file of the data to be compressed is filled in line by line under the column where the character is located in the first row of the compression code table in order to obtain the first The primary compression code table of the data to be compressed.
6. A computer data intelligent management method according to claim 1, characterized in that: The specific method of obtaining the row number threshold of the primary compression coding table of the data to be compressed according to the number of each column of data in the primary compression coding table of the data to be compressed includes: For The primary compression code table of the data to be compressed is used to obtain the row number threshold of the primary compression code table according to the number of columns in the primary compression code table and the amount of data in each column. The specific calculation formula is: In the formula, Indicates The row number threshold of the primary compression code table of the data to be compressed; Indicates The number of character types in the target file of the data to be compressed; Indicates The target file of the data to be compressed is The number of characters; Indicates The character in The column number order value in the primary compression code table of the data to be compressed; Indicates a round-up operation.
7. A computer data intelligent management method according to claim 1, characterized in that: The method of obtaining the advanced compression coding table of the data to be compressed according to the row number threshold of the primary compression coding table of the data to be compressed and the primary compression coding table of the data to be compressed comprises the following specific methods: The first The number of rows in the primary compression code table of the data to be compressed is less than The columns are recorded as short columns, and the number of rows is equal to The column is recorded as the base column, and the number of rows is greater than The columns are recorded as long columns; A long column, The first in a long line The data after the row is recorded as Long columns of data to be transferred; According to The distribution of the base column, short column and long column in the primary compression code table of the data to be compressed, as well as the data to be transferred in the long column, is obtained by filling 0 and transferring the data to be transferred from the long column to the short column until all the long columns are converted into base columns. An intermediate compression code table of the data to be compressed; The first In the intermediate compression code table of the data to be compressed, add several zeros to the end of each column of data until the number of data in each column is , get the An advanced compression code table for the data to be compressed.
8. A computer data intelligent management method according to claim 1, characterized in that: The method of obtaining the final compression coding table of the data to be compressed according to the value of each column of data in the advanced compression coding table of the data to be compressed includes: For The first entry in the advanced compression code table of the data to be compressed Column data, get the The first entry in the advanced compression code table of the data to be compressed The rounded-off result of the mean of all data in the column is recorded as the reference value. The first entry in the advanced compression code table of the data to be compressed Replace all 0s in the column data with the reference value, and then The first entry in the advanced compression code table of the data to be compressed Divide all data in the column by the reference value to get the The first entry in the advanced compression code table of the data to be compressed The remainder and multiple of each data in the column data; For The first entry in the advanced compression code table of the data to be compressed Column data data, the first The first entry in the advanced compression code table of the data to be compressed Column data The sum of the remainder and multiple of the data is taken as the The first entry in the advanced compression code table of the data to be compressed Column data The characterization value of the data is obtained The representation value of each data in the advanced compression coding table of the data to be compressed, and replace the All the data in the advanced compression coding table of the data to be compressed are obtained The final compression code table of the data to be compressed.
9. A computer data intelligent management method according to claim 1, characterized in that: The method of compressing the final compression coding table of the data to be compressed to obtain the compression result of the difference file in the data to be compressed includes: For The final compression code table of the data to be compressed is scanned column by column in an S shape to obtain the The sequence of the final compression code table of the data to be compressed is recorded as the target sequence, and the target sequence is compressed using the LZ77 compression algorithm to obtain the compression result of the target sequence, and the compression result of the target sequence is recorded as the compression result of the difference file in the data to be compressed.
10. A computer data intelligent management method according to claim 1, characterized in that: The method of obtaining a compression log according to the summary short code of the data to be compressed and the compression result of the difference file in the data to be compressed and constructing a storage tree to store the compressed log in the storage tree includes the following specific methods: For The data to be compressed will be The compression result of the summary short code of the data to be compressed and the difference file in the data to be compressed is recorded as Get all compressed logs and take the first piece of data to be compressed as a compressed log. A storage tree is constructed in a time-oriented manner. The node where the newly stored log file is located is used as the query root node. After each new node is inserted, the height difference between the left subtree and the right subtree is made 1. Finally, all compressed logs are stored in the storage tree.
Citation Information
Patent Citations
File compression and decompression method and device
CN106844479A
Distributed log unified management method
CN111274215A
Text compression method, module, chip, electronic equipment and storage medium
CN114492322A
Data processing method and device, storage medium, and electronic device
EP3757815A1
