Log data compression method and device, electronic equipment and medium
By dynamically selecting compression algorithms and indexes in IoT devices, the compression and index structure challenges in high-order magnitude device log data processing are solved, and space saving and retrieval efficiency are improved.
Patent Information
- Application Number
- CN202410926792.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
AI Technical Summary
On the Internet of Things device server side, how to choose the best compression algorithm to process device log data of high orders of magnitude, especially in the process of log filtering and indexing structure processing, the existing technology is difficult to effectively solve.
By obtaining log data, selecting key information to determine the compression algorithm, compressing the log data, and setting an index based on the key information; for the compressed log data, determine whether it is necessary to optimize the compression algorithm, and dynamically select new key information and compression algorithms. , to adapt to the different characteristics and usage of log data.
It realizes efficient compression and indexing of log data in IoT devices, saves storage space, and improves the retrieval efficiency of important logs.
Smart Images

Figure CN118939617A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart home technology, and in particular to a log data compression method and a log data compression device, electronic equipment and medium. Background Art
[0002] Smart home refers to the use of advanced information technology and Internet of Things (IoT) technology to connect traditional home appliances and systems to the Internet to achieve automated, intelligent and convenient home management and control systems. Compression algorithm is a method of encoding data into a smaller volume, the main purpose of which is to reduce data storage space and increase transmission speed. It is based on statistical laws and reduces data volume by eliminating redundant information and encoding data in the data.
[0003] Data analysis is a process of analyzing, summarizing, understanding and digesting a large amount of collected data through appropriate statistical analysis methods, aiming to maximize the development of data functions and play the role of data. Its core goal is to extract useful information and form conclusions from data to help enterprises or institutions make more informed decisions.
[0004] When the IoT device server is faced with processing a high number of device logs, how to select the best compression algorithm in the process of log screening, compression algorithm, index structure, etc. is an urgent problem to be solved. Summary of the invention
[0005] In view of the above problems, embodiments of the present invention are proposed to provide a log data compression method and a corresponding log data compression device that overcome the above problems or at least partially solve the above problems.
[0006] In order to solve the above problem, an embodiment of the present invention discloses a log data compression method, comprising:
[0007] Get log data;
[0008] Select key information according to the log data;
[0009] Determine a compression algorithm according to the key information, compress the log data using the compression algorithm, and set an index for the compressed log data according to the key information;
[0010] For the compressed log data, determine whether the compression algorithm needs to be optimized;
[0011] When the compression algorithm needs to be optimized, new key information is selected, and the step of determining the compression algorithm according to the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information is returned.
[0012] Optionally, judging whether it is necessary to optimize the compression algorithm for the compressed log data includes:
[0013] Obtaining usage of the compressed log data for retrieval;
[0014] Whether the compression algorithm needs to be optimized is determined based on the usage of searching the compressed log data.
[0015] Optionally, judging whether it is necessary to optimize the compression algorithm for the compressed log data includes:
[0016] Obtaining compression performance of compressing the log data using the compression algorithm;
[0017] According to the compression performance, it is determined whether the compression algorithm needs to be optimized.
[0018] Optionally, the selecting new key information includes:
[0019] Obtain the keywords used by users in searching logs;
[0020] New key information is determined according to the keywords used by the user in the process of searching the logs.
[0021] Optionally, determining new key information according to keywords used by the user in the process of searching the logs includes:
[0022] Obtain the weight of the keywords used by the user in the process of searching logs;
[0023] New key information is determined based on the keywords used by the user in the process of searching the logs and the weights of the keywords.
[0024] Optionally, the compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information, includes:
[0025] The log data is compressed using the compression algorithm to obtain a plurality of log blocks;
[0026] According to the key information, indexes are set for the multiple log blocks respectively.
[0027] Optionally, the compressing the log data using the compression algorithm and setting an index for the compressed log data according to the key information further includes:
[0028] If the size of the index of the multiple log blocks is greater than a preset threshold, compression processing is performed on the index of the multiple log blocks.
[0029] An embodiment of the present invention further provides a log data compression device, the device comprising:
[0030] A data acquisition module is used to acquire log data;
[0031] An information selection module, used to select key information according to the log data;
[0032] A data compression and index setting module, used to determine a compression algorithm according to the key information, compress the log data using the compression algorithm, and set an index for the compressed log data according to the key information;
[0033] The first judgment module is used to judge whether the compression algorithm needs to be optimized for the compressed log data;
[0034] The step return module is used to select new key information when the compression algorithm needs to be optimized, and return the step of determining the compression algorithm based on the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data based on the key information.
[0035] Optionally, the first determination module includes:
[0036] A usage status acquisition module, used to acquire usage status of the compressed log data;
[0037] The first judgment submodule is used to judge whether the compression algorithm needs to be optimized according to the usage of searching the compressed log data.
[0038] Optionally, the first determining module further includes:
[0039] A compression performance acquisition module, used to acquire the compression performance of compressing the log data using the compression algorithm;
[0040] The second judgment submodule is used to judge whether the compression algorithm needs to be optimized according to the compression performance.
[0041] Optionally, the information selection module includes:
[0042] The keyword acquisition module is used to obtain the keywords used by users in the process of searching logs;
[0043] The first key information determination module is used to determine new key information according to the keywords used by the user in the process of searching the logs.
[0044] Optionally, the key information determination module includes:
[0045] The weight acquisition module is used to obtain the weights of keywords used by users in the process of searching logs;
[0046] The second key information determination module is used to determine new key information according to the keywords used by the user in the process of searching the logs and the weights of the keywords.
[0047] Optionally, the data compression and index setting module includes:
[0048] A first data compression submodule, configured to compress the log data using the compression algorithm to obtain a plurality of log blocks;
[0049] The index setting submodule is used to set indexes for the multiple log blocks respectively according to the key information.
[0050] Optionally, the data compression and index setting module further includes:
[0051] The second data compression submodule is configured to compress the indexes of the plurality of log blocks if the size of the indexes of the plurality of log blocks is greater than a preset threshold.
[0052] An embodiment of the present invention further provides an electronic device, the device comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program, when executed by the processor, implements the steps of a log data compression method as described above.
[0053] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log data compression method described above are implemented.
[0054] The embodiments of the present invention include the following advantages:
[0055] In an embodiment of the present invention, a log data compression method is introduced, which first obtains log data, selects key information according to the log data, and then determines a compression algorithm according to the key information, compresses the log data, and sets an index for the compressed log data according to the key information; for the compressed log data, it is determined whether the compression algorithm needs to be optimized; when the compression algorithm needs to be optimized, new key information is selected, and the compression algorithm is determined according to the key information, the log data is compressed using the compression algorithm, and the compressed log data is indexed according to the key information. In the process of using a large number of IoT devices, the number of logs generated will be large. The present invention will select different compression methods and set indexes for different key information of these logs, and dynamically select compression methods for log data according to actual usage, so as to save space while improving the retrieval efficiency of important logs. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 is a flow chart of the steps of a log data compression method provided by an embodiment of the present invention;
[0057] Figure 2 is a flowchart of another log data compression method provided by an embodiment of the present invention;
[0058] Figure 3 This is another flow chart of compressing log data provided by an embodiment of the present invention;
[0059] Figure 4 It is a structural block diagram of a log data compression device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0061] One of the core concepts of the embodiments of the present invention is that when a large number of IoT devices are used, a large number of logs will be generated. The present invention will select different compression methods and create index formats based on the different characteristics of these logs, and divide the log data according to actual usage, and then dynamically select the compression method to save space while improving the retrieval efficiency of important logs.
[0062] Smart home refers to the use of advanced information technology and Internet of Things (IoT) technology to connect traditional home appliances and systems to the Internet to achieve automated, intelligent and convenient home management and control systems. Compression algorithm is a method of encoding data into a smaller volume, the main purpose of which is to reduce the storage space of data and increase the transmission speed. It is based on statistical laws and reduces the volume of data by eliminating redundant information and encoding data in the data. Data analysis is a process of analyzing, summarizing, understanding and digesting a large amount of collected data through appropriate statistical analysis methods, aiming to maximize the development of data functions and play the role of data. Its core goal is to extract useful information and form conclusions from data to help companies or institutions make more informed decisions.
[0063] The present invention solves the problem of how to select the best compression method in the process of log screening, compression method, index structure and the like in the process of processing a relatively high number of device logs faced by an IoT device server.
[0064] Reference Figure 1 , shows a flow chart of the steps of a log data compression method provided by an embodiment of the present invention, and the method may specifically include the following steps:
[0065] Step 101, obtaining log data;
[0066] As the number of IoT devices increases, embodiments of the present invention can analyze large amounts of log data generated during daily use of devices. For example, log data generated by devices over a period of time, or log data generated by devices in a certain area, or log data of a certain device, or log data of a certain type of device can be obtained for analysis; different compression algorithms and indexes can be selected based on the analysis results.
[0067] In the smart home scenario, smart devices in the home can be managed through the APP, and the log data generated by each smart device can be obtained through the APP, such as the log data of the sweeping robot and the log data of the monitoring device.
[0068] Step 102, selecting key information according to the log data;
[0069] Key information can be characteristic values or important parameters of log data.
[0070] For example, the structure and content of the log data can be analyzed to identify characteristic values, such as timestamp, error level, specific event keywords, user ID, etc. It should be understood by those skilled in the art that the above key information is only an example of the present invention, and those skilled in the art may use other key information, which is not limited by the present invention. The key information is crucial for the subsequent compression algorithm selection and index setting.
[0071] For another example, important parameters of the log data may be analyzed, such as whether there are a large number of continuous or repeated data segments in the log data, whether the data type is text, number, binary, etc. Those skilled in the art should understand that the above important parameters are only examples of the present invention, and those skilled in the art may use other important parameters, and the present invention is not limited here.
[0072] In one example, the big data model can be used to pre-process the logs to be compressed, and filter out the characteristic values and important parameters of the log data. The AI model + log keywords can be used for training, which will be universal when analyzing user logs; a funnel analysis model can also be created to analyze user usage logs more accurately.
[0073] Step 103, determining a compression algorithm according to the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information;
[0074] A compression algorithm adapted to the log data can be selected based on the key information of the log data. Different compression algorithms can be selected to compress the log data according to different key information.
[0075] For example, for log data with high repetition or containing a large number of similar strings, dictionary encoding or Huffman encoding can be used; if the log data contains continuous numerical data, difference encoding or run-length encoding may be more appropriate; for mixed-type data with low repetition, you can consider using general compression algorithms such as LZ77, LZ77 or Zstandard.
[0076] Data compression: Use the selected compression algorithm to compress the log data. This step may include dividing the original log data into blocks and applying the compression algorithm to each block; according to the compression effect and performance requirements, adjust the parameters of the compression algorithm, such as the compression level, to achieve a balance between storage space saving and decompression speed.
[0077] Set up indexes: Based on the analyzed key information, select fields as index fields for compressed log data. These fields should be able to effectively support future query requirements. Select appropriate index types, such as B-Tree, hash index, full-text index, etc., based on the characteristics of log data and query patterns. Create indexes on compressed log data. In some cases, especially when the index itself takes up a lot of space, consider compressing the index itself.
[0078] Step 104: for the compressed log data, determine whether the compression algorithm needs to be optimized.
[0079] In one embodiment, whether the compression algorithm needs to be optimized can be determined in the following manner:
[0080] Performance evaluation: Test the query performance of compressed log data and indexes to ensure that the compression and indexing strategies do not significantly affect query efficiency.
[0081] Feedback adjustment: Adjust and optimize the compression algorithm, compression level, or indexing strategy based on test results and performance feedback from actual applications.
[0082] Step 105, when the compression algorithm needs to be optimized, new key information is selected, and the step of determining the compression algorithm according to the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information is returned.
[0083] When the performance of the existing compression algorithm no longer meets the requirements, or when the original algorithm is no longer the best choice due to changes in the characteristics of log data, it is necessary to re-evaluate and possibly replace the compression algorithm.
[0084] In one embodiment, the step of optimizing the compression algorithm includes reselecting key information, determining a new algorithm, compression processing, and setting an index.
[0085] In an embodiment of the present invention, a log data compression method is disclosed, which first obtains log data, selects key information according to the log data, and then determines a compression algorithm according to the key information, compresses the log data, and sets an index for the compressed log data according to the key information; for the compressed log data, it is determined whether the compression algorithm needs to be optimized; when the compression algorithm needs to be optimized, new key information is selected, and the compression algorithm is determined according to the key information, the log data is compressed using the compression algorithm, and the compressed log data is indexed according to the key information. During the use of a large number of IoT devices, the number of logs generated will be large. The present invention will select different compression methods and set indexes for different key information of these logs, and dynamically select compression methods for log data according to actual usage, thereby saving space while improving the retrieval efficiency of important logs.
[0086] Reference Figure 2 , shows a flowchart of another log data compression method provided by an embodiment of the present invention, the method may specifically include the following steps:
[0087] Step 201, obtaining log data;
[0088] In the embodiment of the present invention, a household smart device is taken as an example to exemplarily describe the principle of the present invention. However, this description is merely exemplary, and the scope of the present invention is not limited thereto. The principle of the present invention may also be applied to other IoT devices.
[0089] In the embodiment of the present invention, household smart devices and household Internet of Things devices such as sweepers, air conditioners, etc., assume that the main log data are divided into daily reporting logs, usage log records, fault information records, etc.
[0090] Step 202, selecting key information according to the log data;
[0091] In an embodiment of the present invention, an AI model can be used to analyze log data to determine key information; first, taking an IoT device as an example, the model is initialized, assuming that the initialized model is equipped with several compression algorithms, and in the initial state, assuming that the key information is time as the index compression as the optimal algorithm; the log storage volume may be small in the initial state, and time is more convenient for general log search and storage. After initialization, when subsequent logs are generated, the keywords in the logs are screened, such as setting fault codes, HTTP communication fault codes, etc., and these log data are matched through the model, and these logs containing key information are preliminarily screened out as preprocessed data.
[0092] There are many ways to pre-process the logs to be compressed through the big data model and filter the characteristic values and important parameters of the log data. For example, assuming that the current algorithm uses time as an index for block compression, if an important parameter shows regularity within a certain period of time, such as a certain type of fault often occurs in IoT devices within a certain period of time, whether it is related to the stability of the network, etc., then this regularity within the time can be used as a characteristic value, and the type of problem that occurs can be used as key information.
[0093] Step 203, determining a compression algorithm according to the key information; compressing the log data using the compression algorithm to obtain a plurality of log blocks; and setting indexes for the plurality of log blocks respectively according to the key information;
[0094] Take dictionary coding, prefix coding, lookup coding, and LZ77 / LZ78 lossless data compression algorithms as examples. First, analyze the preprocessed data to see if it meets the characteristic conditions. For example, dictionary coding is suitable for log data containing a large number of repeated values, prefix coding is suitable for data with uneven frequency distribution, and if the proportion of plain text in the log data exceeds a certain threshold, the LZ77 / LZ78 lossless data compression algorithm is selected. Select the compression algorithm according to the above rules.
[0095] If the amount of log data is large, it is processed in chunks. The purpose of chunking is that small chunks of data are easier to manage and process, especially in a distributed storage and processing environment; chunking allows for parallel compression and improves processing efficiency; and chunk size can be adjusted dynamically based on storage and performance requirements.
[0096] In one embodiment, the original log data is first cleaned to remove irrelevant information, such as blank lines, comments, garbled characters, etc., and the log format is unified for subsequent processing; then, according to the size or logical division of the log data, the original log is divided into multiple logical blocks or records for block processing. The divided log blocks are compressed using the selected compression algorithm. The compression operation is performed on each log block to generate a compressed data block. In the process, some metadata may need to be recorded or retained, such as compression algorithm information, original data size, compression ratio, etc., for subsequent decompression and management.
[0097] The key word for compression is index. Choose a suitable compression algorithm based on the characteristics of the log data (such as repetition, data type, data distribution, etc.). Possible options include but are not limited to:
[0098] Gzip: A widely used general-purpose compression algorithm suitable for most types of log data;
[0099] Snappy: Suitable for scenarios that require fast compression and decompression, sacrificing a certain compression rate in exchange for speed;
[0100] LZ4: provides very fast decompression speed and is suitable for applications that are sensitive to decompression time;
[0101] Zstandard (Zstd): provides a good balance between compression ratio and speed, and supports adjustable compression level;
[0102] For compressed log blocks, you can create indexes as needed to quickly locate and access specific log data. Indexes can be based on multiple dimensions such as timestamps, log levels, keywords, etc. The design of indexes should take into account the particularity of compressed blocks. It may be necessary to index to a specific location within a compressed block, or use a special structure to record the relationship between data before and after compression.
[0103] In an embodiment of the present invention, the method may further include:
[0104] If the size of the index of the multiple log blocks is greater than a preset threshold, compression processing is performed on the index of the multiple log blocks.
[0105] If the index of the compressed log block is too large, for example, the index is image information, the index needs to be compressed again, and the compression algorithm should be consistent with the log data compression algorithm for subsequent retrieval. The index compression algorithm can also be inconsistent with the log data compression algorithm. For example, for home cameras, the log data can be triggered by "someone appears" or "strangers call the police". When the proportion of such logs is too large, the compression algorithm may be compressed according to event classification. At this time, the index can be a text format of the event type or image information of a specific task. However, when searching with other indexes such as time, the performance requirements are higher.
[0106] Step 204, for the compressed log data, determine whether the compression algorithm needs to be optimized;
[0107] In different situations, the frequency of use by users at a specific time can determine the most suitable compression algorithm for that period of time. For example, log types can be classified and processed, such as text type logs, image type logs, fault type logs, etc. Assuming that the current compression algorithm used is time slices, and the log types are not classified and processed, it is difficult to quickly retrieve all the information associated with a specific person's information during the actual retrieval process, and the performance at this time will exceed a certain threshold; at this time, you can change to distinguish logs by person / importance, and then create a new index structure. Assuming that if a person is indexed, it may be divided into text information and image information. According to the usage of text information and image information, the corresponding compression algorithms are used respectively. If the frequency of use is high, the algorithm with good performance is used for user convenience, and if the frequency is low, the method with high compression rate is used to save space.
[0108] In one embodiment, step 204 may include the following sub-steps:
[0109] Sub-step S11, obtaining usage of searching the compressed log data;
[0110] Use a big data model to monitor the usage of index blocks and adjust the weights of certain index keywords. Specifically, monitoring the usage of certain indexes can be understood as a cache mechanism. In one embodiment, Huffman coding is used to sort the indexes, and the shortest query path is required for the frequently used indexes, that is, this model is required to count the usage of the created indexes, compress the frequently used indexes in the most efficient compression method, and compress other infrequently used indexes in the most space-saving method.
[0111] For example, assuming that the logs stored in the camera device may often search for specific information of people appearing in the camera according to different user needs, in this case, the person information index is put into a higher-performance compression method, assuming that the index type is image; for other less important log data, such as battery reminders and other events, the index uses the text type and is compressed using GZIP. This method has slow compression and decompression speeds but a high compression rate.
[0112] Sub-step S12, judging whether the compression algorithm needs to be optimized according to the usage of searching the compressed log data.
[0113] The threshold of the compression algorithm represents the performance during compression and subsequent log retrieval. Suppose two methods are used to compress certain data, and there is a big performance gap between the two compression algorithms during the actual compression or user retrieval process, that is, the parameters of a certain compression algorithm exceed a certain threshold, which can be assumed to be time. In this case, the algorithm may not be suitable for the compression of such data or may not be suitable for the user's usage habits. The keyword proportion represents the user's usage habits. Taking camera devices as an example, some users may be more accustomed to searching for specific portraits, such as cameras that monitor the elderly, while some users may prefer log data at a specific time, such as recording their lives.
[0114] In another embodiment, step 204 may include the following sub-steps:
[0115] Sub-step S21, obtaining the compression performance of compressing the log data using the compression algorithm;
[0116] In one embodiment, the ratio of the original data size to the compressed data size can be calculated. A high compression ratio means more storage space is saved. In addition to the total time consumption, the speed of compression and decompression is recorded separately to ensure that while the storage requirements are met, no delay is caused to applications with high real-time requirements. If possible, compare the performance of several different compression algorithms on the same data set, including compression efficiency, speed, and resource consumption.
[0117] Sub-step S22, judging whether the compression algorithm needs to be optimized according to the compression performance.
[0118] If the compression or decompression time is too long, affecting the system response speed or user experience, you may need to find a faster compression algorithm or adjust the compression level;
[0119] If the compression process significantly increases the CPU or memory burden, especially in a resource-limited environment, more efficient algorithms or optimization strategies need to be considered.
[0120] If the compression ratio is lower than expected, resulting in insignificant storage savings, explore other algorithms or adjust parameters to achieve better compression results.
[0121] Step 205, when the compression algorithm needs to be optimized, new key information is selected, and the step of using the compression algorithm to compress the log data to obtain multiple log blocks is returned; and indexing is respectively set for the multiple log blocks according to the key information.
[0122] After implementing optimizations, continue to monitor compression performance to ensure that improvements are effective, and adjust strategies as needed based on new data characteristics and system requirements.
[0123] In one embodiment, assuming the initial compression algorithm index is time:
[0124] Normal search: When the user searches for a person's information, they first search for a certain index block, and then they can find the content they want to search for after searching through all the index blocks.
[0125] After setting the optimization algorithm, the character information that users frequently search for will be matched through another compression algorithm and index method. Then, when searching for certain character information, the blocks of the character index method will be searched first, so that the required information can be retrieved more quickly.
[0126] In one embodiment, step 205 may include the following sub-steps:
[0127] Sub-step S31, obtaining keywords used by the user in the process of searching logs;
[0128] When users search for logs and query questions, use big data models to tag keywords. For example, you can obtain the latest log data samples to reflect the current data characteristics and workload; re-evaluate the key attributes of log data, including data type, repetition, data distribution, etc.
[0129] Sub-step S32, determining new key information according to the keywords used by the user in the process of searching the logs.
[0130] Based on the latest data analysis, identify the key information that has the greatest impact on compression efficiency. This may include: data repetitive patterns: such as common keywords, repeated values or time series data; data dynamics: data update frequency and pattern, which affects the stability and efficiency of the compression algorithm; query patterns: frequently searched or filtered fields, which may require special compression processing or avoid over-compression to maintain query performance. Increase the proportion of this keyword in the subsequent log compression and index creation process, and use this keyword as new key information.
[0131] In one embodiment, the sub-step S32 may further include:
[0132] Sub-step S321, obtaining the weight of the keyword used by the user in the process of searching the log;
[0133] The weight setting can be understood as a keyword classification. Assuming that the word cloud is used for statistics, when a keyword accounts for more than a certain proportion, this keyword can be used as an index to establish a new index structure, and then match whether a new compression method is needed. For example, for monitoring logs in some public places, there may be more callbacks for certain monitoring areas, so you can judge the trigger events judged by the camera, such as the appearance of a person, the alarm, etc. At this time, indexing according to the event can facilitate the analysis of the portrait in the subsequent investigation.
[0134] Sub-step S322, determining new key information according to the keywords used by the user in the process of searching the logs and the weights of the keywords.
[0135] In one embodiment, the following rules may be referred to: level classification, such as important ones such as failures, general user response, daily equipment status reporting, etc.; keyword indexing, such as classification according to subsequent usage, frequency, and time distance; chronological order, etc.
[0136] For example, for household smart devices, in theory, daily reporting logs should account for the largest proportion, but their importance should be the lowest, so the weight can be set slightly lower. The most important thing should be the fault information record, which can facilitate maintenance personnel to make a judgment on the equipment status, etc. However, the time period of equipment failure generally accounts for a small proportion. In this case, time or log level can be selected as an index for compression. In an embodiment of the present invention, a log data compression algorithm method is introduced. First, log data is obtained, key information is selected according to the log data, and then a compression algorithm is determined according to the key information, the log data is compressed, and an index is set for the compressed log data according to the key information; for the compressed log data, it is determined whether the compression algorithm needs to be optimized; when the compression algorithm needs to be optimized, new key information is selected, and the compression algorithm is returned to determine the compression algorithm according to the key information, the log data is compressed using the compression algorithm, and the compressed log data is indexed according to the key information. In the use process of large-scale Internet of Things devices, the order of magnitude of the logs generated will be large. The present invention will select different compression methods and set indexes for different key information of these logs, and dynamically select compression methods for log data according to actual usage, so as to save space while improving the retrieval efficiency of important logs.
[0137] Reference Figure 3This is a flow chart of compressing log data in an embodiment of the present invention. In order to enable those skilled in the art to better understand the embodiment of the present invention, the embodiment of the present invention is described below by using an example:
[0138] First, log information (also called log data) is collected, pre-processed, and the features of this part of the log information are screened out. Different compression algorithms are selected based on the features of the log information; an index structure is created based on the log algorithm (also called setting the index), and the index is compressed in blocks; actual usage is subsequently monitored, and the usage of log information containing certain specific situations and specific fields is screened out through a big data algorithm, and the weight of the characteristic value (also called key information) is modified. If the weight exceeds a certain threshold, a better compression algorithm is selected based on the new key information; a new index structure is created based on the new compression algorithm, and the new index is compressed in new blocks. Through continuous iterative optimization, different compression algorithms can be used for different log information of a certain event or a certain log cluster, and different index block sizes and compression algorithms can be created to build a log compression framework with the smallest log space and the fastest retrieval speed.
[0139] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0140] Reference Figure 4 , shows a structural block diagram of a log data compression device provided by an embodiment of the present invention, which may specifically include the following modules:
[0141] Data acquisition module 401, used to acquire log data;
[0142] An information selection module 402, used to select key information according to the log data;
[0143] The data compression and index setting module 403 is used to determine a compression algorithm according to the key information, compress the log data using the compression algorithm, and set an index for the compressed log data according to the key information;
[0144] The first judgment module 404 is used to judge whether the compression algorithm needs to be optimized for the compressed log data;
[0145] Step return module 405 is used to select new key information when the compression algorithm needs to be optimized, and return to the step of determining the compression algorithm according to the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information.
[0146] The first judgment module includes:
[0147] A usage status acquisition module, used to acquire usage status of the compressed log data;
[0148] The first judgment submodule is used to judge whether the compression algorithm needs to be optimized according to the usage of searching the compressed log data.
[0149] The first judgment module also includes:
[0150] A compression performance acquisition module, used to acquire the compression performance (consumed time, consumed resources) of compressing the log data using the compression algorithm;
[0151] The second judgment submodule is used to judge whether the compression algorithm needs to be optimized according to the compression performance.
[0152] The information selection module includes:
[0153] The keyword acquisition module is used to obtain the keywords used by users in the process of searching logs;
[0154] The first key information determination module is used to determine new key information according to the keywords used by the user in the process of searching the logs.
[0155] The key information determination module includes:
[0156] The weight acquisition module is used to obtain the weights of keywords used by users in the process of searching logs;
[0157] The second key information determination module is used to determine new key information according to the keywords used by the user in the process of searching the logs and the weights of the keywords.
[0158] The data compression and index setting module includes:
[0159] A first data compression submodule, configured to compress the log data using the compression algorithm to obtain a plurality of log blocks;
[0160] The index setting submodule is used to set indexes for the multiple log blocks respectively according to the key information.
[0161] The data compression module and the index setting module also include:
[0162] The second data compression submodule is configured to compress the indexes of the plurality of log blocks if the size of the indexes of the plurality of log blocks is greater than a preset threshold.
[0163] The present invention introduces a log data compression method, which first obtains log data, selects key information according to the log data, determines a compression algorithm according to the key information, compresses the log data, and sets an index for the compressed log data according to the key information; for the compressed log data, determines whether the compression algorithm needs to be optimized; when the compression algorithm needs to be optimized, selects new key information, and returns to the step of determining the compression algorithm according to the key information, compresses the log data using the compression algorithm, and sets an index for the compressed log data according to the key information. During the use of large-scale Internet of Things devices, the order of magnitude of the logs generated is large. The present invention selects different compression methods and creates index formats according to the different characteristics of these logs, divides the log data according to actual usage, and then dynamically selects a compression method to save space while improving the retrieval efficiency of important logs.
[0164] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0165] An embodiment of the present invention further provides an electronic device, including:
[0166] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the above-mentioned log data compression method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0167] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned log data compression method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0168] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0169] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0170] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0171] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0173] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0174] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0175] The log data compression method and the log data compression device provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A log data compression method, characterized in that: include: Get log data; Select key information according to the log data; Determine a compression algorithm according to the key information, compress the log data using the compression algorithm, and set an index for the compressed log data according to the key information; For the compressed log data, determine whether the compression algorithm needs to be optimized; When the compression algorithm needs to be optimized, new key information is selected, and the step of determining the compression algorithm according to the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information is returned.
2. The method according to claim 1, characterized in that The step of determining whether the compression algorithm needs to be optimized for the compressed log data includes: Obtaining usage of the compressed log data for retrieval; Whether the compression algorithm needs to be optimized is determined based on the usage of searching the compressed log data.
3. The method according to claim 1, characterized in that: The step of determining whether the compression algorithm needs to be optimized for the compressed log data includes: Obtaining compression performance of compressing the log data using the compression algorithm; According to the compression performance, it is determined whether the compression algorithm needs to be optimized.
4. The method according to claim 1, characterized in that The new key information obtained includes: Obtain the keywords used by users in searching logs; New key information is determined according to the keywords used by the user in the process of searching the logs.
5. The method according to claim 4, characterized in that Determining new key information according to the keywords used by the user in the process of searching the logs includes: Obtain the weight of the keywords used by the user in the process of searching logs; New key information is determined based on the keywords used by the user in the process of searching the logs and the weights of the keywords.
6. The method according to claim 1, characterized in that The compressing the log data using the compression algorithm, and setting an index for the compressed log data according to the key information, includes: The log data is compressed using the compression algorithm to obtain a plurality of log blocks; According to the key information, indexes are set for the multiple log blocks respectively.
7. The method according to claim 6, characterized in that The method of compressing the log data using the compression algorithm and setting an index for the compressed log data according to the key information further includes: If the size of the index of the multiple log blocks is greater than a preset threshold, compression processing is performed on the index of the multiple log blocks.
8. A log data compression device, characterized in that: include: A data acquisition module is used to acquire log data; An information selection module, used to select key information according to the log data; A data compression module and an index setting module, used to determine a compression algorithm according to the key information, compress the log data using the compression algorithm, and set an index for the compressed log data according to the key information; A judgment module is used to judge whether the compression algorithm needs to be optimized for the compressed log data; The step return module is used to select new key information when the compression algorithm needs to be optimized, and return the step of determining the compression algorithm based on the key information, compressing the log data using the compression algorithm, and setting an index for the compressed log data based on the key information.
9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the log data compression method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log data compression method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Log information processing method and device
CN106776954A
An efficient log compression and indexing method
CN109101504A
Intelligent management system based on big data
CN116681400A
Uplink data transmission method, system and device and storage medium
CN118175212A
Cited By
Chip signaling analysis and storage method, computer device and readable storage medium
CN119377180A
