Log data compression and retrieval method and electronic equipment
By combining a trie and a hash table, the problem of flexibility and efficiency in log compression and retrieval is solved, achieving efficient log data compression and fast retrieval, which is suitable for log management in embedded devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-03
AI Technical Summary
Existing log compression technologies suffer from poor flexibility, high algorithm complexity, and the separation of compression and indexing, leading to complex data management and high maintenance costs, which affect log retrieval efficiency.
By employing a data structure combining a trie and a hash table, and through vocabulary partitioning, mapping, and compression mechanisms, we achieve flexible compression and efficient retrieval of log data.
It achieves a compression rate of nearly 50% for log data, while improving retrieval efficiency, reducing memory usage, and supporting fast queries.
Smart Images
Figure CN121786014A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of log processing technology, and in particular to a log data compression and retrieval method and electronic device. Background Technology
[0002] There are many existing log compression technologies, including offline dictionary encoding and dynamic LZ series encoding. Offline dictionary encoding requires pre-defined recognition templates or character libraries for real-time log data compression. This approach requires prior familiarity with and analysis of log content to develop templates, which, once confirmed, cannot be changed, thus lacking flexible control over module granularity.
[0003] LZ series encoding and its optimized encoding scheme can dynamically identify modules, but the algorithm is complex. In extreme cases, the data may not have any duplicate content, which may instead cause the data to bloat.
[0004] Compression algorithms with high compression ratios require additional compression during retrieval, slowing down query results; supporting fast retrieval often requires extensive indexing, consuming significant memory. Separating compression and indexing complicates data management and increases maintenance costs. Summary of the Invention
[0005] The purpose of this application is to provide a method and device for compressing and retrieving log data, which addresses the problems of complex data management and high maintenance costs caused by the inflexibility and complexity of compression algorithms and the separation of compression and indexing.
[0006] In a first aspect, embodiments of this application provide a method for compressing log data, the method comprising: Obtain one log data entry to be compressed from the log file, and perform word segmentation on the log data; The vocabulary is traversed, and the current vocabulary is searched in the trie. The trie is used to establish a mapping from vocabulary to unique identifier ID. If the current word is found, obtain the target ID of the current word mapped in the trie; The corresponding linked list is found based on the target hash value of the target ID. The linked list is used to establish the mapping from ID to vocabulary and to store the storage information of a log data record in the node. If a node exists in the linked list that records the storage information of the log data, compression is triggered; otherwise, compression is triggered after adding a node to the linked list that records the storage information of the log data to be compressed. When compression is triggered, the target ID is used to replace the current word in the log data; Iterate through the next word until the end of the iteration, and update the corresponding storage information after obtaining the compressed log data.
[0007] In some possible embodiments, the method further includes: If the current word is not found, a mapping from the current word to the target ID is created in the trie. If no corresponding linked list is found for the target hash value, a linked list for the target hash value is created. A mapping from the target ID to the current word is established by using the current word as the head of the linked list. The storage information of the log data is recorded in the node under the head of the linked list, and compression is triggered.
[0008] In some possible embodiments, the format of the log data includes a multi-level index header and log content, and also includes: Based on the multiple fixed words corresponding to the multi-level index header, establish a mapping relationship between each fixed word and its ID in the dictionary tree; A linked list is created based on the hash value of the ID mapped to the fixed vocabulary, and the mapping from the ID to the fixed vocabulary is established by using the fixed vocabulary as the head of the linked list.
[0009] In some possible embodiments, the multi-level index header includes any of the following index headers: The log data generated by the project file includes timestamps, error levels, module names, process branches, line numbers, and function names.
[0010] In some possible embodiments, the method further includes: Count the number of times the same word appears in the trie; When an alarm detection is triggered, the statistical frequency of the log data is determined based on the frequency and corresponding weight of the words in the trie. If the statistical frequency is lower than the alarm threshold, an alarm will be triggered according to the configured alarm method; In this process, the weights of the words are set according to their importance.
[0011] In some possible embodiments, the storage information of the log data includes the file storage location, cache storage location, and data length of the log data.
[0012] In some possible embodiments, updating the corresponding storage information after obtaining the compressed log data includes: When compression is triggered, a pointer to the content of the current node is recorded, where the current node is the word position traversed in the log data; After traversing the records, determine the length of the compressed log data based on the pointer length of the records, and update the corresponding data length in the linked list.
[0013] In some possible embodiments, the log data is stored in the following manner: The log data is retrieved and then cached in the cache area; Based on the input command, the compressed log data in the cache is solidified into the log file data area using an active method or a polling method according to a set period. Based on the set deletion criteria, clean up the compressed log data in the cache area that meets the deletion criteria.
[0014] In some possible embodiments, creating the mapping from the current word to the target ID in the trie includes: Based on the characters / numbers contained in the current vocabulary, map each character / number to a node; Establish a path from the root node to the leaf node based on the order of the characters / numbers in the current vocabulary; The leaf nodes record the target ID of the current word mapping and the starting address of the log data of the current word in the log file.
[0015] Secondly, embodiments of this application provide a method for retrieving log data, the method comprising: Retrieve search terms; When the search term is found in the trie, the target ID that the search term maps to in the trie is determined, and the target hash value of the target ID is determined. Based on the target hash value, find the corresponding target linked list and obtain the storage information of the log data recorded by all nodes in the target linked list; Based on the storage information of the log data recorded by any node, read the target decompressed log data; Traverse the target decompressed log data. For any current ID encountered, decompress the current ID into the corresponding word based on the ID-to-word mapping established by all linked lists.
[0016] In some possible embodiments, the storage information of the log data includes the file storage location, cache storage location, and data length of the log data.
[0017] In some possible embodiments, reading the target decompression log data includes: Based on the cache storage location of the log data recorded by any of the nodes, search for a log data in the cache area; If the log data found is the same length as the log data recorded by any of the nodes, then the found log data is read from the cache and used as the target to decompress the log data. Otherwise, based on the file location of the log data recorded by any of the nodes, read one log data from the log file data area as the target decompressed log data.
[0018] Thirdly, embodiments of this application provide a log data compression apparatus, the apparatus comprising: The vocabulary segmentation module is used to obtain a log data entry to be compressed from the log file and perform vocabulary segmentation on the log data. The vocabulary lookup module is used to traverse the vocabulary and search for the current vocabulary in the trie. The trie is used to establish a mapping between vocabulary and unique identifier ID. The ID determination module is used to obtain the target ID of the current word in the trie if the current word is found. The linked list lookup module is used to look up the corresponding linked list based on the target hash value of the target ID. The linked list is used to establish the mapping from ID to words and to store information of a log data record in the node. The compression trigger module is used to trigger compression if there is a node in the linked list that records the storage information of the log data; otherwise, compression is triggered after adding a node to the linked list that records the storage information of the log data to be compressed. The data compression module is used to replace the current word in the log data with the target ID when compression is triggered; The storage update module is used to traverse the next word until the end of the traversal, and then update the corresponding storage information after obtaining the compressed log data.
[0019] Fourthly, embodiments of this application provide a log data retrieval device, the device comprising: The keyword retrieval module is used to retrieve search terms; The hash value determination module is used to determine the target ID that the search term is mapped to in the trie when the search term is found in the trie, and to determine the target hash value of the target ID; The storage information determination module is used to find the corresponding target linked list based on the target hash value and obtain the storage information of the log data recorded by all nodes of the target linked list; The compressed data reading module is used to read the target decompressed log data based on the storage information of the log data recorded by any node; The data decompression module is used to traverse the target decompressed log data. For any current ID encountered during traversal, the current ID is decompressed into the corresponding word based on the ID-to-word mapping established by all linked lists.
[0020] Fifthly, embodiments of this application provide an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect above, or to perform the method provided in the second aspect above.
[0021] In a sixth aspect, embodiments of this application provide a computer storage medium storing a computer program for causing a computer to execute the method provided in the first aspect or the method provided in the second aspect.
[0022] The log data compression and retrieval method and electronic device provided in this application embodiment can balance the contradiction between log compression performance and retrieval efficiency. Combining the characteristics of packet communication device logs, it uses trie and linked lists in hash tables as the main data structures, which not only ensures a compression rate of nearly 50%, but also improves retrieval efficiency. Both the compression process and the retrieval process rely on trie and hash tables. In the scenario of packet communication logs, these two data structures have fast query speed and small memory usage.
[0023] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 A flowchart of a log data compression method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the log data compression process provided in an embodiment of this application. Figure 3 A flowchart illustrating the log data retrieval method provided in this application embodiment; Figure 4 Detailed flowchart of the log data retrieval method provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the original log data output format as an example of an embodiment of this application; Figure 6 This is a schematic diagram illustrating the compressed log data output format as an example of an embodiment of this application; Figure 7 A schematic diagram of the log data compression device provided in the embodiments of this application; Figure 8 A schematic diagram of the log data retrieval device provided in the embodiments of this application; Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0026] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.
[0027] Embedded logging systems are indispensable in development, debugging, and operational monitoring phases. Furthermore, embedded devices have limited memory resources, with most resources allocated to business logic implementation and only a small portion for logging implementation and storage. Simultaneously, embedded devices also have valuable CPU resources, which must be used for business logic implementation. Therefore, logging system implementations should have simple business logic and use as few CPU resources as possible.
[0028] In the field of communications, once a device experiences a field failure, the impact is significant. It is necessary to collect as much effective information as possible as quickly as possible, support rapid log retrieval, and provide effective support for maintenance personnel and developers.
[0029] This application provides a log data compression and retrieval method, apparatus, and device. It uses a trie to extract words, statistical frequencies, and record offset representations of IDs from the log data stream, and uses a hash algorithm to maintain the conversion relationship between IDs and words, innovatively integrating log compression and efficient retrieval together.
[0030] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.
[0031] like Figure 1 As shown in the figure, this application provides a method for compressing log data, the method including: Step 101: Obtain one log data entry to be compressed from the log file and perform word segmentation on the log data; The log file contains multiple log data entries generated based on the execution status of the project file. The log data is read from the log file one by one and compressed. During compression, the log data entry is first divided into multiple words according to the word segmentation rules.
[0032] Step 102: Traverse the vocabulary and search for the current vocabulary in the trie. The trie is used to establish a mapping from vocabulary to unique identifier ID. A trie uses a tree structure to store words. Each root node and leaf node can identify a word. The leaf node is used to record the word's ID and related information. The node above the leaf node is the last character or number in the word. When searching in the trie, the search starts from the root node. If the word above the leaf node is found to be the same as the current word, it means that the word exists in the trie, and the mapped ID is obtained from the leaf node.
[0033] Step 103: If the current word is found, obtain the target ID of the current word mapped in the trie; If the current word is found in the trie, the target ID mapped to the current word in the trie is determined based on the leaf node of the current word in the trie.
[0034] Step 104: Find the corresponding linked list based on the target hash value of the target ID. The linked list is used to establish the mapping from ID to vocabulary and to store the storage information of a log data entry recorded in the node. A hash value can be obtained by using a hash algorithm based on the target ID. Each hash value corresponds to a linked list in the hash table. This linked list is used to establish a mapping from the ID obtained from the hash value to the vocabulary. The head of the linked list records the vocabulary, and the head includes at least one node. Each node records the storage information of a log data entry, which is the location information of the log data.
[0035] In some possible embodiments, the storage information of the log data includes the file storage location Cn, the cache storage location Hn, and the data length Ln. Cn refers to the location of the log data in the log file, and the cache storage location Hn is the location of the log data in the cache area.
[0036] Step 105: If there is a node in the linked list that records the storage information of the log data to be compressed, then compression is triggered; otherwise, after adding a node to the linked list that records the storage information of the log data to be compressed, compression is triggered. As mentioned earlier, a hash table comprises multiple linked lists. The hash value can be used to determine the corresponding linked list. Each linked list includes a head and the nodes under that head. Therefore, after finding the linked list using the target hash value, the head and nodes under that head can be obtained. The head stores the corresponding word, and each node under the head stores the storage information for a log entry containing that word. Based on the nodes under the linked list containing the target hash value, the set of all log data containing the current word recorded during the data compression process can be determined.
[0037] Step 106: When compression is triggered, replace the current word in the log data to be compressed with the target ID; Based on the ID-to-word mapping relationship established by the searched linked list, the target ID of the current word is used to replace the current word in the log data to be compressed, thereby achieving compression of the current word.
[0038] Step 107: Traverse the next word until the traversal ends, and update the corresponding storage information after obtaining the compressed log data.
[0039] At the end of the traversal, all words in the log data are compressed. Since data compression changes the data length, after obtaining the compressed log data, the storage information of the log data is updated in the corresponding node of the linked list. In this embodiment, words are compressed one by one through the traversal process, specifically by replacing words with IDs. When the entire log data has been traversed, the original log data is replaced with IDs to obtain compressed log data. Since the storage information of this compressed log data, such as the starting point and length, and the conversion relationship from ID to word are all recorded in the linked list, it can support fast subsequent lookup.
[0040] This application uses a trie to maintain the mapping from words to IDs, and uses linked lists in a hash table to maintain the mapping from IDs to words. The nodes of these linked lists store the storage information of the compressed log data, effectively improving retrieval efficiency. Specifically, after each word in a log entry is converted into an ID, the corresponding linked list node stores the file storage location Cn, cache storage location Hn, and data length Ln of the compressed log data. This allows for quick location of the compressed log data through the linked list, regardless of the word being retrieved. The compressed log data can then be quickly obtained and decompressed using the mapping relationship established by the linked list, unlike traditional methods that first decompress all logs and then perform the search process.
[0041] In some possible embodiments, the method further includes at least one of the following steps: If the current word is not found, a mapping from the current word to the target ID is created in the trie, thereby adding the current word to the trie; If no corresponding linked list is found for the target hash value, a linked list for the target hash value is created. The target ID is mapped to the current word by using the current word as the head of the linked list. The storage information of the log data to be compressed is recorded in the node under the head of the linked list, and compression is triggered.
[0042] If the current word is not found and is added to the trie, since no corresponding linked list has been created, the linked list will not be found in the hash table based on the target hash value. Therefore, it is necessary to determine the target hash value based on the target ID of the current word, create the corresponding linked list based on the target hash value, record the current word at the head of the linked list, and add a node under the head of the linked list to record the storage information of the log data to be compressed.
[0043] In some possible embodiments, creating a mapping from the current word to the target ID in the trie includes: mapping each character / number to a node based on the characters / numbers contained in the current word; establishing a path from the root node to the leaf node based on the order of the characters / numbers in the current word; using the leaf node to record the target ID mapped to the current word, and the starting address of the log data containing the current word in the log file, wherein the leaf node is the next node after the last character / number.
[0044] The process of establishing the path from the root node to the leaf node described above can start from the first character of the current word, matching from the root node in the trie. If a matching node is found, the next character / number of the current word is matched with the next node of the matching node. This continues until a non-matching node is found. Then, a node with the corresponding character / number is added at the next level of the last matching node. This process continues until the leaf node is inserted, and a unique ID value is assigned to the current word in the leaf node.
[0045] In some possible embodiments, when the storage information of the log data includes the file storage location Cn, cache storage location Hn, and data length Ln, the corresponding storage information is updated in the following way after the log data compression is completed: When compression is triggered, a pointer to the content of the current node is recorded. The current node is the word position traversed in the log data. The data at the corresponding position in the log can be obtained through the word position, such as the nth character / number in the log data. The content at the corresponding position can be accessed through the pointer. After the traversal is completed, the length of the compressed log data is determined according to the recorded pointer length, and the corresponding data length in the linked list is updated. Since the content at the pointer position is replaced by the corresponding ID during the compression process, the length of the compressed log data can be obtained by reversing the pointer length after the traversal is completed.
[0046] In some possible embodiments, the format of the log data includes a multi-level index header and log content, and also includes initialization performed by the software's initialization module as follows: establishing a mapping relationship between each fixed term and ID in a trie based on multiple fixed terms corresponding to the multi-level index header; creating a linked list based on the hash value of the ID mapped to the fixed term, and establishing the mapping from ID to fixed term by using the fixed term as the head of the linked list. The multi-level index header includes any of the following: timestamps, error levels, module names, process branches, line numbers, and function names of the log data generated by the project file.
[0047] In practical implementation, a unified log data format is established based on the characteristics of the transmission and communication equipment: a "six-level index header + actual log content". The index header content includes: [timestamp][error level][module name][process branch][line number][function name], with data following "#:", and words separated by punctuation marks such as spaces, commas, periods, and exclamation marks. During system initialization, the software's initialization module initializes the data, distributes the above configuration information, and loads the dictionary storage area of the log file into the trie.
[0048] Each log entry output by the system, as configured above, is recorded in the form of "six-level index header + actual log content". Based on the aforementioned multi-level index header, the initialization module initializes the trie, adding the fixed words corresponding to the defined index headers and marking them as fixed words. Simultaneously, a linked list is created in the hash table based on the hash value of the fixed word's ID to maintain the mapping from ID to fixed words for fast lookup.
[0049] In some possible embodiments, this application also uses a trie to record the frequency of occurrence of the same word. During the compression process of the log data to be compressed, based on the result of searching for the current word in the trie, the frequency of occurrence of the same word in the trie is counted, and the statistical frequency of the log data is determined based on the frequency of each word. Specifically, this can be achieved using the compression module of software: extracting the words and parameter values (such as the starting address of the log data containing the word in the log file) of each log data from the log data stream according to the trie, and recording the frequency of occurrence of each word.
[0050] During the initialization phase, the initialization module allocates 128KB of memory as a cache area for caching user log data.
[0051] In some possible embodiments, based on the frequency of occurrence of words recorded in the trie, the method of this application embodiment further includes using a software alarm module to implement: when an alarm detection is triggered, the statistical frequency of the log data is determined based on the frequency and corresponding weight of the words in the trie; if the statistical frequency is lower than the alarm threshold, an alarm is triggered according to the configured alarm method; wherein, the weight corresponding to the words is set according to the importance of different words, and the higher the importance of the words, the greater the corresponding weight.
[0052] The initialization module can be used to configure relevant parameters of the alarm module, specifically the alarm threshold, alarm method, and alarm cycle. This allows the alarm module to record every log entry with a statistical frequency below the alarm threshold and, based on the alarm cycle, use the corresponding alarm method to report alarm information to the user, alerting them to the presence of log data with a very low statistical frequency, indicating a potential anomaly and prompting them to investigate.
[0053] In some possible implementations, corresponding thresholds can be set for words. Developers or operations personnel can proactively traverse the trie to check the occurrence count of individual words, obtain low-frequency words whose occurrence count is lower than the threshold, and combine them with business logic to analyze whether there are any anomalies in the existing processes, thereby improving the reliability of the system.
[0054] This application embodiment utilizes a software storage module to store log data. The storage format is divided into two main parts: the first part is a dictionary storage area, which records the words identified on the trie and their IDs, as well as the location of the log data containing those words in the log file; the second part is a compressed data area, which stores the compressed log data. The storage module mainly performs the following: traversing the trie, obtaining all extracted words, and storing them as a dictionary in the dictionary storage area at the beginning of the log file. The log file data area follows the dictionary storage area.
[0055] In some possible embodiments, log data is stored as follows: log data to be compressed is obtained from the log file and cached in the cache area; compressed log data in the cache area is solidified into the log file data area in an active manner according to the input command or in a polling manner according to the set period; compressed log data in the cache area that meets the deletion conditions is cleaned up according to the set deletion conditions.
[0056] The aforementioned log file data area is a non-volatile storage device, a buffer for data persistence awaiting refresh. To enable more flexible data persistence, this embodiment employs a "periodic polling + active" approach to persist logs from memory to the non-volatile storage device. The specific log data persistence process implementation logic is as follows: 1) Initialize the disk flushing semaphore semA, release the semaphore P (semA), and create a background persistent thread; 2) Use a background thread to wait for the disk flushing signal, i.e., P(semA); 3) Obtain the handle of the buffer to be refreshed, use the handle to retrieve the log data in the cache, and asynchronously write it to the physical storage eMMC; 4) Reset the buffer position, occupying semaphores semA and P(semA); 5) Repeat step 2) until the log file is fully compressed.
[0057] Specifically, a timer can be created to periodically release the semaphore semA to activate step 2 of the log solidification process, thereby achieving the purpose of periodically refreshing log data to non-volatile storage devices and realizing periodic polling. Alternatively, the user can trigger V(semA) via the command line, and V(semA) will activate step 2 of the log solidification process.
[0058] like Figure 2 The diagram illustrates the compression process of a single log entry using the log data compression method provided in this embodiment, which mainly includes: Step 201: Perform word segmentation on a single log data entry. A pointer can be used to point to the position of the current word. For specific word segmentation methods, please refer to the description in the above embodiment. Spaces or punctuation marks can be used for segmentation. Step 202: Search for the corresponding word in the trie one by one, and determine whether the current word is found in the trie. If the current word is found, proceed to step 204; otherwise, proceed to step 203. Step 203: If the current word is not found, add the nodes corresponding to the characters / numbers in the word to the trie in turn, and assign a unique ID value n at the leaf node to establish a mapping from the current word to the ID in the trie. The initial word count is 0, and then proceed to step 4. Step 204: Increment the word count by 1, and record the file position Cn where the current log data is to be written, the cache position Hn where the log data is located, and the data length Ln; Step 205: Obtain the hash value and find the linked list through the unique ID value n of the word. Compare the current word, Cn, Hn, Ln obtained in step 204 with the words, Cn, Hn, and Ln in the linked list to determine whether Cn, Hn, and Ln are completely the same. If they are completely the same, proceed to step 207. If they are different, proceed to step 206. Obtaining the hash value and finding the linked list involves two cases. One case is that the hash table contains a linked list with that hash value, corresponding to the case where the current word is found in the trie. In this case, the word, Cn, Hn, and Ln from the linked list with the hash value are directly retrieved for comparison. The other case is that the hash table does not contain a linked list with that hash value, corresponding to the case where the current word is not found in the trie. In this case, a linked list corresponding to the hash value is created, with the current word as the head of the linked list. The nodes under the head of the linked list record the Cn, Hn, and Ln of this log data, and the word, Cn, Hn, and Ln from this linked list are retrieved for comparison. Step 206: The current word, Cn, Hn, Ln is not exactly the same as the word, Cn, Hn, Ln in the linked list. This means that the storage information of the log data containing the current word is not recorded in the linked list. Therefore, add a node to the linked list that records the above Cn, Hn, Ln, initialize the frequency value of the hash node to 0, and execute step 207. Step 207: Increment the frequency value of the hash node by 1, record the pointer Pn of the current node's content, and replace the current word with the ID value n; Step 208: Determine if the current word is the last word. If yes, proceed to step 209; otherwise, move the current pointer Pn to the next word and repeat the subsequent steps from step 202. Step 209: Confirm that the compression of the log data is complete, and update the length of the compressed log data. Specifically, the length Ln of the log data recorded by the node can be obtained by reversing the length of the Pn pointer through the saved Pn pointer.
[0059] Based on the log data compression method provided in the above embodiments, after converting the search terms into IDs and obtaining the head node of the linked list by looking up the hash table, all matching results are contained in the linked list maintained by that head node. The decompressed log data can be obtained by sequentially retrieving the matched compressed log data and parsing each record. The log data retrieval method provided in this application embodiment, such as... Figure 3 As shown, it includes: Step 301: Obtain the search terms; Step 302: When the search term is found in the trie, determine the target ID that the search term maps to in the trie, and determine the target hash value of the target ID; Step 303: Find the corresponding target linked list based on the target hash value, and obtain the storage information of the log data recorded by all nodes of the target linked list; Step 304: Read the target decompressed log data based on the storage information of the log data recorded by any node; Step 305: Traverse the target decompressed log data. For any current ID encountered during traversal, decompress the current ID into the corresponding word based on the ID-to-word mapping established by all linked lists.
[0060] This application embodiment utilizes the software's retrieval module to implement the retrieval process based on the above method. It does not require decompressing the file first and then searching for the corresponding log data; instead, it first obtains the ID based on the search terms, then obtains the head of the linked list of matching logs through a hash table, traverses the nodes under the head of the linked list to find all log data matching that ID, and finds each compressed log data that meets the conditions; through the starting point and length, and through the mapping relationship between ID and terms recorded in the hash table, it translates the IDs in all log data, and presents the searched and decompressed results to the user, thus completing the log data retrieval.
[0061] In some possible embodiments, the storage information of the log data mentioned above includes the file storage location Cn, the cache storage location Hn, and the data length Ln.
[0062] In some possible embodiments, reading the target decompressed log data includes: searching for a log data entry in the cache area based on the cache storage location of the log data recorded by any node; if the found log data has the same data length as the log data recorded by any node, then reading the found log data from the cache area and using it as the target decompressed log data; otherwise, reading a log data entry from the log file data area based on the file location of the log data recorded by any node and using it as the target decompressed log data. Through the above process of locating compressed log data, the compressed log data is first searched in the cache. If the corresponding compressed log data is found, there is no need to read or write from the log file data area, thus improving retrieval efficiency when there is residual data in the cache.
[0063] like Figure 4 The diagram shows the process of a user entering search terms and then starting a search, which includes: Step 401: Obtain the search terms; Step 402: Search for the search term in the trie and determine whether there is a word in the trie that matches the search term. If there is, proceed to step 404; otherwise, proceed to step 403. Step 403: If the search term is not found in the dictionary tree, the search process will be exited directly, and the user will be prompted that no relevant log data was found. Step 404: If the search term is found in the trie, obtain the ID value n corresponding to the term, use the corresponding hash value to look up the hash table, obtain the corresponding linked list and find the head of the linked list; Step 405: Traverse the nodes under the head of the linked list, obtain the Cn, Hn and Ln records of the traversed nodes, and use Hn to find a log data in the cache area; Step 406: Compare the length of the retrieved log data with Ln to determine if they are the same. If they are the same, proceed to step 408; otherwise, proceed to step 407. Step 407: If they are different, it means that the compressed log data is not in the cache area and has been persisted to the log file data area. Use Cn to retrieve a compressed log data from the log file data area and execute step 409. Step 408: If they are the same, then the compressed log data is still in the cache. Retrieve the compressed log data from the cache and proceed to step 409. Step 409: Iterate through the obtained compressed log data, maintain the mapping relationship between ID and words according to the hash table, and translate the ID in the compressed log data into the original words in turn to complete the decompression process of a log data. Step 410: Determine whether the node traversed is the last node of the above linked list. If yes, proceed to step 411; otherwise, return to step 405. Step 411: Summarize the decompressed log data and output it to the user to complete the log data retrieval.
[0064] This application embodiment balances the contradiction between log compression performance and retrieval efficiency. Combining the characteristics of packet communication device logs, it uses trie and hash table as the main data structures, which not only ensures a compression rate of nearly 50% but also improves retrieval efficiency. The compression and retrieval processes in this application embodiment both rely on trie and hash table. In the context of packet communication logs, these two data structures offer fast query speeds and low memory usage.
[0065] This application uses a trie to maintain the mapping from words to IDs, and a hash table to maintain the mapping from IDs to strings. The nodes of the hash table store the location of the compressed log data in the log file and its location in the cache, effectively improving retrieval efficiency in the following two aspects: On the one hand, after each word in the log data is converted into an ID, the cache address Hn of the compressed log, the address Cn in the file, and the length Ln are stored in the node of the linked list. In this way, no matter which word is searched, the location of the compressed log data can be quickly located, thereby quickly obtaining the compressed data, unlike the traditional solution that first decompresses all logs and then searches. When locating compressed log data, it first searches in the cache. If the compressed log data is found, there is no need to read and write from the file. With residual data in the cache, the retrieval efficiency is improved.
[0066] On the other hand, after converting the search terms into IDs and finding the head node of the linked list by looking up the hash table, all the matching results are contained in the linked list maintained by that head node. We only need to obtain the matching compressed log data one by one and parse them.
[0067] Based on the low probability of abnormal processes occurring in packet communication devices, this application innovatively utilizes the frequency of word occurrences to calculate the frequency value of a certain log data. When the frequency of a certain log data or the number of occurrences of a certain word is lower than the corresponding set threshold, an alarm is issued to the user, indicating that the system may have entered an abnormal process.
[0068] The following examples illustrate the beneficial effects of the log data compression and retrieval methods provided in this application.
[0069] The log data compression method provided in this application can effectively compress data. For example... Figure 5 As shown, the data is output in the form of "[timestamp][error level][module name][process branch][line number][function name]+content", with a total of seven lines of data. Before using the compression method of this application embodiment, these seven lines of data occupy 130 bytes, 168 bytes, 170 bytes, 130 bytes, 168 bytes, 170 bytes, and 130 bytes respectively, totaling 1072 bytes.
[0070] After using the log data compression method provided in this application embodiment, data with timestamps greater than seconds is compressed and displayed in the format #TX, where X represents a number; error levels are displayed in the format #LX, where X represents a number; module names are displayed in the format #MX, where X represents a number; process branches are displayed in the format #PX, where X represents a number; function names are displayed in the format #FX, where X represents a number; and content terms are displayed in the format #DX, where X represents a number. To reduce the size of the trie, line numbers are not compressed, and terms smaller than three bytes are also not compressed. The final compressed data is as follows: Figure 6 As shown.
[0071] After compression, the seven rows of data occupy 57 bytes, 87 bytes, 80 bytes, 57 bytes, 87 bytes, 80 bytes, and 57 bytes respectively, totaling 511 bytes; the data compression rates are 43.8%, 51.8%, 47.1%, 43.8%, 51.8%, 47.1%, and 43.8%, respectively, with an overall compression rate of 47.7%.
[0072] Traditional log data retrieval schemes require decompressing all data before using the BM algorithm for traversal and querying. This embodiment of the application, however, does not require decompressing the stored data. It first extracts the query terms from a trie to obtain the corresponding IDs, then queries a hash table to find the starting address and length of each log entry for all matching terms.
[0073] In communication transmission equipment, the number of vocabulary types involved in the log content is not too large, about 300 to 500, and the vocabulary length is not too long, with an average length of about 5 to 8 bytes. Therefore, the size of the trie is not too large. Table 1 shows the comparison results of different algorithms.
[0074] Table 1
[0075] Where N represents the number of elements, L represents the word length, m represents the pattern string length, n represents the text length, and δ represents the character set size.
[0076] Traditional log compression or retrieval schemes do not provide log content analysis, nor do they generate alerts for individual abnormal data. This application's embodiment utilizes a trie query process to record the frequency of each word, calculates the frequency value of each log entry, and reports an alert to the user when the frequency is below a threshold, helping development and operations personnel discover abnormal processes.
[0077] Based on the same inventive concept, embodiments of this application provide a log data compression device, such as... Figure 7 As shown, the device includes: The vocabulary segmentation module 701 is used to obtain a log data to be compressed from the log file and perform vocabulary segmentation on the log data; The vocabulary lookup module 702 is used to traverse the vocabulary and search for the current vocabulary in the trie. The trie is used to establish a mapping from vocabulary to unique identifier ID. ID determination module 703 is used to obtain the target ID of the current word in the trie if the current word is found; The linked list lookup module 704 is used to look up the corresponding linked list based on the target hash value of the target ID. The linked list is used to establish the mapping from ID to words and to store information of a log data record in the node. The compression trigger module 705 is used to trigger compression if there is a node in the linked list that records the storage information of the log data; otherwise, compression is triggered after adding a node in the linked list that records the storage information of the log data to be compressed. The data compression module 706 is used to replace the current word in the log data with the target ID when compression is triggered; The storage update module 707 is used to traverse the next word until the end of the traversal, and update the corresponding storage information after obtaining the compressed log data.
[0078] Based on the same inventive concept, embodiments of this application provide a log data retrieval device, such as... Figure 8 As shown, the device includes: The keyword acquisition module 801 is used to acquire search terms; The hash value determination module 802 is used to determine the target ID mapped to the search term in the trie when the search term is found in the trie, and to determine the target hash value of the target ID; The storage information determination module 803 is used to find the corresponding target linked list based on the target hash value and obtain the storage information of the log data recorded by all nodes of the target linked list; The compressed data reading module 804 is used to read the target decompressed log data based on the storage information of the log data recorded by any node; The data decompression module 805 is used to traverse the target decompressed log data. For any current ID encountered during traversal, the current ID is decompressed into the corresponding word according to the ID-to-word mapping established by all linked lists.
[0079] Having introduced the log data compression and retrieval method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.
[0080] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."
[0081] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps in the log data compression and retrieval methods according to various exemplary embodiments of this application described above.
[0082] The following reference Figure 9 To describe an electronic device 190 according to this embodiment of the present application. Figure 9 The electronic device 190 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0083] like Figure 9As shown, the electronic device 190 is presented in the form of a general-purpose electronic device. The components of the electronic device 190 may include, but are not limited to: at least one processor 191, at least one memory 192, and a bus 193 connecting different system components (including memory 192 and processor 191).
[0084] Bus 193 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures.
[0085] The memory 192 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1921 and / or cache memory 1922, and may further include read-only memory (ROM) 1923.
[0086] The memory 192 may also include a program / utility 1925 having a set (at least one) of program modules 1924, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0087] Electronic device 190 can also communicate with one or more external devices 194 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 190, and / or with any device that enables electronic device 190 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 195. Furthermore, electronic device 190 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 196. As shown, network adapter 196 communicates with other modules used in electronic device 190 via bus 193. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 190, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0088] In some possible implementations, various aspects of the log data compression and retrieval method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the log data compression and retrieval method according to various exemplary embodiments of this application described above.
[0089] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0090] The program product for log data compression and retrieval according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0091] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0092] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0093] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).
[0094] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.
[0095] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0097] This application is described with reference to flowchart illustrations and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block and / or segment of the flowchart illustrations and block diagrams, as well as combinations of blocks and segments in the flowchart illustrations and block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and boxes Figure 1 The function specified in one or more boxes.
[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and boxes Figure 1 The steps of the function specified in one or more boxes.
[0100] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0101] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for compressing log data, characterized in that, The method includes: Obtain one log data entry to be compressed from the log file, and perform word segmentation on the log data; The vocabulary is traversed, and the current vocabulary is searched in the trie. The trie is used to establish a mapping from vocabulary to unique identifier ID. If the current word is found, obtain the target ID of the current word mapped in the trie; The corresponding linked list is found based on the target hash value of the target ID. The linked list is used to establish the mapping from ID to vocabulary and to store the storage information of a log data record in the node. If a node exists in the linked list that records the storage information of the log data, compression is triggered; otherwise, compression is triggered after adding a node to the linked list that records the storage information of the log data to be compressed. When compression is triggered, the target ID is used to replace the current word in the log data; Iterate through the next word until the end of the iteration, and update the corresponding storage information after obtaining the compressed log data.
2. The method according to claim 1, characterized in that, Also includes: If the current word is not found, a mapping from the current word to the target ID is created in the trie. If no corresponding linked list is found for the target hash value, a linked list for the target hash value is created. A mapping from the target ID to the current word is established by using the current word as the head of the linked list. The storage information of the log data is recorded in the node under the head of the linked list, and compression is triggered.
3. The method according to claim 1, characterized in that, The log data format includes a multi-level index header and log content, and also includes: Based on the multiple fixed words corresponding to the multi-level index header, establish a mapping relationship between each fixed word and its ID in the dictionary tree; A linked list is created based on the hash value of the ID mapped to the fixed vocabulary, and the mapping from the ID to the fixed vocabulary is established by using the fixed vocabulary as the head of the linked list.
4. The method according to claim 3, characterized in that, The multi-level index header includes any of the following index headers: The log data generated by the project file includes timestamps, error levels, module names, process branches, line numbers, and function names.
5. The method according to claim 1, characterized in that, Also includes: Count the number of times the same word appears in the trie; When an alarm detection is triggered, the statistical frequency of the log data is determined based on the frequency and corresponding weight of the words in the trie. If the statistical frequency is lower than the alarm threshold, an alarm will be triggered according to the configured alarm method; In this process, the weights of the words are set according to their importance.
6. The method according to any one of claims 1 to 5, characterized in that, The storage information of the log data includes the file storage location, cache storage location, and data length of the log data.
7. The method according to claim 6, characterized in that, The step of updating the corresponding storage information after obtaining the compressed log data includes: When compression is triggered, a pointer to the content of the current node is recorded, where the current node is the word position traversed in the log data; After traversing the records, determine the length of the compressed log data based on the pointer length of the records, and update the corresponding data length in the linked list.
8. The method according to claim 6, characterized in that, The log data is stored in the following manner: The log data is retrieved and then cached in the cache area; Based on the input command, the compressed log data in the cache is solidified into the log file data area using an active method or a polling method according to a set period. Based on the set deletion criteria, clean up the compressed log data in the cache area that meets the deletion criteria.
9. The method according to claim 2, characterized in that, The step of creating a mapping from the current word to the target ID in the trie includes: Based on the characters / numbers contained in the current vocabulary, map each character / number to a node; Establish a path from the root node to the leaf node based on the order of the characters / numbers in the current vocabulary; The leaf nodes record the target ID of the current word mapping and the starting address of the log data of the current word in the log file.
10. A method for retrieving log data, characterized in that, The method includes: Retrieve search terms; When the search term is found in the trie, the target ID that the search term maps to in the trie is determined, and the target hash value of the target ID is determined. Based on the target hash value, find the corresponding target linked list and obtain the storage information of the log data recorded by all nodes in the target linked list; Based on the storage information of the log data recorded by any node, read the target decompressed log data; Traverse the target decompressed log data. For any current ID encountered, decompress the current ID into the corresponding word based on the ID-to-word mapping established by all linked lists.
11. The method according to claim 10, characterized in that, The storage information of the log data includes the file storage location, cache storage location, and data length of the log data.
12. The method according to claim 11, characterized in that, The reading of the target decompression log data includes: Based on the cache storage location of the log data recorded by any of the nodes, search for a log data in the cache area; If the log data found is the same length as the log data recorded by any of the nodes, then the found log data is read from the cache and used as the target to decompress the log data. Otherwise, based on the file location of the log data recorded by any of the nodes, read one log data from the log file data area as the target decompressed log data.
13. An electronic device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-9, or to perform the method as described in any one of claims 10-12.