A log fuzzy retrieval system and method based on time-series data

Abstract

The application provides a log fuzzy retrieval system and method based on time sequence data, and relates to the technical field of log retrieval. The method specifically comprises the following steps: collecting log data, and performing domain processing on the collected log data, so that each piece of log data is divided into a timestamp domain, a log data attribute domain and a log data text domain; performing word segmentation processing on the log data text domain; selecting a plurality of pieces of log data from the collected log data and constructing a prefix dictionary tree in combination with the text of the segmented log data; performing index construction according to the collected log data and the prefix dictionary tree, so as to generate a prefix index tree containing an inverted list and a skip list; a user selects a fuzzy retrieval mode, simultaneously acquires a query statement of the user and a given maximum edit distance threshold; the acquired query statement is subjected to fuzzy retrieval according to the fuzzy retrieval mode selected by the user, so as to generate a fuzzy retrieval result and perform visual display, thereby improving the fuzzy retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of log retrieval technology, and in particular to a log fuzzy retrieval system and method based on time-series data. Background Technology

[0002] Time series data, as a typical data type, is widely present in human society's production and daily life. Arranged in chronological order by timestamps, time series data describes the observation results at a specific moment by recording a series of observations. Especially in the field of log retrieval, time series data is considered a key resource.

[0003] Fuzzy search for time-series data is a key data analysis technique used to effectively retrieve and analyze time-series information containing log data. Fuzzy search allows query results to differ to some extent from a given query pattern, thus enabling the identification of log data similar to the given query within the original dataset. This method can be applied to monitoring, analyzing, and optimizing the performance of various systems and applications, playing a crucial role in troubleshooting, performance analysis, and security monitoring.

[0004] In troubleshooting, log data is used to track errors, faults, or anomalies in a system, helping engineers identify the root causes of problems. In performance analysis, analyzing system performance logs can identify bottlenecks, critical issue points, and improvement opportunities, thereby improving system efficiency. In security monitoring, security logs can be used to detect potential security threats, including intrusion detection and anomaly detection. Furthermore, in trend analysis, comparing historical event sequences with query patterns can reveal trends and periodic patterns in the data, supporting long-term planning and decision-making. The value of time-series data lies in providing in-depth insights over time. Fuzzy retrieval of logs based on time-series data provides crucial support for problem-solving, efficiency improvement, and informed strategic decision-making, helping to better understand and manage information across various fields and driving scientific research and business development.

[0005] Currently, significant work has been done both domestically and internationally to effectively support fuzzy search, typically assuming a given similarity function to measure the similarity between query results and query input. Log data in time-series data is in string format, and common string similarity functions include Edit Distance, the Jaccard metric, and the token-based Cosine metric. Among these, Edit Distance is the most widely used approximation function due to its high applicability in numerous applications. The Levenshtein automaton is a data structure for implementing edit distance matching of strings. It can construct finite-state automata or deterministic automata, define a similarity metric, and perform automaton matching on the input query to obtain fuzzy search results.

[0006] However, existing fuzzy search methods still have shortcomings. For example, the popular open-source search engine ElasticSearch supports fuzzy search for single words, but does not support fuzzy search across multiple words, and the types of fuzzy queries it supports are limited. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention proposes a log fuzzy retrieval system and method based on time-series data by optimizing the prefix filtering algorithm, aiming to improve the efficiency of fuzzy retrieval.

[0008] The first aspect of this invention proposes a log fuzzy retrieval system based on time-series data, which includes, in sequence, a data domain segmentation module, a text segmentation module, a dictionary construction module, an index construction module, a fuzzy retrieval module, and a result output module;

[0009] The data domain segmentation module is used to divide the input log data into data domains to obtain the timestamp, attributes, and text of each log data.

[0010] The text segmentation module is used to segment the text of the log data, divide the strings in the text into several vocabulary unit tokens, and output a vocabulary unit stream consisting of several vocabulary unit tokens to obtain the text of the segmented log data.

[0011] The dictionary construction module is used to construct a prefix trie based on the first n log data entries and the token stream, and to use the prefix trie as the basis for partitioning the log data; wherein the set of tokens on a path from the root node to the leaf node in the prefix trie constitutes a dictionary item;

[0012] The index building module is used to build a prefix index tree for the text of the segmented log data based on the prefix dictionary tree, so as to speed up the query speed of fuzzy search.

[0013] The fuzzy search module is used to perform fuzzy search on the query statement provided by the user based on the prefix index tree and generate fuzzy search results;

[0014] The result output module is used to visualize the generated fuzzy search results;

[0015] The data fields include: timestamp field, log data attribute field, and log data text field;

[0016] The timestamp field is used to indicate the production time of the log data; the log data attribute field is used to indicate the source of the log data; the log data text field is used to indicate the information recorded in the log data; the timestamp and attribute of a single log data are used as the key value representing the log data, and the text of the single log data is used as the value value of the log data.

[0017] The method for constructing the prefix trie is as follows: given the minimum number of tokens tmin contained in a dictionary item, the maximum number of tokens tmax contained in a dictionary item, and the threshold T for dividing dictionary items, for each segmented log data text, the dictionary items are divided according to a sliding window with a length from tmin to tmax, and the obtained dictionary items are inserted into the prefix trie. The frequency of each token is recorded. After all log data is processed, the prefix trie is pruned according to T and the recorded frequency information to obtain the final prefix trie.

[0018] The method for constructing the prefix index tree is as follows: Initialize a prefix index tree, divide the segmented log data text into index items according to the prefix dictionary tree and insert them into the prefix index tree. The index item is the set of tokens on a path from the root node to the leaf node in the prefix index tree. For any index item, insert the key value of the log data and the offset position of the index item in the log data into the inverted linked list of the leaf node of the index item. Then, construct a secondary index for the index item. The secondary index is a jump list in the prefix index tree. Further divide the index item to obtain all suffix substrings and insert them into the prefix index tree. The last token node in the suffix substring points to a jump list. The jump list stores the inverted address of the parent string corresponding to the suffix substring and the offset of the suffix substring in the parent string. Obtain the inverted linked list of the suffix substring according to the jump list, and complete the construction of the prefix index tree.

[0019] The fuzzy search module includes two methods: single-token fuzzy search and multi-token fuzzy search.

[0020] The single-token fuzzy retrieval is as follows: when the user's query statement is a single token, the candidate options are filtered from the prefix index tree using optimized adaptive prefix filtering and length filtering methods, and all candidate options are verified using a dynamic programming algorithm. Based on the verified candidate options, the fuzzy retrieval results of the single-token fuzzy retrieval are generated.

[0021] The fuzzy search result of the single token fuzzy search is as follows: find the token that matches the query statement within the edit distance range in the log data text field after word segmentation, and add the key value of the log data containing the token to the returned result set;

[0022] The multi-token fuzzy search is as follows: when the user's query statement consists of multiple tokens, the query statement is segmented to obtain a set of tokens, and a string matching the set of tokens is found within the edit distance range. The string includes characters and spaces. Fuzzy search results with no edit operation at the space and fuzzy search results with edit operation at the space are calculated separately and merged to obtain the fuzzy search result of the multi-token fuzzy search.

[0023] The fuzzy search result of the multi-token fuzzy search is as follows: in the log data text field after word segmentation, the key value of the log data containing the matched string is added to the returned result set;

[0024] A second aspect of this invention proposes a log fuzzy retrieval method based on time-series data, the method comprising the following steps:

[0025] Step 1: Collect log data;

[0026] Step 2: Perform domain segmentation on the collected log data, dividing each log data into a timestamp field, a log data attribute field, and a log data text field;

[0027] Step 3: Perform word segmentation on the log data text field. Use the Unicode text segmentation algorithm to segment the text in the log data text field into words, and convert all uppercase letters in the text to lowercase letters to obtain the segmented log data text.

[0028] Step 4: Select the first n log data from the collected log data, and construct a prefix trie based on the selected log data and the text of the segmented log data;

[0029] Step 5: Build an index based on the collected log data and the prefix trie, generating a prefix index tree containing an inverted linked list and a jump list;

[0030] Step 6: The user selects a fuzzy search method, and the system simultaneously obtains the user's query statement and the given maximum edit distance threshold τ; the fuzzy search method includes: single-token fuzzy search and multi-token fuzzy search;

[0031] Step 7: Perform single-token or multi-token fuzzy search on the obtained query statement and the given maximum edit distance threshold τ according to the fuzzy search method selected by the user, and generate fuzzy search results;

[0032] Step 8: Visualize the fuzzy search results;

[0033] The single-token fuzzy retrieval process described in step 7 includes:

[0034] Step A1: Perform length filtering and adaptive prefix filtering on all tokens in the first-level nodes of the prefix index tree to obtain candidate tokens;

[0035] Step A2: For each candidate option, use a dynamic programming algorithm to calculate the actual edit distance between the candidate option and the query statement and verify it. If the actual edit distance is less than or equal to the maximum edit distance threshold τ, the candidate option passes the verification and is used as a token that matches the query result; otherwise, the candidate option fails the verification.

[0036] Step A3: Obtain the inverted linked list, jump list, and linked list of all child nodes of the verified candidates in the prefix index tree;

[0037] Step A4: Merge and deduplicate all the obtained linked lists to obtain the key value set of the query result data, and return this set as the fuzzy search result of single token fuzzy search to the user;

[0038] The length filtering in step A1 is as follows: the length of the candidate is greater than or equal to the difference between the length of the query statement and the maximum edit distance threshold, and less than or equal to the sum of the length of the query statement and the maximum edit distance threshold;

[0039] The adaptive prefix filtering is as follows: For each query statement, the prefix length q used by the query statement is determined according to the formula len(Query)=q*τ+q, where Query represents the query statement; len(Query) represents the number of characters in the returned query statement; the query statement is divided according to the calculated prefix length q and the prefix q-grams that best represent the entire query statement are obtained from it;

[0040] The multi-token fuzzy retrieval process described in step 7 includes:

[0041] Step B1: Perform word segmentation on the query statement to obtain the segmented query statement;

[0042] Step B2: Calculate the matching results of the query statement after word segmentation where no editing operation is generated at the spaces, and obtain the result set;

[0043] Step B2.1: The query statement after word segmentation is divided into variable-length index items according to the prefix trie;

[0044] Step B2.2: Calculate the lower bound μ of the common index entries based on the number of variable-length index entries and the maximum edit distance threshold;

[0045] Step B2.3: Based on the lower bound μ of the common index item, arbitrarily select several different sets of index items from the m index items, where each set of index items includes μ index items; for each set of index items, take the intersection of the node linked list of the set of index items and the edit distance, so that the position information of the token and the character position information of the substring within the token are both less than or equal to the maximum edit distance threshold τ, and then filter the safe distance substrings based on the intersection result;

[0046] Step B2.4: Use dynamic programming algorithm to perform edit distance verification on the safe distance substring. Calculate whether the safe distance substring contains a substring whose edit distance to the query statement is within the threshold range. If it contains a substring, add the key value corresponding to the safe distance substring to the result set. If it does not contain a substring, it means that the safe distance substring is not a query result.

[0047] Step B3: Calculate the matching results of the edit operation at the spaces in the query statement after word segmentation, and obtain the result set;

[0048] Step B3.1: Analyze the effects of inserting, deleting, and replacing spaces in the query statement on the maximum edit distance threshold τ. When inserting or deleting spaces in the query statement, the value of τ decreases by 1, while when replacing spaces, the value of τ remains unchanged.

[0049] Step B3.2: Rewrite the query statement according to the impact of inserting, deleting and replacing spaces in the query statement on the maximum edit distance threshold τ, and obtain a new set of query statements, each of which contains its own edit distance threshold;

[0050] Step B3.3: Repeat step B3.2 to rewrite the query statement according to each query statement provided by the user and its edit distance threshold, and obtain several sets of query result sets;

[0051] Step B3.4: Take the union of the obtained query result sets to obtain the result set;

[0052] Step B4: Take the union of the result set obtained in step B2 and the result set obtained in step B3 to obtain the key value set of the query results, and return this set as the fuzzy search result of multi-token fuzzy search to the user.

[0053] The beneficial effects of adopting the above technical solution are as follows:

[0054] Unlike previous fuzzy search technologies, this invention starts with log data from time-series data and designs and implements a log fuzzy search system based on time-series data. It can systematically and structurally collect, store, index, and perform fuzzy search on large-scale time-series log data, displaying the results. It provides more powerful performance for fuzzy search, especially in large-scale datasets and complex query scenarios. Through intelligent prefix filtering and multi-word query support, it effectively improves query efficiency and result quality.

[0055] Traditional prefix filtering uses a uniform prefix length for all queries, and the fixed prefix length may not adequately represent the query, resulting in poor filtering and low query efficiency. Therefore, this invention proposes an adaptive prefix filtering algorithm to address the single-token fuzzy retrieval problem. During fuzzy retrieval, this algorithm adaptively determines the prefix length for each query and divides the query string based on the calculated prefix length to obtain the prefix that best represents the entire query string. By selecting an adaptive prefix length that better represents the query string, the algorithm can efficiently filter results, reduce unnecessary query operations, thereby reducing the number of verifications and significantly lowering query time.

[0056] This invention proposes a problem definition for multi-token fuzzy retrieval, designs a query algorithm for multi-token fuzzy retrieval, considers character-level operations on top of word-level indexing, supports multi-token fuzzy retrieval with arbitrary edit distances, and utilizes variable-length word-level indexes to achieve functionality previously only achievable with character-level indexes. Finally, a complete log fuzzy retrieval system based on time-series data is built to realize fuzzy retrieval of log data. This system, through data domain segmentation, text domain word segmentation, dictionary construction, index construction, fuzzy retrieval, and result output, can achieve efficient fuzzy retrieval in large-scale datasets and complex query scenarios, ensuring result quality and accuracy.

[0057] This invention supports multi-word queries. For fuzzy retrieval of multiple words, the query statement is first segmented into a group of words. Then, results matching these segments within the edit distance range are searched in the original segmented data. Since spaces are generated by removing delimiters during segmentation, the matching results at spaces where editing operations occurred also need to be considered. Character-level operations are supported on the word-level index, enabling fuzzy retrieval of multiple words at arbitrary edit distances. This allows for a more comprehensive consideration of multiple query terms and the relationships between them and their delimiters, reducing index space usage while providing accurate and efficient matching results for complex fuzzy matching scenarios. Attached Figure Description

[0058] Figure 1 This is a structural diagram of a log fuzzy retrieval method based on time-series data in this embodiment;

[0059] Figure 2 This is a schematic diagram illustrating the principle of single-token fuzzy retrieval in this implementation method;

[0060] Figure 3 This is a schematic diagram illustrating the principle of multi-token fuzzy retrieval in this implementation.

[0061] Figure 4 This is a flowchart of a log fuzzy retrieval method based on time-series data in this embodiment;

[0062] Figure 5 This is a schematic diagram illustrating the principle that no editing operation algorithm is generated in the blank areas of this embodiment;

[0063] Figure 6 This is a schematic diagram illustrating the algorithm for generating editing operations in the blank spaces in this embodiment. Detailed Implementation

[0064] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0065] This embodiment provides a log fuzzy retrieval system based on time-series data, such as... Figure 1 As shown, the system includes the following modules connected in sequence: data domain segmentation module, text segmentation module, dictionary construction module, index construction module, fuzzy search module, and result output module.

[0066] The data domain segmentation module is used to divide the input log data into data domains to obtain the timestamp, attributes, and text of each log data.

[0067] The data fields include: timestamp field, log data attribute field, and log data text field.

[0068] The timestamp field is used to indicate the production time of the log data; the log data attribute field is used to indicate the source of the log data; the log data text field is used to indicate the information recorded in the log data; the timestamp and attribute of a single log data are used as the key value representing the log data, and the text of the single log data is used as the value value of the log data.

[0069] In this implementation, since only one log data record can be generated from the same source at any given time, the timestamp field and the log data attribute field uniquely identify each log data record, and the log data text field contains the information of that log data record; the log data text field is also the data field for fuzzy retrieval. By dividing the data into fields, we obtain the timestamp, attribute field, and text field for each log data record. The values ​​of the timestamp and attribute fields are used as the key values ​​for this log data record, and the value of the text field is used as the value value for this log data record. This ensures the structural uniformity of the log data and provides support for subsequent index construction and fuzzy retrieval modules.

[0070] The text segmentation module is used to segment the log data text, divide the strings in the text to obtain several vocabulary unit tokens, and output a vocabulary unit stream consisting of several vocabulary unit tokens to obtain the segmented log data text.

[0071] In this implementation, the text segmentation module is responsible for finding the boundaries between words and outputting the content between all boundaries, that is, performing word segmentation on the text of the log data; the word segmentation method includes: discarding some punctuation marks and other characters, segmenting the text into words, converting uppercase letters to lowercase letters, and outputting a word unit stream token stream based on several word unit tokens obtained from the word segmentation process.

[0072] The dictionary construction module is used to construct a prefix trie based on the first n log data entries and the token stream, and to use the prefix trie as the basis for partitioning the log data; wherein the set of tokens on a path from the root node to the leaf node in the prefix trie constitutes a dictionary item.

[0073] The method for constructing the prefix trie is as follows: given three parameters, namely: the minimum number of tokens tmin contained in a dictionary item, the maximum number of tokens tmax contained in a dictionary item, and the threshold T for dividing dictionary items, for each segmented log data text, the dictionary items are divided according to a sliding window with a length from tmin to tmax, and the obtained dictionary items are inserted into the prefix trie. The frequency of each token is recorded. After all log data is processed, the prefix trie is pruned according to T and the recorded frequency information to obtain the final prefix trie.

[0074] The index building module is used to build a prefix index tree for the text of the segmented log data based on the prefix dictionary tree, so as to speed up the query speed of fuzzy search.

[0075] The method for constructing the prefix index tree is as follows: Initialize a prefix index tree, divide the segmented log data text into index items according to the prefix dictionary tree and insert them into the prefix index tree. The index item is the set of tokens on a path from the root node to the leaf node in the prefix index tree. For any index item, insert the key value of the log data and the offset position of the index item in the log data into the inverted linked list of the leaf node of the index item. Then, construct a secondary index for the index item. The secondary index is a jump list in the prefix index tree. Further divide the index item to obtain all suffix substrings and insert them into the prefix index tree. The last token node in the suffix substring points to a jump list. The jump list stores the inverted address of the parent string corresponding to the suffix substring and the offset of the suffix substring in the parent string. Obtain the inverted linked list of the suffix substring based on the jump list.

[0076] In this embodiment, the offset position of the index item in the log data can be directly obtained, indicating the starting position of the index item in the log data. After processing all the segmented log data text as described above, the prefix index tree is constructed.

[0077] The fuzzy search module is used to perform fuzzy search on the user's query statement based on the prefix index tree and generate fuzzy search results.

[0078] The fuzzy search module includes two modes: single-token fuzzy search and multi-token fuzzy search.

[0079] The single-token fuzzy retrieval is as follows: when the user's query statement is a single token, the candidate options are filtered from the prefix index tree using optimized adaptive prefix filtering and length filtering methods. The dynamic programming algorithm is used to verify all candidate options, and the fuzzy retrieval results of the single-token fuzzy retrieval are generated based on the verified candidate options.

[0080] The fuzzy search result of the single token fuzzy search is as follows: find the token that matches the query statement within the edit distance range in the log data text field after word segmentation, and add the key value of the log data containing the token to the returned result set.

[0081] In this embodiment, such as Figure 2 As shown, when performing single-token fuzzy retrieval, if the user's query is a single token, there is no need to perform word segmentation on the query. Optimized adaptive prefix filtering and length filtering methods are used to filter candidate items from the prefix index tree. A dynamic programming algorithm is used to verify the edit distance of all candidate items. Then, based on the verified candidate items, the inverted list, jump list, and linked lists of all child nodes of the candidate item are obtained from the prefix index tree and merged to obtain the fuzzy retrieval result of single-token fuzzy retrieval.

[0082] The multi-token fuzzy search is as follows: when the user's query statement consists of multiple tokens, the query statement is segmented to obtain a set of tokens, and a string matching the set of tokens is found within the edit distance range. The string includes characters and spaces. Fuzzy search results with no edit operation at the space and fuzzy search results with edit operation at the space are calculated separately and merged to obtain the fuzzy search result of the multi-token fuzzy search.

[0083] In this embodiment, such as Figure 3 As shown, when performing multi-token fuzzy search, if the user's query statement contains multiple tokens, the query statement needs to be segmented into words first.

[0084] The fuzzy search result of the multi-token fuzzy search is as follows: in the log data text field after word segmentation, the key value of the log data containing the matched string is added to the returned result set.

[0085] The result output module is used to visualize the generated fuzzy search results.

[0086] In this embodiment, since the fuzzy search module returns the key value of the matching result, the result output module needs to extract the complete log data and display it to the user. This invention uses a table to represent the output result. The table includes the timestamp field, the log data attribute field, and the log data text field of the query result log data for data visualization.

[0087] This embodiment provides a log fuzzy retrieval system and method based on time-series data, such as... Figure 4 As shown, the method includes the following steps:

[0088] Step 1: Collect log data.

[0089] In this embodiment, the source of log data can be either a log dataset or actual log data generated by the system that produces the log data.

[0090] Step 2: Perform domain segmentation on the collected log data, dividing each log data into a timestamp field, a log data attribute field, and a log data text field.

[0091] In this embodiment, the collected log data is processed by domain segmentation to ensure the uniformity of the log data structure.

[0092] Step 3: Perform word segmentation on the log data text field. Use the Unicode text segmentation algorithm to segment the text in the log data text field into words, and convert all uppercase letters in the text to lowercase letters to obtain the segmented log data text.

[0093] Step 4: Select the first n log data from the collected log data, and construct a prefix trie based on the selected log data and the text of the segmented log data.

[0094] Step 5: Build an index based on the collected log data and the prefix trie, generating a prefix index tree containing an inverted linked list and a jump list.

[0095] Step 6: The user selects a fuzzy search method, and the system obtains the user's query statement and the given maximum edit distance threshold τ. The fuzzy search method includes: single-token fuzzy search and multi-token fuzzy search.

[0096] In this embodiment, the user provides a query statement based on the selected fuzzy search method. This statement is the fuzzy search statement that the user wants to perform using the retrieval system, and it is unrelated to the specific data.

[0097] Step 7: Perform single-token or multi-token fuzzy search on the obtained query statement and the given maximum edit distance threshold τ according to the fuzzy search method selected by the user, and generate fuzzy search results.

[0098] When a user selects a single-token fuzzy search, adaptive prefix filtering and length filtering are used to obtain candidate options, and the candidate options are verified to obtain the result set. Multi-token fuzzy search is used to find matching results where no editing operation was performed at the space, and matching results where editing operation was performed at the space.

[0099] The single-token fuzzy search process includes:

[0100] Step A1: Perform length filtering and adaptive prefix filtering on all tokens in the first-level nodes of the prefix index tree to obtain candidate tokens;

[0101] The length filtering is as follows: the length of the candidate is greater than or equal to the difference between the length of the query statement and the maximum edit distance threshold, and less than or equal to the sum of the length of the query statement and the maximum edit distance threshold.

[0102] The prefix filtering is as follows: if the actual edit distance between two strings is less than or equal to the maximum edit distance threshold τ, then at least one pair of q-grams in the q*τ+1 prefix q-grams after sorting the two strings by letter frequency is the same and the position difference is less than or equal to the maximum edit distance threshold τ, where q is the window size for dividing the prefix q-grams.

[0103] In this implementation, the disadvantage of traditional prefix filtering is that since the parameter q is given in advance, a single q value is used to divide the prefix q-grams for query statements of different lengths. However, in reality, if a very long query statement selects a small q value, then the prefix q-grams cannot fully represent the query statement, which will lead to poor filtering effect and correspondingly more verification times. It can be seen that the selection of a fixed q value limits the efficiency of prefix filtering.

[0104] The adaptive prefix filtering is as follows: for each query statement, the prefix length q used by the query statement is determined according to the formula len(Query)=q*τ+q, where Query represents the query statement; len(Query) represents the number of characters in the returned query statement; the query statement is divided according to the calculated prefix length q and the prefix q-grams that best represent the entire query statement are obtained, resulting in the best filtering effect.

[0105] Step A2: For each candidate option, use a dynamic programming algorithm to calculate and verify the actual edit distance between the candidate option and the query statement. If the actual edit distance is less than or equal to the maximum edit distance threshold τ, the candidate option passes the verification and is used as a token that matches the query result; otherwise, the candidate option fails the verification.

[0106] In this implementation, the industry-classic dynamic programming algorithm is used for each candidate option. A two-dimensional array is constructed to calculate the actual edit distance between the candidate option and the query string. Candidate options whose actual edit distance is less than or equal to the maximum edit distance threshold τ are verified. Verified candidates are tokens that match the query results, while candidates that do not pass verification do not match the query results.

[0107] Step A3: Obtain the inverted linked list, jump list, and linked list of all child nodes of the verified candidates in the prefix index tree.

[0108] Step A4: Merge and deduplicate all the obtained linked lists to obtain the key value set of the query result data, and return this set as the fuzzy search result of single token fuzzy search to the user.

[0109] In this implementation, the set of key values ​​obtained from the query results is returned to the user as the final result set of the single-token fuzzy search.

[0110] The process of the multi-token fuzzy retrieval method includes:

[0111] Step B1: Perform word segmentation on the query statement to obtain the segmented query statement.

[0112] In this embodiment, the same word segmentation algorithm is used as the text field word segmentation module when performing word segmentation on the query statement to ensure the accuracy of the query results.

[0113] Step B2: The query statement after word segmentation did not produce any matching results for editing operations at the spaces.

[0114] Step B2.1: The query statement after word segmentation is divided into variable-length index items according to the prefix trie.

[0115] The method for dividing variable-length index terms is as follows: use a sliding window to find the longest substring that matches the segmented query statement in the prefix trie. If the substring does not exist, divide the segmented query statement according to the length of the minimum number of tokens tmin contained in the dictionary item in the prefix trie. If the current index term is contained in the previous index term, do not divide the current index term.

[0116] Step B2.2: Calculate the lower bound μ of the common index entries based on the number of variable-length index entries and the maximum edit distance threshold.

[0117] The lower bound μ of the common index entries is equal to the number of variable-length index entries m minus the maximum number of index entries that can be affected by τ edit operations.

[0118] The maximum number of index entries that the τ edit operations may affect is determined by: recording the number of index entries affected by an edit operation at each position in the query statement, arranging the recorded numbers from largest to smallest, and summing the first τ values ​​to obtain the maximum number of index entries that the τ edit operations may affect.

[0119] Step B2.3: Based on the lower bound μ of the common index terms, arbitrarily select several different sets of index terms from the m index terms, where each set of index terms includes μ index terms; for each set of index terms, take the intersection of the node linked list of the set of index terms and the edit distance, so that the position information of the token and the character position information of the substring within the token are both less than or equal to the maximum edit distance threshold τ, and then filter the safe distance substrings based on the intersection result.

[0120] In this embodiment, μ index items are randomly selected from m index items to obtain a set of index items. For each set of index items, the node linked lists are intersected within the edit distance range. The position information of the token and the character position information of the substring within the token must satisfy the condition of being less than or equal to the maximum edit distance threshold τ. Finally, the position corresponding to the original data is found based on the result of the intersection. The safe distance substring may contain the result string.

[0121] Step B2.4: Use dynamic programming algorithm to perform edit distance verification on the safe distance substring. Calculate whether the safe distance substring contains a substring whose edit distance to the query statement is within the threshold range. If it contains a substring, add the key value corresponding to the safe distance substring to the result set. If it does not contain a substring, it means that the safe distance substring is not a query result.

[0122] In this embodiment, such as Figure 5 As shown, the query statement after word segmentation is divided into variable-length index items according to the prefix trie. The lower bound μ of the common index items is calculated based on the divided variable-length index items and the maximum edit distance threshold. The node linked list of each group of lower bounds is aggregated and the intersection within the edit distance range is performed to obtain a set of safe distance substrings. The safe distance substrings of each group are combined into a union. The edit distance of each safe distance substring is verified. Finally, the key value of the log data containing the verified safe distance substring is added to the returned result set.

[0123] Step B3: Calculate the matching results of the edit operation at the spaces in the query statement after word segmentation, and obtain the result set.

[0124] Step B3.1: Analyze the effects of inserting, deleting, and replacing spaces in the query statement on the maximum edit distance threshold τ. When inserting or deleting spaces in the query statement, the value of τ decreases by 1, while when replacing spaces, the value of τ remains unchanged.

[0125] In this embodiment, since spaces are an influencing factor of edit distance, it is necessary to analyze the impact of inserting, deleting, and replacing spaces on the maximum edit distance threshold τ.

[0126] Step B3.2: Rewrite the query statement according to the impact of inserting, deleting and replacing spaces in the query statement on the maximum edit distance threshold τ, to obtain a new set of query statements, and each query statement contains its own edit distance threshold.

[0127] In this embodiment, such as Figure 6 As shown, for the query statement after word segmentation, the possible editing operations and threshold changes at the spaces are enumerated, the query statement is rewritten, and a new set of query statements is obtained after rewriting. Each query statement has its own edit distance threshold, and the new query statements and their corresponding edit distance thresholds are used as a set of query results.

[0128] Step B3.3: Repeat step B3.2 to rewrite the query statement according to each query statement provided by the user and its edit distance threshold, and obtain several sets of query result sets.

[0129] Step B3.4: Take the union of the obtained query result sets to obtain the result set.

[0130] Step B4: Take the union of the result set obtained in step B2 and the result set obtained in step B3 to obtain the key value set of the query results, and return this set as the fuzzy search result of multi-token fuzzy search to the user.

[0131] Step 8: Visualize the fuzzy search results.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A log fuzzy retrieval system based on time-series data, characterized in that, The system comprises, in sequence: a data domain segmentation module, a text segmentation module, a dictionary construction module, an index construction module, a fuzzy search module, and a result output module; The data domain segmentation module is used to divide the input log data into data domains to obtain the timestamp, attributes, and text of each log data. The text segmentation module is used to segment the text of the log data, divide the strings in the text into several vocabulary unit tokens, and output a vocabulary unit stream tokenstream composed of several vocabulary unit tokens to obtain the text of the segmented log data. The dictionary construction module is used to construct a prefix trie based on the first n log data entries and the token stream, and to use the prefix trie as the basis for partitioning the log data. The set of tokens along a path from the root node to a leaf node in the prefix trie constitutes a dictionary entry; The method for constructing the prefix trie is as follows: given the minimum number of tokens tmin contained in a dictionary item, the maximum number of tokens tmax contained in a dictionary item, and the threshold T for dividing dictionary items, for each segmented log data text, the dictionary items are divided according to a sliding window with a length from tmin to tmax, and the obtained dictionary items are inserted into the prefix trie. The frequency of each token is recorded. After all log data is processed, the prefix trie is pruned according to T and the recorded frequency information to obtain the final prefix trie. The index building module is used to build a prefix index tree for the text of the segmented log data based on the prefix dictionary tree, so as to speed up the query speed of fuzzy search. The fuzzy search module is used to perform fuzzy search on the query statement provided by the user based on the prefix index tree and generate fuzzy search results; The fuzzy search module includes two search methods: single-token fuzzy search and multi-token fuzzy search. The single-token fuzzy retrieval is as follows: when the user's query statement is a single token, the candidate options are filtered from the prefix index tree using optimized adaptive prefix filtering and length filtering methods, and all candidate options are verified using a dynamic programming algorithm. Based on the verified candidate options, the fuzzy retrieval results of the single-token fuzzy retrieval are generated. The fuzzy search result of the single token fuzzy search is as follows: find the token that matches the query statement within the edit distance range in the log data text field after word segmentation, and add the key value of the log data containing the token to the returned result set; The multi-token fuzzy search is as follows: when the user's query statement consists of multiple tokens, the query statement is segmented to obtain a set of tokens, and a string matching the set of tokens is found within the edit distance range. The string includes characters and spaces. Fuzzy search results with no edit operation at the space and fuzzy search results with edit operation at the space are calculated separately and merged to obtain the fuzzy search result of the multi-token fuzzy search. The fuzzy search result of the multi-token fuzzy search is as follows: in the log data text field after word segmentation, the key value of the log data containing the matched string is added to the returned result set; The result output module is used to visualize the generated fuzzy search results.

2. The log obfuscation search system based on time-series data according to claim 1, wherein, The data fields include: timestamp field, log data attribute field, and log data text field; The timestamp field is used to indicate the production time of the log data; the log data attribute field is used to indicate the source of the log data; the log data text field is used to indicate the information recorded in the log data; the timestamp and attribute of a single log data are used as the key value representing the log data, and the text of the single log data is used as the value value of the log data.

3. The log obfuscation search system based on time-series data according to claim 2, wherein, The method for constructing the prefix index tree is as follows: Initialize a prefix index tree, divide the segmented log data text into index items according to the prefix dictionary tree and insert them into the prefix index tree. The index item is the set of tokens on a path from the root node to the leaf node in the prefix index tree. For any index item, insert the key value of the log data and the offset position of the index item in the log data into the inverted linked list of the leaf node of the index item. Then, construct a secondary index for the index item. The secondary index is a jump list in the prefix index tree. Further divide the index item to obtain all suffix substrings and insert them into the prefix index tree. The last token node in the suffix substring points to a jump list. The jump list stores the inverted address of the parent string corresponding to the suffix substring and the offset of the suffix substring in the parent string. Obtain the inverted linked list of the suffix substring according to the jump list, and complete the construction of the prefix index tree.

4. A method for fuzzy search on time-series data based log, implemented by using the system for fuzzy search on time-series data based log according to any one of claims 1-3, characterized in that, The method includes the following steps: Step 1: Collect log data; Step 2: Perform domain segmentation on the collected log data, dividing each log data into a timestamp field, a log data attribute field, and a log data text field; Step 3: Perform word segmentation on the log data text field. Use the Unicode text segmentation algorithm to segment the text in the log data text field into words, and convert all uppercase letters in the text to lowercase letters to obtain the segmented log data text. Step 4: Select the first n log data from the collected log data, and construct a prefix trie based on the selected log data and the text of the segmented log data; Step 5: Build an index based on the collected log data and the prefix trie, generating a prefix index tree containing an inverted linked list and a jump list; Step 6: The user selects a fuzzy search method, and the system simultaneously obtains the user's query statement and the given maximum edit distance threshold τ; the fuzzy search method includes: single-token fuzzy search and multi-token fuzzy search; Step 7: Perform single-token or multi-token fuzzy search on the obtained query statement and the given maximum edit distance threshold τ according to the fuzzy search method selected by the user, and generate fuzzy search results; Step 8: Visualize the fuzzy search results.

5. The method of claim 4, wherein the time-based data is a time series data. The method includes the following steps: Step 7, the single-token fuzzy retrieval process, includes: Step A1: Perform length filtering and adaptive prefix filtering on all tokens in the first-level nodes of the prefix index tree to obtain candidate tokens; Step A2: For each candidate option, use a dynamic programming algorithm to calculate the actual edit distance between the candidate option and the query statement and verify it. If the actual edit distance is less than or equal to the maximum edit distance threshold τ, the candidate option passes the verification and is used as a token that matches the query result; otherwise, the candidate option fails the verification. Step A3: Obtain the inverted linked list, jump list, and linked list of all child nodes of the verified candidates in the prefix index tree; Step A4: Merge and deduplicate all the obtained linked lists to obtain the key value set of the query result data, and return this set as the fuzzy search result of single token fuzzy search to the user.

6. The method of claim 5, wherein the time-based data is a time series data. The length filtering in step A1 is as follows: the length of the candidate is greater than or equal to the difference between the length of the query statement and the maximum edit distance threshold, and less than or equal to the sum of the length of the query statement and the maximum edit distance threshold; The adaptive prefix filtering is as follows: for each query statement, the prefix length q used by the query statement is determined according to the formula len(Query)=q*τ+q, where Query represents the query statement; len(Query) represents the number of characters in the returned query statement; the query statement is divided according to the calculated prefix length q and the prefix q-grams that best represent the entire query statement are obtained from it.

7. The method of claim 6, wherein the time-based data is a time series data. The multi-token fuzzy retrieval process described in step 7 includes: Step B1: Perform word segmentation on the query statement to obtain the segmented query statement; Step B2: Calculate the matching results of the query statement after word segmentation where no editing operation is generated at the spaces, and obtain the result set; Step B2.1: The query statement after word segmentation is divided into variable-length index items according to the prefix trie; Step B2.2: Calculate the lower bound µ of the common index item based on the number of variable-length index items and the maximum edit distance threshold; Step B2.3: Based on the lower bound µ of the common index item, arbitrarily select several different sets of index items from the m index items, where each set of index items includes µ index items; for each set of index items, take the intersection of the node linked list of the set of index items and the edit distance, so that the position information of the token and the character position information of the substring within the token are both less than or equal to the maximum edit distance threshold τ, and then filter the safe distance substrings based on the intersection result; Step B2.4: Use dynamic programming algorithm to perform edit distance verification on the safe distance substring. Calculate whether the safe distance substring contains a substring whose edit distance to the query statement is within the threshold range. If it contains a substring, add the key value corresponding to the safe distance substring to the result set. If it does not contain a substring, it means that the safe distance substring is not a query result. Step B3: Calculate the matching results of the edit operation at the spaces in the query statement after word segmentation, and obtain the result set; Step B3.1: Analyze the effects of inserting, deleting, and replacing spaces in the query statement on the maximum edit distance threshold τ. When inserting or deleting spaces in the query statement, the value of τ decreases by 1, while when replacing spaces, the value of τ remains unchanged. Step B3.2: Rewrite the query statement according to the impact of inserting, deleting and replacing spaces in the query statement on the maximum edit distance threshold τ, and obtain a new set of query statements, each of which contains its own edit distance threshold; Step B3.3: Repeat step B3.2 to rewrite the query statement according to each query statement provided by the user and its edit distance threshold, and obtain several sets of query result sets; Step B3.4: Take the union of the obtained query result sets to obtain the result set; Step B4: Take the union of the result set obtained in step B2 and the result set obtained in step B3 to obtain the key value set of the query results, and return this set as the fuzzy search result of multi-token fuzzy search to the user.

Images