Log query method and device, medium and program product

Abstract

The invention discloses a log query method and device, a medium and a program product, and relates to the field of log analysis. The log query method comprises the steps of obtaining a current log query statement; when the current log query statement has the hot word query attribute, determining a hot word query bit vector according to the current log query statement and the target hot word language unit dictionary; generating a log index bit vector according to the to-be-queried log and the target hot word language unit dictionary; and based on the hot word query bit vector and the log index bit vector, determining a hit log in the to-be-queried log. According to the technical scheme provided by the embodiment of the invention, the log required to be queried can be accurately and quickly hit on the premise of low resource consumption.

Description

Technical Field

[0001] This invention relates to the field of log analysis technology, and in particular to a log query method, device, medium, and program product. Background Technology

[0002] Internet systems typically generate a large number of operational log records. To quickly locate and analyze these logs, a full-text index is usually built for the original log content to accelerate retrieval. However, according to regulations, log records usually need to be retained for at least 180 days, and some industries even require more than 3 years.

[0003] Currently, mainstream log indexing technologies mainly fall into two categories: 1) Inverted indexes built based on full-text segmentation, allowing for quick index matching with any keyword search. Common traditional log products such as Elasticsearch (an open-source distributed search engine) and Splunk (a machine data analysis and log management platform) currently use inverted indexes for log queries. 2) Fixed-size Bloom filters built based on full-text segmentation, allowing for quick skipping of corresponding raw data if the keyword search does not match the filter. Common cloud-native log products such as Loki (a log aggregation system) currently use Bloom filters for log queries.

[0004] However, building an inverted index requires significant CPU (Central Processing Unit) resources for data processing and writing, and the storage space is typically 1.5 to 3 times the original log size. Furthermore, under regulations requiring at least six months of retention, inverted indexes incur substantial disk costs. Mathematically, Bloom filters can verify falsehoods but not verify truths. Even when a keyword is matched, a secondary filtering and confirmation of the corresponding original log is necessary. Therefore, while highly efficient for querying rare keywords, such as identification numbers, it is inefficient for querying common keywords, such as searching for "GET 200" in HTTP (Hypertext Transfer Protocol) logs, approaching the efficiency of a full disk hard scan. Summary of the Invention

[0005] This invention provides a log query method, device, medium, and program product to solve the problem that existing log queries cannot accurately and efficiently hit logs under the premise of low resource consumption.

[0006] According to one aspect of the present invention, a log query method is provided, comprising: Get the current log query statement; When the current log query statement is a hot word query attribute, determine the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary; Generate a log index bit vector based on the log to be queried and the target hot word language unit dictionary; Based on the hot word query bit vector and the log index bit vector, the hit logs in the log to be queried are determined.

[0007] According to another aspect of the present invention, a log query apparatus is provided, comprising: The query statement retrieval module is used to retrieve the current log query statement; The query bit vector determination module is used to determine the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary when the current log query statement is a hot word query attribute. The log index bit vector determination module is used to generate a log index bit vector based on the log to be queried and the target hot word language unit dictionary; The log query module is used to determine the matching logs in the logs to be queried based on the hot word query bit vector and the log index bit vector.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the log query method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log query method according to any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the log query method described in any embodiment of the present invention.

[0011] The technical solution of this invention obtains the current log query statement. When the current log query statement has a hot word query attribute, it determines the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary. Then, it generates a log index bit vector based on the log to be queried and the target hot word language unit dictionary. Finally, based on the hot word query bit vector and the log index bit vector, it determines the hit logs in the log to be queried. This solution creates a target hot word language unit dictionary that meets the user's actual query needs by querying hot words, which can effectively control the amount of dictionary data. Then, it matches the target hot word language unit dictionary with the current log query statement and the log to be queried, and directly determines the hit logs in the log to be queried based on the secondary comparison of the matching results. This can significantly reduce the complexity of log query while ensuring query accuracy, and solves the problem that existing log queries cannot accurately and efficiently hit logs under low resource consumption. It can accurately and quickly hit the required logs under low resource consumption.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a log query method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a log query method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a log query device provided in Embodiment 3 of the present invention; Figure 4 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "initial," "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1 This is a flowchart illustrating a log query method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations requiring accurate and rapid log retrieval under low disk cost conditions. The method can be executed by a log query device, which can be implemented in hardware and / or software and can be configured in an electronic device. The electronic device may include, but is not limited to, a server or computer. Figure 1 As shown, the method includes: Step 110: Obtain the current log query statement.

[0018] The current log query statement can be a structured query statement that the user has just submitted to retrieve logs. The current log query statement is used to filter logs that match the user's current search requirements.

[0019] In this embodiment of the invention, the current log query statement can be obtained first.

[0020] Step 120: When the current log query statement is a hot word query attribute, determine the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary.

[0021] The hot word query attribute can be a log query statement type that uses hot words based on word frequency statistics for log queries. The target hot word language unit dictionary can be a query dictionary built based on a log hot word database. For example, the target hot word language unit dictionary can include an n-gram dictionary built based on a log hot word database. In the n-gram dictionary, n represents the number of consecutively occurring words, characters, or syllables. n can be a positive integer that can be adjusted as needed. For example, n can be 2, and the 2-gram dictionary elements corresponding to the word "Error" are [Er,rr,ro,or]. The hot word query bit vector can be a vector representing the position of the successfully matched element in the target hot word language unit dictionary for the current log query statement. For example, if the target hot word language unit dictionary includes four elements, and the current log query statement, after being divided according to the granularity of the elements in the target hot word language unit dictionary, successfully matches the first and fourth elements, but fails to match the second and third elements, then the hot word query bit vector can be [1,0,0,1].

[0022] Optionally, the target hot word language unit dictionary can be created online or offline. For example, when the target hot word language unit dictionary is created offline, the time interval for offline mining of the log hot word database can be customized. After the log hot word database mining is completed, the offline update process of the target hot word language unit dictionary is triggered to ensure that the log hot word database corresponds to the user's actual query needs. When the target hot word language unit dictionary is created online, it can be created in real time for hot words in logs within a recent period (e.g., the last minute).

[0023] In this embodiment of the invention, the query attribute of the current log query statement can be determined first. If the current log query statement is a hot word query attribute, it indicates that the log needs to be retrieved through the target hot word language unit dictionary. Then, the current log query statement is matched with the elements in the target hot word language unit dictionary to obtain the hot word query bit vector.

[0024] Optionally, the filter conditions and query statements in the current log query statement can be extracted. Then, the filter conditions and query statements (i.e., the extracted data) can be split according to the granularity of the elements in the target hot word language unit dictionary. The extracted data can then be matched with the target hot word language unit dictionary. The positions of the elements in the target hot word language unit dictionary that successfully match the split results of the extracted data are marked as 1, and the positions of the elements that fail to match are marked as 0, thus obtaining the hot word query bit vector.

[0025] Step 130: Generate a log index bit vector based on the log to be queried and the target hot word language unit dictionary.

[0026] The log to be queried can be any log to be retrieved. The log index bit vector can be a vector determined by the positions of the matching elements in the dictionary of the target hot words and logs in the log to be queried.

[0027] In this embodiment of the invention, each log entry in the log to be queried can be split according to the granularity of the elements in the target hot word language unit dictionary. The splitting results of each log entry in the log to be queried can then be matched with the elements in the target hot word language unit dictionary. Based on the matching results of each log entry in the log to be queried and the elements in the target hot word language unit dictionary, a log index bit vector can be created.

[0028] Optionally, the corresponding positions of elements in the target hot word language unit dictionary that successfully match the log splitting results are marked as 1, and the corresponding positions of elements that fail to match are marked as 0, to obtain the matching results of a single log and the elements in the target hot word language unit dictionary. Then, according to the storage order of each log in the log to be queried, the marked values ​​of the corresponding positions of each log and the elements in the target hot word language unit dictionary are arranged to obtain the log index bit vector.

[0029] For example, suppose the log to be queried includes two logs. The log splitting result of the first log successfully matches the first and fourth elements in the target hot word language unit dictionary, but fails to match the second and third elements. The log splitting result of the second log successfully matches the first and second elements in the target hot word language unit dictionary, but fails to match the third and fourth elements. Then the log index bit vector is [[1,0,0,1],[1,1,0,0]].

[0030] Step 140: Based on the hot word query bit vector and the log index bit vector, determine the hit logs in the logs to be queried.

[0031] Among them, the hit logs can be logs in the logs to be queried that meet the query conditions in the current log query statement.

[0032] In this embodiment of the invention, the hot word query bit vector can be compared with the log index bit vector, and the log corresponding to the data part in the log index bit vector that is the same as the hot word query bit vector can be used as the hit log in the log to be queried.

[0033] The technical solution of this invention obtains the current log query statement. When the current log query statement has a hot word query attribute, it determines the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary. Then, it generates a log index bit vector based on the log to be queried and the target hot word language unit dictionary. Finally, based on the hot word query bit vector and the log index bit vector, it determines the hit logs in the log to be queried. This solution creates a target hot word language unit dictionary that meets the user's actual query needs by querying hot words, which can effectively control the amount of dictionary data. Then, it matches the target hot word language unit dictionary with the current log query statement and the log to be queried, and directly determines the hit logs in the log to be queried based on the secondary comparison of the matching results. This can significantly reduce the complexity of log query while ensuring query accuracy, and solves the problem that existing log queries cannot accurately and efficiently hit logs under low resource consumption. It can accurately and quickly hit the required logs under low resource consumption.

[0034] Example 2 Figure 2 This is a flowchart of a log query method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and provides a specific optional implementation method for creating a target hot word language unit dictionary before obtaining the current log query statement. Figure 2 As shown, the method includes: Step 210: Create the current log hot word library.

[0035] The current log hot keyword library can be a collection of log query keywords.

[0036] In this embodiment of the invention, a current log hot word library can be created based on whether there is a log historical query behavior and according to a preset hot word determination strategy.

[0037] In an optional embodiment of the present invention, creating the current log hot word library may include: when the historical query statement is not empty, parsing the historical query statement according to the syntax parser to determine fixed text words; cleaning the log stop words in the fixed text words to obtain the initial log hot word library; counting the number of times the words appear in the initial log hot word library according to the number of queries of the historical query statement, and determining the current log hot word library according to the number of times the words appear in the initial log hot word library and the second quantity truncation threshold.

[0038] The historical query statements can be query statements corresponding to a user's historical log query behavior. Log stop words can be pre-defined stop words in the internet log domain. For example, log stop words differ from general domain stop words; they do not include "from," "to," "in," "on," "at," "because," "no," and "not." Fixed text words can be text words that represent content related to the original log text in the log query statement. The initial log hot word library can be a hot word library constructed by removing log stop words from the fixed text words. The second quantity truncation threshold can be a pre-defined lower limit value for the word frequency in the current log hot word library.

[0039] In this embodiment of the invention, if the historical query statement is not empty, it indicates that there is a log query behavior. The historical query statement is then parsed based on the syntax parser, and fixed text words are extracted from the filtering conditions and parsed statements in the historical query statement. Log stop words in the fixed text words are then removed to obtain an initial log hot word library. Furthermore, based on the number of times the historical query statement is used, the number of times the words in the initial log hot word library appear is counted. The words in the initial log hot word library are then sorted by frequency according to the number of times the words appear. The current log hot word library is created based on the words in the initial log hot word library that are ranked first and are greater than or equal to the second truncation threshold.

[0040] In an optional embodiment of the present invention, creating the current log hot word library may further include: when the historical query statement is empty and the system log is not empty, clustering the system log to obtain a clustered log set; parsing the log template of the clustered log set based on a pattern learning algorithm to obtain fixed text words of the clustered log set; using the fixed text words of the clustered log set as the current log hot word library; or, when the system log is empty, determining the current log hot word library based on a target classification corpus or log analysis rules.

[0041] System logs can be a collection of files recording the operating system, hardware, software running status, and system events. Clustered log sets can be collections of logs obtained by clustering system logs. Pattern learning algorithms include, but are not limited to, Logmine, Drain, Logppt, and Adaparser. The target classification corpus can be publicly available corpora from the internet industry. For example, a target classification corpus can include, but is not limited to, Chinese text classification corpora for word frequency statistics in the computer field. Log analysis rules can be open-source rules for analyzing logs from different vendors on free code hosting platforms.

[0042] In this embodiment of the invention, if the historical query statement is found to be empty, but the system log is not empty, it indicates that there has been no log query behavior. The system log can be clustered to obtain a clustered log set. Then, based on the pattern learning algorithm, the log template of the clustered log set is parsed, that is, the fixed structure or field of the logs within the same set in the clustered log set is determined to obtain the fixed text words of the clustered log set. The fixed text words of the clustered log set are used as the current log hot word library. If the historical query statement is empty and the system log is empty, it indicates that the newly deployed system has no log access. The current log hot word library can be determined based on the target classification corpus and the empirical threshold of word frequency statistics. Alternatively, the log analysis rules can be regarded as historical query statements, and lexical parsing and extraction can be performed on the log analysis rules provided by different manufacturers to obtain the current log hot word library.

[0043] Optionally, the current log hot words library allows users to manually add and modify them.

[0044] Step 220: Determine the continuous language unit dictionary of the current log hot word library, and create the target hot word language unit dictionary based on the continuous language unit dictionary and the first quantity truncation threshold.

[0045] The continuous language unit dictionary can be an n-gram dictionary that divides each word in the current log hot word database into a preset number of language units. Optionally, since logs are usually composed of English words, based on the vowel characteristics of English words, continuous language units are two consecutive English letters in a word, and will be smaller than a complete word, provided it is a common vocabulary. The first truncation threshold can be a pre-set upper limit on the number of continuous language units in the target hot word language unit dictionary. The continuous language units in the target hot word language unit dictionary are the elements of the target hot word language unit dictionary.

[0046] In this embodiment of the invention, the continuous language units of each word in the current log hot word library can be divided based on the number of words included in the pre-set language unit to obtain a continuous language unit dictionary. Then, the continuous language units in the continuous language unit dictionary are filtered by a first quantity truncation threshold and the word frequency of the continuous language units in the continuous language unit dictionary to obtain a target hot word language unit dictionary.

[0047] In an optional embodiment of the present invention, creating a target hot word language unit dictionary based on a continuous language unit dictionary and a first quantity truncation threshold may include: sorting and indexing the continuous language units in the continuous language unit dictionary according to word frequency to obtain an initial hot word language unit dictionary; and determining the target hot word language unit dictionary based on the initial hot word language unit dictionary and the first quantity truncation threshold.

[0048] The initial hot word language unit dictionary can be an n-gram dictionary obtained by sorting and indexing the continuous language units in the continuous language unit dictionary according to word frequency.

[0049] In this embodiment of the invention, the continuous language units in the continuous language unit dictionary can be sorted according to word frequency, and an index can be assigned to the sorted continuous language units to obtain an initial hot word language unit dictionary. Then, the continuous language units in the initial hot word language unit dictionary can be truncated according to a first quantity truncation threshold to obtain a target hot word language unit dictionary.

[0050] Step 230: Obtain the current log query statement.

[0051] In an optional embodiment of the present invention, after obtaining the current log query statement, the method may further include: when the current log query statement is a time query attribute, determining the hit logs in the logs to be queried based on the time query range of the current log query statement; when the current log query statement is a universal identifier query attribute, determining the hit logs in the logs to be queried based on the UUID in the current log query statement.

[0052] The time query attribute can be a log query statement type that queries logs based on a time range. The time query range can be a time range for filtering logs. The universal identifier query attribute can be a log query statement type that queries logs based on a UUID (Universally Unique Identifier).

[0053] In this embodiment of the invention, if the current log query statement is a time-based query, the log to be queried can be retrieved through the time range of the current log query statement to obtain the hit log. If the current log query statement is a generic identifier query, a Bloom filter index is built based on the UUID in the current log query statement, and the log to be queried can be retrieved through the Bloom filter index to obtain the hit log.

[0054] UUID-based and time-range-based queries are common in internet log analysis, but the UUID and time range for each query are different, so frequency-based queries will never identify the hottest terms. Therefore, when log data is received, the earliest and latest times for each log segment are recorded independently to accelerate time-range filtering. Simultaneously, lexical analysis is used to identify high-cardinality UUIDs, and a separate Bloom filter index is built for each UUID, which is then used in conjunction with a dictionary of target hot-term language units for joint queries.

[0055] Step 240: When the current log query statement is a hot word query attribute, determine the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary.

[0056] Step 250: Generate a log index bit vector based on the log to be queried and the target hot word language unit dictionary.

[0057] In an optional embodiment of the present invention, generating a log index bit vector based on the log to be queried and the target hot word language unit dictionary may include: determining the continuous language unit of the log to be queried; matching the continuous language unit of the log to be queried with the target hot word language unit dictionary to obtain the continuous language unit matching result; and generating a log index bit vector based on the continuous language unit matching result.

[0058] The log continuous language unit can be a continuous language unit of words in the log. The continuous language unit matching result can be the comparison result of the log continuous language unit of the query log with the continuous language unit in the target hot word language unit dictionary.

[0059] In this embodiment of the invention, each word in the log to be queried can be divided into continuous language units according to the granularity of continuous language units in the target hot word language unit dictionary to obtain the log continuous language units of the log to be queried. Then, the log continuous language units of the log to be queried are matched with the continuous language units in the target hot word language unit dictionary to obtain the continuous language unit matching result. Based on the continuous language unit matching result, a log index bit vector is created.

[0060] Step 260: Based on the hot word query bit vector and the log index bit vector, determine the hit logs in the logs to be queried.

[0061] In a specific example, the log query method may include the following key steps: 1) Offline mining of hot keywords a1) Log query statements typically include filtering conditions, parsing statements, and statistical statements. Only the content in the filtering conditions and parsing statements that is relevant to the original log text needs to be included in the term frequency and popularity calculation. For example, a log query statement might be: The parser retrieves the relevant log text content as (exception OR out of memory) and at (?).<code_path\S+> ):\d+. Extract the fixed text phrase "exception out of memory at" from the relevant content of the original log. To make the hot words better reflect the query intent, clean the fixed text phrases and exclude some stop words. It should be noted that "from," "to," "in," "on," "at," "because," "no," and "not" are generally considered stop words in general fields, but they are very useful in internet logs. In the example above, after stop word filtering, the final result only includes the four words "exception out memory at."

[0062] a2) Based on the usage frequency of historical queries, aggregate the statistics for each word obtained in step a1 to obtain the total number of times each word has appeared in the query history. Sort the words according to their total occurrence frequency to obtain a list of common words, i.e., the current log hot word library. For example, the current log hot word library can be determined based on a second quantity truncation threshold, such as the word frequency threshold of common words. For example, the first quantity truncation threshold can be determined by calculating the inflection point of the word frequency histogram curve, calculating the abnormal quartiles of the word frequency array, or selecting the top 100 based on historical experience. Hot word statistics can be processed independently for different log types; however, for the sake of generalization and migration, it is calculated uniformly for all logs by default.

[0063] (b) Before new access logs have been queried, log templates for the new access logs are obtained through log clustering and pattern learning algorithms. The fixed text words in the log templates are used as a cold start vocabulary.

[0064] For example, the original text of a log is: Dec 11 00:46:29 localhost syslogd

[137] :Configuration Notice: ASL Module "com.apple.mkb" sharing output destination" / private / var / log / keybagd.log" with ASL Module "com.apple.mkb.internal".

[0065] The corresponding log template may be parsed by the algorithm as: <datetime> <hostorip> < >[ <number>]:Configuration Notice: ASL Module < > sharing output destination < with ASL Module < The obtained fixed text words are configuration notice asl module sharingoutput destination.

[0066] c) Publicly available corpora come from different sources and require different cleaning processes. For example, for Chinese text classification corpora used in the computer science field, common threshold calculations can be performed directly. However, log analysis rules open-sourced by different vendors on free code hosting platforms typically include regular expressions and query statements, which can be considered equivalent to "historical query statements." However, the query syntax varies slightly between vendors, requiring targeted lexical parsing and extraction. Furthermore, hot word libraries allow users to manually add and modify them.

[0067] Among them, steps a1 and a2 constitute one hot word determination strategy, step b is another hot word determination strategy, and step c is yet another hot word determination strategy.

[0068] 2) n-gram dictionary construction 2.1) Construct a continuous language unit dictionary for the current log hot word database. After sorting the continuous language unit dictionary by word frequency, assign corresponding indices. Assume the continuous language unit dictionary is a 2-gram dictionary. Continuous language units for different words may be the same; for example, both "exception" and "notice" include "ce". Therefore, it is necessary to globally sort the continuous language units of hot words in the current log hot word database by word frequency. For example, in the above example, the index number of "ce" would be higher. The final 2-gram dictionary will be much larger than the current log hot word database. For better cost-effectiveness, the continuous language units in the 2-gram dictionary are truncated according to a first truncation threshold, such as retaining only the top 100 continuous language units and indexing them to obtain the target hot word language units. For example, the final target hot word language unit dictionary can be {ce:0,vm:1,no:2,er:3...}.

[0069] 2.2) The log to be queried is also processed using n-grams to obtain continuous log language units. These continuous log language units are then compared with the target hot word language unit dictionary. For each position in the dictionary, a non-matching position is marked as 0, and a matching position is marked as 1. Assume the target hot word language unit dictionary is {ce:0,vm:1,no:2,er:3}. The original log to be queried is 2024-10-20 14:30 Error: vm123 memory overflow. The continuous log language units would then be 20,02,24,41,10...er,rr,ro,or,vm,m1...ve,er,rf... Since the target hot word language unit dictionary contains both vm and er that can be successfully matched, the log index bit vector would be [0,1,0,1]. If there are 100 log entries, a log index bit vector of length 100 would be generated: [[0,1,0,1],...].

[0070] 2.3) It should be noted that the logs to be queried are usually stored using segmented compression, such as compressing logs into 10MB, 10,000-logs, or 1-minute logs into a single compressed file. For each segment of logs, the index of consecutive language units in the n-gram dictionary is calculated. This calculation fully utilizes the parallelism of multi-core CPUs, and the disk I / O interface pressure is significantly reduced during query decompression.

[0071] The target hot word language unit dictionary is calculated from historical query statements mined offline, ensuring global consistency and eliminating the need for changes based on log data. Therefore, the target hot word language unit dictionary can be built online or independently offline after compressing and storing the original log text. For log segments where offline construction is not yet complete, queries should revert to a hard disk scan method.

[0072] Users can customize the time interval for offline mining of target hot word language unit dictionaries and trigger the offline construction process when the dictionary is updated to ensure that the target hot word language unit dictionary can correspond to the user's actual query business needs.

[0073] 3) Online query stage 3.1) After obtaining the current log query statement, use a syntax parser to extract fixed text words from the current log query statement and compare them with the target hot word language unit dictionary to generate a hot word query bit vector.

[0074] 3.2) Compare the hot word query bit vector and the log index bit vector. If neither the hot word query bit vector nor the current segment in the log index bit vector matches, skip the current segment without processing. If a match is found, decompress the log segment based on the successfully matched segment position to obtain the corresponding log text. Further processing of the query can then be performed, such as field extraction and field statistics.

[0075] This solution can filter only the hot keywords relevant to the actual query, achieving a better balance between storage cost and query efficiency. Offline hot keyword mining and asynchronous index building ensure that the log writing phase involves no processing other than compression, maximizing write performance. The offline index building process can also be scaled up or down as needed, with extremely low CPU costs. Testing shows that this solution achieves a very high index compression rate, requiring only 2-5% of the original log text space to build the corresponding target hot keyword language unit dictionary, covering over 75% of query hits.

[0076] The technical solution of this invention involves creating a current log hot word library to determine the continuous language unit dictionary of the current log hot word library, and creating a target hot word language unit dictionary based on the continuous language unit dictionary and a first quantity truncation threshold. Then, the current log query statement is obtained, and when the current log query statement has a hot word query attribute, a hot word query bit vector is determined based on the current log query statement and the target hot word language unit dictionary. A log index bit vector is generated based on the log to be queried and the target hot word language unit dictionary. Finally, based on the hot word query bit vector and the log index bit vector, the matching logs in the log to be queried are determined. This solution creates a target hot word language unit dictionary that meets the user's actual query needs by querying hot words. It controls the number of elements in the dictionary based on a first quantity truncation threshold, which can further compress the dictionary data volume. Then, it matches the target hot word language unit dictionary with the current log query statement and the log to be queried. Based on the secondary comparison of the matching results, it directly determines the hit logs in the logs to be queried. It can significantly reduce the complexity of log query while ensuring the accuracy of the query. It solves the problem that existing log queries cannot accurately and efficiently hit logs under the premise of low resource consumption. It can accurately and quickly hit the required logs under the premise of low resource consumption.

[0077] Example 3 Figure 3 This is a schematic diagram of a log query device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: The query statement acquisition module 310 is used to acquire the current log query statement; The query bit vector determination module 320 is used to determine the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary when the current log query statement is a hot word query attribute. The log index bit vector determination module 330 is used to generate a log index bit vector based on the log to be queried and the target hot word language unit dictionary; The log query module 340 is used to determine the hit logs in the logs to be queried based on the hot word query bit vector and the log index bit vector.

[0078] The technical solution of this invention obtains the current log query statement. When the current log query statement has a hot word query attribute, it determines the hot word query bit vector based on the current log query statement and the target hot word language unit dictionary. Then, it generates a log index bit vector based on the log to be queried and the target hot word language unit dictionary. Finally, based on the hot word query bit vector and the log index bit vector, it determines the hit logs in the log to be queried. This solution creates a target hot word language unit dictionary that meets the user's actual query needs by querying hot words, which can effectively control the amount of dictionary data. Then, it matches the target hot word language unit dictionary with the current log query statement and the log to be queried, and directly determines the hit logs in the log to be queried based on the secondary comparison of the matching results. This can significantly reduce the complexity of log query while ensuring query accuracy, and solves the problem that existing log queries cannot accurately and efficiently hit logs under low resource consumption. It can accurately and quickly hit the required logs under low resource consumption.

[0079] Optionally, the log query device further includes a hot word dictionary creation module, used to create a current log hot word library; determine the continuous language unit dictionary of the current log hot word library, and create a target hot word language unit dictionary based on the continuous language unit dictionary and a first quantity truncation threshold.

[0080] Optionally, the hot word dictionary creation module includes a hot word library creation unit, used to parse the historical query statement according to the syntax parser when the historical query statement is not empty, determine fixed text words; clean the log stop words in the fixed text words to obtain an initial log hot word library; count the number of times the words appear in the initial log hot word library according to the number of queries of the historical query statement, and determine the current log hot word library according to the number of times the words appear in the initial log hot word library and a second quantity truncation threshold.

[0081] Optionally, the hot word library creation unit is further configured to, when the historical query statement is empty and the system log is not empty, cluster the system log to obtain a clustered log set; based on a pattern learning algorithm, parse the log template of the clustered log set to obtain fixed text words of the clustered log set; use the fixed text words of the clustered log set as the current log hot word library; or, when the system log is empty, determine the current log hot word library based on a target classification corpus or log analysis rules.

[0082] Optionally, the hot word dictionary creation module further includes a hot word dictionary creation unit, used to sort and index the continuous language units in the continuous language unit dictionary according to word frequency to obtain an initial hot word language unit dictionary; and to determine the target hot word language unit dictionary based on the initial hot word language unit dictionary and the first quantity truncation threshold.

[0083] Optionally, the log query device further includes a special query module, used to determine the hit logs in the logs to be queried based on the time query range of the current log query statement when the current log query statement is a time query attribute; and to determine the hit logs in the logs to be queried based on the universally unique identifier (UUID) in the current log query statement when the current log query statement is a universally unique identifier (UUID) query attribute.

[0084] Optionally, the log index bit vector determination module 330 is used to determine the log continuous language unit of the log to be queried; match the log continuous language unit of the log to be queried with the target hot word language unit dictionary to obtain the continuous language unit matching result; and generate the log index bit vector based on the continuous language unit matching result.

[0085] The log query device provided in this embodiment of the invention can execute the log query method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] Example 4 Figure 4 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0087] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as ROM 12, RAM 13, etc., communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14. The ROM 12 is a read-only memory, the RAM 13 is a random access memory, and the I / O interface 15 is an input / output interface.

[0088] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the log query method.

[0090] In some embodiments, the log query method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log query method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the log query method by any other suitable means (e.g., by means of firmware).

[0091] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0095] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0096] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS servers, such as high management difficulty and weak business scalability.

[0097] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the log query method provided in any embodiment of this application. This program product shares the same inventive concept as the log query methods disclosed in the embodiments of this application, and therefore will not be described in detail here.

[0098] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / number> < / hostorip> < / datetime>

Claims

1. A log query method, characterized in that, include: Get the current log query statement; When the current log query statement is a hot word query attribute, the hot word query bit vector is determined based on the current log query statement and the target hot word language unit dictionary; Based on the log to be queried and the target hot word language unit dictionary, generate a log index bit vector; Based on the hot word query bit vector and the log index bit vector, the hit logs in the logs to be queried are determined.

2. The method according to claim 1, characterized in that, Before retrieving the current log query statement, the following is also included: Create a current log hot keyword database; Determine the continuous language unit dictionary of the current log hot word library, and create a target hot word language unit dictionary based on the continuous language unit dictionary and a first quantity truncation threshold.

3. The method according to claim 2, characterized in that, Create the current log hot word library, including: When the historical query statement is not empty, the historical query statement is parsed by the syntax parser to determine the fixed text words; Clean the log stop words from the fixed text words to obtain the initial log hot word library; Based on the number of queries in the historical query statements, the number of times words appear in the initial log hot word library is counted, and based on the number of times words appear in the initial log hot word library and the second quantity truncation threshold, the current log hot word library is determined.

4. The method according to claim 2, characterized in that, Creating the current log hot word library also includes: When the historical query statement is empty and the system log is not empty, the system log is clustered to obtain a clustered log set; based on a pattern learning algorithm, the log template of the clustered log set is parsed to obtain fixed text words of the clustered log set; the fixed text words of the clustered log set are used as the current log hot word library; or, When the system log is empty, the current log hot word library is determined based on the target classification corpus or log analysis rules.

5. The method according to claim 2, characterized in that, Based on the continuous language unit dictionary and the first quantity truncation threshold, a target hot word language unit dictionary is created, including: The continuous language units in the continuous language unit dictionary are sorted by word frequency and indexed to obtain the initial hot word language unit dictionary; The target hot word language unit dictionary is determined based on the initial hot word language unit dictionary and the first quantity truncation threshold.

6. The method according to claim 1, characterized in that, After retrieving the current log query statement, it also includes: When the current log query statement is a time query, the matching logs in the logs to be queried are determined according to the time query range of the current log query statement; When the current log query statement is a generic identifier query attribute, the matching log in the log to be queried is determined according to the generic unique identifier (UUID) in the current log query statement.

7. The method according to claim 1, characterized in that, Based on the log to be queried and the target hot word language unit dictionary, a log index bit vector is generated, including: Determine the log continuum language unit of the log to be queried; The continuous language units of the log to be queried are matched with the target hot word language unit dictionary to obtain the continuous language unit matching result; The log index bit vector is generated based on the continuous language unit matching results.

8. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the log query method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log query method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the log query method according to any one of claims 1-7.

Images