Log processing methods, log processing devices and electronic equipment

Abstract

This invention provides a log processing method, a log processing device, and an electronic device. The log processing method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; extracting features from the raw variables based on variable group sub-patterns to obtain storage units; and compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags, and the compressed storage units are used to perform searches on the raw logs. The log processing method of this invention can effectively improve the log compression rate and significantly reduce the latency during interactive log retrieval.

Description

Technical Field

[0001] This invention relates to the field of log processing technology, and in particular to a log processing method, log processing device, and electronic device. Background Technology

[0002] In modern cloud systems, a massive amount of log data is generated daily. This data is used for functions such as error diagnosis, user behavior modeling, system performance analysis, and early warning of potential security risks. As the system scales up, the corresponding log data volume also grows rapidly. In some cases, the size of system logs can even reach petabytes (PB) daily. These logs can be categorized into three types based on their access frequency: offline logs, near-line logs, and online logs. Offline logs are primarily used for archiving and, once stored in the system, cannot be retrieved in the future. Near-line logs are mainly used for querying, with a typical monthly access frequency between 0 and 100 times. Online logs are primarily used for monitoring system status and are accessed more frequently, typically nearly a thousand times per month. Statistics show that near-line logs constitute the largest proportion of the logs.

[0003] To reduce storage costs, a common approach is to compress these logs. For offline logs, only their storage overhead needs to be considered, so existing high-density log compression methods can be used (such as lzma, Logzip, LogReducer, etc.). For online logs, only their retrieval efficiency needs to be considered, so the raw logs can be stored directly and retrieved using existing efficient text retrieval tools (such as ElasticSearch, Splunk, etc.).

[0004] However, for nearline logs, which account for the largest proportion, offline compression methods cannot provide efficient retrieval functions, while using online retrieval tools directly cannot effectively save space. Therefore, how to better realize log storage has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a log processing method, log processing device, and electronic device to solve the shortcomings of existing technologies that cannot effectively achieve high-density compression and fast retrieval of logs, thereby improving the log compression rate and reducing the latency during interactive log retrieval.

[0006] This invention provides a log processing method, characterized in that the method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; extracting features from the raw variables based on variable group sub-patterns to obtain storage units; and compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags, and the compressed storage units are used to enable searching of the raw logs.

[0007] According to a log processing method provided by the present invention, the method further includes: determining a target compressed storage unit corresponding to the search keyword among a plurality of compressed storage units based on the search keyword and by means of the storage unit tag; determining a target element corresponding to the search keyword in the target compressed storage unit based on a fixed-length retrieval strategy; and reconstructing a target log corresponding to the target element based on the target element.

[0008] According to a log processing method provided by the present invention, the variable group sub-pattern is determined by the following method: determining the position of the original variable in the original log; dividing the original variables with the same position into original variable groups; and determining the variable group sub-pattern based on the original variable groups.

[0009] According to a log processing method provided by the present invention, the method further includes: dividing the original variable group into a dictionary variable group and a non-dictionary variable group based on the repetition rate of elements in the original variable group; determining the variable group sub-pattern based on the original variable group includes: determining a first variable group sub-pattern based on the dictionary variable group, wherein the first variable group sub-pattern is a variable group sub-pattern related to the dictionary variable group; and determining a second variable group sub-pattern based on the non-dictionary variable group, wherein the second variable group sub-pattern is a variable group sub-pattern related to the non-dictionary variable group.

[0010] According to a log processing method provided by the present invention, the step of determining a first variable group sub-pattern based on the dictionary variable group includes: dividing each dictionary item in the dictionary variable group into multiple dictionary sub-segments based on dictionary special characters; determining a segment type label for the dictionary sub-segments based on the character type of the dictionary sub-segments; determining a variable sub-pattern for the dictionary item based on the segment type label and the dictionary special characters; and obtaining the first variable group sub-pattern based on the variable sub-pattern of the dictionary item.

[0011] According to a log processing method provided by the present invention, the step of determining a second variable group sub-pattern based on the non-dictionary variable group includes: randomly selecting a first non-dictionary item in the non-dictionary variable group and determining whether a preset number of non-dictionary items include a non-dictionary special character at a preset position in the first non-dictionary item; if the preset number of non-dictionary items include the non-dictionary special character, then cutting each non-dictionary item in the non-dictionary variable group into non-dictionary sub-fragments based on the non-dictionary special character; if the preset number of non-dictionary items do not include the non-dictionary special character, then obtaining the longest common substring of any two non-dictionary items, and cutting each non-dictionary item in the non-dictionary variable group based on the longest common substring according to the preset number of non-dictionary items including the longest common substring to obtain non-dictionary sub-fragments; obtaining a tree structure about the non-dictionary variable group based on the non-dictionary sub-fragments, wherein the leaf nodes of the tree structure are the non-dictionary sub-fragments; performing the above-mentioned cutting operation on the leaf nodes, and determining the second variable group sub-pattern based on the cutting result.

[0012] According to a log processing method provided by the present invention, the step of determining the second variable group sub-pattern based on the cutting result includes: if the leaf node is cut, then determining the variable sub-pattern of the non-dictionary item based on the node type label, and obtaining the second variable group sub-pattern based on the variable sub-pattern of the non-dictionary item; if the leaf node is not cut, and each of the leaf nodes is the same constant, then taking the constant as the constant quantum part of the second variable group sub-pattern, and obtaining the second variable group sub-pattern based on the constant quantum part; if the leaf node is not divisible, and each of the leaf nodes is not the same constant, then determining the second variable group sub-pattern based on the type and length of the leaf node.

[0013] According to a log processing method provided by the present invention, the original variables include dictionary variables and non-dictionary variables. The step of extracting features from the original variables based on variable group sub-patterns to obtain storage units includes: if the original variable is a dictionary variable, obtaining the dictionary value and dictionary offset of the dictionary variable based on the first variable group sub-pattern, and using the dictionary value and dictionary offset as the storage unit; if the original variable is a non-dictionary variable, obtaining the sub-variables of the non-dictionary variable based on the second variable group sub-pattern, and using the sub-variables as the storage unit.

[0014] According to a log processing method provided by the present invention, the step of determining a target compressed storage unit corresponding to the search keyword among a plurality of compressed storage units based on the search keyword and through the storage unit tag includes: matching a plurality of compressed storage units through the search keyword based on a finite state automaton and the variable group sub-pattern, wherein the storage unit tag includes sub-pattern shape information and sub-pattern type information; and determining the target compressed storage unit among the plurality of compressed storage units based on the sub-pattern shape information and the sub-pattern type information.

[0015] According to a log processing method provided by the present invention, the step of determining the target element corresponding to the search keyword in the target compressed storage unit includes: retrieving multiple candidate target elements in the target compressed storage unit based on the search keyword, wherein a Bitmap records multiple candidate target elements, and the Bitmap is composed of a Bitmap and an Index; and determining the target element corresponding to the search keyword among the multiple candidate target elements based on the Bitmap.

[0016] According to a log processing method provided by the present invention, the method further includes: storing a plurality of candidate target elements in the Bitmap according to a fixed-length storage method; and determining the target element corresponding to the search keyword among the plurality of candidate target elements based on the Bitmap, including: determining the target element corresponding to the search keyword among the plurality of candidate target elements based on the Boyer-Moore fixed-length retrieval strategy and the Bitmap.

[0017] According to a log processing method provided by the present invention, the method further includes: returning the retrieved target element to the upper-layer application in real time, and returning a pointer array to the upper-layer application when reconstructing the target log. The present invention also provides a log processing apparatus, characterized in that the apparatus includes: an acquisition module for acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; an extraction module for extracting features from the raw variables based on variable group sub-patterns to obtain storage units; and a processing module for compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags, and the compressed storage units are used to perform searches on the raw logs.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the log processing methods described above.

[0019] The log processing method, log processing device, and electronic device provided by this invention extract features from the original variables based on variable grouping sub-patterns to obtain storage units, and then compress the storage units to obtain compressed storage units. In application, the original logs can be searched based on the compressed storage units, thereby effectively improving the log compression rate and significantly reducing latency during interactive log retrieval. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the log processing method provided by the present invention;

[0022] Figure 2 This is the second flowchart of the log processing method provided by the present invention;

[0023] Figure 3 This is one of the schematic diagrams of the search and reconstruction target log provided by the present invention;

[0024] Figure 4 This is the second schematic diagram of the search and reconstruction target log provided by the present invention;

[0025] Figure 5 This is one of the flowcharts illustrating the determination of a variable group sub-pattern provided by the present invention;

[0026] Figure 6 This is one of the flowcharts provided by the present invention for determining the sub-pattern of the first variable group based on dictionary variable groups;

[0027] Figure 7 This is a schematic diagram of determining the sub-pattern of the first variable group provided by the present invention;

[0028] Figure 8 This is one of the flowcharts provided by the present invention for determining the sub-pattern of the second variable group based on dictionary variable groups;

[0029] Figure 9 This is a schematic diagram of determining the sub-pattern of the second variable group provided by the present invention;

[0030] Figure 10 This is a schematic diagram of the log processing device provided by the present invention;

[0031] Figure 11This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0033] Figure 1 This is one of the flowcharts of the log processing method provided by the present invention.

[0034] In an exemplary embodiment of the present invention, such as Figure 1 As shown, the log processing method may include steps 110 to 130, which will be described in detail below.

[0035] In step 110, the raw log is obtained and divided into raw template and raw variables based on the log parsing tool.

[0036] In one embodiment, existing general strategies can be used, such as log parsing, to divide the raw log into raw templates and raw variables. It should be noted that this embodiment does not specifically limit the log parsing process; appropriate general strategies can be used to divide the raw log into raw templates and raw variables according to the actual situation. In this embodiment, dividing the raw log into raw templates and raw variables is mainly based on the inherent structure of the raw log to reduce storage overhead and can also be used to support the extraction of subsequent variable group sub-patterns.

[0037] In step 120, based on the variable group sub-pattern, features are extracted from the original variables to obtain storage units.

[0038] In step 130, the storage unit is compressed to obtain a compressed storage unit, wherein the compressed storage unit includes a storage unit label and is used to enable searching of the original log.

[0039] In one embodiment, based on variable grouping sub-patterns, the original variables can be represented in multiple different forms. During application, feature extraction is performed on the original variables based on the variable grouping sub-patterns, dividing the original variables into several smaller, independent storage units. When searching target logs, storage units that may contain the target search string can be determined based on the tags of each storage unit. In one example, storage units can also be compressed, thereby reducing storage overhead. Furthermore, since independent compression techniques are used when compressing storage units, independent decompression is supported during storage unit decompression, greatly reducing the workload of decompression and significantly improving retrieval speed. In this embodiment, the original variables are organized into independent storage units according to their respective characteristics, which improves the compression ratio and significantly reduces the search scope during retrieval, thus improving retrieval efficiency.

[0040] The log processing method provided by this invention extracts features from the original variables based on variable group sub-patterns to obtain storage units, and then compresses these storage units to obtain compressed storage units. In application, the original logs can be searched based on the compressed storage units, thereby effectively improving the log compression rate and significantly reducing latency during interactive log retrieval.

[0041] During application, target logs can be quickly searched based on the compressed storage unit. The following embodiments will illustrate another log processing method.

[0042] Figure 2 This is the second flowchart of the log processing method provided by the present invention.

[0043] In an exemplary embodiment of the present invention, such as Figure 2 As shown, the log processing method may include steps 210 to 260, wherein steps 210 to 230 are the same as steps 110 to 130 described above. For specific implementation methods and beneficial effects, please refer to the previous description. In this embodiment, they will not be repeated. Steps 240 to 260 will be described in detail below.

[0044] In step 240, based on the search keyword, the target compressed storage unit corresponding to the search keyword is determined from multiple compressed storage units by using storage unit tags.

[0045] In one embodiment, the target compressed storage unit corresponding to the search keyword can be determined from multiple compressed storage units based on search keywords, such as target retrieval strings, and using storage unit tags provided by the storage units. In another example, the compressed storage units can be parsed into memory, and the search engine can perform fast matching on the compressed storage units to determine the target compressed storage unit corresponding to the search keyword from multiple compressed storage units.

[0046] In another embodiment, storage unit labels can be used to narrow down the search to quickly locate the target compressed storage unit. In one example, variables that may contain the target string can be identified through matching on a template. When retrieving variables, the target compressed storage unit to be retrieved is determined by the shape and type of the subpattern. It should be noted that the original variable group can include dictionary variable groups and non-dictionary variable groups. During application, if the variable group subpattern belongs to a dictionary variable group, it can be checked whether it exists in the dictionary directory. If it does not exist, the decompression retrieval of dictionary items can be skipped. If the variable group subpattern belongs to a non-dictionary variable group, it can be checked whether the variable pattern matches. If it matches completely or does not match completely, the decompression retrieval of subvariables can be avoided, and compressed storage units that do not meet the conditions can be pruned according to the size and label of the compressed storage unit, thereby avoiding the overhead of decompression. At the log fragment item level, by combining with BITMAP, a fixed-length retrieval algorithm can be used to skip the retrieval of a portion of characters, thereby achieving the goal of pruning.

[0047] It should be noted that, as Figure 4 As shown, a BITMAP can consist of a Bmap that records whether corresponding positions in the log match and an Index array that identifies the positions marked as 1 in the Bmap. The Index can quickly indicate the matching position in the log, while the Bmap can directly determine whether a position matches through index operations. The combination of these two basic data structures can significantly improve the efficiency of fixed-length retrieval. The multi-step retrieval process can be divided into two processes: merging and pruning. For ease of explanation, the merging process will be combined with... Figure 4 The section corresponding to Figure a will be described, and the pruning process will be combined with... Figure 4 The part corresponding to Figure b in the middle is described.

[0048] During the merging process, Bmap can be used to directly determine whether a matching position in BITMAP2 (from left to right, corresponding to the second group of BITMAPs in Figure a) matches in BITMAP1 (from left to right, corresponding to the first group of BITMAPs in Figure a). If a match is found, the next matching position is checked. If no match is found, the corresponding position in Bmap1 is directly updated and added to the end of its index, thereby reducing the retrieval overhead at each step of the merging process.

[0049] During the pruning process, the corresponding positions of log fragments in the target compressed storage unit can be directly retrieved based on the Index information in BITMAP1 (only positions 1, 3, 5, and 7 need to be retrieved). If a match is found, the matching positions are directly updated to the Index from front to back (ensuring that subsequent data is not overwritten), and the corresponding Bmap information is updated, thereby reducing the retrieval overhead at each step of the pruning process.

[0050] In step 250, based on a fixed-length retrieval strategy, the target element corresponding to the search keyword is determined in the target compressed storage unit.

[0051] In one embodiment, the target element corresponding to the search keyword can be determined in the compressed target storage unit based on the fixed-length Boyer-Moore (BM) algorithm. It is understood that determining the target element based on a fixed-length retrieval strategy, besides allowing for fast location based on offsets, also allows the use of the faster BM algorithm instead of the KMP algorithm. In one example, if each line in the file is not fixed-length, a delimiter (such as \n) is needed to distinguish boundaries. Each character must be scanned to determine the delimiter's position, in which case only the faster KMP algorithm can be used for retrieval. With fixed-length retrieval, there is no need to worry about skipping multiple boundary positions, thus allowing the use of the more efficient BM algorithm. The fixed-length BM algorithm can skip multiple positions at once. It first matches whether the last character is the same. If they are different, a certain length of characters can be skipped, and matching can start directly from the next end position. Since the boundary positions can be calculated, each matching check determines whether a cross-line match has occurred. If a cross-line match occurs, it is considered a bad match, and the current match is canceled, and matching starts again from the next segment. After a successful match, the line number position can be calculated based on the offset and the fixed length, thus significantly improving retrieval efficiency.

[0052] In step 260, the target log corresponding to the target element is reconstructed based on the target element.

[0053] In one embodiment, a target element corresponding to the search keyword can be determined in at least one target compressed storage unit based on a fixed-length retrieval strategy. Furthermore, once the target element is determined, the target log corresponding to the target element can be identified and reconstructed. It is understood that the target log is the log corresponding to the search keyword. In this embodiment, storage units are extracted and independently compressed into compressed storage units. When retrieving target logs, the search string is analyzed, and the variable groups to be retrieved are locked according to the template. Furthermore, each variable group to be retrieved is matched on a sub-pattern, and the range of compressed storage units to be retrieved is continuously narrowed through search pruning to obtain the target compressed storage unit. In application, only the target compressed storage unit is decompressed, and a fixed-length strategy is used within the target compressed storage unit to complete fast matching, determine the log containing the target string, and complete reconstruction. This embodiment can effectively improve the log compression rate and significantly reduce the latency during interactive log retrieval.

[0054] To further illustrate the log processing method provided by this invention, the following embodiments will be used for explanation.

[0055] In another embodiment, metadata and extracted pattern data, such as compressed storage units, can be pre-parsed into memory before providing the retrieval service. During pattern matching, the main pattern provides log-level matching templates, and the sub-patterns provide variable-level matching templates.

[0056] In one example, pattern matching can be performed using a finite automatic state machine algorithm. During the main pattern matching, the pattern string and search string can be segmented into multiple fields based on special characters. Each field can be labeled with static characters, special characters, and variables, and the label of the feature value is recorded. During application, common substring matching can be performed in a forward-to-back order. If the labels of corresponding fields are the same, the label value of the pattern string contains the label value of the search string, and the corresponding positions of static characters and special characters match completely, then a successful match is recorded. This process may result in multiple successful matches, and a single match may require recording the starting position and length of matches for multiple fields. The matching results can be temporarily stored by creating a "result matrix."

[0057] In another example, when performing subpattern matching, the pattern string and search string can be segmented into multiple fields based on invariant characters (strings), and the matching process is the same as the main pattern. To pre-prune illegal matching positions, the matching can be categorized into "any position matching," "left-aligned matching," "right-aligned matching," and "full-word matching" based on the matching characteristics of the pattern string and search string. In one example, during matching, the first field in the search string can be considered a right-aligned match, the middle field can be considered a full-word match, and the last field must be a left-aligned match. This method allows for the rapid pruning of illegal matching branches.

[0058] Furthermore, based on the matching results of the main mode and sub-mode, the decompression target, i.e., the compressed storage unit, can be determined. Before decompressing the target compressed storage unit, it can be checked whether intermediate retrieval results can be obtained. If not, it is further checked whether the data to be decompressed already exists in the data cache pool. If it does not exist, the new target compressed storage unit is decompressed.

[0059] When decompressing the target data into storage, the decompressed data can be temporarily stored in a data cache pool and evicted using the LRU algorithm. Furthermore, metadata such as the BITMAP of each query result can also be temporarily stored. Understandably, since adjacent queries often have similarities or repetitions, caching technology can effectively reduce the overhead of repeated queries and decompression, thereby improving overall retrieval efficiency.

[0060] Understandably, variables within the compressed target storage unit are organized in a fixed-length or locally fixed-length manner, thus eliminating the need for delimiters between variables. During searching, the more efficient fixed-length Boyer-Moore (BM) algorithm can be used for string matching, utilizing offsets to calculate variable boundaries and matching line numbers. This allows the identification of the target element corresponding to the search keyword within the compressed target storage unit. In application, this fixed-length retrieval strategy effectively complements BITMAP for rapid location, significantly improving retrieval efficiency by directly calculating offsets instead of searching for matching positions.

[0061] It should be noted that since the BITMAP records the result of the previous match, this result can guide the decompression of the next target compressed storage unit. If the result of the previous BITMAP is not available, the compression of subsequent target compressed storage units can be avoided. If the result of the previous BITMAP is sparse, involving only a limited number of target compressed storage units, then only the corresponding target compressed storage units need to be decompressed, thus significantly reducing decompression overhead.

[0062] In one embodiment, when reconstructing the target log corresponding to the search keywords, although a small number of target compressed storage units are identified, all target compressed storage units involved in the target log need to be decompressed to reassemble the complete data. During application, due to the use of fixed-length storage, the corresponding location of the target compressed storage unit can be quickly located based on the BITMAP results, thereby enabling rapid acquisition of log fragment information. In one example, the target element can be quickly located based on the BITMAP results, and the target log can be reconstructed based on the target element.

[0063] In yet another embodiment, such as Figure 3 As shown, to provide a good retrieval experience for upper-layer applications, the retrieved partial results should be returned to the upper layer promptly after the retrieval, requiring a timely streaming return strategy. To accelerate reconstruction, the overhead of memory copying should be avoided. Therefore, for static log segments (e.g., common parts of logs with the same pattern), pointers are used to avoid copying. For dynamic log segments (e.g., different parts of the log), they can be copied into a contiguous array, with pointers pointing to the starting positions. During reconstruction, only the pointer array is returned to the upper-layer user, thereby reducing memory copying.

[0064] In one embodiment, a unified platform and service interface can also be established. During application, based on the distributed deployment of the program, each distributed node uses a scheduler to schedule the log storage and retrieval units in parallel. Furthermore, after completing a block of data tasks, the log storage and retrieval units return the results to the scheduler, which then returns them to the upper layer in a streaming manner.

[0065] The present invention will describe the process of determining the sub-pattern of variable groups in conjunction with the following embodiments.

[0066] Figure 5 This is one of the flowcharts illustrating the determination of variable group sub-patterns provided by the present invention.

[0067] In an exemplary embodiment of the present invention, such as Figure 5 As shown, determining the variable group sub-pattern may include steps 510 to 530, which will be described in detail below.

[0068] In step 510, the location of the original variable in the original log is determined.

[0069] In step 520, original variables with the same position are divided into original variable groups.

[0070] In step 530, a variable group sub-pattern is determined based on the original variable group.

[0071] In one embodiment, the location of the original variable in the original log can be determined, and variables appearing in the same template and at the same location are considered to be in the same variable group. It is understood that original variables with the same location are grouped into original variable groups. Further, based on the original variable groups, variable group sub-patterns are determined. In one example, sampling can be performed on the original variable groups, and training can be performed based on the sampling results to obtain variable group sub-patterns.

[0072] To further illustrate the process of determining the sub-pattern of the variable group based on the original variable group, the present invention will be described in conjunction with the following embodiments.

[0073] In an exemplary embodiment of the present invention, the log processing method may further include dividing the original variable group into dictionary variable groups and non-dictionary variable groups based on the repetition rate of elements within the original variable group. In one example, the original variable group is divided into dictionary variable groups and non-dictionary variable groups according to the repetition rate of elements within the group. In one example, the repetition rate can be obtained by subtracting the number of unique elements from the total number of elements and then dividing by the total number of elements. In application, original variable groups with a repetition rate higher than 50% can be referred to as dictionary variable groups; original variable groups with a repetition rate lower than 50% can be referred to as non-dictionary variable groups.

[0074] Furthermore, different variable group sub-pattern extraction strategies can be applied to different types of original variable groups (dictionary variable groups and non-dictionary variable groups).

[0075] In one embodiment, a first variable group sub-pattern can be determined based on a dictionary variable group, wherein the first variable group sub-pattern is a variable group sub-pattern relating to the dictionary variable group. Alternatively, a second variable group sub-pattern can be determined based on a non-dictionary variable group, wherein the second variable group sub-pattern is a variable group sub-pattern relating to the non-dictionary variable group.

[0076] The process of determining the first variable group sub-pattern based on the dictionary variable group will be described below with reference to the following embodiments.

[0077] Figure 6 This is one of the flowcharts provided by the present invention for determining the sub-pattern of the first variable group based on dictionary variable groups.

[0078] In an exemplary embodiment of the present invention, such as Figure 6 As shown, determining the first variable group sub-pattern based on the dictionary variable group may include steps 610 to 640, which will be described in detail below.

[0079] In step 610, based on dictionary special characters, each dictionary item in the dictionary variable group is divided into multiple dictionary sub-fragments.

[0080] In step 620, the fragment type label of the dictionary fragment is determined based on the character type of the dictionary fragment.

[0081] In step 630, the variable subpattern of the dictionary item is determined based on the fragment type label and dictionary special characters.

[0082] In step 640, the first variable group subpattern is obtained based on the variable subpattern of the dictionary entry.

[0083] In one embodiment, each dictionary entry can be segmented into multiple dictionary sub-segments based on dictionary special characters, such as non-alphanumeric characters. The type label of each dictionary sub-segment is then obtained based on the character types it contains. The type label of each dictionary sub-segment is concatenated with the special string used for segmentation to obtain a variable sub-pattern for the current dictionary entry. Furthermore, dictionary entries with the same pattern are merged to obtain several variable sub-patterns for the entire dictionary variable group. It is understood that several variable sub-patterns can constitute a first variable group sub-pattern. In one example, dictionary entries with the same variable sub-pattern can be stored together for later lookup. Each variable sub-pattern is recorded, noting the maximum length and number of dictionary entries corresponding to each variable sub-pattern.

[0084] In another embodiment, all unique elements in the original variable set can be extracted to form a dictionary. Each dictionary entry is then segmented according to a special character to obtain the variable sub-pattern corresponding to that entry. Merging identical variable sub-patterns results in a variable sub-pattern group for the entire dictionary, i.e., the first variable group sub-pattern. During application, for each variable sub-pattern in the first variable group sub-pattern, the dictionary value of the corresponding dictionary entry and its maximum length can be recorded. Furthermore, the dictionary variable set can be reordered, and dictionary entries belonging to the same pattern can be stored contiguously. The offset corresponding to each element in the dictionary variable set is calculated to obtain an offset array, and its maximum length is recorded.

[0085] The process of determining the first variable group sub-pattern will be described below with reference to the following embodiments.

[0086] Figure 7 This is a schematic diagram of determining the sub-pattern of the first variable group provided by the present invention.

[0087] In one embodiment of the present invention, such as Figure 7 As shown, the process of determining the first variable group sub-pattern based on the dictionary variable group can include six nodes, namely nodes 710 to 760. The process of determining the first variable group sub-pattern will be described below.

[0088] Node 710 is a dictionary variable group. First, by extracting all unique values ​​from it, we obtain node 720. For each dictionary item in node 720, we extract its own variable subpattern. For example, for "ERR#404", we first use the special character "#" to split it into two parts, "ERR" and "404". Thus, the variable subpattern for the current dictionary item is "". <sv1> # <sv2>This yields the group consisting of all patterns (corresponding to node 730). Merging identical variable sub-patterns in node 730, if a variable is found to be identical in all instances of the current pattern, then that variable becomes a constant within the pattern. This yields the two patterns "ERR#" of the dictionary group variable.<typ=1> "and "SUCC" (corresponding to node 740). Instances of all dictionary items belonging to the same pattern are stored contiguously together, resulting in node 750. Furthermore, the number of items corresponding to each pattern needs to be recorded, for example, "ERR#".<typ=1> This corresponds to two instances, with the longest instance having 7 characters. Finally, each dictionary entry corresponds to an offset within the dictionary. Each element in the original dictionary variable group is converted into an offset, resulting in an offset array with a maximum length of 1, as shown in node 760. As described above, the first variable group sub-pattern of this embodiment can be extracted and represented as: ERR#<typ=1> (cnt=2, len=7); SUCC(cnt=1, len=4); IdxLen=1. In this embodiment, the first variable group sub-pattern is obtained by extracting the original dictionary variable group, and the dictionary and dictionary offset are obtained based on the first variable group sub-pattern. Subsequent searches can directly jump to the corresponding position in the dictionary based on the first variable group sub-pattern, and the dictionary offset is only decompressed when the search is found in the dictionary, thereby reducing the search latency.

[0089] The process of determining the second variable group sub-pattern based on the non-dictionary variable group will be described below with reference to the following embodiments.

[0090] Figure 8 This is one of the flowcharts provided by the present invention for determining the sub-pattern of the second variable group based on dictionary variable groups.

[0091] In an exemplary embodiment of the present invention, such as Figure 8 As shown, determining the second variable group sub-pattern based on the dictionary variable group may include steps 810 to 850, which will be described in detail below.

[0092] In step 810, the first non-dictionary item in the non-dictionary variable group is randomly selected, and it is determined whether a preset number of non-dictionary items include non-dictionary special characters at preset positions in the first non-dictionary item.

[0093] In step 820, if a preset number of non-dictionary items include non-dictionary special characters, then each non-dictionary item in the non-dictionary variable group is segmented into non-dictionary sub-fragments based on the non-dictionary special characters.

[0094] In step 830, if the preset number of non-dictionary items does not include non-dictionary special characters, the longest common substring of any two non-dictionary items is obtained, and based on the preset number of non-dictionary items including the longest common substring, each non-dictionary item in the non-dictionary variable group is segmented based on the longest common substring to obtain non-dictionary sub-segments.

[0095] It is understood that the preset number of non-dictionary items can be adjusted according to actual circumstances. In this invention, the preset number of non-dictionary items is not specifically limited. The preset position can also be adjusted according to actual circumstances. In this invention, the preset position is not specifically limited.

[0096] In step 840, a tree structure for the non-dictionary variable group is obtained based on the non-dictionary sub-fragments, wherein the leaf nodes of the tree structure are non-dictionary sub-fragments.

[0097] In step 850, the above-described cutting operation is performed on the leaf nodes, and the second variable group sub-pattern is determined based on the cutting result.

[0098] In one embodiment, the second variable group subpattern is determined based on the cutting results as follows: If the leaf nodes are cut, the non-dictionary item variable subpatterns are determined based on the node type labels, and the second variable group subpattern is obtained based on the non-dictionary item variable subpatterns. If the leaf nodes are not cut and all leaf nodes are the same constant, the constant is used as the constant quantum part of the second variable group subpattern, and the second variable group subpattern is obtained based on the constant quantum part. If the leaf nodes are not divisible and all leaf nodes are not the same constant, the second variable group subpattern is determined based on the type and length of the leaf nodes.

[0099] In one embodiment, a non-dictionary variable can be randomly selected from the current non-dictionary variable group. It is then determined whether its first non-alphanumeric character (from left to right) is simultaneously contained in at least 90% of the other non-dictionary variables. If so, the current non-dictionary variable group is divided into three parts based on this character. In one example, all parts before this character form one variable group, the parts after this character form another variable group, and the character itself forms a third group. If no matching character is found, another non-dictionary variable is selected from the non-dictionary variable group. The longest common substring between the two non-dictionary variables is calculated, and it is determined whether this substring exists in 90% of the non-dictionary variables. If so, this substring is used to segment the non-dictionary variable group. Furthermore, the non-dictionary variable group can form a tree structure through the segmentation of special characters and common substrings. The above operation is performed on each leaf node of the current tree structure. If three consecutive segmentation attempts fail, the current leaf node is considered indivisible. If all leaf nodes are indivisible, each leaf node is analyzed. If all the variables within a node are the same, then the node constitutes a constant; otherwise, it constitutes a sub-variable group. The type and maximum length of the sub-variable group are recorded to obtain a label. By concatenating all the constants and labels, the second variable group sub-pattern of the non-dictionary variable group can be obtained.

[0100] The process of determining the second variable group sub-pattern will be described below with reference to the following embodiments.

[0101] Figure 9 This is a schematic diagram of determining the sub-pattern of the second variable group provided by the present invention.

[0102] In one embodiment of the present invention, such as Figure 9 As shown, the process of determining the first variable group sub-pattern based on the dictionary variable group can include six nodes, namely nodes 910 to 960. The process of determining the second variable group sub-pattern will be described below.

[0103] Node 910 corresponds to the original non-dictionary variable group. First, a random element "block_1F81F" is selected. Each special character "_" is examined. If this character is included in elements with a% or higher (where a is a threshold, for example, a can be 70%), then it is considered suitable for segmenting the variable group. The non-dictionary variable group is then segmented. Specifically, the part before the "_" is extracted to form one variable group (corresponding to node 920), and the part after the "_" is extracted to form another variable group (corresponding to node 930). For the group corresponding to node 930, the longest common substring "F8" can be calculated by randomly selecting two strings. Based on "F8", the non-dictionary variable group is divided into three parts. These three parts include the non-dictionary sub-segment before the common substring (corresponding to node 940), the non-dictionary sub-segment after the common substring (corresponding to node 960), and the common substring itself (corresponding to node 950). At this point, all leaf nodes are indivisible, and the algorithm terminates. All elements within nodes 920 and 950 are identical, forming the constant part; the elements within nodes 940 are calculated to obtain their corresponding type information (node ​​940 corresponds to numbers, node 960 corresponds to hexadecimal numbers) and maximum length. This forms a similar structure.<typ=1,len=1> The label is linked to the sub-template. As described above, the second variable group sub-pattern of this embodiment can be extracted and can be represented as: block_<typ=1,len=1> F8<typ=5,len=4> In this embodiment, the original non-dictionary variable group is extracted into several sub-variable groups. Subsequent searches only need to decompress those sub-variable groups that may contain the target string, thus reducing search latency.

[0104] The present invention will describe the process of extracting features from original variables to obtain storage units in conjunction with the following embodiments.

[0105] In an exemplary embodiment of the present invention, the original variables may include dictionary variables and non-dictionary variables. Based on the variable group sub-pattern, feature extraction of the original variables to obtain storage units can be implemented in the following manner.

[0106] If the original variable is a dictionary variable, then the dictionary value and dictionary offset of the dictionary variable are obtained based on the first variable group sub-pattern, and the dictionary value and dictionary offset are used as storage units. If the original variable is a non-dictionary variable, then the sub-variables of the non-dictionary variable are obtained based on the second variable group sub-pattern, and the sub-variables are used as storage units.

[0107] In one embodiment, after extracting the sub-patterns of the first variable group, unique elements in the dictionary variable group can be extracted to form dictionary values, and dictionary items belonging to the same template are stored contiguously for convenient subsequent retrieval. Furthermore, the dictionary offset of each element in the dictionary variable group is calculated based on the dictionary values, and the dictionary values ​​and dictionary offsets are set and compressed independently to obtain storage units.

[0108] In another embodiment, after completing the extraction of the sub-pattern of the second variable group, each element in the non-dictionary variable group can be extracted according to the sub-pattern of the second variable group, and the part belonging to the sub-variable can be extracted to form a separate sub-variable group. Each sub-variable group can be independently compressed into a storage unit.

[0109] In this embodiment, the original variable group is divided into several independent storage units based on the variable group sub-pattern. During retrieval, the storage units that may contain the target search string can be determined based on the tags of each storage unit. Because independent compression technology is used, these storage units can support independent decompression, greatly reducing the amount of decompression and significantly improving retrieval speed.

[0110] In an exemplary embodiment of the present invention, determining the target compressed storage unit corresponding to the search keyword among multiple compressed storage units based on the search keyword and by using storage unit tags can be achieved in the following way:

[0111] Based on finite state automata and variable group subpatterns, multiple compressed storage units are matched by searching keywords. The storage unit labels include subpattern shape information and subpattern type information. Based on the subpattern shape and type information, the target compressed storage unit is determined from among the multiple compressed storage units.

[0112] In one embodiment, a finite state automaton can be used to match subpatterns of a variable group to determine variables that may contain the target string. When retrieving variables, the shape and type of the subpattern can be used to determine the target compressed storage unit to be retrieved.

[0113] In an exemplary embodiment of the present invention, determining the target element corresponding to the search keyword in the target compressed storage unit may include the following:

[0114] Based on the search keywords, multiple candidate target elements are retrieved from the compressed target storage unit. Among them, a Bitmap records multiple candidate target elements and is composed of a Bitmap and an Index. Based on the Bitmap, the target element corresponding to the search keywords is determined from the multiple candidate target elements.

[0115] In one embodiment, during the retrieval of the target element in the compressed target storage unit, a Bitmap composed of a Bmap and an Index can be used to record the retrieval results of the storage unit (including multiple candidate target elements). Subsequent retrievals will be carried out based on this, and further, the target element will be determined from the multiple candidate target elements. In this embodiment, the Bitmap supports efficient merging and pruning operations, thereby significantly reducing the overhead of fixed-length retrieval.

[0116] In an exemplary embodiment of the present invention, the log processing method further includes storing multiple candidate target elements into a Bitmap according to a fixed-length storage method. Further, the target element corresponding to the search keyword can be determined from the multiple candidate target elements based on the Boyer-Moore fixed-length retrieval strategy and the Bitmap.

[0117] In one embodiment, fixed-length storage can be used during the retrieval process in the storage unit, eliminating the need to worry about skipping multiple boundary positions and enabling the use of the more efficient Boyer-Moore (BM) fixed-length retrieval algorithm. The BM algorithm can skip multiple positions at once. It first matches whether the last character is the same; if not, it skips a certain length of characters and starts matching directly from the next end position. Since the boundary positions can be calculated, it checks whether a cross-line match has occurred during each match. If a cross-line match occurs, it is considered a bad match, and the match is canceled, starting again from the next segment. After a successful match, the line number position can be calculated based on the offset and the fixed length, significantly improving retrieval efficiency.

[0118] In an exemplary embodiment of the present invention, the log processing method may further include returning the retrieved target element to the upper-layer application in real time, and returning the pointer array to the upper-layer application when reconstructing the target log.

[0119] In one embodiment, after retrieving the target element, the retrieved partial results should be returned to the upper layer promptly, i.e., a timely streaming return strategy is adopted. Simultaneously, to accelerate reconstruction and minimize memory copy overhead, only the pointer array is returned to the upper-layer user during actual reconstruction, significantly reducing the latency of retrieval and reconstruction. As described above, the log processing method provided by this invention, based on a variable group sub-pattern, extracts features from the original variables to obtain storage units, and then compresses these storage units to obtain compressed storage units. In application, the original logs can be searched based on the compressed storage units, thereby effectively improving the log compression rate and significantly reducing the latency during interactive log retrieval.

[0120] Based on the same concept, the present invention also provides a log processing device.

[0121] The log processing apparatus provided by the present invention is described below. The log processing apparatus described below and the log processing method described above can be referred to in correspondence.

[0122] Figure 10 This is a schematic diagram of the log processing device provided by the present invention.

[0123] In an exemplary embodiment of the present invention, such as Figure 10 As shown, the log processing device may include an acquisition module 1010, an extraction module 1020, and a processing module 1030. Each module will be described in detail below.

[0124] The acquisition module 1010 can be configured to: acquire raw logs and divide the raw logs into raw templates and raw variables based on a log parsing tool.

[0125] The extraction module 1020 can be configured to: extract features from the original variables based on the variable group sub-pattern to obtain storage units.

[0126] The processing module 1030 can be configured to: compress the storage unit to obtain a compressed storage unit, wherein the compressed storage unit includes a storage unit label and is used to perform a search on the original log.

[0127] In an exemplary embodiment of the present invention, the log processing apparatus may further include a retrieval module, wherein the retrieval module may be configured to: determine a target compressed storage unit corresponding to the search keyword among a plurality of compressed storage units based on the search keyword and by means of storage unit tags; determine a target element corresponding to the search keyword in the target compressed storage unit based on a fixed-length retrieval strategy; and reconstruct a target log corresponding to the target element based on the target element.

[0128] In an exemplary embodiment of the present invention, the extraction module 1020 may determine the variable group sub-pattern in the following manner: determine the location of the original variable in the original log; divide the original variables with the same location into original variable groups; and determine the variable group sub-pattern based on the original variable groups.

[0129] In an exemplary embodiment of the present invention, the log processing apparatus may further include a partitioning module, wherein the partitioning module may be configured to: partition the original variable group into a dictionary variable group and a non-dictionary variable group based on the repetition degree of elements in the original variable group. The extraction module 1020 may determine variable group sub-patterns based on the original variable group in the following manner: determining a first variable group sub-pattern based on the dictionary variable group, wherein the first variable group sub-pattern is a variable group sub-pattern relating to the dictionary variable group; and determining a second variable group sub-pattern based on the non-dictionary variable group, wherein the second variable group sub-pattern is a variable group sub-pattern relating to the non-dictionary variable group.

[0130] In an exemplary embodiment of the present invention, the extraction module 1020 may determine the first variable group sub-pattern based on the dictionary variable group in the following manner: based on dictionary special characters, each dictionary item in the dictionary variable group is divided into multiple dictionary sub-segments; based on the character type of the dictionary sub-segments, the segment type label of the dictionary sub-segments is determined; based on the segment type label and dictionary special characters, the variable sub-pattern of the dictionary item is determined; based on the variable sub-pattern of the dictionary item, the first variable group sub-pattern is obtained.

[0131] In an exemplary embodiment of the present invention, the extraction module 1020 may determine the second variable group sub-pattern based on the non-dictionary variable group in the following manner: randomly select the first non-dictionary item in the non-dictionary variable group, and determine whether a preset number of non-dictionary items include a non-dictionary special character at a preset position in the first non-dictionary item; if the preset number of non-dictionary items include a non-dictionary special character, then cut each non-dictionary item in the non-dictionary variable group into non-dictionary sub-fragments based on the non-dictionary special character; if the preset number of non-dictionary items do not include a non-dictionary special character, then obtain the longest common substring of any two non-dictionary items, and cut each non-dictionary item in the non-dictionary variable group based on the longest common substring according to the preset number of non-dictionary items including the longest common substring to obtain non-dictionary sub-fragments; based on the non-dictionary sub-fragments, obtain a tree structure about the non-dictionary variable group, wherein the leaf nodes of the tree structure are non-dictionary sub-fragments; perform the above-mentioned cutting operation on the leaf nodes, and determine the second variable group sub-pattern based on the cutting result.

[0132] In an exemplary embodiment of the present invention, the extraction module 1020 may determine the second variable group sub-pattern based on the cutting result in the following manner: if the leaf node is cut, then the variable sub-pattern of the non-dictionary item is determined based on the node type label, and the second variable group sub-pattern is obtained based on the variable sub-pattern of the non-dictionary item; if the leaf node is not cut and each leaf node is the same constant, then the constant is taken as the constant quantum part of the second variable group sub-pattern, and the second variable group sub-pattern is obtained based on the constant quantum part; if the leaf node is not divisible and each leaf node is not the same constant, then the second variable group sub-pattern is determined based on the type and length of the leaf node.

[0133] In an exemplary embodiment of the present invention, the original variables include dictionary variables and non-dictionary variables. Based on the variable group sub-pattern, the extraction module 1020 can extract features from the original variables in the following manner to obtain storage units: if the original variable is a dictionary variable, the dictionary value and dictionary offset of the dictionary variable are obtained based on the first variable group sub-pattern, and the dictionary value and dictionary offset are used as storage units; if the original variable is a non-dictionary variable, the sub-variables of the non-dictionary variable are obtained based on the second variable group sub-pattern, and the sub-variables are used as storage units.

[0134] In an exemplary embodiment of the present invention, the retrieval module may determine the target compressed storage unit corresponding to the search keyword from multiple compressed storage units based on the search keyword and through storage unit tags in the following manner: based on a finite state automaton and variable group sub-patterns, multiple compressed storage units are matched through the search keyword, wherein the storage unit tag includes sub-pattern shape information and sub-pattern type information; the target compressed storage unit is determined from multiple compressed storage units based on the sub-pattern shape information and sub-pattern type information.

[0135] In an exemplary embodiment of the present invention, the retrieval module may determine the target element corresponding to the search keyword in the target compressed storage unit in the following manner: based on the search keyword, multiple candidate target elements are retrieved in the target compressed storage unit, wherein a Bitmap records multiple candidate target elements, and the Bitmap is composed of a Bmap and an Index; based on the Bitmap, the target element corresponding to the search keyword is determined from the multiple candidate target elements.

[0136] In an exemplary embodiment of the present invention, the log processing apparatus may further include a storage module, wherein the storage module may be configured to: store multiple candidate target elements into a Bitmap according to a fixed-length storage method. The retrieval module may determine the target element corresponding to the search keyword from the multiple candidate target elements based on the Bitmap in the following manner: determining the target element corresponding to the search keyword from the multiple candidate target elements based on a Boyer-Moore fixed-length retrieval strategy and the Bitmap.

[0137] In an exemplary embodiment of the present invention, the retrieval module may also be configured to: return the retrieved target element to the upper-layer application in real time, and return the pointer array to the upper-layer application when reconstructing the target log.

[0138] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140. The processor 1110, communications interface 1120, and memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a log processing method. This method includes: acquiring the original log and dividing the original log into original templates and original variables based on a log parsing tool; extracting features from the original variables based on variable group sub-patterns to obtain storage units; and compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags, and the compressed storage units are used to search the original logs.

[0139] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log processing method provided by the above methods. The method includes: obtaining the original log and dividing the original log into original templates and original variables based on a log parsing tool; extracting features from the original variables based on variable group sub-patterns to obtain storage units; and compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags and are used to perform searches on the original logs.

[0141] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the log processing methods provided by the above methods. The method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; extracting features from the raw variables based on variable group sub-patterns to obtain storage units; and compressing the storage units to obtain compressed storage units, wherein the compressed storage units include storage unit tags, and the compressed storage units are used to perform searches on the raw logs.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. < / sv1>

Claims

1. A log processing method, characterized in that, The method includes: Obtain the raw logs and divide them into raw templates and raw variables based on the log parsing tool; Based on the variable group sub-pattern, feature extraction is performed on the original variables to obtain storage units; The storage unit is compressed to obtain a compressed storage unit, wherein the compressed storage unit includes a storage unit label, and the compressed storage unit is used to search the original log. The method further includes: dividing the original variable group into a dictionary variable group and a non-dictionary variable group based on the repetition degree of elements in the original variable group. The variable group sub-pattern is determined in the following way: Determine the location of the original variable in the original log; The original variables with the same position are divided into groups of original variables; Based on the dictionary variable group, a first variable group sub-pattern is determined, wherein the first variable group sub-pattern is a variable group sub-pattern related to the dictionary variable group; Randomly select the first non-dictionary item in the non-dictionary variable group, and determine whether a preset number of non-dictionary items include non-dictionary special characters at preset positions in the first non-dictionary item; If the preset number of non-dictionary items includes the non-dictionary special characters, then each non-dictionary item in the non-dictionary variable group is cut into non-dictionary sub-fragments based on the non-dictionary special characters; If the preset number of non-dictionary items does not include the non-dictionary special characters, then the longest common substring of any two non-dictionary items is obtained, and based on the preset number of non-dictionary items including the longest common substring, each non-dictionary item in the non-dictionary variable group is cut based on the longest common substring to obtain non-dictionary sub-fragments. Based on the non-dictionary sub-fragments, a tree structure for the non-dictionary variable group is obtained, wherein the leaf nodes of the tree structure are the non-dictionary sub-fragments; Perform the above cutting operation on the leaf nodes, and determine the second variable group sub-pattern based on the cutting result.

2. The log processing method according to claim 1, characterized in that, The method further includes: Based on the search keywords, the target compressed storage unit corresponding to the search keywords is determined from among the multiple compressed storage units using the storage unit tags; Based on a fixed-length retrieval strategy, the target element corresponding to the search keyword is determined in the target compressed storage unit; Based on the target element, the target log corresponding to the target element is reconstructed.

3. The log processing method according to claim 1, characterized in that, The step of determining the first variable group sub-pattern based on the dictionary variable group includes: Based on special characters in the dictionary, each dictionary item in the dictionary variable group is divided into multiple dictionary sub-fragments; Based on the character type of the dictionary sub-fragment, determine the fragment type label of the dictionary sub-fragment; Based on the fragment type label and the dictionary special characters, determine the variable sub-pattern of the dictionary item; Based on the variable subpatterns of the dictionary entries, the first variable group subpattern is obtained.

4. The log processing method according to claim 1, characterized in that, The step of determining the second variable group sub-pattern based on the segmentation results includes: If the leaf node is slicable, then based on the node type label, determine the variable sub-pattern of the non-dictionary item, and based on the variable sub-pattern of the non-dictionary item, obtain the second variable group sub-pattern; If the leaf nodes cannot be cut and each leaf node is the same constant, then the constant is taken as the constant quantum part of the second variable group sub-pattern, and the second variable group sub-pattern is obtained based on the constant quantum part. If the leaf nodes are indivisible and each leaf node is not the same constant, then the second variable group sub-pattern is determined based on the type and length of the leaf nodes.

5. The log processing method according to claim 1, characterized in that, The original variables include dictionary variables and non-dictionary variables. The process of extracting features from the original variables based on variable group sub-patterns to obtain storage units includes: If the original variable is the dictionary variable, then the dictionary value and dictionary offset of the dictionary variable are obtained based on the first variable group sub-pattern, and the dictionary value and dictionary offset are used as the storage unit; If the original variable is a non-dictionary variable, then the sub-variable of the non-dictionary variable is obtained based on the second variable group sub-pattern, and the sub-variable is used as the storage unit.

6. The log processing method according to claim 2, characterized in that, The step of determining the target compressed storage unit corresponding to the search keyword from among multiple compressed storage units based on the search keyword and the storage unit tag includes: Based on the finite state automaton and the variable group sub-pattern, multiple compressed storage units are matched using the search keywords, wherein the storage unit label includes sub-pattern shape information and sub-pattern type information; Based on the sub-pattern shape information and the sub-pattern type information, the target compressed storage unit is determined among the plurality of compressed storage units.

7. The log processing method according to claim 2 or 6, characterized in that, The step of determining the target element corresponding to the search keyword in the target compressed storage unit includes: Based on the search keywords, multiple candidate target elements are retrieved in the target compressed storage unit, wherein a Bitmap records multiple candidate target elements, and the Bitmap is composed of a Bmap and an Index; Based on the Bitmap, the target element corresponding to the search keyword is determined from among the multiple candidate target elements.

8. The log processing method according to claim 7, characterized in that, The method further includes: According to the fixed-length storage method, multiple candidate target elements are stored in the Bitmap; The step of determining the target element corresponding to the search keyword from a plurality of candidate target elements based on the Bitmap includes: Based on the Boyer-Moore fixed-length retrieval strategy and the Bitmap, the target element corresponding to the search keyword is determined from a plurality of candidate target elements.

9. The log processing method according to claim 8, characterized in that, The method further includes: The retrieved target element is returned to the upper-layer application in real time, and the pointer array is returned to the upper-layer application when the target log is reconstructed.

10. A log processing device, characterized in that, The apparatus is used to implement the log processing method according to any one of claims 1 to 9, the apparatus comprising: The acquisition module is used to acquire the raw logs and divide the raw logs into raw templates and raw variables based on the log parsing tool; The extraction module is used to extract features from the original variables based on the variable group sub-pattern to obtain storage units; A processing module is used to compress the storage unit to obtain a compressed storage unit, wherein the compressed storage unit includes a storage unit label, and the compressed storage unit is used to perform a search on the original log.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the log processing method as described in any one of claims 1 to 9.

Images