A log processing method, apparatus and electronic device
By obtaining the position and frequency information of characters in the logs, building a frequent pattern tree, and extracting log templates, the problem of low log processing efficiency is solved, and more efficient and accurate log processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2021-12-14
- Publication Date
- 2026-06-09
AI Technical Summary
Log processing is inefficient, and existing technologies mainly rely on manual rule matching, resulting in low efficiency.
By obtaining the position and frequency information of each character in the log, a target frequent pattern tree is built, the target log template is extracted, and the log to be processed is processed using the template.
It improves the efficiency and accuracy of log processing, supports incremental learning, reduces computing resources, and enhances processing performance.
Smart Images
Figure CN116303875B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network transmission technology, and in particular to a log processing method, apparatus and electronic device. Background Technology
[0002] The complexity of log data presents significant difficulties and challenges for log processing and feature extraction. Traditionally, operations and maintenance personnel have relied on manually writing rules to filter and check for anomalies in log data, and then processing logs using regular expressions to match keywords—a method that is inefficient. Summary of the Invention
[0003] This application provides a log processing method, apparatus, and electronic device to solve the problem of low log processing efficiency.
[0004] In a first aspect, embodiments of this application provide a log processing method, including:
[0005] Obtain parameter information for each character in the log set, the parameter information including the character's position information and the frequency information corresponding to the position information;
[0006] A target frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0007] Obtain the target log template based on the target frequent pattern tree;
[0008] The target log template is used to process the logs to be processed.
[0009] Secondly, embodiments of this application also provide a log processing apparatus, including:
[0010] The first acquisition module is used to acquire parameter information for each character in the log set. The parameter information includes the character's position information and the frequency information corresponding to the position information.
[0011] A module is established to build a target frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0012] The second acquisition module is used to acquire the target log template based on the target frequent pattern tree;
[0013] The processing module is used to process the logs to be processed using the target log template.
[0014] Thirdly, embodiments of this application also provide an electronic device, including: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the method described in the first aspect of embodiments of this application.
[0015] Fourthly, embodiments of this application also provide a readable storage medium storing a program that, when executed by a processor, implements the steps of the method described in the first aspect of embodiments of this application.
[0016] In this embodiment of the application, a target frequent pattern tree is established based on the position information of each character in the log set and the frequency information corresponding to the position information. That is, each node in the target frequent pattern tree can correspond to a character and the position information of the character. A target log template is extracted based on the target frequent pattern tree, and the target log template is used to process the log to be processed, thereby improving the efficiency of log processing. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of this application, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a schematic flowchart of a log processing method provided in an embodiment of this application;
[0019] Figure 2 This is one of the schematic diagrams of a frequent pattern tree provided in the embodiments of this application;
[0020] Figure 3 This is a schematic diagram of frequent pattern tree node splitting provided in an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of frequent pattern tree node merging provided in an embodiment of this application;
[0022] Figure 5 This is a schematic diagram of a log cluster provided in an embodiment of this application;
[0023] Figure 6 This is a second schematic diagram of a frequent pattern tree provided in an embodiment of this application;
[0024] Figure 7 This is a schematic diagram of frequent pattern tree pruning provided in an embodiment of this application;
[0025] Figure 8 This is a third schematic diagram of a frequent pattern tree provided in an embodiment of this application;
[0026] Figure 9 This is a schematic diagram of the structure of a log processing device provided in an embodiment of this application;
[0027] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0029] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.
[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating a log processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, it includes the following steps:
[0031] Step 101: Obtain parameter information for each character in the log set. The parameter information includes the character's position information and the frequency information corresponding to the position information.
[0032] The aforementioned log set can be obtained in advance. For example, multiple historical logs for a specific application scenario can be obtained in advance, or historical logs from multiple scenarios can be obtained to adapt to different scenarios.
[0033] The characters in the logs mentioned above can be letters, words, or numbers, excluding punctuation marks. The position information of these characters indicates their location within the log. For example, if the log set contains the message "GET / v2 / 54fadb412c4e40cdbaed9335e4c35a9e / servers / detail HTTP / 1.1"status:200len:1893time:0.2477829", the character "GET" is the first character in the log (position 1), "v2" is the second character (position 2), and so on. If the log set also contains the message "GET / openstack / 2013-10-17 / user_data", then the position of the character "GET" in this log is the first character (position 1), the position of the character "v2" is the second character (position 2), and so on. The HTTP / 1.1 "status:404len:176time:0.0010660" output corresponds to the character "GET" appearing twice at position 1, once at position 8, and once at position 10. This means each character has a corresponding frequency at its position. If a character never appears at a certain position, its frequency is 0 (meaning there is no frequency corresponding to that position). Therefore, the parameter information for each character in the log set can include the frequency information of the same character appearing at different positions.
[0034] Step 102: Using the position information of each character and the frequency information corresponding to the position information, establish a target frequent pattern tree. In the target frequent pattern tree, for characters with at least two position information, there are at least two nodes, and each node corresponds to one position information.
[0035] It is understandable that a node in the above frequent pattern tree can correspond to a character. Since the above target frequent pattern tree is built using the position information of each character and the frequency information corresponding to the position information, the characters corresponding to different nodes in the above target frequent pattern tree can be the same. When building the frequent pattern tree, it is first necessary to traverse the logs in the log set to obtain the frequency of each character at the corresponding position. Then, when building the frequent pattern tree according to the logs one by one, the nodes are constructed in descending order according to the frequency of the characters at the corresponding positions in the logs. For example, when the same character has corresponding frequency information in different positions, different nodes will be generated during the construction of the above frequent pattern tree. Taking the character "servers" in the log set as having a frequency of 3 times at position 4 and a frequency of 2 times at position 5 as an example, during the construction of the above frequent pattern tree, the character "servers" with a frequency of 3 times at position 4 can correspond to node 1, and the character "servers" with a frequency of 2 times at position 5 can correspond to node 2. It can be understood that the character corresponding to node 1 and node 2 is "servers", but the position information of the character corresponding to node 1 and the position information of the character corresponding to node 2 are different.
[0036] Step 103: Obtain the target log template based on the target frequent pattern tree.
[0037] It can be understood that each branch in the above target frequent pattern tree corresponds to a log template. By extracting the characters corresponding to each node in each branch, a log template can be formed. Furthermore, the above target frequent pattern tree includes the position information of the characters corresponding to each node. Therefore, based on the above target frequent pattern tree, the characters and character order contained in each branch can be quickly obtained to form a log template.
[0038] Step 104: Process the log to be processed using the target log template.
[0039] The aforementioned processing of logs to be processed may include matching the logs to be processed with the aforementioned log template to determine the log template corresponding to the logs to be processed. The process of matching logs to be processed with log templates can be understood as a process of classifying logs to be processed. By using the log templates to process the logs to be processed, the category of the logs to be processed can be determined, and corresponding processing can be carried out.
[0040] In this embodiment of the application, a target frequent pattern tree is established based on the position information of each character in the log set and the frequency information corresponding to the position information. That is, each node in the target frequent pattern tree can correspond to a character and the position information of the character. A target log template is extracted based on the target frequent pattern tree, and the target log template is used to process the log to be processed, thereby improving the efficiency of log processing.
[0041] Furthermore, in the target frequent pattern tree used to extract the log template, characters with at least two position information include at least two nodes, each node corresponding to one position information. This ensures that in the target log template obtained based on the target frequent pattern tree, the position of the character corresponding to each node in the target log template can be directly obtained based on the target frequent pattern tree, thereby improving the accuracy of the log processing.
[0042] Furthermore, the target frequent pattern tree supports incremental learning. That is, after the target frequent pattern tree is established, it can be updated based on newly added logs. For example, after processing the log to be processed using the log template, the character parameter information in the log to be processed can be directly added to the target frequent pattern tree. This enables rapid updating of the target frequent pattern tree and the corresponding extracted log template, reducing computation time and computational resources, and improving log processing efficiency.
[0043] Optionally, the step 102, which involves using the position information of each character and the frequency information corresponding to that position information to establish a target frequent pattern tree, may specifically include the following steps:
[0044] A first frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the first frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0045] Obtain the first node in the first frequent pattern tree. The first node is located in the first branch and the second branch. The previous node of the first node is located in the first branch, the second branch and the third branch. The second branch and the third branch include nodes with the same position information corresponding to the same character.
[0046] The first node is split into a second node located in the first branch and a third node located in the second branch;
[0047] The nodes with the same position information corresponding to the same character in the second branch and the third branch are merged to obtain the target frequent pattern tree.
[0048] The aforementioned first frequent pattern tree can be understood as a tree built based on the Frequent Pattern Tree (FP-Tree) algorithm, using the position information of each character in the log set and the frequency information corresponding to that position information. It can be understood that in the process of building the first frequent pattern tree, each log entry in the log set is inserted sequentially, and each character is inserted in descending order of frequency during each log insertion. Therefore, the same character at the same position in different logs may correspond to different nodes in the first frequent pattern tree, resulting in a large number of branches in the first frequent pattern tree. For example, each branch in the first frequent pattern tree can extract a log template, such as... Figure 2 As shown, under the node containing the character "GET", there are two branches, both containing the characters "V2", "5a9e", "servers", and "detail". This means that both branches include nodes corresponding to the same character ("V2", "5a9e", "servers", or "detail") at the same location. The two branches could belong to the same log template, but because the character "200" appears frequently in multiple branches and is closer to the root node in the frequent pattern tree, it appears on different branches. Taking the first node as an example, where the character "200" is located... Figure 3 As shown, the character "200" can be divided into nodes located in two branches (both corresponding to the character "200"), and then the nodes corresponding to the same character in the two branches can be merged (e.g., Figure 4 As shown), the node positions of the frequent pattern tree are adjusted according to the parameter information of the character corresponding to each node.
[0049] It is understandable that the nodes in the first frequent pattern tree can be split and merged multiple times until there are no more nodes in the first frequent pattern tree that can be split and merged.
[0050] In this embodiment, when the first node exists in the first frequent pattern tree, by splitting the first node into a second node located in the first branch and a third node located in the second branch, and merging the nodes in the second branch and the third branch that correspond to the same position information of the same character, the number of branches in the frequent pattern tree that can actually be represented as the same log template can be reduced, thereby improving the extraction accuracy of the log template.
[0051] Optionally, the step 102, which involves using the position information of each character and the frequency information corresponding to that position information to establish a target frequent pattern tree, may specifically include the following steps:
[0052] A second frequent pattern tree is constructed using the position information of each character and the frequency information corresponding to the position information. In the second frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0053] Obtain the fourth node in the second frequent pattern tree, wherein the fourth node includes multiple leaf nodes and the position information of the characters corresponding to the multiple leaf nodes is consistent;
[0054] The multiple leaf nodes are pruned to obtain the target frequent pattern tree.
[0055] The fact that the position information of the characters corresponding to the multiple leaf nodes is consistent can be understood as the fact that the characters corresponding to the multiple leaf nodes are different, but the position information of each character is consistent. That is, the character that appears at a certain position in the log is not fixed and has a low frequency, which can be identified as a log parameter.
[0056] The aforementioned second frequent pattern tree can be understood as a tree built based on the FP-Tree algorithm, using the position information of each character in the log set and the frequency information corresponding to the position information. It is understandable that, during the process of building the first frequent pattern tree, certain characters in the logs of the aforementioned log set are log parameters and should not appear as characters in the log template. Taking the log "POST / v2 / e9746973ac574c6b8a9e8857f56a7608 / os-server-external-events HTTP / 1.1"status:200len:380time:0.1043310" as an example, the character "0.1043310" is understood to be a log parameter. If characters with high frequency are determined as log templates, a threshold needs to be set in advance. The frequency information of each character is compared with the set threshold to determine whether the character is a log parameter. Furthermore, setting the threshold needs to consider the data volume of the log set and the application scenario. The setting of the threshold can also easily affect the extraction efficiency of the log template and the accuracy of the logs to be processed when using the log template.
[0057] In this embodiment, the characters corresponding to the leaf nodes in the second frequent pattern tree are the characters with the lowest frequency of occurrence in the log. By determining whether the position information of the characters corresponding to multiple leaf nodes of the fourth node is consistent, if they are consistent, it can be determined that the character at that position is always changing, and it can be identified as a log parameter. This allows the corresponding multiple leaf nodes to be pruned from the frequent pattern tree, thereby improving the efficiency of log template extraction. Furthermore, it avoids pre-setting thresholds to judge the parameter information of the nodes, ensuring the accuracy of pruning while also reducing the amount of data processing.
[0058] Furthermore, before obtaining the fourth node in the second frequent pattern tree, the nodes of the second frequent pattern tree can also be split and merged as described above.
[0059] Optionally, the step 103 of obtaining the target log template based on the target frequent pattern tree may specifically include the following steps:
[0060] Obtain the character and parameter information corresponding to each node in the target frequent pattern tree;
[0061] The target log template is obtained by filling in the preset log template based on the character and parameter information corresponding to each node.
[0062] In this embodiment, each branch in the target frequent pattern tree corresponds to a log template, and each node includes corresponding characters and parameter information, that is, the corresponding characters and the position information of the characters. Thus, the characters corresponding to each branch node can be filled into the corresponding positions of the preset log template according to the position information of the characters, thereby realizing the extraction of the target log template.
[0063] Optionally, before obtaining the parameter information of each character in the log set in step 101, the method may further include the following steps:
[0064] Obtain multiple log entries, along with the punctuation information for those log entries;
[0065] Based on the punctuation information of the multiple log entries, the multiple log entries are classified to obtain the log set, wherein the punctuation information of each log entry in the log set is consistent.
[0066] The punctuation information mentioned above refers to the punctuation marks and their corresponding positions in each log entry. By grouping logs with consistent punctuation information into the same log set, the punctuation information of each log entry in the unified log set is consistent, and the number of characters is also consistent, which facilitates the acquisition and matching of the log template.
[0067] It is understood that the multiple logs obtained above can be divided into multiple log sets based on the punctuation information, so as to achieve pre-classification of multiple logs. That is, each log set can be used to build a frequent pattern tree, thereby improving the accuracy of the frequent pattern tree.
[0068] In this embodiment, before obtaining the parameter information of each character in the log set, the logs are classified based on the punctuation information of the logs to obtain the log set. The punctuation information of each log in the log set is consistent, so that the logs can be pre-classified before building the frequent pattern tree.
[0069] Optionally, the preset log template is determined based on the punctuation information of each log entry in the log set;
[0070] The process of filling a preset log template with the character and parameter information corresponding to each node to obtain the target log template includes:
[0071] Based on the character and parameter information corresponding to each node, the target position of the preset log template is filled with the character corresponding to each node.
[0072] The target log template is obtained by filling in the positions in the preset log template except for the target position with preset characters.
[0073] It is understood that the character and parameter information corresponding to each node includes the character's position information. When filling the preset log template with each character, the target position can be determined based on the character's position information, and each character can be filled according to the position information. Furthermore, the length of the preset log template is predetermined. After filling the target position of the preset log template with the characters corresponding to each node, the positions of the unfilled characters in the preset log template are the positions of the log parameters. Preset characters can be used to fill these unfilled positions. When processing the log to be processed using the target log template, the position of the preset characters indicates that the character at that position in the log to be processed can be arbitrary, thereby achieving the extraction of the target log template.
[0074] It is understandable that the above log template includes characters and punctuation marks. If there are many punctuation marks in the log, the granularity of the extracted log template can be refined by including the punctuation marks in the log template, thereby reducing the loss of useful parameter information.
[0075] The various optional implementation methods described in the embodiments of this application can be combined with each other or implemented individually without conflict. The embodiments of this application do not limit this.
[0076] For ease of understanding, the specific implementation method is as follows:
[0077] This application provides a method for mining network device operation log templates, which may specifically include the following steps:
[0078] Step 1: Iterate through all the logs in the given set, extract the "log signature" for each log, and classify each log into the corresponding log cluster based on the "log signature".
[0079] It is understandable that massive network device logs consist of a fixed template section and a variable parameter section. Regardless of how the log parameters change, the relative positions of punctuation marks remain unchanged. Each type of log template has the same punctuation mark pattern. This application uses the punctuation marks in the logs as the "log signature". Figure 5 As shown, punctuation marks are first extracted from each log entry and combined into a "log signature" using a specific method. Different log entries are then grouped into different log clusters based on their signatures. For example, the log signature of the original log "GET / v2 / 54fadb412c4e40cdbaed9335e4c35a9e / servers / detail HTTP / 1.1"status:200len:1893time:0.2477829" is "- / - / - / - / - / -"-:--:--:- (where "-" is used to replace characters in the log entry). Logs with the same signature are thus grouped into the same log cluster.
[0080] Among them, by mining and extracting "log signatures", the extracted "log signatures" can be used as the first-level log classification, which can initially classify logs with different punctuation patterns; logs with the same log pattern contain the same number of words (tokens), and each word will have position information (if all the tokens contained in the log can be formed into a list, the position information of the token is the index position of the token in the list). Combining the position information of each token, the logs in the log cluster can be further classified.
[0081] Step 2: Referring to the Frequent Pattern-Growth (FP-Growth) algorithm, for the logs in each log cluster in Step 1, first perform a traversal, and count the frequency of each token appearing at each position during the traversal.
[0082] For example, if "GET" appears twice and "DELETE" appears three times at position 1, it can be represented as "position 1: [GET, 2], [DELETE, 3]".
[0083] When counting the frequency of each token at each location, only the frequency of the token appearing at the corresponding location is counted. By introducing the location information dimension, it is possible to more accurately assess whether the token is a log template at that location.
[0084] Step 3: For each log cluster, perform a second traversal. First, sort the tokens in each log entry according to the frequency counted in Step 2. Then, insert the sorted list of log tokens into the frequent pattern tree under that log cluster, with each token corresponding to a node in the tree. During insertion, record the frequency of each tree node. For example, if the original logs are:
[0085] "GET,v2,54fadb412c4e40cdbaed9335e4c35a9e,servers,detail,HTTP,1.1,status,200,len,1893,time,0.2648301";
[0086] Based on the results obtained in step 2, sorting each token according to the frequency of its appearance in the corresponding position yields the sorted log list:
[0087] "HTTP,1.1,status,len,time,GET,54fadb412c4e40cdbaed9335e4c35a9e,servers,200,detail,1893,0.2648301";
[0088] Finally, the sorted logs are inserted sequentially into the frequent item pattern tree under the current log cluster. Each token corresponds to a tree node, and the tree node has two attributes: position information and frequency information. Figure 2 The frequent item pattern tree shown shows that if the same token appears in two different branches (e.g., ... Figure 2 In V2), the tokens are interconnected via pointers. Each class of log signature corresponds to a log cluster, and each log cluster corresponds to a frequent item pattern tree.
[0089] Step 4: Starting from the leaf node, traverse upwards. If, in different branches under a certain node in the tree, there are multiple consecutive (greater than or equal to 2) identical nodes (i.e., ... Figure 3 If nodes are connected by dashed lines, then the branches containing these identical nodes will be merged.
[0090] In log mining using frequent itemsets, a branch of a tree represents a log pattern. However, since a single node may appear in multiple branches simultaneously (i.e., ... Figure 4The "200" node in the tree appears frequently and is located closer to the root node, resulting in multiple essentially identical branches (the branches connected by dashed lines). These branches should actually belong to the same template. To eliminate the branch explosion caused by this node (i.e., the "200" node), it is necessary to separate the node and then merge multiple branches of the same type, such as... Figure 4 As shown, the merging process is as follows:
[0091] First, node separation is performed: When traversing to the "GET" node, it is found that the "200" node causes multiple consecutive identical nodes in its two branches, so the node separation process is initiated. For example... Figure 4 As shown, the node “200” is first split and its frequency in each branch is counted. The frequency of the split “200” node is equal to the sum of the frequencies of its next node in the tree, that is, the frequency of “latest” and “V2”.
[0092] Then, branch merging is performed: first, the two branches are separated from the original tree; then, the sum of the frequencies of multiple consecutive identical nodes in the two branches is calculated; then, the other nodes in the two branches are merged into a new branch in descending order; finally, the new branch is inserted into the original tree.
[0093] Step 5: Prune the frequent pattern tree after merging branches, removing log parameter nodes. For example... Figure 7 As shown, starting from the leaf node, traverse upwards. If a node has multiple leaf nodes (greater than or equal to 2) and the leaf nodes are in the same position, then prune the tree by removing all leaf nodes in the same position under that node. The pruned frequent pattern tree is as follows: Figure 8 As shown.
[0094] Step 6: Starting from the root node, traverse downwards, collecting the tokens left after pruning. These tokens form the log template. Finally, the collected tokens are assembled with the log signature of the log cluster to form the log template for that log cluster. The pruned positions in the log template are represented by "*".
[0095] Taking the log template from one of the branches as an example, the extracted log template is as follows:
[0096] "GET / v2 / 54fadb412c4e40cdbaed9335e4c35a9e / servers / detail HTTP / 1.1" status:*len:*time:*
[0097] The log template obtained directly using the FT-Tree algorithm is as follows:
[0098] "GET / v2 / 54fadb412c4e40cdbaed9335e4c35a9e / servers / detail HTTP / 1.1" status:503len:323time:0.0080531
[0099] "GET*HTTP / 1.1"status:200len:*time:*
[0100] It is understandable that the log templates extracted by the running log template mining method provided in this application are more in line with the actual situation.
[0101] In this embodiment, by assembling log punctuation marks into a string as a log signature in step 1, logs with the same signature are grouped into the same log cluster, meaning logs in the same cluster have the same number of tokens, facilitating log template extraction. Furthermore, by combining location information with the frequency of token occurrences, a frequent pattern tree is established, resulting in more accurate node frequency information. In step 4, the frequent pattern tree is merged, meaning multiple branches with the same log template can be merged, reducing the possibility of inclusion or conflict between log templates and improving the accuracy of log template extraction. In step 5, the frequent pattern tree can be pruned based on the location information of each node, eliminating the need to pre-set multiple thresholds and further improving the accuracy of log template extraction.
[0102] See Figure 9 , Figure 9 This is a schematic diagram of the structure of a log processing device provided in an embodiment of this application. Figure 9 As shown, the log processing device 900 includes:
[0103] The first acquisition module 901 is used to acquire parameter information for each character in the log set. The parameter information includes the character's position information and the frequency information corresponding to the position information.
[0104] The module 902 is used to establish a target frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0105] The second acquisition module 903 is used to acquire the target log template based on the target frequent pattern tree;
[0106] The processing module 904 is used to process the log to be processed using the target log template.
[0107] Optionally, the establishment module 902 may specifically include:
[0108] The first establishment unit is used to establish a first frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the first frequent pattern tree, for characters with at least two position information, there are at least two nodes, and each node corresponds to one position information.
[0109] The first acquisition unit is used to acquire the first node in the first frequent pattern tree, wherein the first node is located in the first branch and the second branch, the previous node of the first node is located in the first branch, the second branch and the third branch, and the second branch and the third branch include nodes with the same position information corresponding to the same character.
[0110] A splitting unit is used to split the first node into a second node located in the first branch and a third node located in the second branch;
[0111] The merging unit is used to merge nodes with the same position information corresponding to the same character in the second branch and the third branch to obtain the target frequent pattern tree.
[0112] Optionally, the establishment module 902 may specifically include:
[0113] The second establishment unit is used to establish a second frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the second frequent pattern tree, for characters with at least two position information, there are at least two nodes, and each node corresponds to one position information.
[0114] The second acquisition unit is used to acquire the fourth node in the second frequent pattern tree, wherein the fourth node includes multiple leaf nodes and the position information of the characters corresponding to the multiple leaf nodes is consistent.
[0115] The pruning unit is used to prune the multiple leaf nodes to obtain the target frequent pattern tree.
[0116] Optionally, the second acquisition module 903 may specifically include:
[0117] The third acquisition unit is used to acquire the character and parameter information corresponding to each node in the target frequent pattern tree;
[0118] The filling unit is used to fill a preset log template based on the character and parameter information corresponding to each node to obtain the target log template.
[0119] Optionally, the log processing device 900 may also include:
[0120] The third acquisition module is used to acquire multiple log entries and the punctuation information of the multiple log entries;
[0121] The classification module is used to classify the multiple logs based on their punctuation information to obtain the log set, wherein the punctuation information of each log in the log set is consistent.
[0122] Optionally, the preset log template is determined based on the punctuation information of each log entry in the log set;
[0123] The filling unit may specifically include:
[0124] The first filling subunit is used to fill the target position of the preset log template with the characters corresponding to each node based on the character and parameter information corresponding to each node.
[0125] The second filling subunit is used to fill the positions in the preset log template other than the target position with preset characters to obtain the target log template.
[0126] The log processing device 900 can implement the embodiments of this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.
[0127] This application also provides an electronic device. Because the principle by which the electronic device solves the problem is similar to that in the embodiments of this application... Figure 1 The log processing method shown is similar; therefore, the implementation of this electronic device can be found in the method implementation section, and repeated details will not be elaborated upon. For example... Figure 10 As shown, the electronic device according to an embodiment of this application includes: a processor 1000, configured to read a program from a memory 1020 and execute the following processes:
[0128] Obtain parameter information for each character in the log set, the parameter information including the character's position information and the frequency information corresponding to the position information;
[0129] A target frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0130] Obtain the target log template based on the target frequent pattern tree;
[0131] The target log template is used to process the logs to be processed;
[0132] Transceiver 1010 is used to receive and send data under the control of processor 1000.
[0133] Among them, Figure 10 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 1000) and memory (memory 1020). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1010 may be multiple elements, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. The processor 1000 is responsible for managing the bus architecture and general processing, and the memory 1020 may store data used by the processor 1000 during operation.
[0134] Optionally, the step of using the position information of each character and the frequency information corresponding to the position information to build a target frequent pattern tree may include:
[0135] A first frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the first frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0136] Obtain the first node in the first frequent pattern tree. The first node is located in the first branch and the second branch. The previous node of the first node is located in the first branch, the second branch and the third branch. The second branch and the third branch include nodes with the same position information corresponding to the same character.
[0137] The first node is split into a second node located in the first branch and a third node located in the second branch;
[0138] The nodes with the same position information corresponding to the same character in the second branch and the third branch are merged to obtain the target frequent pattern tree.
[0139] Optionally, the step of using the position information of each character and the frequency information corresponding to the position information to build a target frequent pattern tree may include:
[0140] A second frequent pattern tree is constructed using the position information of each character and the frequency information corresponding to the position information. In the second frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information.
[0141] Obtain the fourth node in the second frequent pattern tree, wherein the fourth node includes multiple leaf nodes and the position information of the characters corresponding to the multiple leaf nodes is consistent;
[0142] The multiple leaf nodes are pruned to obtain the target frequent pattern tree.
[0143] Optionally, obtaining the target log template based on the target frequent pattern tree may include:
[0144] Obtain the character and parameter information corresponding to each node in the target frequent pattern tree;
[0145] The target log template is obtained by filling in the preset log template based on the character and parameter information corresponding to each node.
[0146] Optionally, the processor 1000 is also used to read the program from the memory 1020 and perform the following steps:
[0147] Obtain multiple log entries, along with the punctuation information for those log entries;
[0148] Based on the punctuation information of the multiple log entries, the multiple log entries are classified to obtain the log set, wherein the punctuation information of each log entry in the log set is consistent.
[0149] Optionally, the preset log template is determined based on the punctuation information of each log entry in the log set;
[0150] The process of filling a preset log template with the character and parameter information corresponding to each node to obtain the target log template may include:
[0151] Based on the character and parameter information corresponding to each node, the target position of the preset log template is filled with the character corresponding to each node.
[0152] The target log template is obtained by filling in the positions in the preset log template except for the target position with preset characters.
[0153] The electronic device provided in this application embodiment can perform the above-described functions. Figure 1 The method embodiments shown are similar in principle and technical effect, and will not be described again here.
[0154] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the following... Figure 1 The various processes in the Chinese method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.
[0155] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0157] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0158] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A log processing method, characterized in that, include: Obtain parameter information for each character in the log set, the parameter information including the character's position information and the frequency information corresponding to the position information; A target frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information. Obtain the target log template based on the target frequent pattern tree; The target log template is used to process the logs to be processed; The step of building a target frequent pattern tree using the position information of each character and the frequency information corresponding to the position information includes: A first frequent pattern tree is established using the position information of each character and the frequency information corresponding to the position information. In the first frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information. Obtain the first node in the first frequent pattern tree. The first node is located in the first branch and the second branch. The previous node of the first node is located in the first branch, the second branch and the third branch. The second branch and the third branch include nodes with the same position information corresponding to the same character. The first node is split into a second node located in the first branch and a third node located in the second branch; Merge the nodes of the same position information corresponding to the same character in the second branch and the third branch to obtain the target frequent pattern tree; Alternatively, the step of using the position information of each character and the frequency information corresponding to the position information to build a target frequent pattern tree includes: A second frequent pattern tree is constructed using the position information of each character and the frequency information corresponding to the position information. In the second frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information. Obtain the fourth node in the second frequent pattern tree, wherein the fourth node includes multiple leaf nodes and the position information of the characters corresponding to the multiple leaf nodes is consistent; The multiple leaf nodes are pruned to obtain the target frequent pattern tree.
2. The method as described in claim 1, characterized in that, The step of obtaining the target log template based on the target frequent pattern tree includes: Obtain the character and parameter information corresponding to each node in the target frequent pattern tree; The target log template is obtained by filling in the preset log template based on the character and parameter information corresponding to each node.
3. The method as described in claim 2, characterized in that, Before obtaining the parameter information for each character in the log set, the method further includes: Obtain multiple log entries, along with the punctuation information for those log entries; Based on the punctuation information of the multiple log entries, the multiple log entries are classified to obtain the log set, wherein the punctuation information of each log entry in the log set is consistent.
4. The method as described in claim 3, characterized in that, The preset log template is determined based on the punctuation information of each log entry in the log set; The process of filling a preset log template with the character and parameter information corresponding to each node to obtain the target log template includes: Based on the character and parameter information corresponding to each node, the target position of the preset log template is filled with the character corresponding to each node. The target log template is obtained by filling in the positions in the preset log template except for the target position with preset characters.
5. A log processing device, characterized in that, include: The first acquisition module is used to acquire parameter information for each character in the log set. The parameter information includes the character's position information and the frequency information corresponding to the position information. A module is established to build a target frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the target frequent pattern tree, characters with at least two position information include at least two nodes, and each node corresponds to one position information. The second acquisition module is used to acquire the target log template based on the target frequent pattern tree; The processing module is used to process the logs to be processed using the target log template; The establishment module includes: The first establishment unit is used to establish a first frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the first frequent pattern tree, for characters with at least two position information, there are at least two nodes, and each node corresponds to one position information. The first acquisition unit is used to acquire the first node in the first frequent pattern tree, wherein the first node is located in the first branch and the second branch, the previous node of the first node is located in the first branch, the second branch and the third branch, and the second branch and the third branch include nodes with the same position information corresponding to the same character. A splitting unit is used to split the first node into a second node located in the first branch and a third node located in the second branch; The merging unit is used to merge nodes with the same position information corresponding to the same character in the second branch and the third branch to obtain the target frequent pattern tree; Alternatively, the establishment module includes: The second establishment unit is used to establish a second frequent pattern tree using the position information of each character and the frequency information corresponding to the position information. In the second frequent pattern tree, for characters with at least two position information, there are at least two nodes, and each node corresponds to one position information. The second acquisition unit is used to acquire the fourth node in the second frequent pattern tree, wherein the fourth node includes multiple leaf nodes and the position information of the characters corresponding to the multiple leaf nodes is consistent. The pruning unit is used to prune the multiple leaf nodes to obtain the target frequent pattern tree.
6. An electronic device, comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, The processor is configured to read a program from memory to implement the steps of the method as described in any one of claims 1 to 4.
7. A readable storage medium, characterized in that, A program is stored on the readable storage medium, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 4.