A method for tracing the entire process of internal data circulation in an enterprise based on log auditing
Through the uncertain length log analysis method based on the tree framework, combined with the largest public sequence and hierarchical clustering, the problem of low accuracy of non-fix length log analysis in the existing technology is solved, and the visual display of automated log analysis and data circulation traceability is realized. , improve analysis efficiency and accuracy.
Patent Information
- Application Number
- CN202411038164.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The existing log audit method has low accuracy when processing non-fixed log data, and cannot achieve fully automated analysis, and the complex log forms and template analysis is inaccurate, which affects efficiency and security control.
Adopting an indefinite-length log analysis method based on a tree-like framework, the similarity is calculated through the maximum common sequence formula, combining hierarchical clustering and the longest common subsequence fusion, and automatically setting thresholds to achieve accurate analysis of non-fixed-length logs and traceability diagram construction .
It improves the accuracy and automation of log analysis, breaks the log length limit, adapts to complex log data analysis, and realizes clear visual display and traceability of data circulation.
Smart Images

Figure CN118568309B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of log auditing, and in particular to a method for tracing the entire process of internal data circulation in an enterprise based on log auditing. Background Art
[0002] Data in enterprise application systems are frequently transmitted across domains, generating a large amount of internal log data, which is large and messy. Data transmission may be subject to security threats, internal and external attacks, and system failures. These will cause data information leakage, data transmission failure, data transmission anomalies, and other data failures, affecting the normal operation of the system. A large number of complex system logs contain relevant information about the data. Therefore, auditing and tracing the log data is crucial, and log parsing is a key step.
[0003] The existing log parsing methods mainly include the following: regular expression matching, log template matching, statistical analysis, dictionary-based parsing methods, and machine learning and deep learning-based methods. Regular expression matching is highly flexible and can customize patterns to match almost any type of log format, but regular expressions are complex, resulting in reduced performance and difficult maintenance. Statistical analysis provides quantitative analysis results that can reveal trends, peaks, and abnormal patterns, but it will cause the loss of contextual information. Log template matching is simple to implement, easy to understand and use, and has high efficiency. Compared with complex regular expressions, template matching usually has better performance, but it has low flexibility and requires a large number of templates to be customized. Templates need to be added manually for new log formats. Dictionary-based parsing is simple to implement and has a fast processing speed. It is suitable for real-time log analysis scenarios, but it has poor flexibility and adaptability and may not be able to accurately parse log data with diverse formats. For complex log structures, it may not provide in-depth parsing. Machine learning and deep learning-based methods can parse complex log patterns, realize automated log parsing, reduce manual intervention, and improve performance and accuracy. However, current machine learning and deep learning-based methods fail to take into account the three aspects of non-fixed-length log parsing, parsing accuracy, and fully automated parsing.
[0004] However, the log mode of internal log data is not always fixed-length, so it is necessary to apply the method of non-fixed-length log parsing. And because the internal log data is huge, it is necessary to design an automated log audit method. For the parsed structured log, it is necessary to search for the starting node, intermediate circulation node and final node of the data, and build a traceability diagram to show the relationship between the data circulation nodes, providing a basis for security management and control.
[0005] The analysis shows that the current log auditing method has many shortcomings and limitations: (1) Log parsing relies on manually input regular expressions or grok patterns to extract event templates and key parameters. Although it is simple, parsing a large number of logs is time-consuming and prone to errors; (2) The accuracy of non-fixed-length log parsing methods is significantly reduced, the threshold relies on manual setting, and full automatic parsing is not achieved, which affects the parsing results and efficiency; (3) Complex, unknown log formats and log templates cannot be correctly parsed; (4) The data flow process is complex and lengthy, making it difficult to view and understand the transmission path. Summary of the invention
[0006] The purpose of the present invention is to provide a method for tracing the entire process of internal enterprise data circulation based on log auditing, so as to solve at least one problem existing in the above-mentioned prior art when auditing and tracing the internal enterprise data.
[0007] To achieve the above purpose, the present invention adopts the following technical solution.
[0008] A method for tracing the entire process of internal data circulation in an enterprise based on log auditing includes the following steps:
[0009] S1: Obtain internal log data of cross-domain enterprises;
[0010] S2: After extracting the timestamp, level and component information of the log message, the timestamp, level and component information in the log message are deleted to complete the log data preprocessing;
[0011] S3: Construct an indefinite-length log parsing tree based on a tree-like framework, parse the preprocessed log data, and achieve preliminary identification of log templates and log parameters;
[0012] S4: Set an intra-group similarity threshold. If the intra-group similarity of the log group corresponding to the initially identified log template is lower than the intra-group similarity threshold, the corresponding log group is re-analyzed using hierarchical clustering.
[0013] S5: Set the inter-group similarity threshold. If the similarity between two log groups is higher than the inter-group similarity threshold, the two log groups are merged based on the longest common subsequence.
[0014] S6: Extract log messages about data flow and timestamps corresponding to the log messages, convert the log messages into dictionary data sets, and then form a directed traceability graph according to the data transmission direction.
[0015] Furthermore, in step S2, a regular expression is used to extract the timestamp, level and component information of the log message.
[0016] Furthermore, step S3 specifically includes the following steps:
[0017] S31: Set a fixed depth, and use the first depth-2 tokens of the log message as the search criteria; and set the following two rules: (1) Set the maximum number of child nodes of a node; (2) If the token of the log message contains a number, replace it with a wildcard;
[0018] S32: Construct a parsing tree and parse the preprocessed log message starting from the root node; the root node is divided into two nodes, a number and a word, according to the first token of the log message, and then traverse from the second-level nodes to the leaf nodes;
[0019] S33: Clustering based on similarity. After traversing to the leaf node, there will be a corresponding log group list under the leaf node; calculate the similarity between the log message and the log event of each log group :
[0020]
[0021] In the formula, LCS represents the length of the maximum common subsequence, seq1 represents the log message, and seq2 represents the log event;
[0022] Find out the The largest log group and its corresponding similarity ,like , then this log group is returned, the log message is added to this log group, and the log event is updated; otherwise, a prompt message that no suitable log group is found is returned, and a new log group is created based on the log message. The log message will be used as the initial log event of the log group, and the log group is added to the log group list; where, Represents the similarity threshold.
[0023] Furthermore, the similarity threshold The initial value of , and its calculation formula is as follows:
[0024]
[0025] In the formula, Indicates the length of the log message. It is the number of tokens containing numbers in the log event;
[0026] Similarity Threshold Automatically updated during parsing, when log events are updated, similarity threshold The similarity threshold will also be updated. The update formula is as follows:
[0027] }
[0028]
[0029] In the formula, The initial value is 0, which indicates the cumulative number of tokens replaced by wildcards when updating log events. When the token in a log event is replaced by a wildcard when updating, .
[0030] Furthermore, step S4 specifically includes the following steps:
[0031] S41: Calculate the distance between all log messages in each log group:
[0032]
[0033] In the formula, Indicates the distance between log message P and log message Q in the log group. and Represents the first Tokens, and Respectively represent the token length of log message P and log message Q; The function is expressed as follows:
[0034]
[0035] in, is an adjustable parameter;
[0036] S42: Setting the intra-group similarity threshold , the calculation formula of the intra-group similarity of the log group is: ; If there is a similarity between two log messages in the log group , it is considered that there is a problem with the log parsing of the log group, and the process goes to step S43; otherwise, no processing is performed;
[0037] S43: hierarchical clustering is used to analyze the log group, and the number of cluster categories is calculated from low to high. First, it is calculated whether the intra-group similarities of the two log groups clustered into two categories are both higher than the threshold, and so on, until the intra-group similarity of each log group formed by the cluster is higher than the intra-group similarity threshold, and the clustering is terminated.
[0038] Furthermore, in step S5, the similarity between the two log groups is expressed by temSim as follows:
[0039]
[0040] In the formula, It indicates the number of tokens of the longest common subsequence of log events in two log groups; lenseqn and lenseqe indicate the number of tokens of log events in two log groups respectively.
[0041] Furthermore, in step S6, the directed traceability graph is formed as follows: t is set as the data transmission execution time, A is the IP address of the starting node of the data transmission, and B is the IP address of the destination node of the data transmission, which are converted into a dictionary form {("A", "t"): ("B", "t")}, and the data transmission process of the directed edge A->B is represented by the dictionary form; the data flow node at node B is searched until the termination node is found, and the directed edge links represented in the dictionary form are formed into a directed traceability graph.
[0042] The present invention proposes a method for tracing the entire process of internal data circulation in an enterprise based on log audit, which has at least the following beneficial effects:
[0043] 1. The tree-like framework on which the present invention relies selects the maximum common sequence formula as the similarity calculation formula, which can complete the recognition of non-fixed-length logs; adopts an automated threshold setting method to reduce the intervention of human factors; adds hierarchical clustering verification and strengthens the accuracy of log parsing; and finally performs log fusion to prevent over-parsing, which can achieve correct parsing of complex, unknown log forms and log templates. After these steps, log data is converted from semi-structured data to structured data, which facilitates the subsequent search and extraction of information related to transmission data. This method improves the accuracy, realizes automated log parsing, breaks the limitation of log length, and is more suitable for the parsing of complex cross-domain log data within an enterprise.
[0044] 2. A visual display of the traceability of internal enterprise log data is realized. Data on the IP of data circulation is selected from the parsed data, and the data flow process between nodes is simplified into a directed vector, represented by a dictionary, and searched along the flow node to find the final node, and the directed vectors are linked to form a directed graph, which is the data traceability graph. The data flow is clearly visible, and the address, time and flow process of the flow can be checked. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1It is a flow chart of a method for tracing the entire process of internal data circulation in an enterprise based on log auditing provided by an embodiment of the present invention;
[0047] Figure 2 It is a framework diagram of log collection and auditing provided by an embodiment of the present invention;
[0048] Figure 3 It is a schematic diagram of automatic indefinite-length log parsing based on a tree framework provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0049] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0050] like Figure 1 As shown, the embodiment of the present invention discloses a method for tracing the entire process of internal data circulation in an enterprise based on log audit, including the following steps:
[0051] S1: Obtain internal log data of cross-domain enterprises. Figure 2 The following is a framework diagram for log collection and auditing. A unified network entity identity resource system is established to ensure that the same user is presented as a unique entity identifier in different business systems, thus achieving security auditing across application domains. When a user accesses the system to extract logs, the user first needs to authenticate through the unified identity authentication system. When the user accesses a business with a token, the token needs to be verified. After the authentication, the log collection is completed through the interface call, and then the log is transferred for log auditing.
[0052] S2: After extracting the timestamp, level and component information of the log message, the timestamp, level and component information in the log message are deleted to complete the log data preprocessing and reduce the complexity of log parsing.
[0053] Among them, regular expressions are used to extract the timestamp, level, and component information of the log message. Exemplarily, in this embodiment, \d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2} is used to extract the timestamp (following the "2021-03-15 13:45:30" format); \b(INFO|WARN|ERROR|DEBUG)\b is used to extract the level information, b represents a capture group (INFO|WARN|ERROR|DEBUG); \[\w+\]\s\[(\w+)\] is used to extract the component information (following the Component: [ComponentName] format).
[0054] S3: Construct a log parsing tree of indefinite length based on the tree framework, parse the preprocessed log data, and achieve preliminary identification of log templates and log parameters.
[0055] Specifically, step S3 includes the following steps:
[0056] S31: Set a fixed number of layers depth. Excluding the root node and leaf nodes, the number of internal nodes actually used for template extraction is depth-2. The first depth-2 codes of the log message are used as the search criteria. In order to prevent branch explosion, the following two rules are set: (1) Set the maximum number of child nodes of a node to limit the maximum number of child nodes of a node; (2) If the token of the log message contains a number, it is replaced by a wildcard. For log messages such as "120 bytes receive", branch explosion may occur because "120" will be encoded into three nodes. Therefore, in this embodiment, if the log token contains a number, it is replaced by "*".
[0057] S32: Construct a parsing tree and parse the preprocessed log message from the root node; the root node is divided into two nodes, a number node and a word node, according to the first token of the log message, and then traverse from the second-level nodes to the leaf nodes. For example, "Receive from node 4", reach the first-level word node, the second-level receive node, until the last token of the log message is found or the fixed depth is reached.
[0058] S33: Clustering based on similarity. After traversing to the leaf node, there will be a corresponding log group list under the leaf node. The log messages of these log groups are consistent with the tree path content. The log event is the best template to describe the log message in the log group. The similarity between the log message and the log event of each log group is calculated. :
[0059]
[0060] In the formula, LCS represents the length of the maximum common subsequence, seq1 represents the log message, and seq2 represents the log event;
[0061] Find out the The largest log group and its corresponding similarity ,like , then this log group is returned, the log message is added to this log group, the log event is updated, and in the new log event, the largest common subsequence is retained, and the content not in the largest common subsequence is replaced by the wildcard "*" (for example, the original log event is: "receive node * from node", the log message is "receive node n from node n & q", and the updated log event is "receive node * from node *"); otherwise, it returns a prompt message that no suitable log group is found, and a new log group is created based on the log message. The log message will be used as the initial log event of the log group, and the log group is added to the log group list; where, Represents the similarity threshold.
[0062] Among them, the similarity threshold The initial value of , and its calculation formula is as follows:
[0063]
[0064] In the formula, Indicates the length of the log message. It is the number of digital tokens in the log event. If a log message is considered to belong to a log group, it is the same as the number of digital tokens contained in the log event. The digital tokens in the log are generally present as parameters rather than template information. For each newly created log group, its log event is the first log message in the group.
[0065] Similarity Threshold Automatically updated during parsing, when log events are updated, similarity threshold The similarity threshold will also be updated. The update formula is as follows:
[0066] }
[0067]
[0068] In the formula, The initial value is 0, which indicates the cumulative number of tokens replaced by wildcards when updating log events. When the token in a log event is replaced by a wildcard when updating, .
[0069] S4: setting an intra-group similarity threshold. If the intra-group similarity of the log group corresponding to the initially identified log template is lower than the intra-group similarity threshold, the corresponding log group is re-analyzed using hierarchical clustering.
[0070] Specifically, step S4 includes the following steps:
[0071] S41: Calculate the distance between all log messages in each log group:
[0072]
[0073] In the formula, Indicates the distance between log message P and log message Q in the log group. and Represents the first Tokens, and Respectively represent the token length of the log message P and log message Q within the group; The function is expressed as follows:
[0074]
[0075] in, It is an adjustable parameter and is set to 1 in this embodiment;
[0076] S42: Setting the intra-group similarity threshold , the calculation formula of the intra-group similarity of the log group is: ; If there is a similarity between two log messages in the log group , it is considered that there is a problem with the log parsing of the log group, and the process goes to step S43; if , no processing is done;
[0077] S43: hierarchical clustering is used to analyze the log groups. The number of cluster categories is calculated from low to high. First, the intra-group similarities of the two log groups clustered into two categories are calculated to see whether they are both higher than the threshold. This is repeated until the intra-group similarities of each log group formed by the cluster are higher than the intra-group similarity threshold, and the clustering is terminated.
[0078] S5: Set the inter-group similarity threshold. If the similarity between two log groups (templates) is higher than the inter-group similarity threshold, the two log groups (templates) are merged based on the longest common subsequence.
[0079] Among them, the similarity between two log groups is expressed by temSim as follows:
[0080]
[0081] In the formula, It indicates the number of tokens of the longest common subsequence of log events in two log groups; lenseqn and lenseqe indicate the number of tokens of log events in two log groups respectively.
[0082] S6: Extract log messages about data flow and timestamps corresponding to the log messages, convert the log messages into dictionary data sets, and then form a directed traceability graph according to the data transmission direction.
[0083] The process of forming a directed traceability graph is as follows: set t as the execution time of data transmission, A as the IP address of the starting node of data transmission, and B as the IP address of the destination node of data transmission, and convert them into dictionary form {("A", "t"): ("B", "t")}, and use the dictionary form to represent the data transmission process of the directed edge A->B; find the data flow from node B to node C, B->C, until the end node is found, and then link the directed edges represented in dictionary form to form a directed traceability graph.
[0084] The above embodiment provides a method for tracing the entire process of internal data circulation in an enterprise based on log auditing, which has at least the following beneficial effects: a log parsing method for internal log data in an enterprise is provided, which relies on a tree-like framework, selects the maximum common sequence formula as the similarity calculation formula, and can complete the identification of non-fixed-length logs; adopts an automated threshold setting method to reduce the intervention of human factors; adds hierarchical clustering verification and enhances the accuracy of log parsing; finally, log fusion is performed to prevent over-parsing, and correct parsing of complex and unknown log forms and log templates can be achieved. After these steps, the log data is converted from semi-structured data to structured data, which facilitates the subsequent search and extraction of information related to the transmission data. This method improves the accuracy, realizes automated log parsing, breaks the limitation of log length, and is more suitable for the parsing of complex cross-domain log data within an enterprise. A visual display of the traceability of internal log data in an enterprise is realized. The data of the data circulation IP is filtered out from the parsed data, and the data flow process between nodes is simplified into a directed vector, represented by a dictionary, and searched along the flow node to find the final node, and the directed vector is linked to form a directed graph, which is the data traceability graph. The data flow is clearly visible, and the address, time and flow process can be checked.
[0085] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A method for tracing the entire process of internal data circulation in an enterprise based on log auditing. It is characterized in that The steps include: S1: Obtain internal log data of cross-domain enterprises; S2: After extracting the timestamp, level and component information of the log message, the timestamp, level and component information in the log message are deleted to complete the log data preprocessing; S3: Construct an indefinite-length log parsing tree based on a tree-like framework, parse the preprocessed log data, and achieve preliminary identification of log templates and log parameters; S4: Set an intra-group similarity threshold. If the intra-group similarity of the log group corresponding to the initially identified log template is lower than the intra-group similarity threshold, the corresponding log group is re-analyzed using hierarchical clustering. S5: Set the inter-group similarity threshold. If the similarity between two log groups is higher than the inter-group similarity threshold, the two log groups are merged based on the longest common subsequence. S6: extract log messages about data flow and timestamps corresponding to the log messages, convert the log messages into a dictionary data set, and then form a directed traceability graph according to the data transmission flow; Step S3 specifically includes the following steps: S31: Set a fixed depth, and use the first depth-2 tokens of the log message as the search criteria; and set the following two rules: (1) Set the maximum number of child nodes of a node; (2) If the token of the log message contains a number, replace it with a wildcard; S32: Construct a parsing tree and parse the preprocessed log message starting from the root node; the root node is divided into two nodes, a number and a word, according to the first token of the log message, and then traverse from the second-level nodes to the leaf nodes; S33: Clustering is performed based on similarity. After traversing to a leaf node, there will be a corresponding log group list under the leaf node; Calculate the similarity between the log message and the log event of each log group : ; In the formula, LCS represents the length of the maximum common subsequence, seq1 represents the log message, and seq2 represents the log event; Find out the The largest log group and its corresponding similarity ,like , then this log group is returned, the log message is added to this log group, and the log event is updated; otherwise, a prompt message that no suitable log group is found is returned, and a new log group is created based on the log message. The log message will be used as the initial log event of the log group, and the log group is added to the log group list; where, represents the similarity threshold; Similarity Threshold The initial value of , and its calculation formula is as follows: ; In the formula, Indicates the length of the log message. It is the number of tokens containing numbers in the log event; Similarity Threshold Automatically updated during parsing, when log events are updated, similarity threshold The similarity threshold will also be updated. The update formula is as follows: }; ; In the formula, The initial value is 0, which indicates the cumulative number of tokens replaced by wildcards when updating log events. When the token in a log event is replaced by a wildcard when updating, ; In step S6, the directed traceability graph is formed as follows: t is set as the data transmission execution time, A is the IP address of the starting node of the data transmission, and B is the IP address of the destination node of the data transmission, which are converted into a dictionary form {("A", "t"): ("B", "t")}, and the data transmission process of the directed edge A->B is represented in dictionary form; the data flow node at node B is searched until the end node is found, and the directed edge links represented in dictionary form are formed into a directed traceability graph.
2. According to claim 1, the method for tracing the entire process of internal data circulation in an enterprise based on log auditing, It is characterized in that In step S2, a regular expression is used to extract the timestamp, level, and component information of the log message.
3. According to claim 1, the method for tracing the entire process of internal data circulation in an enterprise based on log auditing, It is characterized in that Step S4 specifically includes the following steps: S41: Calculate the distance between all log messages in each log group: ; In the formula, Indicates the distance between log message P and log message Q in the log group. and Represents the first Tokens, and Respectively represent the token length of log message P and log message Q; The function is expressed as follows: ; in, is an adjustable parameter; S42: Setting the intra-group similarity threshold , the calculation formula of the intra-group similarity of the log group is: ; If there is a similarity between two log messages in the log group , it is considered that there is a problem with the log parsing of the log group, and the process goes to step S43; otherwise, no processing is performed; S43: hierarchical clustering is used to analyze the log group, and the number of cluster categories is calculated from low to high. First, it is calculated whether the intra-group similarities of the two log groups clustered into two categories are both higher than the threshold, and so on, until the intra-group similarity of each log group formed by the cluster is higher than the intra-group similarity threshold, and the clustering is terminated.
4. According to claim 1, the method for tracing the entire process of internal data circulation in an enterprise based on log auditing, It is characterized in that In step S5, the similarity between the two log groups is expressed by temSim as follows: ; In the formula, It indicates the number of tokens of the longest common subsequence of log events in two log groups; lenseqn and lenseqe indicate the number of tokens of log events in two log groups respectively.
