Log analysis system based on a custom rules detection engine

Abstract

The application discloses a log analysis system based on a self-defined rule detection engine, and comprises a log collection module, a log aggregation module, a self-defined rule detection engine module, a log processing module and a log storage and display module.The log collection module is used for collecting real-time and reliable user behavior operation log data in a SpringBoot system.The log aggregation module is used for guaranteeing reliable transmission, caching and convergence of the log data.The self-defined rule detection engine module is used for analyzing and converting user self-defined rules, and then matching the log data by using a Hyperscan high-performance regular expression matching library, so that self-defined rule detection and analysis of the user behavior operation log data are realized.The log processing module is used for processing the log by interacting with a dynamic link library formed by compiling the self-defined rule detection engine module.The log storage and display module is used for storing, managing and displaying the log data.The application designs a meta syntax expression of a self-defined rule based on a JSONPointer standard for analyzing user self-defined rules, enhances the analysis capability of the log, and provides an alarm function for users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log data management and log analysis, in particular to a log analysis system based on a JSONPointer standard and a custom rule detection engine. BACKGROUND

[0002] With the continuous improvement of modern informatization level, the log data of various systems is increasing, and how to effectively manage the log data has become a relatively urgent problem. At present, the existing data management systems, such as ELK, can store log data, but the processing of data, such as data extraction and matching, is not flexible enough, and needs to be further improved. SUMMARY

[0003] The purpose of the present application is to propose a log analysis system based on a JSONPointer standard and a custom rule detection engine for the json format log data of user behavior in a SpringBoot system, design a meta syntax expression based on the JSONPointer standard, analyze the user-defined rules by using a lexical and syntax analyzer, thereby enhancing the analysis capability of the log, and at the same time, providing an alarm function for the user behavior log, and enhancing the log abnormality detection capability.

[0004] To achieve the above purpose, the technical scheme provided by the present application is: a log analysis system based on a custom rule detection engine, comprising:

[0005] A log collection module is used to collect real-time and reliable user behavior operation log data in a SpringBoot system.

[0006] A log aggregation module is used as an intermediate link from log data collection to storage, and is used to ensure reliable transmission, caching and convergence of log data.

[0007] A custom rule detection engine module is used to parse and convert the user-defined rules by using a lexical and syntax analyzer, and then match the log data by using a Hyperscan high-performance regular expression matching library, so as to finally realize custom rule detection and analysis of the user behavior operation log data.

[0008] A log processing module is used to interact with a dynamic link library formed by compiling the custom rule detection engine module, so as to realize processing of the log.

[0009] A log storage and display module is used for storage, management and display of the log data.

[0010] Further, the log collection module collects real-time and reliable user behavior operation log data in the SpringBoot system by integrating Logstash, wherein the user behavior operation log data is in json format, and the log collection module performs the following operations:

[0011] 1) In the pom.xml file of SpringBoot, Logstash dependency needs to be added; in the SpringBoot application, Logback is used as a log framework, and Logback configuration needs to be added in the application.properties or application.yml file, which is used as a system collection component to output logs to Logstash;

[0012] 2) Install the json_lines plug-in in the Logstash component of step 1);

[0013] 3) Configure the input and output of Logstash to read log data from the SpringBoot application and send it to the subsequent module.

[0014] Further, the log aggregation module realizes log data caching and guarantees high availability of data through kafka.

[0015] Further, the self-defined rule detection engine module includes a compiler and a matcher, and the self-defined rule engine module compiles the compiler and the matcher into a dynamic link library as a whole and interacts with the log processing module; wherein:

[0016] The compiler includes a lexical analysis module and a syntax analysis module, and finally the two modules are compiled into a static link library as a whole and interact with the matcher; wherein:

[0017] The lexical analysis module is used to parse and split the input character stream S using the lexical analyzer Lexical(·) into individual lexical units, also called tokens, including the following steps:

[0018] a1) define the meta-grammar expression of the self-defined rule based on the JSON Pointer standard, using the syntax of BNF;

[0019] a2) define a finite state machine FSM for identifying whether the input character stream conforms to the meta-grammar of the self-defined rule based on the JSON Pointer standard, which is composed of states and transition functions; assume that there are n states s1, s2,..., s n , wherein the state s1 is the starting state, s nThe state is a terminal state, each state can accept one or more input characters, and perform state transition, state transition function is represented as:

[0020] (s i ,c)→s j

[0021] In the formula, c represents an input character, s i and s j represent the current state and the next state; if the current state is s i , after reading the input character c, the state transition function will calculate the next state s j , and record c as one of the characters accepted by the current state; wherein, a state can have multiple out edges, i.e. for the same state and different input characters, there can be different next states;

[0022] a3) define each lexical unit Token as a set of expressions, assuming there are m lexical units t1, t2,..., tm m , where tm m is the mth lexical unit, and each token has a corresponding expression Where is the mth lexical unit tm m corresponding expression, each expression can be expressed as the meta-expression expression of the custom rule based on the JSON Pointer standard defined in step a1) above and the action embedding operator group;

[0023] a4) convert each regular expression into an equivalent finite automaton, assuming that the i-th lexical unit t i defined in step a3) has a corresponding expression regular expression The finite automaton corresponding to the expression is M i , and the finite automaton is represented by the state and transition function defined in step a2) above;

[0024] a5) combine all the finite automata in step a4) into a large finite automaton, assuming that the large finite automaton is M, by connecting the start state of each finite automaton to a new start state, and converting the terminal state of each finite automaton into a new terminal state;

[0025] a6) use the finite automaton M constructed in step a5) as input, run the lexical analyzer, read characters from the start position b of the input string, and send them to M, starting from the start state of M, for each input character, calculate the next state through the state transition function until the terminal state is reached; if the current state is a certain token corresponding to tm ithe end state, then the corresponding substring in b is identified as t i If none of the end states is reached, or the complete input string has been processed, the recognition process is stopped.

[0026] a7) output the recognized tokens Tokens;

[0027] The syntax analysis module takes the tokens Tokens output by the lexical analysis module as input, and then converts them into corresponding data structures and performs syntax rule detection, including the following steps:

[0028] b1) define the grammar G = (N, E, P, S) according to the syntax specification and semantic information, where N is the set of non-terminal symbols, E is the set of terminal symbols, P is the set of production rules, and S is the start symbol; each symbol in the grammar has a leftmost terminal symbol set FIRST and a successor terminal symbol set FOLLOW, which are defined according to the following syntax rules:

[0029] For each non-terminal symbol A ∈ N, calculate FIRST(A): if A is a terminal symbol, then FIRST(A) = {A}; if A is a non-terminal symbol, then FIRST(A) is the union of all possible FIRST sets of the first symbols that can appear in A, specifically: if there is a rule, then all elements in FIRST(α) except ε are added to FIRST(A); if FIRST(α) contains ε for all rules A→α, then ε is added to FIRST(A);

[0030] For each non-terminal symbol A, B ∈ N, calculate FOLLOW(A): FOLLOW(S) = {$}, where S is the start symbol of the grammar; if there is a rule A→αBβ, then all elements in FOLLOW(A) are added to FOLLOW(B); if there is a rule A→αBβ and β can derive ε, then all elements in FOLLOW(A) are added to FOLLOW(B);

[0031] where α and β are symbol strings, which are the right part of some production rule of non-terminal symbol A; ε represents an empty symbol string; $ represents the end symbol of the input string;

[0032] b2) generate LALR(1) analysis table according to the syntax rules and the assumed grammar, FIRST set, and FOLLOW set in step b1) above, which is used to track the new states that each state and each non-terminal symbol can reach, and guide the execution of specified operations;

[0033] b3) performing syntax analysis, reading in from the input token stream and processing the tokens according to the LALR(1) analysis table generated in step b2) above, until a syntax error is encountered or all input tokens are processed;

[0034] b4) if the syntax analysis is completed without encountering a syntax error, converting the matched tokens into corresponding data structures and storing them into a cache file to form a rule library cache file;

[0035] The matcher utilizes the Hyperscan regular expression engine, loads the rule library cache file processed by the compiler above, including the following steps:

[0036] c1) creating a matched result storage vector matched and a pointer pointing to it;

[0037] c2) loading the json format log data to be matched and parsing the json format log data using the rapidjson technology;

[0038] c3) loading the cache file generated by the compiler above and initializing the regular expression set;

[0039] c4) compiling the regular expressions to generate an automaton state containing all regular expressions and constructing a matching tree for each regular expression;

[0040] c5) after generating the automaton state, converting it into a DFA state diagram, where each state corresponds to a matching result of a regular expression;

[0041] c6) performing matching on the automaton;

[0042] c7) when the matching is successful, storing the id of the regular expression in the matched variable.

[0043] Further, the log processing module performs the following steps:

[0044] 1) pulling the log data in the log aggregation module into the custom rule detection engine module for detection to obtain a matched rule id array;

[0045] 2) finding the corresponding rule according to the rule id array, finding the specified strategy of the corresponding rule, and executing the action on the log data according to the specified strategy.

[0046] Further, the log storage and display module stores the log data and visualizes the log data, and performs the following operations:

[0047] 1) In the pom.xml file of SpringBoot, the dependencies of Elasticsearch and Kibana need to be added;

[0048] 2) In the web interface of Kibana, the log data of the SpringBoot application can be viewed, and the data analysis and visualization can be performed using the visualization tools of Kibana.

[0049] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0050] 1. The ELK system is integrated in the SpringBoot system, and the special storage and display of the json format are realized, which is beneficial to the management of logs.

[0051] 2. The log is detected and analyzed by the custom rule detection engine module, and then is imported into the ELK system again, which greatly reduces the difficulty of log management.

[0052] 3. The present application is aimed at the most common json format log in the SpringBoot system, and analyzes the jsonpointer standard conforming to the json format, and fully utilizes the characteristics of the json format for log analysis and detection.

[0053] 4. The custom rule detection engine module is compiled to form a dynamic link library, which further decouples the custom detection engine module and the log processing module, reduces the interaction complexity of log processing, and decouples the code management repository of the custom detection engine module and the log processing module, which is convenient for maintenance.

[0054] 5. In the custom rule detection engine module, the compiler and the matcher are further decoupled to reduce complexity and be more general. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The figure is a schematic diagram of the system architecture of the present application.

[0056] Figure 2 The figure is a schematic diagram of the overall design of the custom rule detection engine module. DETAILED DESCRIPTION

[0057] The present application will be further described below in conjunction with specific embodiments.

[0058] As shown in Figure 1 and Figure 2 , the present embodiment discloses a log analysis system based on a custom rule detection engine, which comprises:

[0059] A log collection module is used to collect real-time and reliable user behavior operation log data in the SpringBoot system.

[0060] A log aggregation module, as an intermediate link from log data collection to storage, is used to guarantee reliable transmission, caching and convergence of log data;

[0061] A custom rule detection engine module, which utilizes a lexical and syntax analyzer to parse and convert user-defined rules, and then utilizes a Hyperscan high-performance regular expression matching library to match log data, finally realizes custom rule detection and analysis of user behavior operation log data;

[0062] A log processing module, which interacts with a dynamic link library formed by compiling the custom rule detection engine module, realizes processing of logs;

[0063] A log storage and display module, which is used for storage, management and display of log data.

[0064] Specifically, the log collection module collects real-time and reliable user behavior operation log data in the SpringBoot system by integrating Logstash, wherein the user behavior operation log data is in json format, and the log collection module performs the following operations:

[0065] 1) In the pom.xml file of SpringBoot, Logstash dependency needs to be added; in the SpringBoot application, Logback is used as a log framework, and Logback configuration needs to be added in the application.properties or application.yml file, which is used as a system collection component to output logs to Logstash;

[0066] 2) Install the json_lines plug-in in the Logstash component of step 1);

[0067] 3) Configure the input and output of Logstash to read log data from the SpringBoot application and send it to the subsequent module.

[0068] Specifically, the log aggregation module realizes caching of log data and guarantees high availability of data through kafka.

[0069] Specifically, the custom rule detection engine module includes a compiler and a matcher, and the custom rule engine module compiles the compiler and the matcher into a dynamic link library and interacts with the log processing module; wherein:

[0070] The compiler includes a lexical analysis module and a syntax analysis module, and finally the two modules are compiled into a static link library and interact with the matcher; wherein:

[0071] The lexical analysis module is used to parse and split the input character stream S into individual lexical units, also called tokens, using the lexical analyzer Lexical(·). This includes the following steps:

[0072] a1) Define meta-syntax expressions for custom rules based on the JSON Pointer standard, using BNF syntax representation;

[0073] a2) A finite state machine (FSM) is defined to identify whether the input character stream conforms to the metasyntax of custom rules based on the JSONPointer standard. It consists of states and transition functions; assuming there are n states s1, s2, ..., sn. n Where state s1 is the initial state, s n The state is a terminating state. Each state can accept one or more input characters and perform a state transition. The state transition function is expressed as:

[0074] (s i ,c)→s j

[0075] In the formula, c represents the input character, and s i and s j Indicates the current state and the next state; if the current state is s i After reading the input character 'c', the state transition function will calculate the next state 's'. j And record c as one of the characters accepted by the current state; where a state can have multiple outgoing edges, that is, for the same state and different input characters, there can be different next states;

[0076] a3) Define each lexical unit Token as a set of expressions. Assume there are m lexical units t1, t2, ..., t m , where t m It is the m-th lexical unit, and each token has a corresponding expression. in It is the m-th lexical unit t m The corresponding expressions, each of which can be represented as an expression composed of a meta-syntax expression of a custom rule based on the JSON Pointer standard defined in step a1) above and an action embedding operator;

[0077] a4) Convert each regular expression into an equivalent finite automaton, assuming the i-th lexical unit t defined in step a3) i The corresponding expression is a regular expression. The corresponding finite automaton for this expression is M. iThe finite automaton is represented by the states and the transition function defined in step a2) above;

[0078] a5) All finite automata from step a4) are merged into one large finite automaton, denoted as M, by connecting the start state of each finite automaton with a new start state and transforming the end states of each finite automaton into a new end state;

[0079] a6) Using the finite automaton M constructed in step a5) as input, a lexical analyzer is run, starting at the beginning of the input string b and feeding the characters into M, starting at the start state of M, for each input character the next state is calculated by the state transition function until the end state is reached; if the current state is the end state of a certain token t i , the corresponding substring in b is recognized as t i ; if no end state is reached or the complete input string has been processed, the recognition process is stopped;

[0080] a7) The recognized tokens are output;

[0081] The syntax analysis module takes the tokens Tokens output by the lexical analysis module as input, converts them into the corresponding data structure and performs syntax rule detection according to the lexical and syntax rules, including the following steps:

[0082] b1) According to the syntax specification and semantic information, define the grammar G = (N, E, P, S), where N is the set of non-terminal symbols, E is the set of terminal symbols, P is the set of production rules, and S is the start symbol; each symbol of the grammar has a leftmost terminal symbol set FIRST set and a successor terminal symbol FOLLOW set, which are defined according to the following syntax rules:

[0083] For each non-terminal symbol A ∈ N, calculate FIRST(A): if A is a terminal symbol, then FIRST(A) = {A}; if A is a non-terminal symbol, then FIRST(A) is the union of all possible FIRST sets of the first symbols that can appear in A, specifically: if there exists a rule A → α, then all elements in FIRST(α) except ε are added to FIRST(A); if FIRST(α) contains ε for all rules A → α, then ε is added to FIRST(A);

[0084] For each nonterminal A, B e N, compute FOLLOW(A): FOLLOW(S) = {$}, where S is the start symbol of the grammar; if there is a rule A -> aBp, add all elements of FOLLOW(A) to FOLLOW(B); if there is a rule A -> aBp and p derives e, add all elements of FOLLOW(A) to FOLLOW(B);

[0085] where a and b are a string of symbols, the right part of some production of the nonterminal A; e represents an empty symbol string; $ represents the end symbol of the input string;

[0086] b2) generating an LALR(1) analysis table according to the syntax rules and the assumed grammar, the FIRST set and the FOLLOW set in step b1) above, to track the new state that each state and each nonterminal can reach, and to guide the execution of the specified operation;

[0087] b3) performing syntax analysis, starting from the input token stream and processing the tokens according to the LALR(1) analysis table generated in step b2) above, until a syntax error is encountered or all input tokens are processed;

[0088] b4) if the syntax analysis is completed and no syntax error is encountered, converting the matched tokens into corresponding data structures and storing them into the cache file to form a rule library cache file;

[0089] The matcher loads the rule library cache file processed by the compiler output using the Hyperscan regular expression engine, including the following steps:

[0090] c1) creating a matched result storage vector matched and a pointer pointing to it;

[0091] c2) loading the json format log data to be matched and parsing the json format log data using the rapidjson technology;

[0092] c3) loading the cache file generated by the compiler and initializing the regular expression set;

[0093] c4) compiling the regular expressions to generate an automaton state containing all regular expressions, and constructing a matching tree for each regular expression;

[0094] c5) after generating the automaton state, converting it into a DFA state diagram, where each state corresponds to a matching result of a regular expression;

[0095] c6) performing matching on the automaton;

[0096] c7) When the matching is successful, the id of the regular expression is stored in the matched variable.

[0097] Specifically, the log processing module performs the following steps:

[0098] 1) Pull the log data in the log aggregation module into the custom rule detection engine module for detection to obtain an array of matched rule ids;

[0099] 2) According to the rule id array, find the corresponding rule, find the specified strategy of the corresponding rule, and execute the action according to the specified strategy.

[0100] Specifically, the log storage and display module stores log data and visualizes the log data, and performs the following operations:

[0101] 1) In the pom.xml file of SpringBoot, the dependencies of Elasticsearch and Kibana need to be added;

[0102] 2) In the Web interface of Kibana, the log data of the SpringBoot application can be viewed, and the data analysis and visualization can be performed using the visualization tools of Kibana.

[0103] Next, we will apply the above log analysis system to the log data management process of SpringBoot integrated ELK, including the following steps:

[0104] 1) By adding the logstash-logback-encoder dependency in the SpringBoot system, the log in the SpringBoot system is connected to Logstash;

[0105] 2) By installing the json_lines plugin in Logstash, the json format log in the SpringBoot system is connected;

[0106] 3) Compile the custom rule detection engine module to form a dynamic link library;

[0107] 4) The log processing module interacts with the custom rule detection engine module through the dynamic link library compiled in step 3);

[0108] 5) The log processing module sends the custom rule file to the compiler in the custom rule detection engine module by calling the dynamic link library, and compiles to generate the corresponding rule library.

[0109] Example of custom rule file content:

[0110] ` / src / ip` = '106.120.188.39' {alert: true; rid: 1;}

[0111] ` / dst / port` < 90 {alert: true rid: 2;}

[0112] 6) SpringBoot produces a piece of json format log content example:

[0113] {"dst": {"ip": "106.120.188.39", "ipv6": "::", "mac": "6cb3113100e6", "port": 80,}, "src": {"ip": "192.168.1.1", "ipv6": "::", "mac": "509a4c084438", "port": 60856,}}

[0114] This json format data is sent to the log processing module through the kafka message queue.

[0115] 7) The log processing module sends this json format log to the matcher of the custom rule detection engine module through the dynamic link library. As can be seen, the ` / src / ip` of this json log = '192.168.1.1', which does not meet the rule rid = 1; ` / dst / port` = 80, which meets the rule rid = 2. Therefore, the matcher returns the rid set as: rid[0] = 2.

[0116] 8) The log processing module receives the id set of the hit rule returned by the matcher of the custom rule detection engine module, and then scans the rule file to query the command required by the corresponding rule: ` / dst / port` < 90 {alert: true rid: 2;}, so here the alert command is executed. The final json log is:

[0117] {"dst": {"ip": "106.120.188.39", "ipv6": "::", "mac": "6cb3113100e6", "port": 60856,

[0118] },"src": {"ip": "192.168.1.1", "ipv6": "::", "mac": "509a4c084438", "port": 80,}, "myr

[0119] ule": {"alert": true}}

[0120] 9) The log processing module sends the final json file back through kafka into the Logstash pipeline and finally into the ELK log analysis system.

[0121] The above-described embodiments are only one of the expression cases of the meta-syntax expression designed based on the JSONPointer standard of the present application, and do not limit the scope of the embodiments of the present application, and thus any changes made in the shape and principle of the present application should be covered within the scope of protection of the present application.

Claims

1. A log analysis system based on a custom rule detection engine, characterized in that, include: The log collection module is used to collect real-time and reliable user behavior operation log data in the Spring Boot system; The log aggregation module, as an intermediate link from log data collection to storage, is used to ensure reliable transmission, caching, and aggregation of log data; The custom rule detection engine module uses lexical and syntax analyzers to parse and transform user-defined rules, and then uses the Hyperscan high-performance regular expression matching library to match log data, ultimately realizing custom rule detection and analysis of user behavior operation log data. The log processing module interacts with the dynamic link library compiled from the custom rule detection engine module to process logs. The log storage and display module is used for storing, managing, and displaying log data. The custom rule detection engine module includes a compiler and a matcher. The custom rule engine module compiles the compiler and matcher into a dynamic link library, which interacts with the log processing module. Wherein: The compiler includes a lexical analysis module and a syntax analysis module. These two modules are then compiled together to form a static library that interacts with the matcher. Wherein: The lexical analysis module is used to analyze the input character stream S using a lexical analyzer. The process involves parsing and breaking down the data into individual lexical units, also known as tokens. The syntax analysis module takes the tokens output by the lexical analysis module as the token stream input of this module through lexical and grammatical rules, then transforms them into the corresponding data structure and performs grammatical rule detection. The matcher utilizes the Hyperscan regular expression engine to load and process the rule base cache file output by the compiler, including the following steps: c1) Create a matching result storage vector `matched`, and a pointer to it; c2) Load the JSON format log data to be matched and parse the JSON format log data using RapidJSON technology; c3) Load the cache file generated by the compiler and initialize the regular expression set; c4) Compile the regular expressions to generate an automaton state containing all regular expressions and build a matching tree for each regular expression; c5) After generating the automaton states, convert the automaton states into a DFA state diagram, where each state corresponds to a matching result of a regular expression; c6) Matching is performed on the automaton; c7) When a match is successful, store the id of the regular expression into the matched variable.

2. The log analysis system based on a custom rule detection engine according to claim 1, characterized in that, The log collection module integrates Logstash to collect real-time and reliable user behavior log data from the Spring Boot system. The user behavior log data is in JSON format. The log collection module performs the following operations: 1) In the pom.xml file of Spring Boot, you need to add the Logstash dependency; in the Spring Boot application, if you use Logback as the logging framework, you need to add the Logback configuration in the application.properties or application.yml file as a system collection component to output logs to Logstash; 2) Install the json_lines plugin in the Logstash component from step 1); 3) Configure Logstash's input and output to read log data from the Spring Boot application and send it to subsequent modules.

3. The log analysis system based on a custom rule detection engine according to claim 2, characterized in that, The log aggregation module uses Kafka to cache log data and ensure high availability of data.

4. The log analysis system based on a custom rule detection engine according to claim 3, characterized in that: The lexical analysis module includes the following steps: a1) Define metasyntactic expressions for custom rules based on the JSON Pointer standard, using BNF syntax representation; a2) Define a finite state machine (FSM) to identify whether the input character stream conforms to the metasyntax of custom rules based on the JSON Pointer standard. It consists of states and transition functions; assume there are n states. , where the state It is the initial state. The state is a terminating state. Each state can accept one or more input characters and perform a state transition. The state transition function is expressed as: ； In the formula, Indicates the input character. and Indicates the current state and the next state; if the current state is Read in the input characters Then, the state transition function will calculate the next state. and will Record one of the characters accepted by the current state; where a state can have multiple outgoing edges, that is, for the same state and different input characters, there can be different next states; a3) Define each lexical unit (Token) as a set of expressions, assuming there are... Lexical unit ,in It is the first Each lexical unit has a corresponding expression. ,in It is the first Lexical unit The corresponding expressions, each of which can be represented as an expression composed of a meta-syntax expression of a custom rule based on the JSON Pointer standard defined in step a1) above and an action embedding operator; a4) Convert each regular expression into an equivalent finite automaton, assuming the first finite automaton defined in step a3) Lexical unit The corresponding expression is a regular expression. The finite automaton corresponding to this expression is Finite automata are represented by the state and transition functions defined in step a2) above; a5) Combine all the finite automata from step a4) into a larger finite automaton, assuming the larger finite automaton is... This is achieved by linking the initial state of each finite automaton to a new initial state and transforming the final state of each finite automaton into a new final state. a6) Using the finite automata constructed in step a5) As input, the lexical analyzer is run, starting from the beginning of the input string. Start reading characters and sending them to the input. From Starting from the initial state, for each input character, the next state is calculated using a state transition function until the termination state is reached; if the current state corresponds to a certain token... If the termination state is reached, then... The corresponding substring is identified as If no termination state is reached, or the entire input string has been processed, the recognition process stops. a7) Output the identified tokens; The syntax analysis module includes the following steps: b1) Define the grammar based on grammatical rules and semantic information. ,in It is a set of non-terminal symbols. For the set of terminal symbols, For the production set, The first symbol is denoted by . Each grammar symbol has a leftmost terminal set (FIRST) and a successor terminal set (FOLLOW), defined according to the following grammar rules: For each nonterminal symbol ,calculate :if If it is a terminal symbol, then ;if If it is a non-terminal symbol, then For all possible occurrences The union of the first FIRST sets of the first symbols, specifically: if a rule exists, then Except Add all other elements If for all rules , All include Then join in ; For each nonterminal symbol ,calculate : ,in It is the start symbol of the grammar; if a rule exists. Then Add all elements If rules exist and It can be deduced Then Add all elements ; in, and They are all a string of symbols, which are non-terminal symbols. The right-hand side solution of a certain production rule; Represents an empty string; This indicates the end-of-string character of the input string; b2) Generate an LALR(1) parsing table based on the grammar rules and the assumed grammar, FIRST set, and FOLLOW set in step b1) above. This table is used to track each state and the new states that each non-terminal symbol can reach, and to guide the execution of specified operations. b3) Perform syntax analysis, starting from the input token stream and processing the tokens according to the LALR(1) parsing table generated in step b2) above, until a syntax error is encountered or all input tokens have been processed; b4) If all syntax analysis has been completed and no syntax errors have been encountered, the matched tags are converted into corresponding data structures and stored in the cache file to form the rule base cache file.

5. The log analysis system based on a custom rule detection engine according to claim 4, characterized in that: The log processing module performs the following steps: 1) Pull log data from the log aggregation module and send it to the custom rule detection engine module for detection to obtain an array of matching rule IDs; 2) Find the corresponding rule based on the rule ID array, find the specified policy of the corresponding rule, and perform actions on the log data according to the specified policy.

6. The log analysis system based on a custom rule detection engine according to claim 5, characterized in that: The log storage and display module stores log data and visualizes the log data, performing the following operations: 1) By adding Elasticsearch and Kibana dependencies to the pom.xml file in Spring Boot; 2) In the Kibana web interface, you can view the log data of the Spring Boot application and use Kibana's visualization tools for data analysis and visualization.

Images