A method for detecting logs based on an aggregation mode of stream processing

Abstract

The application relates to the technical field of electric digital data processing, and discloses a method for detecting logs of an aggregation mode based on flow processing, which comprises the following steps: merging multiple detection modes into mode groups according to aggregation fields, constructing an AC automatic machine and an inverted index table for each mode group; inputting an original log stream into a flow processing engine, dividing the log into micro batches through a sliding window, and discarding logs exceeding the window duration by using a watermark mechanism; traversing each mode group, sending fields in the current micro batch log into the AC automatic machine of the corresponding mode group to perform multi-mode matching, and obtaining matching nodes; performing secondary verification on the matching nodes, mapping the nodes passing the verification to mode identifiers through the inverted index table, and outputting corresponding alarm results. The method solves the problems of poor real-time performance, unstable performance and low mode detection efficiency, and achieves the purposes of real-time detection, high stability, high performance and high compatibility.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method for detecting logs based on aggregation patterns using stream processing. Background Technology

[0002] Most log auditing systems rely on offline analysis, typically saving logs to a database such as Elasticsearch or PostgreSQL and then using SQL statements combined with pattern queries to detect alerts. This method suffers from poor real-time performance in many scenarios (such as industrial control), failing to detect logs immediately. Furthermore, offline analysis is performance-instantaneous, as CPU and memory are concentrated on querying all detection patterns within a set timer period. When the log volume is continuous, the distribution of CPU and memory usage for analysis becomes uneven, leading to poor stability. Therefore, this invention aims to perform real-time analysis on uninterrupted logs, distributing the analysis and computation time more evenly.

[0003] Regarding real-time data analysis, many inventions, based solely on simple analytical applications of Spark Streaming, haven't delved deeply into the algorithmic level. Once the number of detected patterns reaches hundreds or thousands, the algorithm's time complexity and detection performance don't significantly improve. With a continuous influx of logs, detection time exceeds log reception time, leading to blocking. Furthermore, these inventions haven't integrated with existing pattern libraries, lacking practical application scenarios. Therefore, this invention will implement a detection framework that combines existing performance-enhancing algorithms such as multi-pattern matching and inverted indexes with Spark Streaming's detection logic, integrating it with the Sigma pattern ecosystem to achieve significant improvements in both performance and application. For example, Chinese patent CN119322717B discloses a workflow log collection and visualization method, providing the following technical solutions: S1: Receiving log data from all microservices; S2: Employing AI retrieval, using natural language processing and machine learning for log pattern recognition and correlation analysis; S3: Using a tracing framework to maintain request context information throughout the microservice architecture; S4: Using real-time analysis tools for real-time monitoring and anomaly detection of the log stream; S5: Combining an event-driven architecture, automatically triggering alarm notifications to relevant team members for logs or patterns deemed important by analysis; S6: Auditing the current log level and content, optimizing log output, and ensuring the quality and relevance of log data. This invention, by implementing a centralized log aggregation and standardization platform, not only ensures the integrity and consistency of logs but also significantly reduces the difficulty of log retrieval. However, the aforementioned workflow log collection and visualization method only uses Spark Streaming for routine real-time monitoring, does not involve pattern detection performance optimization, relies on machine learning leading to a high false positive rate, and lacks integration with pattern libraries such as Sigma, resulting in insufficient detection efficiency and practicality. Summary of the Invention

[0004] This invention solves the problems of poor real-time performance, unstable performance, and low pattern detection efficiency in the prior art. It proposes a method for aggregated pattern detection logs based on stream processing, which achieves the goals of real-time detection, high stability, high performance, and high compatibility.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting logs based on aggregation patterns using stream processing includes: Multiple detection patterns are merged into pattern groups based on the aggregation field, and an AC automaton and inverted index table are built for each pattern group. The raw log stream is input into the stream processing engine, which divides the log into micro-batches using a sliding window and discards logs that exceed the window duration using a watermarking mechanism. Traverse each pattern group, and send the fields in the current micro-batch log into the AC automaton of the corresponding pattern group for multi-pattern matching to obtain the matching node; A second verification is performed on the matching nodes. The nodes that pass the verification are mapped to the pattern identifier through the inverted index table, and the corresponding alarm results are output.

[0006] Preferably, in the merged pattern group, if multiple patterns contain the same filtering condition, the filtering condition is merged into one, and different patterns are connected by a logical OR operator; the merged pattern group uses the aggregation field as the grouping key, and logs in the same window are grouped and detected by this field in the Spark stream processing engine.

[0007] Preferably, the Spark stream processing engine divides the raw log stream into multiple micro-batches using a sliding window, and each micro-batch is converted into an elastic distributed dataset. For each partition of the dataset, a user-defined function that encapsulates the matching logic of the AC automaton and the inverted index table is called. The matching is performed in parallel for each partition, and the matching results are merged and output on the driver side.

[0008] Preferably, in the construction of the AC automaton and the inverted index table, the construction of the AC automaton includes adding the filter condition values ​​of all patterns under the same pattern group as pattern strings to the trie, and adding a failure pointer to each node; during the matching process, when the log field value matches a node in the trie, the matching is continued by jumping through the failure pointer, and multiple pattern strings are matched in a single scan.

[0009] Preferably, the construction of the AC automaton and the inverted index table includes extracting key-value pairs of all filtering conditions under each pattern group, merging filtering conditions with the same key, and combining all values ​​under the same key into a list to form a mapping structure from string keys to string arrays. After matching is completed, the corresponding pattern identifier list is quickly located according to the matched key through the mapping structure.

[0010] Preferably, when performing secondary verification on the matching node, it is determined whether the matched filter condition value contains a wildcard. If it contains a wildcard and satisfies the wildcard matching mode, the matching result is retained and the inverted index mapping step is entered. If it contains a wildcard but does not satisfy the wildcard matching mode, or does not contain a wildcard and cannot completely match the log field value, the matching result is discarded.

[0011] Preferably, after mapping the verified nodes to pattern identifiers via an inverted index table, the corresponding complete pattern content is extracted based on the mapped pattern identifiers, and a second verification is performed with the log fields to confirm whether a complete match has been achieved; the second verification is only performed on the mapped parts of the patterns that require second verification.

[0012] Preferably, the detection pattern adopts the Sigma pattern format, which describes the log field matching conditions in the form of key-value pairs, including equality matching, wildcard matching, and regular expression matching; in the pattern parsing stage, the detection fields and condition values ​​of the Sigma pattern are extracted, and the key-value mapping of the pattern string of the AC automaton and the inverted index table is constructed.

[0013] Preferably, the sliding window uses a fixed window length and sliding step size. The window length is equal to the time span of the log data contained in each micro-batch, and the sliding step size is equal to the time offset of the window moving forward. The watermarking mechanism sets a delay threshold based on the window length. When the log event time is earlier than the current window start time minus the delay threshold, the log is discarded.

[0014] Preferably, after traversing each pattern group, when the number of log fields exceeds a preset threshold, the log fields are filtered, and only fields that match the keys existing in the current pattern group are retained and sent to the AC automaton for matching.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows.

[0016] 1. This invention divides the continuous log stream into uniform small batches and completes pattern detection in memory, avoiding the CPU and memory spikes caused by centralized processing in traditional offline analysis. It can also reduce alarm detection latency to the second level, with stable resource consumption, meeting the dual requirements of stability and real-time performance.

[0017] 2. The AC automaton of this invention achieves multi-pattern string matching in a single scan, with time complexity linearly related to text length. The inverted index quickly maps the matching results to pattern IDs, avoiding line-by-line comparison. The combination of these two features significantly improves the detection performance for thousands of patterns, eliminating log backlog and blocking.

[0018] 3. This invention seamlessly integrates the Sigma pattern into the Spark-Streaming detection engine. The pattern describes log matching conditions using key-value pairs, making it easy to extend and maintain. The pattern parsing stage completes merging, grouping, and index building, and can be directly called during the detection stage, improving the accuracy of security detection. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of a method for detecting logs based on aggregation patterns using stream processing, according to the present invention. Detailed Implementation

[0020] See Figure 1 As shown, a method for detecting logs based on stream processing aggregation patterns includes: Multiple detection patterns are merged into pattern groups based on the aggregation field, and an AC automaton and inverted index table are built for each pattern group. The raw log stream is input into the stream processing engine, which divides the log into micro-batches using a sliding window and discards logs that exceed the window duration using a watermarking mechanism. Traverse each pattern group, and send the fields in the current micro-batch log into the AC automaton of the corresponding pattern group for multi-pattern matching to obtain the matching node; A second verification is performed on the matching nodes. The nodes that pass the verification are mapped to the pattern identifier through the inverted index table, and the corresponding alarm results are output.

[0021] The main problems with existing technologies are: most rely on static detection, resulting in poor stability and real-time performance; they lack integration with existing mature and open-source pattern libraries, limiting their applicability; they lack performance optimization, leading to unreliable performance when integrating thousands of patterns; and their heavy reliance on machine learning is unsuitable for scenarios requiring low false positives and high detection rates. To address these shortcomings, this invention, based on the Spark platform, utilizes the sliding window feature of Spark-Streaming to achieve micro-batch processing of real-time logs. The pattern detection process is performed in memory, reducing I / O while increasing concurrency. Patterns with the same aggregation fields are merged, and pattern matching using the AC automaton is used to detect alarms, improving algorithm time complexity. Finally, an inverted index is used to match the corresponding pattern library pattern ID, enabling alarm output.

[0022] This invention proposes a security detection method for real-time log streams. Relying on a distributed stream processing framework and combining efficient string matching and indexing structures, it enables fast, stable, and low-latency aggregation analysis of log events even with large-scale pattern sets. First, in the system preparation phase, multiple predefined detection patterns need to be merged according to specific aggregation fields to form several pattern groups. The purpose of this operation is to group patterns with the same detection dimension together, so that subsequent stream processing can perform centralized matching based on the same grouping key. During the merging process, if multiple patterns contain completely identical filtering conditions, only one copy is retained, and different patterns are connected using a logical "OR" operator. The merged pattern groups will use the aggregation field as the grouping identifier. In the subsequent Spark stream processing engine, the system will perform grouping operations on log data within the same time window according to this field, thereby ensuring that logs belonging to the same detection dimension can be processed uniformly.

[0023] After the pattern groups are constructed, two core data structures need to be built for each pattern group: the Aho-Corasick automaton and an inverted index table. When building the Aho-Corasick automaton, the system extracts the filter condition values ​​from all patterns within the same pattern group and adds these values ​​as pattern strings to a trie structure. Based on the trie, a fail pointer is added to each node, thus forming the complete Aho-Corasick automaton. In this way, during subsequent matching, when a field value in the log enters the automaton for scanning, it can not only match multiple pattern strings simultaneously in a single traversal, but also jump to matching via the fail pointer, greatly improving the efficiency of multi-pattern matching.

[0024] To construct the inverted index table, the system extracts the key-value pairs of all filtering conditions under each pattern group, merges filtering conditions with the same key, and organizes all values ​​under the same key into a list, ultimately forming a mapping structure from string keys to string arrays. The purpose of this inverted index table is to quickly locate the corresponding pattern identifier list based on the matched key after matching, thereby avoiding traversing each pattern one by one and significantly speeding up the pattern matching process.

[0025] After the above construction process is completed, the system can enter the real-time processing stage of log data. The raw log data is first sent to the stream processing engine, which uses a sliding window mechanism to divide the continuous log stream into multiple micro-batches. Each micro-batch corresponds to a set of logs within a time segment. The window length and sliding step size can be configured according to actual needs: the window length determines the time span of the log data contained in each micro-batch, and the sliding step size defines the time offset by which the window moves forward. For example, a window can be set to 5 minutes, and the window can slide once every 1 minute. In this way, the system will generate a new micro-batch every 1 minute, containing log data from the most recent 5 minutes.

[0026] After each micro-batch is generated, the system also uses a watermarking mechanism to manage the event time of the logs. The watermarking mechanism sets a delay threshold based on the window length. When the event time of a log is earlier than the current window start time minus this delay threshold, the log will be judged as expired data and discarded. This design effectively prevents already processed logs from occupying memory space again, while reducing unnecessary matching counts and improving overall processing performance.

[0027] Micro-batch data is transformed into a Resilient Distributed Dataset (RDD) within the stream processing engine. Subsequently, the system calls a user-defined function (UDF) encapsulating the matching logic of an AC automaton and an inverted index table for each partition of this dataset. The matching tasks can be executed in parallel across partitions. After matching is complete, the results from each partition are merged at the driver end, ultimately outputting a unified alarm result. This partitioned parallel processing method fully leverages the advantages of distributed computing, ensuring real-time performance under high throughput.

[0028] In the specific matching process, the system iterates through the pre-built pattern groups. For the log data in the current micro-batch, it first checks if the number of fields exceeds a preset threshold. If it does, the fields are filtered, retaining only those fields that exist in the key set of the current pattern group. The filtered fields are then fed into the AC automaton for matching. When a field value in the log matches a node in the AC automaton's trie, the system obtains that matching node. Because the AC automaton supports multi-pattern matching, the entire log field scanning process can be completed in linear time complexity.

[0029] After a matching node is obtained, the system performs a secondary validation operation. During validation, it checks whether the matched filter value contains a wildcard. If the value contains a wildcard, it further checks whether it meets the wildcard matching pattern (e.g., the wildcard appears at the beginning or end of the string). If it does, the matching result is retained and the process continues to the inverted index mapping step. If it does not contain a wildcard, it checks whether the matched value can completely match the field value in the original log. If it completely matches, it is retained; otherwise, the matching result is discarded. This secondary validation mechanism effectively filters out candidate results that only partially match but do not actually conform to the pattern semantics.

[0030] Matching nodes that pass the secondary validation are then sent to the inverted index table for mapping. The system quickly finds the list of pattern identifiers associated with the key in the inverted index's mapping structure based on the key corresponding to the matching node. Because the inverted index uses a direct mapping from key to pattern identifier list, the entire search process is highly efficient, avoiding the inefficient operation of traversing each pattern one by one.

[0031] After obtaining the list of pattern identifiers, the system does not immediately output an alarm. To ensure accuracy, the system extracts the corresponding complete pattern content based on the mapped pattern identifiers and performs a second verification against the log fields. This verification is only performed on patterns that have been mapped and are designated to require secondary verification, rather than rescanning all patterns. This layered verification mechanism ensures both detection accuracy and avoids unnecessary computational overhead.

[0032] This invention adopts the Sigma schema format for defining the detection pattern. Sigma schema is an alarm detection pattern format that describes the matching conditions of log fields in key-value pairs, supporting various matching methods such as equality matching, wildcard matching, and regular expression matching. During the schema parsing phase, the system extracts the detection fields and condition values ​​from the Sigma schema, using this information to construct the schema string of the AC automaton and the key-value mapping of the inverted index table. This design allows this invention to seamlessly integrate into the Sigma schema ecosystem, possessing good scalability and practical application value.

[0033] Both window length and sliding step are configurable parameters. The window length determines the time span of log data contained in each micro-batch, while the sliding step determines the time offset by which the window moves forward. The delay threshold of the watermarking mechanism is usually set based on the window length to ensure that late data is processed within a reasonable range, while data exceeding the threshold is discarded. This design ensures both real-time performance and control over memory usage.

[0034] Furthermore, during the traversal of different pattern groups, the system dynamically optimizes based on the number of log fields. If the current log contains a large number of fields, the system compares it with the keys existing in the current pattern group, retaining only those fields whose keys are defined in the pattern group, and then sending these fields to the AC automaton for matching. This field filtering strategy can further reduce unnecessary data processing and improve matching efficiency.

[0035] This invention departs from the traditional offline analysis model of storing logs first and then scanning them centrally. Instead, it employs Spark Streaming's sliding window and watermarking mechanism to divide the continuous log stream into uniform micro-batches for processing. Each micro-batch consumes only limited memory and CPU resources, and the analysis and computation are evenly distributed over time, avoiding the problems of sudden load spikes and resource peaks when timers are triggered. This allows the system to maintain smooth operation even in environments with extremely high stability requirements, such as industrial control and finance. Simultaneously, the watermarking mechanism automatically discards old logs that exceed the window duration, preventing unlimited memory growth and reducing invalid matches, further ensuring the reliability of long-term operation.

[0036] In terms of detection performance and scalability, this invention integrates a collaborative algorithm of AC automaton and inverted index into the user-defined function of the stream processing engine. The AC automaton enables a single log field scan to simultaneously match all pattern strings in the pattern group, with time complexity linearly related to log length, replacing the inefficient polling method of pattern-by-pattern. The inverted index table, after the matching node passes secondary verification, directly maps from the key to the pattern identifier list, eliminating the need to traverse the pattern pool. When the number of patterns increases to thousands or more, the detection time no longer increases linearly with the number of patterns, thus avoiding blocking caused by the log inflow rate exceeding the detection rate.

[0037] The Sigma schema of this invention describes log field matching conditions in key-value pair format, supporting multiple patterns such as equality, wildcards, and regular expressions, and boasts a large and mature community schema library. By automatically parsing the Sigma schema to construct the AC automaton and inverted index, it allows direct application of thousands of existing security detection patterns without requiring users to redefine the format, significantly lowering the implementation threshold and enhancing the product's security detection capabilities. Compared to methods relying solely on machine learning, this invention employs precise pattern matching, making it more reliable in scenarios such as industrial control systems that require extremely low false positive rates and high detection rates, avoiding the uninterpretability and generalization risks associated with black-box models.

[0038] like Figure 1 In one embodiment shown, Figure 1This is an overall flowchart of a method for detecting logs based on aggregated patterns using stream processing, according to the present invention. The invention relates to the construction process of the pattern parsing and detection architecture, and the log detection process, all based on the Spark-Streaming platform. First, the pattern is parsed into a concrete data structure, and a detection engine is built in conjunction with an algorithm architecture (UDF). This is the configuration-side process. Then, the data-side process, i.e., the specific processing of the logs, is implemented.

[0039] The specific method for constructing the pattern parsing and detection architecture involved in this invention is as follows: Step 1: Merge all the thousands of patterns together according to the aggregation field. If a certain filter condition exists in multiple patterns, merge them into one. Use OR to connect different patterns.

[0040] Step 2: Add all patterns in the same pattern group to the AC automaton and use the AC automaton's multi-pattern matching to accelerate pattern search.

[0041] Step 3: Extract the keys from the key-value pairs of all filter conditions under this pattern group, such as "process.command_line", to form a list. The key in each list is used to identify all the values ​​under that key, that is, to attach all the values ​​of this key in the pattern group to this key.

[0042] Step 4: Construct a trie tree for all values ​​of each key under the pattern group, and combine string matching and wildcard matching (* before and after the string) into the same tree.

[0043] Step 5: Construct an inverted index table, extract the key-value pairs of all filter conditions for each group, merge filter conditions with the same key, and merge all values ​​under the same key into a list, thereby constructing a dictionary table structure of [String, Array].

[0044] Step 6: Add secondary validation to each leaf of the trie tree to distinguish between a complete string match and a match with wildcards. Since each leaf of the trie tree represents a filter condition, the results of the trie matching and secondary validation are pointed to the inverted index table, mapping all results (keys) of each filter matching pattern in the pattern group to the relevant pattern.

[0045] The log detection process involved in this invention is as follows: Step 1: The raw logs enter the Spark-Streaming data stream batch microprocessing engine.

[0046] Step 2: After the raw logs enter Spark, they are first grouped by a sliding window (e.g., every 5 minutes is a window, with a sliding interval of 1 minute). Then, logs older than 5 minutes are discarded by the watermark mechanism, thereby removing previously checked logs, preventing them from consuming Spark's memory, saving memory space, and reducing the number of matching operations to improve performance.

[0047] Step 3: Iterate through the pattern groups for different aggregation fields and examine each pattern group, combining... Figure 1 The example given is the detection of pattern groups aggregated by the host.name field.

[0048] Step 4: Send all existing fields in the log to the AC automaton of the current pattern group for detection, match them on the Trie tree, and obtain the matching results, which are the leaves of the Trie tree.

[0049] Step 5: Perform a second matching on the matched results to determine whether the value of the matched filter contains wildcards. If it does not contain wildcards and cannot completely match the value of the original log, it is discarded. Then, the matched leaves are mapped in the inverted index table to collect the hit pattern IDs.

[0050] Step 6: Find the matching pattern ID, extract the pattern of the ID, and perform a second matching, thereby avoiding matching all patterns in the pattern group and speeding up the matching efficiency.

[0051] Spark Streaming is a real-time stream processing framework based on Apache Spark. It divides data streams into micro-batches based on time, achieving low-latency, high-throughput distributed stream computing through a batch processing engine. Sliding window is a dynamic time-range limiting mechanism where the window moves forward in fixed steps, allowing repeated statistical or aggregation operations on the latest batch of data in a continuous data stream. Sigma pattern is an alarm detection pattern format used in the security field (especially in SIEM / SOC scenarios). AC automaton is a multi-pattern string matching algorithm that, by constructing a Trie tree and adding failure pointers, simultaneously matches all pattern strings in a single text scan, with time complexity linearly related to text length. Inverted index is a mapping structure from words (terms) to a list of document IDs used in full-text search systems. It quickly locates the collection of documents containing a term, supporting efficient Boolean queries and relevance ranking. UDF is any Spark code written in Python, Scala, or Java, then encapsulated into a function that can be executed on each row or group of data in a DataFrame.

[0052] In summary, the advantages of this invention are as follows: 1. Using Spark for log analysis and machine learning is a static analysis method that requires centralized data processing, leading to high and unstable CPU and memory usage, making it unsuitable for stable environments such as industrial control systems. This invention employs real-time window analysis and micro-batch processing of log streams, avoiding the processing of large amounts of logs at once and maintaining stable and reliable memory and CPU usage.

[0053] 2. By utilizing Spark for real-time data processing, this invention improves detection performance from the perspective of specific performance optimization algorithms.

[0054] 3. This invention combines the Sigma model and uses a pattern detection method, which is more reasonable and accurate. Compared with the machine learning method for detecting logs, there is a contradiction between high false alarms and low detection, and it is not applicable in certain specific scenarios (such as industrial control systems that require accurate detection).

[0055] 4. This invention integrates the Sigma model into the Spark-Streaming detection engine, expanding the applicability of the technology and enhancing its security capabilities, thereby enabling better productization.

[0056] All data collection and extraction in this invention are carried out under compliant and legal conditions.

Claims

1. A method for detecting logs based on aggregation patterns using stream processing, characterized in that, include: Multiple detection patterns are merged into pattern groups based on the aggregation field, and an AC automaton and inverted index table are built for each pattern group. The raw log stream is input into the stream processing engine, which divides the log into micro-batches using a sliding window and discards logs that exceed the window duration using a watermarking mechanism. Traverse each pattern group, and send the fields in the current micro-batch log into the AC automaton of the corresponding pattern group for multi-pattern matching to obtain the matching node; A second verification is performed on the matching nodes. The nodes that pass the verification are mapped to the pattern identifier through the inverted index table, and the corresponding alarm results are output.

2. The method for detecting logs based on aggregation patterns using stream processing according to claim 1, characterized in that, In the process of merging into a pattern group, if multiple patterns contain the same filtering condition, the filtering condition is merged into one, and different patterns are connected by a logical OR operator. The merged pattern group uses the aggregation field as the grouping key, and the logs in the same window are grouped and detected by this field in the Spark stream processing engine.

3. The method for detecting logs based on aggregation patterns using stream processing according to claim 2, characterized in that, The Spark stream processing engine divides the raw log stream into multiple micro-batches using a sliding window, and each micro-batch is converted into an elastic distributed dataset. For each partition of the dataset, a user-defined function that encapsulates the matching logic of the AC automaton and the inverted index table is called. The matching is performed in parallel for each partition, and the matching results are merged and output on the driver side.

4. The method for detecting logs based on aggregation patterns using stream processing according to claim 3, characterized in that, The construction of the AC automaton and the inverted index table includes adding the filter condition values ​​of all patterns under the same pattern group as pattern strings to the trie, and adding a failure pointer to each node. During the matching process, when the log field value matches a node in the trie, the matching continues by jumping to the failure pointer, matching multiple pattern strings in a single scan.

5. A method for detecting logs based on aggregation patterns using stream processing, as described in claim 3 or 4, characterized in that... The construction of the AC automaton and the inverted index table includes extracting key-value pairs of all filtering conditions under each pattern group, merging filtering conditions with the same key, and combining all values ​​under the same key into a list to form a mapping structure from string keys to string arrays. After matching is completed, the corresponding pattern identifier list is quickly located according to the matched key through the mapping structure.

6. The method for detecting logs based on aggregation patterns using stream processing according to claim 5, characterized in that, When performing secondary verification on the matching node, it is determined whether the matched filter condition value contains a wildcard. If it contains a wildcard and meets the wildcard matching mode, the matching result is retained and the inverted index mapping step is entered. If it contains a wildcard but does not meet the wildcard matching mode, or does not contain a wildcard and cannot completely match the log field value, the matching result is discarded.

7. The method for detecting logs based on aggregation patterns using stream processing according to claim 6, characterized in that, After mapping the verified nodes to pattern identifiers via the inverted index table, the corresponding complete pattern content is extracted based on the mapped pattern identifiers, and a second verification is performed with the log fields to confirm whether a complete match has been achieved. The second verification is only performed on the mapped parts of the patterns that require second verification.

8. The method for detecting logs based on aggregation patterns using stream processing according to claim 1, characterized in that, The detection pattern adopts the Sigma pattern format, which describes the matching conditions of log fields in the form of key-value pairs, including equality matching, wildcard matching and regular expression matching. In the pattern parsing stage, the detection fields and condition values ​​of the Sigma pattern are extracted, and the key-value mapping of the pattern string of the AC automaton and the inverted index table is constructed.

9. The method for detecting logs based on aggregation patterns using stream processing according to claim 8, characterized in that, The sliding window uses a fixed window length and sliding step size. The window length is equal to the time span of the log data contained in each micro-batch, and the sliding step size is equal to the time offset of the window moving forward. The watermarking mechanism sets a delay threshold based on the window length. When the log event time is earlier than the current window start time minus the delay threshold, the log is discarded.

10. A method for detecting logs based on aggregation patterns using stream processing, as described in claim 1 or 9, characterized in that, After traversing each pattern group, when the number of log fields exceeds a preset threshold, the log fields are filtered, and only fields that match the keys existing in the current pattern group are retained and sent to the AC automaton for matching.

Images