A method and system for dynamic matching of real-time data stream alarm rules
By migrating alarm rule matching and statistical calculation in the security monitoring system to memory execution, and adopting a memory sliding window and asynchronous persistence mechanism, the problem of existing security systems being unable to respond quickly to dynamic security situations is solved, achieving efficient, stable real-time alarms and low-cost operation and maintenance.
Patent Information
- Application Number
- CN202611301180.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
Existing security monitoring systems suffer from several problems, including inability to respond quickly to dynamic changes in the security situation, lagging rule adjustments, insufficient system processing capacity, limited device access scale, inability to guarantee continuous 24/7 protection, alarm interruptions during system maintenance, and high operation and maintenance costs.
The alarm rule matching and statistical calculation are migrated to memory execution, adopting a memory sliding window structure isolated by quadruples, an asynchronous batch persistence mechanism, a cold start automatic replay mechanism, lazy and global cleanup strategies, a two-level write invalidation cache, a four-layer decoupled architecture, and an embedded database.
It achieves millisecond-level alarm response, improves statistical calculation efficiency, supports large-scale device concurrency, ensures stable system operation, reduces operation and maintenance costs and deployment complexity, and meets the requirements for 24/7 uninterrupted operation.
Smart Images

Figure CN123115255A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of security monitoring technology, and specifically relates to a method and system for dynamic matching of real-time data stream alarm rules. Background Technology
[0002] With the deepening of the construction of safe cities and smart parks, security monitoring systems have fully shifted from the traditional human-monitored screen mode to digitalization and intelligence. A typical medium-to-large-sized park security system usually includes hundreds to thousands of video surveillance cameras, dozens to hundreds of access control devices, a large number of environmental monitoring sensors (temperature, humidity, smoke detectors, vibration, etc.), and video AI analysis servers. These devices and systems continuously generate structured event data: access control card swipe records, personnel behavior events identified by video AI, environmental data collected by sensors, etc.
[0003] The core responsibility of security monitoring and maintenance personnel is to promptly detect anomalies and trigger alarms based on these massive amounts of real-time event data, notifying security personnel to take appropriate measures. This business scenario places four rigid requirements on the underlying alarm system: real-time performance, rule flexibility, continuous reliability, and scalability.
[0004] With the widespread adoption of surveillance, its application scenarios have become more extensive, leading to a greater number of problems. In the security surveillance industry, the core challenge faced by frontline maintenance personnel every day is not the technology itself, but rather the severe mismatch between the dynamic changes in the security situation and the ability to adjust alarm rules. A park's security level may need to switch from normal mode to the highest alert mode within minutes due to an emergency; video AI identifying new abnormal behavior patterns requires the immediate addition of corresponding alarm rules; seasonal changes cause fluctuations in the normal operating parameters of equipment, necessitating the synchronous adjustment of threshold alarms for temperature, vibration, etc. These are the real-world work scenarios that maintenance personnel face daily.
[0005] Current security monitoring systems face a significant challenge: the speed of rule adjustments cannot keep pace with the ever-changing security landscape. Maintenance personnel encounter the following issues in their daily work: severely delayed rule adjustment responses, inability to match dynamic security situations, insufficient system processing capacity, limited device connectivity, alarm interruptions during system maintenance, inability to guarantee 24 / 7 continuous protection, long-term system instability, and high maintenance costs. Therefore, providing security monitoring maintenance personnel with a real-time data stream matching system that allows for flexible configuration and dynamic adjustment of alarm rules is a core business problem that needs to be addressed. Summary of the Invention
[0006] The purpose of this invention is to provide a method and system for dynamic matching of real-time data stream alarm rules, addressing the aforementioned problems, in order to improve and solve the problem that existing security systems cannot apply rules that require flexible adjustment.
[0007] The technical solution adopted in this invention is as follows: A method for dynamic matching of real-time data stream alarm rules, the method comprising the following steps: Step S100: Receive event data reported by security monitoring equipment, read the dataset format definition corresponding to the event data from the preset dataset cache, and perform format verification on the event data; Step S200: Flatten the validated event data into a single-layer format data to ensure that the field structure of the event data is consistent with the database table structure and rule definition; Step S300: Read all enabled alarm rules in the current dataset from the preset rule cache. Alarm rules include simple conditions and / or statistical conditions. Step S400: Perform alarm rule matching in memory. For each alarm rule, perform simple condition matching and / or statistical condition matching in sequence to determine whether the event data meets the triggering conditions of the alarm rule. Statistical condition matching involves grouping the event data by specified dimensions within a preset time window, performing aggregated statistical calculations, and comparing the statistical results with a preset threshold. Step S500: After writing the event data and alarm matching results into the asynchronous persistent queue, return the alarm results to the caller; the background thread reads data in batches from the asynchronous persistent queue and writes it into the database; Step S600: When the system restarts, historical event data is loaded from the database and replayed in chronological order to the memory statistics window to restore the statistical state required to match the statistical conditions.
[0008] Furthermore, the statistical condition matching in step S400 includes: A unique window identifier is created for each quadruple, consisting of dataset name, rule ID, statistical condition index, and grouping field value. An independent memory statistical window is established for each quadruple. The memory statistics window uses a double-ended queue to store time-series data, and each record contains an event timestamp and statistical field values. When a new event arrives, the new event is appended to the tail of the double-ended queue according to the memory statistics window corresponding to the quadruple location. Based on a preset time window, valid data is filtered from the double-ended queue, aggregated statistical calculations are performed on the valid data, the calculation results are compared with a preset threshold, and the matching results are output.
[0009] Furthermore, lazy cleanup and global cleanup are configured to manage memory usage in the memory statistics window, specifically including the following: Lazy cleanup: When the number of valid data in the deque is less than half of the total number of records, replace all data in the deque with valid data and release the memory occupied by expired data. Global cleanup: Whenever the number of statistical calculations reaches a preset threshold, a global window scan is triggered to clean up expired data in all memory statistical windows that exceeds the global maximum retention time, and to delete empty windows that no longer contain data.
[0010] Furthermore, in step S500, the background thread reads data in batches from the asynchronous persistent queue and writes it to the database, specifically including: A background thread continuously listens to the asynchronous persistent queue at preset timeout intervals. A batch write operation is triggered when the accumulated data in the asynchronous persistent queue reaches a preset batch threshold, or when the time interval since the last batch write reaches a preset time interval. The batch write operation includes: reading data in batches from the asynchronous persistent queue, detecting the database table structure, automatically performing table structure expansion if the data contains new fields, and writing the data in batches to the database; When the system shuts down normally, all remaining data in the asynchronous persistent queue is written to the database, and the database connection is closed after the data has been written to disk.
[0011] Furthermore, step S600 specifically includes: Iterate through all created datasets and filter out alert rules that contain statistical conditions; For each dataset, take the longest time window among all statistical rules in that dataset, compare it with the global maximum retention time, and take the smaller value as the playback historical time range; Retrieve all historical events within the historical time range from the database and sort them in ascending chronological order. Historical events are written to the corresponding memory statistics window one by one. Only the window data state is restored. Alarm matching calculation is not performed and alarm records are not generated.
[0012] Furthermore, in step S200, the event data is flattened into a single-layer format, specifically including: Identify the nested structure in the event data, connect the nested fields in hierarchical order with a fixed delimiter to form flat field names, and convert them into a single-level key-value pair format, so that the field structure of the event data is consistent with the storage field structure of the database and the reference field structure of the alarm rules.
[0013] Furthermore, in step S300, the rule cache and dataset cache adopt a write-on-failure strategy: When processing a certain type of device event data for the first time, the corresponding dataset format definition and alarm rule configuration are loaded from the database and stored in the dataset cache and rule cache, respectively. When a dataset or alarm rule is created, modified, deleted, or started / stopped, the corresponding cached item in the dataset cache or rule cache should be cleared immediately. During the execution of steps S100 to S400, the rule cache and dataset cache are always hit when they are not invalidated, and no database query operation is generated.
[0014] Furthermore, the method also includes step S700: mode switching, in response to an external command, switching to full processing mode, storage mode only, or matching test mode only; In the complete processing mode, the complete process from step S100 to step S500 is executed; In storage-only mode, steps S300 and S400 are skipped, and event data is directly written to the asynchronous persistent queue. In the test-only matching mode, the matching process from step S100 to step S400 is executed, the matching result is returned but not written to the asynchronous persistent queue, and the memory statistics window state is not modified.
[0015] Furthermore, a real-time data stream alarm rule dynamic matching system is provided, which adopts a four-layer layered architecture, including: The interface service layer is used to receive event data reported by the device, as well as to receive configuration operations of datasets and alarm rules by operation and maintenance personnel; The engine control layer and the interface service layer are used to manage rule caching and dataset caching, schedule alarm rule matching processes, and manage asynchronous persistent queues. The memory matching layer connects to the engine's main control layer and is used to perform simple condition matching and statistical condition matching of alarm rules in memory. Statistical condition matching includes aggregation and statistical calculations by grouping according to specified dimensions within a preset time window. The persistent storage layer connects to the engine control layer and includes an asynchronous writer and an embedded database. The asynchronous writer is used to read data in batches from the asynchronous persistent queue and write it to the embedded database. The cold start management module connects the persistent storage layer and the memory matching layer. It is used to load historical event data from the embedded database when the system restarts and replay it in chronological order to the memory statistics window of the memory matching layer to restore the statistical status.
[0016] Furthermore, the memory matching layer includes: The simple condition matching module is used to perform conditional judgments on the field values of a single event data entry. The sliding window aggregation module uses a quadruple consisting of dataset name, rule ID, statistical condition index, and grouping field value as a unique window identifier, stores time-series data in a double-ended queue, and performs aggregation statistical calculations within a preset time window. The memory management module is used to perform lazy cleanup and global cleanup. Lazy cleanup releases expired data when the number of valid data in the double-ended queue is less than half of the total number of records. Global cleanup scans and cleans up expired data in all memory statistics windows that exceeds the global maximum retention time at a preset number of statistical calculations. The event replay module is used to receive historical event data during cold start, write the historical events one by one into the corresponding memory statistics window, and only restore the window data state without performing matching calculations.
[0017] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention migrates all alarm matching and statistical calculations to memory execution, eliminating database read / write operations on the critical path of event processing, thus fundamentally eliminating the defect of database synchronous read / write locking up system throughput in the prior art.
[0018] 2. This invention employs a dedicated memory sliding window structure with quadruple key isolation, storing time-series data in a double-ended queue. This completely replaces existing database query and statistical methods, fundamentally eliminating the inefficient process of disk read / write, query parsing, and full table scans required for each statistical calculation. This improves statistical calculation efficiency, with no significant performance degradation even when multiple statistical rules run in parallel. Maintenance personnel can simultaneously configure multiple statistical alarm rules for different devices, regions, and time windows, and the system can respond in real time, no longer limited by finite preset templates.
[0019] 3. This invention completely decouples alarm calculation from data storage through an asynchronous batch persistence mechanism. After the main thread completes alarm calculation, it only needs to perform a non-blocking enqueue operation, while the background thread independently completes the batch write, and a forced flush is performed when the service is shut down. This fundamentally eliminates the shortcomings of existing technologies where synchronous writes lead to a trade-off between performance and reliability. It achieves zero loss of event data and alarm records without affecting the millisecond-level response of real-time alarms. While enjoying ultra-fast alarms, operations and maintenance personnel ensure that all event data and alarm records are completely preserved, supporting post-event auditing and problem tracing.
[0020] 4. This invention employs a cold-start automatic replay mechanism to automatically load historical events from the database and restore the statistical window state upon system startup. It only restores the state without repeating matching, thus avoiding duplicate alarms. This fundamentally eliminates the defects of existing technologies, such as data loss in the statistical window and gaps in alarm capabilities after service restarts. After system upgrades, maintenance, or fault recovery, all statistical alarm rules immediately resume normal operation without any alarm gaps. Maintenance personnel do not need to worry about missed alarms during system maintenance, fully meeting the rigid requirement of 24 / 7 uninterrupted operation of security systems.
[0021] 5. This invention employs a two-tiered memory management strategy combining lazy cleanup and global timed cleanup, coupled with a global maximum window duration limit. This fundamentally eliminates the system crashes caused by the continuous accumulation of expired data and unlimited memory growth inherent in existing technologies. System memory usage remains controllable, eliminating the need for manual data cleanup or periodic restarts by operations personnel during long-term operation. Operations personnel can focus on the security aspects of the system without being distracted by system stability issues.
[0022] 6. This invention employs a two-level write-invalidation caching mechanism to immediately clear the cache when rules or datasets change, ensuring the latest configuration is loaded the next time an event is processed. This fundamentally eliminates the shortcomings of existing technologies, such as querying the database for each event and uncertain delays in rule changes taking effect. After operations and maintenance personnel modify alarm rules through the interface, the new rules take effect immediately upon the arrival of the next event, without requiring a system restart or waiting for a refresh. When the security situation changes, operations and maintenance personnel can adjust the rules and put them into use immediately within seconds.
[0023] 7. This invention adopts a four-layer decoupled modular architecture, with each layer having a single responsibility and unidirectional data flow. This fundamentally eliminates the shortcomings of existing technologies where strong coupling between alarm calculation, data storage, and interface services prevents independent optimization. Optimization and upgrades of any module will not affect the normal operation of other modules, significantly improving the system's maintainability and scalability.
[0024] 8. This invention uses an embedded DuckDB database, eliminating the need for independent deployment of database services or big data clusters. It fundamentally eliminates the strong dependence of existing technologies on external database services and the resulting deployment and maintenance complexity. The entire system can run on a single machine with a single process, making installation and deployment simple and quick, requiring no professional database maintenance personnel. For the numerous small and medium-sized projects in the security industry, deployment and maintenance costs are significantly reduced, perfectly suited to the actual situations of limited project budgets and simple on-site conditions. Attached Figure Description
[0025] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation
[0026] The present invention will now be described in detail with reference to the accompanying drawings.
[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0028] In the current security monitoring industry, most alarm system solutions adopt a synchronous processing architecture based on relational databases. The business process of this existing solution is as follows: After a device reports event data, the system first synchronously writes the event data to the database. Then, it executes database queries to perform conditional checks and statistical calculations for the alarm rules. For example, to determine whether a user has failed to swipe their card more than 5 times in 5 minutes, the system needs to query the database for all card swipe records of that user in the last 5 minutes, count the number of failures, and then compare it with a threshold. Throughout this process, from rule matching to statistical calculations to data storage, each step requires waiting for the database read and write operations to complete before proceeding to the next step.
[0029] The key technical features of this existing solution include: all alarm calculation logic relies on database execution and has no independent in-memory computing capability; event data is written to the database synchronously one by one, and business threads need to wait for the writing to be completed before returning results; statistical alarms rely on database SQL queries and have no dedicated in-memory statistical data structure; there is no alarm status recovery capability after system restart; rule and dataset configuration information is not cached, and the database needs to be queried for each event processing.
[0030] The problems faced by the existing solutions mentioned above are as follows: First, alarm response is severely delayed. Because processing each event requires multiple database reads and writes, the system can only handle approximately 35 to 54 events per second. In a campus with 500 cameras, the number of events generated per second by the video AI may far exceed this limit, causing a large backlog of events in the queue. Alarm latency worsens from milliseconds to seconds or even longer, meaning that by the time maintenance personnel see the alarm, the abnormal event may have already occurred long ago.
[0031] Secondly, statistical alerts are extremely inefficient. The most commonly used statistical alert rules by operations and maintenance personnel (such as frequency statistics and average statistics) require scanning a large amount of historical data in the database each time a calculation is triggered. When multiple statistical rules are in effect simultaneously, the database query pressure increases dramatically, further deteriorating system performance and potentially leading to database response timeouts and complete paralysis of the alerting function.
[0032] Third, system maintenance leads to alarm gaps. This solution lacks any mechanism to save and restore the ongoing statistical calculations. After each system restart, all historical data within the statistical window is lost, and maintenance personnel face the risk of statistical alarms becoming completely ineffective for a period after the restart (equal to the duration of the maximum statistical window). This is a fatal flaw in security scenarios requiring 24 / 7 uninterrupted protection.
[0033] Fourth, the system slows down or even crashes after long-term operation. This solution lacks an effective historical data cleanup mechanism, leading to a continuous accumulation of event records in the database and a sustained decline in query performance. Maintenance personnel experience the system becoming increasingly slow with use, necessitating periodic manual data cleanup or service restarts to maintain availability.
[0034] Fifth, the deployment and maintenance threshold is high. This solution typically requires independent deployment of database services or even big data clusters, with complex installation and configuration, requiring professional database maintenance personnel. For the numerous small and medium-sized projects in the security industry, the deployment and maintenance costs of this heavy-duty architecture far exceed the project budget.
[0035] To address the aforementioned problems with existing solutions, this invention provides a real-time data stream alarm rule dynamic matching method and system. It migrates all alarm rule matching and statistical calculations to memory, completely eliminating the constraints of database read / write operations on event processing speed. This enables a single server to support real-time processing of over 1500 events per second, with alarm latency controlled at the millisecond level, meeting the rigid real-time requirements of security scenarios. Furthermore, it designs a dedicated in-memory statistical data structure, allowing maintenance personnel to flexibly define statistical dimensions, time windows, aggregation methods, and thresholds. Statistical data from different dimensions are completely isolated and do not interfere with each other, significantly improving the computational efficiency and configuration flexibility of statistical alarms. It also achieves asynchronous persistent storage of event data without affecting... While ensuring real-time alarm performance, the system guarantees zero loss of all event data and alarm records, while providing a data foundation for system recovery after restarts. An automatic recovery mechanism is built to automatically restore the statistical window state from historical data upon service restart, ensuring alarm rules immediately resume normal operation without any alarm gaps, meeting the 24 / 7 uninterrupted operation requirements of the security system. A multi-layered memory management strategy is designed to automatically clean up expired data and limit memory growth, ensuring long-term stable system operation without manual intervention from maintenance personnel. An embedded database replaces external database services, enabling lightweight, out-of-the-box deployment and reducing the deployment threshold and maintenance costs of security projects.
[0036] Example 1 One embodiment of the present invention is a dynamic matching method for real-time data stream alarm rules. When a device event (such as an access control card swipe record) arrives in the system, the processing flow is as follows: Step 1: The interface service layer receives event data pushed by the device through the HTTP interface, supporting both single push and batch push modes.
[0037] Step 2: Retrieve the format definition of this type of device data from the dataset cache, and verify whether the required fields for the event are complete and whether the field types are correct. If the verification fails, return an error message directly. The entire verification process reads from the cache and does not query the database.
[0038] Step 3: Flatten event data that may contain nested structures into a single-level format. For example, the id field under the user object in the nested data structure reported by the device will be converted into a flat field name of user__id to maintain consistency with the database table structure and rule definition.
[0039] Step 4: Read all enabled alerting rules for the current dataset from the rule cache. If no rules are enabled, skip the matching and proceed directly to the data storage step. The entire rule loading process reads from the cache and does not query the database.
[0040] Step 5: Perform alarm rule matching in memory. For each rule, simple condition matching and time window statistical matching are performed sequentially. Both conditions must be met simultaneously for an alarm to be considered triggered. All calculations are completed entirely in memory, without any database operations.
[0041] Statistical alerts are the most critical alerting capability in security operations and maintenance. When operations and maintenance personnel configure a statistical rule (e.g., grouped by user_id, count(...) within 300 seconds...),... (where count is the number of times the event is counted) When the system receives each new event, it performs statistical calculations according to the following steps: Step 501, Parse Event Timestamp: Parse the business occurrence time of the event based on the time field configured in the rules. If not specified, the current system time will be used. Automatic recognition of multiple time formats is supported.
[0042] Step 502, construct the window isolation key: Generate a quadruple consisting of the dataset name, rule ID, statistical condition index, and grouping field value as a unique window identifier. This design ensures that statistical data between different datasets, rules, statistical conditions, and grouping values are completely isolated and do not interfere with each other. For example, the card-swiping failure count for user A and the count for user B are stored in their respective independent windows.
[0043] Step 503, extract statistical field values: For count-type statistics, there is no need to extract field values; for statistical types that require numerical calculations, such as summation and average, extract the values of the specified fields from the event and convert them into numerical format.
[0044] Step 504, Write to Statistics Window: Append the event timestamp and field value as a record to the tail of the corresponding window's deque. A deque is a data structure where both ends can be efficiently added to and deleted from, making it ideal for sliding window scenarios.
[0045] Step 505, Filter valid data: Based on the time window length configured in the rules, filter out valid records within the time range of one window before the current time point.
[0046] Step 506, Lazy Expiration Cleanup: If the number of valid records in the window is less than half of the total number of records, it means that a large number of records have expired. At this time, the original window data is replaced with valid records, releasing the memory occupied by expired data. This on-demand cleanup method avoids the overhead of performing a cleanup operation for every calculation.
[0047] Step 507, Perform statistical calculations: Perform statistical operations on valid records. The system supports seven statistical functions: count, sum, average, maximum, minimum, count_distinct, and standard deviation, which can cover most statistical needs in security operation and maintenance scenarios.
[0048] Step 508, Threshold Comparison: Compare the statistical calculation results with the threshold configured in the rule, and return whether the alarm condition has been met.
[0049] The accompanying global window cleanup mechanism: After every 5000 statistical calculations, the system automatically triggers a global window scan to clean up expired data in all windows that exceeds the global maximum retention time (default 30 days) and delete empty windows with no data, thereby achieving global control over memory usage.
[0050] Step 6: Write the raw event data and alarm matching results to the thread-safe queue of the asynchronous writer. The write operation is non-blocking, and the main thread does not need to wait for the data to be actually written to the database.
[0051] After completing alarm calculations, the main thread places the event data and alarm results into a thread-safe queue using a non-blocking enqueue operation, incurring almost no additional latency. A background write thread continuously listens to the queue, reading data in a loop with a 50-millisecond timeout period. Batch writing is triggered when either of the following conditions is met: the accumulated data in the queue reaches a preset batch limit (e.g., 500 records), or the preset maximum waiting interval (e.g., 1 second) since the last write is reached. During batch writing, the background thread automatically checks the database table structure; if new fields appear in the event, it automatically performs a table structure expansion operation. Any write exceptions are captured and logged, without affecting subsequent data processing.
[0052] When the system shuts down normally, the asynchronous writer, upon receiving a stop signal, will first write all remaining data in the queue to the database at once. Only after confirming that all data has been written to disk will it close the database connection. This graceful shutdown mechanism ensures that no data is lost during normal shutdown.
[0053] When the system restarts, historical event data is loaded from the database and replayed in chronological order to the memory statistics window to restore the statistical state required to match the statistical conditions.
[0054] Cold start playback is the core mechanism for ensuring uninterrupted 24 / 7 alarm operation in security systems. The following process is automatically executed upon system startup: Step 601: Traverse all created datasets; Step 602: Filter out the enabled rules containing statistical conditions for each dataset; skip datasets without statistical rules. Step 603: Take the largest time window duration among all statistical rules in the dataset, and the smaller value between it and the global maximum retention duration, as the historical time range that needs to be replayed; Step 604: Query all historical events within this time range from the database and sort them in ascending chronological order; Step 605: Write the historical events one by one into the memory statistics window of the corresponding rule.
[0055] A key point about cold start replay is that the replay process only writes event data to the statistics window; it does not perform alarm matching calculations or generate alarm records. This is because these historical events have already triggered alarms during the initial processing (if they met the criteria). If matching were performed again during replay, it would generate a large number of duplicate alarms, interfering with normal business operations. The sole purpose of replay is to restore the data state of the statistics window, allowing for accurate statistical judgments based on the complete historical context when new events arrive.
[0056] After actual testing and verification: Create a group by user_id, and count (...) within 120 seconds... The statistical rule (>=3) is used to push 5 events from the same user and then restart the service. After restarting, the 6th event from that user is pushed, and the rule is immediately hit (there are already 5 replay history records in the window). At the same time, an event from a new user is pushed, but the rule is correct and the event is not hit. This proves that the cold start replay mechanism can completely restore the statistical window state.
[0057] Experimental results show that single-machine processing capacity has increased from 35-54 QPS in existing technologies to over 1500 QPS, an improvement of approximately 30 times; P50 latency is as low as 12 milliseconds, and P99 latency is controlled within 35 milliseconds. For operations and maintenance personnel, this means that a single server can support concurrent event reporting from thousands of security devices, with alarm pushes being almost real-time, eliminating missed opportunities for response due to insufficient system processing capacity. Alarm response time has been improved from seconds to milliseconds, meeting real-time alarm requirements while also supporting large-scale concurrent device access.
[0058] Step 7: Immediately return the alarm result to the caller, including the event's unique identifier, whether a rule was hit, the number of rules hit, and detailed alarm information. The entire delay from the event to the return of the result is controlled within milliseconds.
[0059] Step 8 (in parallel with the main process): The background thread of the asynchronous writer reads data in batches from the queue and writes it to the DuckDB database to achieve data persistence. This process runs independently of the main thread and does not affect the alarm processing speed.
[0060] To ensure that no database queries are generated throughout the entire event handling process, the system implements a two-level cache: Rule Cache: Stores the complete configuration of all enabled rules for a given dataset, using the dataset name as the key. It loads from the database and caches the rule configuration the first time a device event is processed; subsequent processing uses the cache directly. When operations personnel create, modify, delete, or enable / disable rules via the interface, the rule cache for the corresponding dataset immediately becomes invalid, and the latest configuration is automatically reloaded the next time an event is processed.
[0061] Dataset Cache: Stores the field structure definition of a dataset using the dataset name as the key. It is loaded from the database and cached when a certain type of device event is validated for the first time; subsequent validations use the cache directly. The cache is immediately invalidated when the dataset is created or modified.
[0062] Both levels of caching employ a write-on-invalidation strategy: the cache is only cleared when the configuration changes, and used at all other times. Since the frequency of rule and dataset modifications (several times per day) is much lower than the event processing frequency (thousands of times per second), this strategy achieves zero database queries for hot paths at a very low cost, while ensuring strong configuration consistency. After operations and maintenance personnel modify the rules, the latest configuration can be used for the next event processing.
[0063] This embodiment executes alarm rule condition matching and time window statistical calculation entirely in memory. There are no database read / write operations on the critical path of event processing, completely eliminating the constraints of database read / write on alarm real-time performance at the architectural level, enabling security operations personnel to obtain millisecond-level alarm responses. A memory-based sliding window statistical structure is used, isolated by a four-tuple key. The dataset name, rule ID, statistical condition index, and grouping field value tuple serve as unique window identifiers. A double-ended queue is used to store time-series data, achieving complete physical isolation of statistical data between different device types, different alarm rules, different statistical dimensions, and different grouping objects. A dual-layer memory management strategy of lazy cleanup and global timed cleanup prevents unlimited memory growth. Simultaneously, an asynchronous batch write mechanism completely decouples the main thread from persistence. An independent background write thread and a thread-safe queue are used. The main thread only needs a non-blocking enqueue operation to complete data transfer; the background thread performs batch writes based on dual trigger conditions of batch number and time interval; dynamic expansion of the database table structure is supported; forced disk flushing ensures zero data loss when the service is shut down. An automatic state recovery mechanism is also included for cold starts. Upon service startup, a historical event replay process is automatically executed, loading recent historical events from the database and writing them to the in-memory statistics window in ascending chronological order. The key design feature is that the replay process only restores the window data state without repeating matching calculations, thus fully restoring the statistical context while avoiding duplicate alarms and ensuring seamless alarm capability continuity after a security system restart. A two-level write invalidation caching mechanism is employed. A two-level cache system—rule cache and dataset cache—is built, with configuration information retrieved from the cache throughout event processing. The corresponding cached item is immediately invalidated only when a rule or dataset changes, balancing extreme performance with strong configuration consistency, ensuring that rule modifications by operations personnel take effect immediately.
[0064] Example 2 Another embodiment of the present invention is a real-time data stream alarm rule dynamic matching system, the system adopting a four-layer layered architecture, including: The interface service layer is used to receive event data reported by the device, as well as to receive configuration operations of datasets and alarm rules by operation and maintenance personnel; The engine control layer and the interface service layer are used to manage rule caching and dataset caching, schedule alarm rule matching processes, and manage asynchronous persistent queues. The memory matching layer connects to the engine's main control layer and is used to perform simple condition matching and statistical condition matching of alarm rules in memory. Statistical condition matching includes aggregation and statistical calculations by grouping according to specified dimensions within a preset time window. The memory matching layer includes: The simple condition matching module is used to perform conditional judgments on the field values of a single event data entry. The sliding window aggregation module uses a quadruple consisting of dataset name, rule ID, statistical condition index, and grouping field value as a unique window identifier, stores time-series data in a double-ended queue, and performs aggregation statistical calculations within a preset time window. The memory management module is used to perform lazy cleanup and global cleanup. Lazy cleanup releases expired data when the number of valid data in the double-ended queue is less than half of the total number of records. Global cleanup scans and cleans up expired data in all memory statistics windows that exceeds the global maximum retention time at a preset number of statistical calculations. The event replay module is used to receive historical event data during cold start, write the historical events one by one into the corresponding memory statistics window, and only restore the window data state without performing matching calculations.
[0065] The persistent storage layer connects to the engine control layer and includes an asynchronous writer and an embedded database. The asynchronous writer is used to read data in batches from the asynchronous persistent queue and write it to the embedded database. The cold start management module connects the persistent storage layer and the memory matching layer. It is used to load historical event data from the embedded database when the system restarts and replay it in chronological order to the memory statistics window of the memory matching layer to restore the statistical status.
[0066] The system in this embodiment is configured for standalone deployment, with a mainstream multi-core processor (such as Intel Core i7 or equivalent performance), 16GB or more of memory, solid-state drive storage, and a mainstream operating system (Windows / Linux). The actual hardware configuration can be flexibly adjusted according to the scale of connected devices and the number of alarm rules.
[0067] This embodiment provides three switchable event handling modes for different work scenarios in security operations and maintenance: Full processing mode (default): Performs the complete process of memory matching and asynchronous persistence, suitable for production environment alarm processing in daily operation and maintenance, taking into account both real-time alarms and data storage.
[0068] Storage-only mode: No rule matching is performed; event data is simply stored in the database. This is suitable for scenarios where operations and maintenance personnel need to accumulate historical data over a period of time before designing and validating alarm rules based on this data.
[0069] Test-only matching mode: Only memory rule matching is performed; no data is written to the database, and the state of the memory statistics window is not modified. This mode is suitable for operations personnel to test whether new rules meet expectations without affecting the running production environment. This mode allows operations personnel to confidently try various rule configurations without worrying about impacting existing alarm services.
[0070] This embodiment adopts a four-layer decoupled lightweight system architecture. Each layer—interface service layer, engine control layer, memory matching layer, and persistent storage layer—has a single responsibility and is unidirectionally decoupled. Optimization or upgrades of any module will not affect the normal operation of other modules, thus increasing the system's maintainability and scalability. Based on an embedded database, it achieves single-machine deployment without requiring external database services, enabling out-of-the-box operation. A multi-mode adaptive processing method is provided for operation and maintenance scenarios, offering three switchable modes: complete processing, storage only, and matching test only, covering the full range of needs for daily alarms, data accumulation, and rule testing in security operations and maintenance.
[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for dynamic matching of real-time data stream alarm rules, characterized in that, The method includes the following steps: Step S100: Receive event data reported by security monitoring equipment, read the dataset format definition corresponding to the event data from the preset dataset cache, and perform format verification on the event data; Step S200: Flatten the validated event data into a single-layer format data to ensure that the field structure of the event data is consistent with the database table structure and rule definition; Step S300: Read all enabled alarm rules in the current dataset from the preset rule cache. Alarm rules include simple conditions and / or statistical conditions. Step S400: Perform alarm rule matching in memory. For each alarm rule, perform simple condition matching and / or statistical condition matching in sequence to determine whether the event data meets the triggering conditions of the alarm rule. Statistical condition matching involves grouping event data by specified dimensions within a preset time window, performing aggregated statistical calculations, and comparing the statistical results with preset thresholds. Step S500: After writing the event data and alarm matching results into the asynchronous persistent queue, return the alarm results to the caller; A background thread reads data in batches from an asynchronous persistent queue and writes it to the database; Step S600: When the system restarts, historical event data is loaded from the database and replayed in chronological order to the memory statistics window to restore the statistical state required to match the statistical conditions.
2. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, The statistical condition matching in step S400 includes: A unique window identifier is created for each quadruple, consisting of dataset name, rule ID, statistical condition index, and grouping field value. An independent memory statistical window is established for each quadruple. The memory statistics window uses a double-ended queue to store time-series data, and each record contains an event timestamp and statistical field values. When a new event arrives, the new event is appended to the tail of the double-ended queue according to the memory statistics window corresponding to the quadruple location. Based on a preset time window, valid data is filtered from the double-ended queue, aggregated statistical calculations are performed on the valid data, the calculation results are compared with a preset threshold, and the matching results are output.
3. The method for dynamic matching of real-time data stream alarm rules according to claim 2, characterized in that, Configure lazy cleanup and global cleanup to manage memory usage in the memory statistics window, specifically including the following: Lazy cleanup: When the number of valid data in the deque is less than half of the total number of records, replace all data in the deque with valid data and release the memory occupied by expired data. Global cleanup: Whenever the number of statistical calculations reaches a preset threshold, a global window scan is triggered to clean up expired data in all memory statistical windows that exceeds the global maximum retention time, and to delete empty windows that no longer contain data.
4. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, In step S500, the background thread reads data in batches from the asynchronous persistent queue and writes it to the database, specifically including: A background thread continuously listens to the asynchronous persistent queue at preset timeout intervals. A batch write operation is triggered when the accumulated data in the asynchronous persistent queue reaches a preset batch threshold, or when the time interval since the last batch write reaches a preset time interval. The batch write operation includes: reading data in batches from the asynchronous persistent queue, detecting the database table structure, automatically performing table structure expansion if the data contains new fields, and writing the data in batches to the database; When the system shuts down normally, all remaining data in the asynchronous persistent queue is written to the database, and the database connection is closed after the data has been written to disk.
5. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, Step S600 specifically includes: Iterate through all created datasets and filter out alert rules that contain statistical conditions; For each dataset, take the longest time window among all statistical rules in that dataset, compare it with the global maximum retention time, and take the smaller value as the playback historical time range; Retrieve all historical events within the historical time range from the database and sort them in ascending chronological order. Historical events are written to the corresponding memory statistics window one by one. Only the window data state is restored. Alarm matching calculation is not performed and alarm records are not generated.
6. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, In step S200, the event data is flattened into a single-layer format, specifically including: Identify the nested structure in the event data, connect the nested fields in hierarchical order with a fixed delimiter to form flat field names, and convert them into a single-level key-value pair format, so that the field structure of the event data is consistent with the storage field structure of the database and the reference field structure of the alarm rules.
7. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, In step S300, the rule cache and dataset cache adopt a write-on-failure policy: When processing a certain type of device event data for the first time, the corresponding dataset format definition and alarm rule configuration are loaded from the database and stored in the dataset cache and rule cache, respectively. When a dataset or alarm rule is created, modified, deleted, or started / stopped, the corresponding cached item in the dataset cache or rule cache should be cleared immediately. During the execution of steps S100 to S400, the rule cache and dataset cache are always hit when they are not invalidated, and no database query operation is generated.
8. The method for dynamic matching of real-time data stream alarm rules according to claim 1, characterized in that, The method also includes step S700: mode switching, in response to an external command, switching to full processing mode, storage mode only, or matching test mode only; In the complete processing mode, the complete process from step S100 to step S500 is executed; In storage-only mode, steps S300 and S400 are skipped, and event data is directly written to the asynchronous persistent queue. In the test-only matching mode, the matching process from step S100 to step S400 is executed, the matching result is returned but not written to the asynchronous persistent queue, and the memory statistics window state is not modified.
9. A real-time data stream alarm rule dynamic matching system, using the real-time data stream alarm rule dynamic matching method according to any one of claims 1-8, characterized in that, The system adopts a four-layer layered architecture, including: The interface service layer is used to receive event data reported by the device, as well as to receive configuration operations of datasets and alarm rules by operation and maintenance personnel; The engine control layer and the interface service layer are used to manage rule caching and dataset caching, schedule alarm rule matching processes, and manage asynchronous persistent queues. The memory matching layer connects to the engine's main control layer and is used to perform simple condition matching and statistical condition matching of alarm rules in memory. Statistical condition matching includes aggregation and statistical calculations by grouping according to specified dimensions within a preset time window. The persistent storage layer connects to the engine control layer and includes an asynchronous writer and an embedded database. The asynchronous writer is used to read data in batches from the asynchronous persistent queue and write it to the embedded database. The cold start management module connects the persistent storage layer and the memory matching layer. It is used to load historical event data from the embedded database when the system restarts and replay it in chronological order to the memory statistics window of the memory matching layer to restore the statistical status.
10. A real-time data stream alarm rule dynamic matching system according to claim 9, characterized in that, The memory matching layer includes: The simple condition matching module is used to perform conditional judgments on the field values of a single event data entry. The sliding window aggregation module uses a quadruple consisting of dataset name, rule ID, statistical condition index, and grouping field value as a unique window identifier, stores time-series data in a double-ended queue, and performs aggregation statistical calculations within a preset time window. The memory management module is used to perform lazy cleanup and global cleanup. Lazy cleanup releases expired data when the number of valid data in the double-ended queue is less than half of the total number of records. Global cleanup scans and cleans up expired data in all memory statistics windows that exceeds the global maximum retention time at a preset number of statistical calculations. The event replay module is used to receive historical event data during cold start, write the historical events one by one into the corresponding memory statistics window, and only restore the window data state without performing matching calculations.
