Internet of Things event center distributed alarm processing method based on Redis Stream

Abstract

The invention provides an Internet of Things event center distributed alarm processing method based on Redis Stream, belongs to the field of Internet of Things (IoT) data processing, and realizes millisecond-level event processing by utilizing the high-performance streaming consumption capability of the Redis Stream and combining a distributed rule matching engine and a cross-device data dynamic association mechanism. Comprising the following steps that: Internet of Things terminal equipment writes data into Redis Stream according to a stream: Model: DeviceId mode to form an independent data stream; an event consumer reads the data from the stream, performs matching based on a preloaded rule, and supports cross-device linkage calculation through a metadata relation table and an SQL template; real-time duplicate removal and full-life-cycle management are carried out on alarm events through a dual convergence mechanism based on a database and a cache, and repeated alarms are eliminated; and finally, through multi-channel push and push state write-back, a closed-loop alarm system is constructed. The system adopts a layered architecture, comprises an event acquisition layer, a rule processing layer, an event management layer, a push service layer and an asynchronous processing layer, has the characteristics of high throughput, strong orderliness and high reliability, and can effectively improve the alarm processing efficiency and accuracy in a large-scale Internet of Things scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Internet of Things (IoT) data processing, and in particular to a distributed alarm processing method for an Internet of Things event center based on Redis Stream. BACKGROUND

[0002] With the wide application of Internet of Things technology, the number of terminal devices in the fields of industrial control, smart city, intelligent building, environmental monitoring, energy management, etc. grows exponentially. Taking a smart factory as an example, hundreds of thousands of sensors, actuators and control devices will report state data with a millisecond-level or even shorter period, and the cloud platform needs to process tens of billions of data per day. Under this background, how to complete data collection, rule matching, cross-device logic calculation, event convergence and alarm pushing within a millisecond-level time window has become a core technical challenge for Internet of Things platforms.

[0003] The prior art mainly has the following problems:

[0004] High latency problem: At present, a large number of systems use message queues such as Kafka and RabbitMQ in combination with batch processing mode, and event data at the consumer end often needs to wait for the batch to be complete before being processed, and the end-to-end delay is usually in seconds or even minutes. This obviously cannot meet the needs of millisecond-level response scenarios such as fire alarm, medical rescue and industrial abnormal shutdown.

[0005] Insufficient scalability of rule engine: The rule engine of most platforms relies on single-machine operation (such as Drools and Esper), and rule updates require reloading of configuration files or even restarting of services, which cannot achieve dynamic addition, deletion and modification of large-scale rules in a distributed environment. Especially when logical association between multiple device models is involved, the rule maintenance cost is high and the flexibility is poor.

[0006] Lack of event convergence: In an industrial site, the same device may report dozens or even hundreds of repeated events for the same abnormality, and existing systems lack a unified convergence mechanism, resulting in "alarm storm", which not only consumes computing resources but also increases the processing pressure on operation and maintenance personnel.

[0007] Life cycle status fragmentation: State information such as event triggering, escalation, degradation, recovery and closure is scattered in multiple subsystems, and there is a lack of unified life cycle management and view, making it difficult for operation and maintenance personnel to have a global view of the event status.

[0008] Incomplete message push closed loop: Push channels such as SMS, email, WebSocket and HTTP callback are usually independent of each other, and lack unified push state management and failure retry mechanism. Once the message sending fails, it may cause missed alarm and affect system reliability.

[0009] Therefore, there is an urgent need for an Internet of Things event center solution that can achieve low latency, high scalability, cross-device rule calculation, event convergence and deduplication, and multi-channel closed-loop push in a distributed environment. SUMMARY

[0010] To solve the above technical problems, the present application provides a method for collecting, distributing, matching rules, calculating across devices, converging and deduplicating events, and pushing through multiple channels to achieve full-link closed-loop control of high-frequency real-time event streams generated by large-scale Internet of Things devices using Redis Stream streaming data characteristics.

[0011] The technical solution of the present application is:

[0012] A Redis Stream-based Internet of Things alarm center rule linkage processing method uses the high-performance streaming consumption capability of Redis Stream, combines a distributed rule matching engine and a cross-device data dynamic association mechanism to achieve millisecond-level event processing. The method includes: Internet of Things terminal devices write data into Redis Stream in the stream:Model:DeviceId mode to form independent data streams; event consumers read data from the stream and match based on preloaded rules, supporting cross-device linkage calculation through metadata relationship tables and SQL templates; through a dual convergence mechanism based on databases and caches, real-time deduplication and full-life-cycle management of alarm events are performed to eliminate duplicate alarms; finally, a closed-loop alarm system is built through multi-channel push and push state writeback.

[0013] Further,

[0014] The method includes the following steps:

[0015] Event collection and streaming processing step: Internet of Things terminal devices write device event data into Redis Stream message queues in real time according to the pre-defined stream:{Model}:{DeviceId} key-value mode, where {Model} is the device model identifier and {DeviceId} is the device instance identifier, forming independent data streams with device models and device instances as dimensions.

[0016] Rule loading and caching step: when the system starts, all valid rules are loaded from the rule library to the local memory cache, and the cache structure is a mapping table with device models as keys and rule sets sorted by priority as values.

[0017] Distributed consumption and rule matching step: the event consumer reads event data from the Redis Stream and acquires a corresponding rule set from the local memory cache for matching according to a device model in the event data; the rule matching includes parsing a rule expression and performing cross-device data association to obtain data required for linkage calculation.

[0018] Event convergence and management step: when a rule matching successfully triggers an alarm event, a dual convergence mechanism is used to process the alarm event, including: first convergence (cache deduplication): within a set time window, a deduplication set in Redis is used to record triggered event IDs to prevent repeated processing and pushing of the same event; second convergence (lifecycle management): alarm event information is persisted to an event convergence table in a database to record a start time, a current level, a duration, and a trend change of the event, and to implement lifecycle management of the event from triggering to ending; when the event level is reduced to zero, the event is marked as an end state.

[0019] Further,

[0020] In the distributed consumption and rule matching step, the event consumer runs in an independent consumer mode or a consumer group mode according to a business scenario:

[0021] In a single-device high-frequency reporting scenario, the independent consumer mode is used to ensure real-time processing of a single-device data stream; in a multi-device concurrent reporting scenario, the consumer group mode is used to horizontally expand the consumption nodes to improve the overall throughput of the system.

[0022] Further,

[0023] The rule expression uses a "four-segment variable syntax", and its format is Model$Index$Variable$RelatedDeviceId, where: Model represents a target device model name; Index represents a data index, 0 represents the latest state data of the device, and an integer n greater than 0 represents the nth historical stream data before the latest state of the device; Variable represents a device attribute name; and RelatedDeviceId is used to identify a cross-device association relationship, and when a rule logic involves other devices, the ID is used to query associated devices.

[0024] Further,

[0025] The step of performing cross-device data association specifically comprises: when the rule resolves a variable containing a non-current device RelatedDeviceId, the system queries the metadata relationship table to obtain the actual associated device ID corresponding to the RelatedDeviceId; according to the Index and Variable, the latest state data or historical flow data of the associated device is queried from the data source by using a predefined SQL template; and the queried associated device data is merged with the current event data for subsequent rule evaluation calculation.

[0026] Further,

[0027] In the rule matching step, the system compiles the rule expression into bytecode when the rule is loaded or first matched; at runtime, the current event data is filled into a predefined context, and efficient Boolean evaluation is performed by executing the bytecode.

[0028] Further,

[0029] In the event convergence and management step, the lifecycle management of the event specifically comprises:

[0030] When a new event is first triggered, a new record is inserted in the event convergence table, the start time, the current level are set, and the trend field is set to "UP"; if the event continues to exist and is triggered again, the system compares the old and new event levels, and if the level is raised, the trend is updated to "UP", and if the level is lowered, the trend is updated to "DOWN", and the duration of the event is accumulated; when the triggered event level is reduced to zero, the system records the end time of the event, and sets its state to "INACTIVE", completing the lifecycle loop of the event.

[0031] Further,

[0032] A layered architecture is adopted, including an event collection layer, a rule processing layer, an event management layer, a push service layer, and an asynchronous processing layer, and high-performance operation is achieved in cooperation with metadata storage and state caching; in terms of event collection and stream processing, the Internet of Things terminal device writes data into Redis Stream in a key-value mode of stream:{Model}:{DeviceId}, and different device models and specific device instances form independent data streams; the event consumer can run in two modes according to the scene: in the scene of single-device high-frequency reporting, an independent consumer is used to ensure the real-time performance of data processing; in the scene of multi-device concurrent reporting, a consumer group mode is used, and the total throughput of the system is improved by horizontally expanding the consumption nodes.

[0033] In the start-up phase, all valid rules are loaded from the rule base and written to the local cache. The cache structure is designed as Map<String, TreeSet <eventrule>>, taking the device model as the grouping basis, and being sorted by priority in each rule set; the rule expression adopts a "four-segment variable syntax", and the format is Model$Index$Variable$RelatedDeviceId, wherein Model represents a device model name, Index represents a data index, 0 represents shadow data, i.e., the latest state, and a value greater than 0 represents historical flow data, n represents the n th data before the latest state, Variable represents a device attribute name, and RelatedDeviceId is used to identify a cross-device association relationship.

[0034] When the rule logic involves other device models, the associated device ID is queried through a metadata relationship table, and a predefined SQL template is used to obtain the latest or historical data, thereby supporting cross-model and cross-device linkage calculation; after rule analysis, the system calls the AviatorEvaluator.compile() method to compile the rule into bytecode, and fills the current event data into valueMap during runtime for Boolean evaluation, so as to ensure calculation efficiency while maintaining flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a workflow schematic diagram of the present application. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0037] The present application provides an Internet of Things alarm center rule linkage processing method based on Redis Stream, which utilizes the high-performance stream consumption capability of Redis Stream, combines a distributed rule matching engine and a cross-device data dynamic association mechanism, and realizes millisecond-level event processing; through a double-event convergence mechanism and full-life-cycle management, repeated alarms are eliminated; meanwhile, multi-channel pushing and pushing state write-back are introduced, and a closed-loop alarm system is constructed, thereby solving the problems of high delay, poor scalability, convergence loss, state fragmentation and unreliable pushing in the prior art.

[0038] The application adopts a distributed deployment architecture, which is composed of an event collection layer, a rule processing layer, an event management layer, a push service layer and an asynchronous processing layer, and cooperates with metadata storage and state cache to realize high-performance operation. In the aspects of event collection and stream processing, the Internet of Things terminal device writes data into Redis Stream in the key-value mode of stream:{Model}:{DeviceId}, different device models and specific device instances form independent data streams, thereby effectively avoiding interference between different business data. The event consumer (Listener) can run in two modes according to the scene: in the scene of single-device high-frequency reporting, an independent consumer is used to ensure the real-time performance of data processing; in the scene of multi-device concurrent reporting, a consumer group mode is used, and the total throughput of the system is improved by horizontally extending the consumption nodes. Relying on the sequential reading mechanism of Redis Stream based on the cursor, the system can guarantee the strict order of event processing, and the blocking reading mode is used to reduce the additional overhead caused by empty polling.

[0039] In the starting stage, all valid rules in the rule library are loaded and written into the local cache, and the cache structure is designed as Map<String,TreeSet <eventrule>>, and each rule set is sorted by priority. The rule expression adopts a "four-segment variable syntax" in the format of Model$Index$Variable$RelatedDeviceId, where Model represents the device model name, Index represents the data index (0 represents shadow data, i.e., the latest state, and greater than 0 represents historical flow data, and n represents the nth data before the latest state), Variable represents the device attribute name, and RelatedDeviceId is used to identify the cross-device association relationship. When the rule logic involves other device models, the system queries the associated device ID through the metadata relationship table and obtains its latest or historical data using a predefined SQL template, thereby supporting cross-model and cross-device linkage calculation. After rule analysis, the system calls the AviatorEvaluator.compile() method to compile the rule into bytecode, and fills the current event data into valueMap at runtime for Boolean evaluation, thereby maintaining flexibility while ensuring computational efficiency.

[0040] In terms of event management, the application designs a dual convergence mechanism based on database and cache to solve the problem of repeated alarms. The event convergence table (EventConvergence) in the database is used to record the start and end time, current level, trend change and other information of the event, and to realize full-process management of the event life cycle; the de-duplication set (WARN_DEVICE:{DeviceId}) in Redis is used to mark the event ID that has been triggered within a set time window, thereby avoiding repeated entry into the database and repeated pushing. When a new event is triggered, the system inserts the corresponding record in the convergence table and sets the trend field to "UP"; if the event continues to exist, the trend will be adjusted according to the difference between the new and old levels, and the duration will be accumulated; when the event level drops to zero, the system records the end time and sets the state to "INACTIVE", thereby forming a complete life cycle loop.

[0041] The above only describes the preferred embodiments of the application, which are used to illustrate the technical solutions of the application, and are not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.< / eventrule> < / eventrule>

Claims

1. A distributed alarm processing method for an IoT event center based on Redis Streams. Its features are, This includes: IoT terminal devices writing data to Redis Stream in the stream:Model:DeviceId pattern to form independent data streams; event consumers reading data from the stream and matching based on pre-loaded rules, supporting cross-device linkage calculations through metadata relationship tables and SQL templates; real-time deduplication and full lifecycle management of alarm events through a dual convergence mechanism based on database and cache, eliminating duplicate alarms; and finally, building a closed-loop alarm system through multi-channel push and push status write-back.

2. The method according to claim 1, characterized in that, Includes the following steps: Event Acquisition and Streaming Processing: IoT terminal devices write device event data to the Redis Stream message queue in real time according to the predefined stream:{Model}:{DeviceId} key-value pattern, where {Model} is the device model identifier and {DeviceId} is the device instance identifier, forming mutually independent data streams based on device model and device instance. Rule loading and caching: At startup, all valid rules are loaded from the rule base into the local memory cache. The cache structure is a mapping table with the device model as the key and the rule set sorted by priority as the value. Distributed consumption and rule matching: The event consumer reads event data from the Redis Stream and, based on the device model in the event data, retrieves the corresponding rule set from the local memory cache for matching; The rule matching includes parsing the rule expression and performing cross-device data association to obtain the data required for the linkage calculation; Event Convergence and Management: When a rule match successfully triggers an alarm event, the alarm event is processed through a dual convergence mechanism, including: First convergence, namely cache deduplication: Within a set time window, the deduplication set in Redis is used to record the triggered event IDs to prevent duplicate processing and push of the same event; Second convergence, namely lifecycle management: Alarm event information is persisted to the event convergence table in the database, recording the event's start time, current level, duration, and trend changes, realizing full lifecycle management of the event from triggering to ending; When the event level drops to zero, the event is marked as ending.

3. The method according to claim 2, characterized in that, In the distributed consumption and rule matching step, the event consumer selects either independent consumer mode or consumer group mode to operate based on the business scenario: In scenarios involving high-frequency reporting from a single device, an independent consumer mode is adopted to ensure real-time processing of data streams from a single device. In scenarios involving concurrent reporting from multiple devices, a consumer group mode is adopted to improve the overall throughput of the system by horizontally scaling consumer nodes.

4. The method according to claim 2, characterized in that, The rule expression adopts a "four-segment variable syntax", with the format Model$Index$Variable$RelatedDeviceId, where: Model represents the target device model name; Index represents the data index, where 0 represents the latest status data of the device, and an integer n greater than 0 represents the nth historical stream data before the latest status of the device; Variable represents the device attribute name; and RelatedDeviceId is used to identify cross-device relationships. When the rule logic involves other devices, the associated devices are queried through this ID.

5. The method according to claim 2, characterized in that, The specific steps of performing cross-device data association include: when a rule parses a variable containing a non-current device RelatedDeviceId, querying the metadata relationship table to obtain the actual associated device ID corresponding to the RelatedDeviceId; based on the Index and Variable, using a predefined SQL template, querying the latest status data or historical stream data of the associated device from the data source; merging the queried associated device data with the current event data for subsequent rule evaluation calculations.

6. The method according to claim 2, characterized in that, In the rule matching step, the rule expression is compiled into bytecode when the rule is loaded or during the first match; at runtime, the current event data is filled into a predefined context, and efficient Boolean evaluation is performed by executing the bytecode.

7. The method according to claim 2, characterized in that, In the event convergence and management steps, the event lifecycle management specifically includes: When a new event is triggered for the first time, a new record is inserted into the event convergence table, setting its start time, current level, and trend field to "UP". If the event persists and is triggered again, the system compares the levels of the old and new events. If the level increases, the trend is updated to "UP"; if the level decreases, the trend is updated to "DOWN", and the event duration is incremented. When the triggered event level drops to zero, the system records the event's end time and sets its status to "INACTIVE", completing the event's lifecycle loop.

8. The method according to claim 1, characterized in that, It adopts a layered architecture, including an event acquisition layer, a rule processing layer, an event management layer, a push service layer, and an asynchronous processing layer, and works with metadata storage and state caching to achieve high-performance operation. In terms of event acquisition and streaming processing, IoT terminal devices write data to Redis Stream according to the key-value pattern of stream:{Model}:{DeviceId}, and different device models and specific device instances form independent data streams. Event consumers can choose to operate in two modes depending on the scenario: in scenarios where a single device reports frequently, an independent consumer is used to ensure the real-time performance of data processing; in scenarios where multiple devices report concurrently, a consumer group mode is used to improve the overall throughput of the system by horizontally scaling consumer nodes.

9. The method according to claim 8, characterized in that, During the startup phase, all valid rules are loaded from the rule base and written to the local cache, which is designed as a Map. <String,TreeSet <eventrule> > The rules are grouped by device model and sorted by priority within each rule set. The rule expression adopts a "four-segment variable syntax", which is in the format Model$Index$Variable$RelatedDeviceId, where Model represents the device model name, Index represents the data index, 0 represents shadow data, i.e., the latest state, and greater than 0 represents historical stream data, n represents the nth data before the latest state, Variable represents the device attribute name, and RelatedDeviceId is used to identify cross-device associations.< / eventrule> 10. The method according to claim 9, characterized in that, When the rule logic involves other device models, the associated device ID is queried through the metadata relationship table, and the latest or historical data is obtained using a predefined SQL template, thereby supporting cross-model and cross-device linkage calculations. After the rule is parsed, the system calls the AviatorEvaluator.compile() method to compile the rule into bytecode, and fills the current event data into the valueMap for Boolean evaluation at runtime, thereby ensuring computational efficiency while maintaining flexibility.

Images