A high-concurrency real-time data processing and hierarchical intelligent alarm method and system for mass internet of things devices

Abstract

The application discloses a kind of massive internet of things equipment-oriented high concurrency real-time data processing and hierarchical intelligent alarm method and system, comprising: constructing double cache separation rule index;According to rule frequency parameter, hierarchical alarm calculation is carried out;Based on time series database super table model, create independent subtable with the combination of device identification and index identification as dimension, construct the multidimensional inverted index of memory resident for TAG field, realize query complexity O (T), wherein T is the number of hit subtable and T is much smaller than total data D;And using incremental update and atomic replacement mechanism guarantees the state consistency in the process of rule change.The application effectively solves the technical problems of memory explosion, real-time loss and slow query in the scene of million devices and hundred billion time series data, significantly reduces memory occupation and storage cost, and improves system processing capacity and stability.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) big data processing technology, and in particular to a method and system for high-concurrency real-time data processing, rule calculation optimization, and state consistency assurance for tens of thousands of devices and hundreds of billions of time-series data. Background Technology

[0002] With the rapid development of IoT technology, systems in typical scenarios such as underground space engineering and industrial IoT require the integration of large-scale sensor devices. Taking underground integrated pipe corridors, subway tunnels, and mine roadways as examples, a single project often needs to deploy hundreds of monitoring points and tens of thousands of sensor devices. Each device continuously reports data at a frequency of seconds or tens of seconds, accumulating into hundreds of billions of historical time-series data over time. At the same time, the alarm rules for different monitoring points and different types of equipment are different, posing severe technical challenges to data processing and intelligent alarm.

[0003] Traditional data processing and alarm solutions face the following technical bottlenecks in the aforementioned large-scale IoT scenarios: (i) The full deployment of alarm rules leads to exponential memory growth. Traditional alarm rule storage methods employ a full object expansion strategy of "rule × device," meaning that a corresponding rule object is instantiated for each metric of each device. If the number of rules is M and the number of devices is N, then N × M rule instance objects need to be constructed, resulting in a memory complexity of O(N × M). When the number of devices reaches tens of thousands (N=10...), the complexity increases further. 4 When the number of rules reaches thousands (M=10³), 10 rules need to be constructed. 7 The sheer number of rule instances leads to an exponential increase in memory usage, which in turn triggers frequent Full GC or even memory overflow, making the system unable to run stably.

[0004] (ii) Serial processing cannot meet real-time requirements.

[0005] IoT data acquisition is characterized by high frequency and continuous operation, requiring the system to complete data processing within the acquisition cycle. However, traditional solutions employ a serial processing mode, processing sensor-reported data one by one. Assuming the processing time for a single data entry is approximately 50ms, the serial computation time for processing 10,000 data entries in a single batch is... This data collection time far exceeds the typical collection cycle (e.g., 10 seconds). This results in new data overwriting old data, loss of real-time performance, and severe delays in alarms, failing to meet the basic requirements of engineering monitoring for real-time alarms.

[0006] (iii) Single-table storage leads to resource waste and slow query speed.

[0007] Traditional time-series databases employ a "wide table" design, storing all metric data for all devices in a single table. Querying data for a specific device or metric requires a full table scan, resulting in a query complexity of O(D), where D is the total data volume. When the data volume reaches billions or even hundreds of billions, full table scans cause I / O spikes, query response times of several minutes or even hours, and significant waste of disk storage space, failing to meet the demands of engineering operations for rapid data retrieval.

[0008] In summary, existing technologies face multiple technical bottlenecks in large-scale IoT scenarios, such as memory explosion in rule matching, loss of real-time data processing, and slow data storage and querying. There is an urgent need for a high-concurrency real-time data processing and intelligent alarm technology solution that can solve the above problems simultaneously. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for high-concurrency real-time data processing, rule calculation optimization, and state consistency assurance for tens of thousands of devices and hundreds of billions of time-series data, in order to solve the multiple technical bottlenecks mentioned in the background art, such as rule matching memory explosion, loss of real-time data processing, and slow data storage and querying, which are currently faced in large-scale Internet of Things scenarios.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A method for high-concurrency real-time data processing and hierarchical intelligent alarm for massive IoT devices includes the following steps: S1. Dual-Cache Separated Rule Index: Construct rule metadata cache C1 and device-metric-rule index cache C2. C1 stores rule objects with rule identifier as the key, and C2 stores the corresponding rule identifier list with the combination of device identifier and metric identifier as the key. This ensures that only the hit rules are traversed during the rule matching process, and the matching complexity is O(k), where k is the number of hit rules. S2, Hierarchical alarm: According to the alarm frequency parameter of the rule, if the frequency f=1, the single-point alarm pass-through algorithm is used to directly compare the current value with the threshold and trigger the alarm. If the frequency f>1, the sliding window alarm algorithm based on the circular buffer is used to maintain the window data by indexing and looping over, so as to realize the alarm judgment with O(1) time complexity. S3, TAG-driven sub-table routing: Based on the super table model of time-series database, independent sub-tables are created with the combination of device identifier and indicator identifier as the dimension. Static attributes are stored as TAG tags, and a memory-resident multi-dimensional inverted index is built for the TAG field. During the query, the sub-table set is located through this index and only the sub-tables that are hit are scanned, achieving a query complexity of O(T), where T is the number of sub-tables hit. S4. State consistency guarantee: When rules change, incremental update and atomic replacement mechanisms are used to update the cache to ensure the consistency of data state during rule matching. Wherein, the rule frequency is defined as the number of times the threshold condition must be met consecutively before triggering an alarm; the device identifier is a string that uniquely identifies an IoT device; the indicator identifier is the name of a certain type of physical quantity collected by the device; the circular buffer is a fixed-length circular array data structure, which achieves ordered storage and cyclic overwriting of data through index modulo operation.

[0011] Furthermore, in step S1, the data structure of the rule metadata cache C1 is as follows: The space complexity is O(M), where M is the total number of rules. The RuleObject includes at least a threshold, frequency, comparison operator, and window size. The data structure of the device-metric-rule index cache C2 is as follows: The space complexity is O(N), where N is the total number of devices and K is a combination string of device identifier and indicator identifier; the dual-caching separation rule indexing step loads all rule data from the database during the initialization phase, traverses the rule set R, writes the rules into C1, and constructs an index of K→ruleId according to the rule binding relationship before writing it into C2.

[0012] Furthermore, in the hierarchical alarm calculation steps, the single-point alarm pass-through algorithm is as follows: when the rule frequency f=1, the construction and maintenance of the sliding window is skipped, and the current reported value is directly compared with the rule threshold. If the alarm condition is met, the alarm is triggered immediately, and the single calculation complexity is reduced from O(n) to O(1). The sliding window alarm algorithm is as follows: when the rule frequency f>1, a circular buffer with the rule identifier and device identifier as the key is constructed as a sliding window. The old data is overwritten by index loop to achieve data writing and window maintenance with O(1) time complexity. A comparison function is executed on all values ​​in the window. If all alarm conditions are met, the alarm is triggered.

[0013] Furthermore, the circular buffer is implemented using a fixed-length array, and window data is managed through an index-based circular overwrite mechanism. The window size is dynamically determined by the frequency parameter or window size parameter in the rules. When writing data, the oldest data is overwritten according to the current index position, and the index is incremented by modulo operation.

[0014] Furthermore, in the TAG-driven sub-table routing step, the sub-table creation function is... Total number of sub-tables Where N is the total number of devices, and Q is the total number of metrics associated with each device; the memory-resident multidimensional inverted index structure is as follows: ,in To associate the set of all sub-table identifiers associated with the tag value, the index adopts a hybrid structure of hash table and skip list, supporting exact matching with an average time complexity of O(1) for a single tag, intersection and union operations of multiple tag combinations, and dynamic incremental updates when a sub-table is created or deleted.

[0015] Furthermore, in the TAG-driven sub-table routing step, the query process includes: receiving the TAG filtering conditions in the query request, locating and calculating the set of hit sub-tables through the in-memory inverted index, performing parallel data scanning on each sub-table in the set of hit sub-tables, the scanning time complexity being positively correlated with the number of hit sub-tables T and T being much smaller than the total data volume D, and finally aggregating the scanning results of each sub-table and returning them.

[0016] Furthermore, in the state consistency guarantee step, the incremental update and atomic replacement mechanism specifically includes: when the front-end calls the API interface to initiate a rule change, it obtains the rule change set ΔR, and reconstructs a new cache copy according to the changed rule identifier. and After the cache copy is built, an atomic pointer replacement operation is performed, atomically pointing C1 and C2 to the newly built cache copy. and This avoids memory spikes caused by full updates and ensures that a consistent rule view is used throughout the rule matching process.

[0017] A high-concurrency real-time data processing and hierarchical intelligent alarm system for massive IoT devices includes: The dual-caching separate rule index module is used to build and maintain the rule metadata cache C1 and the device-metric-rule index cache C2, so that only the hit rules are traversed during the rule matching process, and the matching complexity is O(k). The hierarchical alarm calculation module is used to determine alarms based on the rule frequency parameter. If the frequency f=1, the single-point alarm pass-through algorithm is used to directly compare the current value with the threshold and trigger an alarm. If the frequency f>1, the sliding window alarm algorithm based on the circular buffer is used. The window data is maintained by index loop overwriting, and alarm determination is achieved in O(1) time complexity. The TAG tag-driven sub-table routing module is used to create device-indicator level independent sub-tables based on the time series database super table model, build an in-memory inverted index for the TAG field, and perform sub-table routing and queries based on the index, achieving a query complexity of O(T). State consistency guarantee module: used to update the cache with incremental update and atomic replacement mechanism when rules change, to ensure state consistency during the rule matching process; Wherein, the rule frequency is defined as the number of times the threshold condition must be met consecutively before triggering an alarm; the device identifier is a string that uniquely identifies an IoT device; the indicator identifier is the name of a certain type of physical quantity collected by the device; the circular buffer is a fixed-length circular array data structure, which achieves ordered storage and cyclic overwriting of data through index modulo operation.

[0018] Furthermore, in the hierarchical alarm calculation module, the sliding window alarm algorithm adopts a circular buffer data structure, using a combination of rule identifier and device identifier as the key, and achieves O(1) time complexity window data writing and maintenance through index cyclic overlay; in the TAG-driven sub-table routing module, the in-memory inverted index is It adopts a hybrid structure of hash table and skip list, supports precise matching with an average time complexity of O(1) for single label, intersection and union operations of multiple label combinations, and dynamic incremental updates when sub-tables are created or deleted.

[0019] Furthermore, the dual-caching separate rule index module supports incremental updates and atomic replacement mechanisms when rules change. By constructing a new cache copy and then performing atomic pointer replacement, it avoids memory spikes caused by full updates. In the TAG-driven sub-table routing module, the number of sub-tables is the product of the number of devices and the number of metrics. During queries, only sub-tables are scanned after being located by the inverted index in memory. The amount of data scanned, T, is much smaller than the total amount of data, D, and the query complexity is O(T). The state consistency guarantee module ensures the data consistency of rule matching during rule changes through an atomic replacement mechanism.

[0020] Compared with the prior art, the beneficial effects of the present invention are: (1) Reduce memory usage. By using dual cache to separate the rule index, the complexity of rule matching is reduced from O(N×M) to O(N+M), and the memory usage is reduced from 4GB to 200MB, a reduction of 95%, completely eliminating the risk of Full GC.

[0021] (2) Improve real-time processing. By adopting hierarchical alarm calculation, the complexity of single alarm judgment is reduced from O(n) to O(1), the processing time of tens of thousands of data in a single batch is reduced from 70s to 5s, and the alarm latency is reduced from 800ms to less than 100ms.

[0022] (3) Optimize query performance. Based on TAG-driven sub-table routing, the query complexity is reduced from O(D) to O(T), where T is the number of hit sub-tables and T is much smaller than the total data volume D. The query time for billions of data is reduced from 120-180s to less than 5s.

[0023] (4) Save storage space. Static attributes are stored independently as TAG tags, avoiding duplicate writing of metadata. The storage space for billion-level data is reduced from 23.4G to 10.8G, with a compression rate of 53.8%.

[0024] (5) Ensure system stability. By adopting incremental update and atomic replacement mechanism, the system availability has been improved from 80% to 99.99%, with no downtime and GC pause time controlled within 500ms. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the overall method of the present invention for high-concurrency real-time data processing, rule calculation optimization, and state consistency assurance for tens of thousands of devices and hundreds of billions of time-series data. Figure 2 This is a flowchart illustrating a specific algorithm in a particular embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figure 1 This embodiment provides a high-concurrency real-time data processing and hierarchical intelligent alarm method for massive IoT devices, which can be applied to underground space engineering monitoring scenarios.

[0028] The following is combined Figure 2 and one To illustrate with a specific scenario, we'll describe a scenario involving 200 engineering sites and 10,000 sensor devices. Each device reports data every 10 seconds, generating approximately 432 million data entries daily, with historical data reaching hundreds of billions. The alarm rules vary depending on the engineering site and device type, totaling approximately 100 rules.

[0029] I. System Initialization

[0030] When the system starts, the initial construction of the dual-caching separation rule index is performed.

[0031] First, load all alarm rule data from the database to obtain the rule set. Each rule object (RuleObject) must include at least: threshold, frequency, operator, and window size.

[0032] Secondly, iterate through the rule set R, and write each rule into the rule metadata cache C1 using the rule identifier (ruleId) as the key. The data structure of C1 is a Map.<ruleId, RuleObject> The space complexity is O(100).

[0033] Then, based on the device range bound to each rule, the device set is traversed. Build an index key for each metric of each device The corresponding rule identifier (ruleId) is added to the rule identifier list corresponding to the key, and finally written to the device-metric-rule index cache C2, whose data structure is a Map. <K, List <ruleid>The space complexity is O(10000).

[0034] Through the above initialization, the system reduces the rule matching complexity from the traditional O(N×M) = O(10000×100) = O(10^22) / 20^22. 6 The memory usage is reduced to O(N+M)=O(10100), and the memory usage is reduced from 4GB in the traditional solution to 200MB.

[0035] II. Real-time Data Processing and Alarm Calculation

[0036] When the sensor device reports data in 10-second cycles, the system processes each data entry as follows.

[0037] Taking the temperature data reported by the air sensor device sensor_001 as an example, let its device identifier be "LS_001", its index identifier be "temp", and its reported value be 35℃. The system first queries the list of associated rule identifiers from C2 according to K = "LS_001:temp" to obtain the hit rule set R_k.

[0038] In this embodiment, it is assumed that a temperature alarm rule is triggered: threshold = 30 degrees Celsius, frequency = 3, operator = greater than (>), and window size = 3.

[0039] Since the frequency f=3>1, the system enters the sliding window alarm process. The system constructs the window key. Use this key to retrieve the current window data set from the circular buffer cache Cw. The circular buffer is implemented using a fixed-length array with a capacity equal to 3 times the window size, and is achieved through modulo operations on the index. Achieve cyclic overwriting of data.

[0040] In this embodiment, the three consecutive reported temperatures were 35 degrees Celsius, 32 degrees Celsius, and 38 degrees Celsius, all exceeding the threshold of 30 degrees Celsius. The system sequentially writes the three values ​​into a circular buffer, and after each write, performs a comparison function on all values ​​within the window. Once the third value is entered, all three values ​​in the window meet the condition of being greater than 30 degrees Celsius, and the system triggers an alarm.

[0041] If the frequency f=1, the system skips the construction and maintenance of the sliding window and directly compares the current value with the threshold. If the condition is met, an alarm is triggered immediately, and the single calculation complexity is reduced from O(n) to O(1).

[0042] Experimental data shows that in this embodiment, the processing time for 50,000 data entries reported by 10,000 devices in a single batch is reduced from 70 seconds in the traditional solution to 5 seconds, and the alarm latency is reduced from 800 ms to less than 100 ms.

[0043] III. Data Storage and Sub-table Routing

[0044] This embodiment uses the supertable model of the TDengine time-series database for data storage.

[0045] First, create a super table. Define dynamic time series (such as timestamp, quota_data) and static TAG columns (such as device_id, device_name, install_position, quota_name, quota_unit, gcid).

[0046] Then, create a separate sub-table for each combination of device and metric. Taking the temperature metric of device sensor_001 as an example, create a sub-table... The super table structure is inherited through the USING clause, and the TAG values ​​are set as follows: device_id='sensor_001', device_name='temperature sensor', install_position='workshop A', quota_name='temperature', quota_unit='℃', gcid='GC_001'.

[0047] When the super table is created, the system automatically builds a memory-resident multidimensional inverted index for all TAG fields, with the index structure being TagIndex = Map.<TagValue, SubTableList> It employs a hybrid structure of hash tables and skip lists. Taking device_id as an example, the inverted index is: .

[0048] When a user needs to query the historical temperature data of device sensor_001, the TAG filter condition in the query request is: The system first locates the target sub-table set using an in-memory inverted index, and then scans only that sub-table instead of the entire table. For multi-label combination queries, the system performs intersection or union operations on the inverted index results of each label, prioritizing the optimization of scanning smaller sets.

[0049] For query scenarios with billions of data points, traditional solutions require scanning the entire table, which takes 120-180 seconds. This embodiment only scans the hit sub-tables, taking less than 5 seconds. The query complexity is reduced from O(D) to O(T), where D is the total data volume and T is the number of hit sub-tables, and T is much smaller than D.

[0050] Regarding storage space, traditional solutions repeatedly store static metadata such as device identifier, device name, installation location, and unit for each record. Taking 100,000 devices reporting every 10 seconds as an example, the daily metadata redundancy reaches 46GB. In this embodiment, the TAG value is stored only once in the sub-table metadata, and the time-series data file only stores the timestamp and dynamic value. The storage space for billions of data points is reduced from 23.4GB to 10.8GB, achieving a compression rate of 53.8%.

[0051] IV. Dynamic Rule Updates and State Consistency Guarantee

[0052] When operations and maintenance personnel modify alarm rules through the front-end API interface, the system executes an incremental update and atomic replacement mechanism.

[0053] Let the original rule set R be changed to ΔR. The system reconstructs a new cache copy based on the rule identifiers in ΔR. and During the construction process, the existing caches C1 and C2 continue to serve the ongoing rule matching requests without causing any blocking.

[0054] Once C1' and C2' are fully constructed, the system performs an atomic pointer replacement operation: This operation employs a lock-free design, with the replacement process taking only one pointer assignment, ensuring a consistent rule view is used throughout the rule matching process and avoiding memory spikes caused by full updates.

[0055] Experimental data shows that the system in this embodiment runs continuously without downtime, with GC pauses controlled within 500ms, and the system availability reaches 99.99%. In contrast, the traditional solution has a downtime frequency of 1-2 times per day, GC pauses of more than 5 seconds, and availability of only 80%.

[0056] To better illustrate the technical effects of the present invention, this embodiment sets up a pair of ratios. This pair of ratios adopts a traditional alarm rule storage and data processing scheme, as detailed below: (a) Alarm rule storage method The comparison adopts a full object expansion strategy of "rule × device", that is, instantiating a corresponding rule object for each metric of each device. If the number of rules M = 100 and the number of devices N = 10000, then N × M = 10000 rules need to be constructed. 6 There are 10 rule instance objects, and the memory complexity is O(N×M). All rule objects reside in memory, resulting in a memory usage of up to 4GB and triggering frequent Full GC, leading to system instability.

[0057] (II) Data Processing Method

[0058] The comparative method uses a serial processing mode, processing the sensor-reported data one by one. Processing a single data entry takes approximately 50ms, and processing a batch of 10,000 data entries takes [time missing]. The data collection cycle is far longer than 10 seconds. This results in new data overwriting old data, causing severe delays in alarms and compromising real-time performance.

[0059] (III) Data storage method

[0060] The comparison uses a single-table "wide table" storage model, storing all indicator data for all devices in a single table. When querying data for a specific device or indicator, the database needs to perform a full table scan. Taking TDengine as an example, the complexity of a traditional single-table query is O(D), where D is the total data volume. When the data volume reaches billions, a full table scan causes a surge in I / O, resulting in query response times of 120-180 seconds. Simultaneously, each record repeatedly stores static metadata such as device identifier, device name, installation location, and organization. Taking 10,000 devices reporting every 10 seconds as an example, approximately 432 million records are generated daily, resulting in severe metadata redundancy and a data storage space consumption of 23.4GB for billions of records.

[0061] (iv) Alarm Calculation Method

[0062] The comparative model uniformly adopts the sliding window algorithm for all alarm rules, maintaining a complete sliding window data structure regardless of whether the rule frequency f is 1. For rules with a frequency f=1, maintaining the window causes unnecessary memory and CPU overhead, with a single calculation complexity of O(W), where W is the window size, CPU usage reaching 310%, and alarm latency as high as 800ms.

[0063] (v) Rule update method

[0064] The comparison uses a full update strategy, reloading all rule data and rebuilding all rule instance objects when rules change. During the update process, memory quickly fills up, causing system lag or even crashes. The system crashes 1-2 times per day, GC pauses are longer than 5 seconds, and availability is only 80%.

[0065] This invention was compared with a comparative example (traditional scheme) under the same experimental environment. The experimental environment configuration was as follows: 16-core CPU, 32GB memory, 10,000 devices, 100 rules, and a data acquisition frequency of 10 seconds / time. The experiment simulated four sensor types: 2,000 water immersion sensors, 2,000 wind speed sensors, 4,000 air quality sensors (each with more than 10 indicators), and 2,000 tilt sensors (each with 3 indicators). The single data upload volume was 50,000 records, and the daily data generation volume was 432 million records. The following are the comparison results after three days of continuous system operation:

[0066] The above comparison shows that the comparative method suffers from multiple technical bottlenecks in large-scale IoT scenarios, such as memory explosion, loss of real-time performance, slow query, storage redundancy, and poor stability. The technical solution of this invention effectively solves the above problems.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / ruleid>

Claims

1. A method for high-concurrency real-time data processing and hierarchical intelligent alarming for massive numbers of IoT devices, characterized in that, Includes the following steps: S1. Dual-Cache Separated Rule Index: Construct rule metadata cache C1 and device-metric-rule index cache C2. C1 stores rule objects with rule identifier as the key, and C2 stores the corresponding rule identifier list with the combination of device identifier and metric identifier as the key. This ensures that only the hit rules are traversed during the rule matching process, and the matching complexity is O(k), where k is the number of hit rules. S2, Hierarchical alarm: According to the alarm frequency parameter of the rule, if the frequency f=1, the single-point alarm pass-through algorithm is used to directly compare the current value with the threshold and trigger the alarm. If the frequency f>1, the sliding window alarm algorithm based on the circular buffer is used to maintain the window data by indexing and looping over, so as to realize the alarm judgment with O(1) time complexity. S3, TAG-driven sub-table routing: Based on the super table model of time-series database, independent sub-tables are created with the combination of device identifier and indicator identifier as the dimension. Static attributes are stored as TAG tags, and a memory-resident multi-dimensional inverted index is built for the TAG field. During the query, the sub-table set is located through this index and only the sub-tables that are hit are scanned, achieving a query complexity of O(T), where T is the number of sub-tables hit. S4. State consistency guarantee: When rules change, incremental update and atomic replacement mechanisms are used to update the cache to ensure the consistency of data state during rule matching. Wherein, the rule frequency is defined as the number of times the threshold condition must be met consecutively before triggering an alarm; the device identifier is a string that uniquely identifies an IoT device; the indicator identifier is the name of a certain type of physical quantity collected by the device; the circular buffer is a fixed-length circular array data structure, which achieves ordered storage and cyclic overwriting of data through index modulo operation.

2. The method according to claim 1, characterized in that, In step S1, the data structure of the rule metadata cache C1 is as follows: The space complexity is O(M), where M is the total number of rules. The RuleObject includes at least a threshold, frequency, comparison operator, and window size. The data structure of the device-metric-rule index cache C2 is as follows: The space complexity is O(N), where N is the total number of devices and K is a combination string of device identifier and indicator identifier; the dual-caching separation rule indexing step loads all rule data from the database during the initialization phase, traverses the rule set R, writes the rules into C1, and constructs an index of K→ruleId according to the rule binding relationship before writing it into C2.

3. The method according to claim 1, characterized in that, In the hierarchical alarm calculation steps, the single-point alarm pass-through algorithm is as follows: when the rule frequency f=1, the construction and maintenance of the sliding window are skipped, and the current reported value is directly compared with the rule threshold. If the alarm condition is met, the alarm is triggered immediately, and the single calculation complexity is reduced from O(n) to O(1). The sliding window alarm algorithm is as follows: when the rule frequency f>1, a circular buffer with the rule identifier and device identifier as the key is constructed as a sliding window. The old data is overwritten by index loop to achieve data writing and window maintenance with O(1) time complexity. The comparison function is executed on all values ​​in the window. If all alarm conditions are met, the alarm is triggered.

4. The method according to claim 3, characterized in that, The circular buffer is implemented using a fixed-length array. Window data is managed through an index-based cyclic overwrite mechanism. The window size is dynamically determined by the frequency parameter or window size parameter in the rules. When writing data, the oldest data is overwritten according to the current index position, and the index is incremented by modulo operation.

5. The method according to claim 1, characterized in that, In the TAG-driven sub-table routing step, the sub-table creation function is: Total number of sub-tables Where N is the total number of devices, and Q is the total number of metrics associated with each device; the memory-resident multidimensional inverted index structure is as follows: ,in To associate the set of all sub-table identifiers associated with the tag value, the index adopts a hybrid structure of hash table and skip list, supporting exact matching with an average time complexity of O(1) for a single tag, intersection and union operations of multiple tag combinations, and dynamic incremental updates when a sub-table is created or deleted.

6. The method according to claim 5, characterized in that, In the TAG-driven sub-table routing step, the query process includes: receiving the TAG filtering conditions in the query request, locating and calculating the set of hit sub-tables through the inverted index in memory, performing parallel data scanning on each sub-table in the set of hit sub-tables, the scanning time complexity being positively correlated with the number of hit sub-tables T and T being much smaller than the total data volume D, and finally aggregating the scanning results of each sub-table and returning them.

7. The method according to claim 1, characterized in that, In the state consistency guarantee steps, the incremental update and atomic replacement mechanism specifically includes: when the front-end calls the API interface to initiate a rule change, obtaining the rule change set ΔR, and reconstructing a new cache copy based on the changed rule identifier. and After the cache copy is built, an atomic pointer replacement operation is performed, atomically pointing C1 and C2 to the newly built cache copy. and This avoids memory spikes caused by full updates and ensures that a consistent rule view is used throughout the rule matching process.

8. A high-concurrency real-time data processing and hierarchical intelligent alarm system for massive IoT devices, characterized in that, include: The dual-caching separate rule index module is used to build and maintain the rule metadata cache C1 and the device-metric-rule index cache C2, so that only the hit rules are traversed during the rule matching process, and the matching complexity is O(k). The hierarchical alarm calculation module is used to determine alarms based on the rule frequency parameter. If the frequency f=1, the single-point alarm pass-through algorithm is used to directly compare the current value with the threshold and trigger an alarm. If the frequency f>1, the sliding window alarm algorithm based on the circular buffer is used. The window data is maintained by index loop overwriting, and alarm determination is achieved in O(1) time complexity. The TAG tag-driven sub-table routing module is used to create device-indicator level independent sub-tables based on the time series database super table model, build an in-memory inverted index for the TAG field, and perform sub-table routing and queries based on the index, achieving a query complexity of O(T). State consistency guarantee module: used to update the cache with incremental update and atomic replacement mechanism when rules change, to ensure state consistency during the rule matching process; Wherein, the rule frequency is defined as the number of times the threshold condition must be met consecutively before triggering an alarm; the device identifier is a string that uniquely identifies an IoT device; the indicator identifier is the name of a certain type of physical quantity collected by the device; the circular buffer is a fixed-length circular array data structure, which achieves ordered storage and cyclic overwriting of data through index modulo operation.

9. The system according to claim 8, characterized in that, In the hierarchical alarm calculation module, the sliding window alarm algorithm adopts a circular buffer data structure, using a combination of rule identifier and device identifier as the key, and achieves O(1) time complexity window data writing and maintenance through index cyclic overlay; in the TAG-driven sub-table routing module, the in-memory inverted index is It adopts a hybrid structure of hash table and skip list, supports precise matching with an average time complexity of O(1) for single label, intersection and union operations of multiple label combinations, and dynamic incremental updates when sub-tables are created or deleted.

10. The system according to claim 8, characterized in that, The dual-caching separate rule index module supports incremental updates and atomic replacement mechanisms when rules change. By constructing a new cache copy and then performing atomic pointer replacement, it avoids memory spikes caused by full updates. In the TAG-driven sub-table routing module, the number of sub-tables is the product of the number of devices and the number of metrics. During queries, only sub-tables are scanned after being located by the inverted index in memory. The amount of data scanned, T, is much smaller than the total amount of data, D, and the query complexity is O(T). The state consistency guarantee module ensures the data consistency of rule matching during rule changes through an atomic replacement mechanism.

Images