Real-time data processing method for Internet of Things based on dynamic rules

Abstract

The present invention discloses a real-time data and processing method of the Internet of Things based on dynamic rules, which has fast calculation speed and less resource occupation. The method of the present invention comprises: (10) dynamic rule configuration information loading: loading device dynamic calculation rule configuration information; (20) device data queuing: putting real-time device related data into the Internet of Things data message queue; (30) device data forwarding: obtaining device data from the Internet of Things data message queue and sending it to the computing node; (40) computing node health check: the computing node authenticates and registers the service with the computing center, reports regularly, and performs health check on each computing node; (50) computing rule configuration acquisition: the computing node obtains the dynamic rule configuration according to the device data content; (60) data calculation processing according to the dynamic rule configuration: the computing node performs parallel calculation according to the device data and the dynamic rule configuration; (70) computing result push: pushing the calculation result to the calculation result queue.

Description

Technical Field

[0001] The invention belongs to the technical field of Internet of Things, and in particular to a real-time data processing method of the Internet of Things based on dynamic rules. Background Art

[0002] In the field of Internet of Things, the real-time data of IoT devices must be calculated and processed before use according to different businesses. For example, different icons of the devices are displayed based on the various status data of the devices; multiple device data status values ​​within a certain value range trigger a behavior, etc.

[0003] Due to the ability to support a massive number of access devices, the current data access concept of IoT platform devices is generally to use queue access and perform calculations sequentially or concurrently; computer computing resources are limited, and the timeliness of data calculation results is restricted.

[0004] In actual use, the efficiency of sequential execution using queues is slow; or threads are used for concurrent calculations, but during the concurrent calculation process, the calculation parameters existing in the computing resources have complex dependencies, and the large-scale use of threads will cause multi-threading to suspend, resulting in high resource overhead.

[0005] Another solution is to deploy multiple computing nodes. The computing center sends messages to the computing nodes, and the computing nodes receive the messages and execute the calculations sequentially or concurrently. This deployment can share the computing pressure of massive real-time data, but the above problems will still exist in the concurrent process.

[0006] Therefore, the problem existing in the prior art is: how to utilize limited hardware computing resources and allocate them reasonably to provide fast and stable computing capabilities without affecting the data calculation results. Summary of the invention

[0007] The purpose of the present invention is to provide a real-time data and processing method of the Internet of Things based on dynamic rules, which has fast calculation speed and small resource occupation.

[0008] The technical solution for achieving the purpose of the present invention is:

[0009] A method for real-time data processing of the Internet of Things based on dynamic rules comprises the following steps:

[0010] (10) Dynamic rule configuration information loading: Load the device dynamic calculation rule configuration information and store it in the distributed cache;

[0011] (20) Device data queuing: Obtain real-time device-related data from the IoT system and put the device-related data into the IoT data message queue as a message producer;

[0012] (30) Device data forwarding: The computing center obtains device data from the IoT data message queue and sends the device data to the computing nodes assigned by the computing center according to the forwarding strategy configured by the computing center;

[0013] (40) Computing node health check: The computing node authenticates and registers the service with the computing center and reports regularly. The computing center regularly performs health checks on each computing node.

[0014] (50) Obtaining calculation rule configuration: The calculation node obtains the dynamic rule configuration from the distributed cache according to the device data content. If the dynamic rule configuration exists, the next step of calculation is performed. If the dynamic rule configuration does not exist, no operation is performed.

[0015] (60) Data calculation and processing according to dynamic rule configuration: the computing nodes perform parallel calculations according to the device data and dynamic rule configuration;

[0016] (70) Pushing calculation results: Pushing calculation results to the calculation result queue, and the business module consumers get the result data to perform corresponding business processing.

[0017] Compared with the prior art, the present invention has the following significant advantages:

[0018] 1. Fast computing speed: The computing center obtains data and sends it to the appropriate computing node for calculation according to the specified strategy. When massive real-time data enters, the data will be distributed to different computing nodes for calculation, which distributes the computing traffic, increases data concurrency, and improves execution speed.

[0019] 2. Less resource usage: During the computing process of computing nodes, multi-consumer parallel computing is adopted. If the computing parameters involve dependencies, the calculation is performed through temporary variables + scheduling, which improves the execution efficiency, occupies less CPU resources, and improves the resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the main flow chart of the real-time data processing method of the Internet of Things based on dynamic rules of the present invention.

[0021] Figure 2 This is the structural block diagram of the Internet of Things system.

[0022] Figure 3 for Figure 1 Flowchart of the data forwarding steps of the device in FIG.

[0023] Figure 4 for Figure 1 Flowchart of the compute node health check steps.

[0024] Figure 5 for Figure 1Flowchart of the steps for configuring data calculation and processing based on dynamic rules. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, the real-time data processing method of the Internet of Things based on dynamic rules of the present invention is used as follows Figure 2 In the IoT system shown in the figure, the real-time data of the massive connected devices is processed in a timely manner. Figure 2 As shown, the IoT system includes an IoT platform, an IoT data collection module, a distributed cache, an IoT data message queue, a computing center, multiple computing nodes, a computing result message queue, and other business modules. The IoT data collection module is connected to the IoT platform through a network connection to complete data collection, and pushes device data to the IoT data message queue. The computing center obtains device data from the IoT data message queue. The computing center forwards device data to the computing node according to the forwarding strategy. The computing node obtains dynamic rule configuration from the distributed cache according to the device data information. After the computing node completes the calculation, it pushes the result to the calculation result message queue. Other business modules obtain the result from the calculation result message queue to perform their respective businesses.

[0026] like Figure 1 As shown, the real-time data processing method of the Internet of Things based on dynamic rules of the present invention comprises the following steps:

[0027] (10) Dynamic rule configuration information loading: Load the device dynamic calculation rule configuration information and store it in the distributed cache;

[0028] The device dynamic rule configuration information refers to the logical rules used for data calculation of IoT devices.

[0029] (20) Device data queuing: Obtain real-time device-related data from the IoT system and put the device-related data into the IoT data message queue as a message producer;

[0030] The IoT acquisition module can use MQTT or API interface to connect to the IoT system to obtain real-time device-related data, and put the device data into the IoT data message queue as a producer (the queue uses a first-in-first-out method).

[0031] (30) Device data forwarding: The computing center obtains device data from the IoT data message queue and sends the device data to the computing nodes assigned by the computing center according to the forwarding strategy configured by the computing center;

[0032] The configuration forwarding strategy includes: optimal algorithm, random, weighted, polling, etc.

[0033] The (30) device data forwarding step includes:

[0034] (31) The computing center obtains device data: The computing center accesses the IoT data message queue as a consumer and obtains device data from the IoT data message queue in sequence;

[0035] (32) Device data forwarding scheduling: The computing center is the main node, connecting multiple computing nodes. The computing center does not participate in the calculation, but is only responsible for scheduling IoT messages to different computing nodes according to the forwarding strategy. The computing nodes are responsible for the calculation.

[0036] (40) Computing node health check: The computing node authenticates and registers the service with the computing center and reports regularly. The computing center regularly performs health checks on each computing node.

[0037] The (40) computing node health check step includes:

[0038] (41) Computing node authentication, registration and reporting: After the computing node is started, the computing node authenticates and registers the service with the computing center, regularly reports the computing node's computing capacity and load status, including CPU utilization and memory usage, and provides a computing node health check interface;

[0039] The health check interface is used to query whether the computing node service is online;

[0040] (42) Computing center health check: The computing center regularly checks the health status of each computing node through the health check interface;

[0041] The computing center regularly performs health checks on downstream computing nodes to ensure that data is sent to reliable computing nodes for calculation. The computing center sends IoT device data to different computing nodes based on optimal algorithms, randomness, weights (the computing power of computing nodes), polling, customization and other strategies.

[0042] (50) Obtaining calculation rule configuration: The calculation node obtains the dynamic rule configuration from the distributed cache according to the device data content. If the dynamic rule configuration exists, the next step of calculation is performed. If the dynamic rule configuration does not exist, no operation is performed.

[0043] The device data content includes a device code.

[0044] The computing node receives IoT device data sent by the computing center and obtains the specified dynamic rule configuration from the distributed cache based on the data content.

[0045] (60) Data calculation and processing according to dynamic rule configuration: the computing nodes perform parallel calculations according to the device data and dynamic rule configuration;

[0046] The step (60) of calculating and processing data according to dynamic rule configuration includes:

[0047] (61) Loading device data and dynamic rule configuration: The built-in module rule scheduler in the computing node loads device data and dynamic rule configuration;

[0048] (62) Input parameter storage: The rule scheduler stores the input parameters required for each rule to execute calculations in the form of variables in memory, with the default value being empty;

[0049] (63) Execution of calculation rules: The rule scheduler assigns values ​​to the input parameters of each calculation dynamic rule in turn. When the calculation rule input parameters meet the calculation conditions, the rule calculation consumers are scheduled to start executing the calculation rules that meet the conditions.

[0050] (64) Reporting of rule calculation results: After the rule calculation consumer completes execution, it reports the result to the rule scheduler;

[0051] (65) Obtaining results and sending results: The rule scheduler records the results and reports the completion status of the dynamic rules. The rule scheduler determines whether the entire dynamic rule is calculated. If not, it jumps to (63) the calculation rule execution step; if so, the calculation result is sent by the rule scheduler to the calculation result queue;

[0052] (70) Pushing calculation results: Pushing calculation results to the calculation result queue, and the business module consumers get the result data to perform corresponding business processing.

[0053] The rule scheduler pushes the calculation results to the calculation result queue, and the business module consumers obtain the result data for corresponding business processing.

[0054] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for real-time data processing of the Internet of Things based on dynamic rules, comprising the following steps: (10) Dynamic rule configuration information loading: Load the device dynamic calculation rule configuration information and store it in the distributed cache; (20) Device data queuing: Obtain real-time device-related data from the IoT system and put the device-related data into the IoT data message queue as a message producer; (30) Device data forwarding: The computing center obtains device data from the IoT data message queue and sends the device data to the computing nodes assigned by the computing center according to the forwarding strategy configured by the computing center; (40) Computing node health check: The computing node authenticates and registers the service with the computing center and reports regularly. The computing center regularly performs health checks on each computing node. (50) Obtaining calculation rule configuration: The calculation node obtains the dynamic rule configuration from the distributed cache according to the device data content. If the dynamic rule configuration exists, the next step of calculation is performed. If the dynamic rule configuration does not exist, no operation is performed. (60) Data calculation and processing according to dynamic rule configuration: The computing nodes perform parallel calculations according to the device data and dynamic rule configuration; (70) Push calculation results: Push the calculation results to the calculation result queue, and the business module consumers get the result data to perform corresponding business processing; Characterized in that the (30) device data forwarding step includes: (31) The computing center obtains device data: The computing center accesses the IoT data message queue as a consumer and obtains device data from the IoT data message queue in sequence; (32) Device data forwarding scheduling: The computing center dispatches IoT messages to different computing nodes based on the forwarding strategy, and the computing nodes are responsible for the calculation; The step (60) of calculating and processing data according to dynamic rule configuration includes: (61) Loading device data and dynamic rule configuration: The built-in module rule scheduler in the computing node loads device data and dynamic rule configuration; (62) Input parameter storage: The rule scheduler stores the input parameters required for each rule to execute calculations in the form of variables in memory, with the default value being empty; (63) Execution of calculation rules: The rule scheduler assigns values ​​to the input parameters of each calculation dynamic rule in turn. When the calculation rule input parameters meet the calculation conditions, the rule calculation consumers are scheduled to start executing the calculation rules that meet the conditions. (64) Reporting of rule calculation results: After the rule calculation consumer completes execution, it reports the results to the rule scheduler; (65) Obtaining results and sending them: The rule scheduler records the results and reports the completion status of the dynamic rules. The rule scheduler determines whether the entire dynamic rule has been calculated. If not, it jumps to (63) the calculation rule execution step; if so, the calculation results are sent by the rule scheduler to the calculation result queue.

2. The method for real-time data processing of the Internet of Things according to claim 1, characterized in that: The (40) computing node health check step includes: (41) Computing node authentication, registration and reporting: After the computing node is started, the computing node authenticates and registers the service with the computing center, regularly reports the computing capacity and load status of the computing node, including CPU utilization and memory usage, and provides a computing node health check interface; (42) Computing center health check: The computing center regularly checks the health status of each computing node through the health check interface.

Images