Load-Adaptive IoT Data Processing Method and System

Abstract

This invention provides a load-adaptive IoT data processing method and system, comprising: Step S1: acquiring device cycle information of IoT devices and classifying IoT devices, receiving device data uploaded by IoT devices, and publishing the device data to a message queue according to the IoT device classification; Step S2: processing the device data according to the data congestion level of the message queue; Step S3: analyzing the device data in an automated manner. This invention proposes a method that includes a producer load-adaptive strategy and a consumer load-adaptive strategy for IoT message passing systems, respectively targeting periodic data traffic and bursty data traffic of IoT devices, thus solving the technical problem of IoT data congestion.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and more specifically, to an IoT data processing method and system based on load adaptation. Background Technology

[0002] With advancements in broadband access networks, cloud / edge computing, big data analytics, and machine learning, the scale and capabilities of the Internet of Things (IoT) are growing rapidly. Smart homes, smart healthcare, smart transportation, smart cities, smart agriculture, and smart grids, all based on IoT, are flourishing. Within the IoT, numerous heterogeneous devices continuously exchange massive amounts of traffic between the network and servers. This presents both opportunities and challenges. The massive amounts of data generated by the IoT are rendering traditional IoT data processing capabilities increasingly ineffective, necessitating the integration of big data technologies to drive IoT development. IoT data processing systems face two main challenges. Firstly, these systems need to provide powerful analytical capabilities to comprehensively analyze device data from multiple sources. However, current IoT data processing systems either have limited application scenarios or require cumbersome manual operations, resulting in unsatisfactory efficiency. Secondly, IoT data processing systems need to handle the massive data streams from various IoT devices. However, the characteristics of IoT data include varying and fluctuating device loads; different devices generate different data flows, and even the same device can experience sudden changes in data flow. This can lead to data congestion, impacting the efficiency of subsequent data analysis and processing.

[0003] To address the challenges of IoT data analysis, machine learning techniques are typically used for predictive analysis. Machine learning leverages hidden information within large datasets, extracting value from these data sources with minimal human intervention. Machine learning is well-suited for the IoT environment due to the diverse and massive amounts of data from various sources, requiring comprehensive analysis that is beyond the capabilities of human resources. However, existing IoT machine learning technologies are insufficient to meet the demands of IoT data analysis.

[0004] To address the data congestion problem in the Internet of Things (IoT), two main technical approaches are typically employed: one is to rationally allocate resources available to different data streams through load balancing strategies, and the other is to reduce the traffic volume of high-speed data streams through resampling or data reduction techniques. Network resource load balancing strategies aim to improve the transmission efficiency of data streams from devices through the network to the IoT system ports, and have achieved good performance in resolving data congestion issues. Adaptive sampling and data reduction techniques reduce the amount of data in device streams at different stages of data stream generation and transmission; both techniques achieve good performance on most data streams. However, existing IoT data congestion techniques cannot completely solve the problem of IoT data congestion.

[0005] Existing IoT machine learning and IoT data blocking techniques suffer from at least the following technical problems:

[0006] 1. For IoT data analysis problems, existing IoT machine learning technologies require participants to master a lot of professional knowledge, compare the performance of different models, consider different algorithms, perform tedious hyperparameter tuning, and spend a long time training.

[0007] 2. Regarding the IoT data congestion problem, existing network resource load balancing strategies focus more on data congestion on the network. Data flow from devices to the IoT system is only part of IoT data transmission; data congestion can also occur within the IoT data processing system itself. In scenarios where network transmission is not the bottleneck, the effectiveness of network load balancing strategies is not significant. IoT data processing systems need to focus more on addressing data congestion issues within the system itself.

[0008] 3. Regarding the data congestion problem in the Internet of Things (IoT), it is difficult to accurately capture every sudden change in data flow with high variability. Furthermore, the reduction in data volume also affects the accuracy of subsequent data analysis stages, which is more pronounced in the presence of highly variable data flows.

[0009] Therefore, it can be seen that existing methods suffer from problems related to IoT machine learning and IoT data blocking.

[0010] Patent document CN115865982A discloses an Internet of Things (IoT) system and an IoT data processing method. The IoT system includes a user terminal configured to obtain a private key from an IoT server based on the user's account, and generate a data packet using the private key based on the user's uploaded information. The data packet contains the uploaded information and a signature generated based on the uploaded information. The user terminal is also configured to send the data packet to a blockchain platform. The IoT server is used to distribute asymmetric encryption keys. However, this invention does not solve the problem of IoT data congestion. Summary of the Invention

[0011] In view of the shortcomings of the prior art, the purpose of this invention is to provide an IoT data processing method and system based on load adaptation.

[0012] According to the present invention, an IoT data processing method based on load adaptation includes:

[0013] Step S1: Obtain the device lifecycle information of IoT devices and classify IoT devices, receive device data uploaded by IoT devices, and publish the device data to the message queue according to the IoT device classification;

[0014] Step S2: Process device data according to the data blocking level of the message queue;

[0015] Step S3: Analyze equipment data using automated methods.

[0016] Preferably, in step S1:

[0017] The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification.

[0018] The message producer load adaptive strategy method specifically includes:

[0019] Step S1.1: Calculate the device communication frequency based on the IoT device cycle information, and store the communication frequency set of all IoT devices in the background;

[0020] Step S1.2: Detect outliers in the communication frequency cluster of the devices, classify the devices, and store the device classification;

[0021] Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including:

[0022] Step S1.2.1: Calculate the outlier threshold of the device communication frequency concentration device frequency;

[0023] Step S1.2.2: Check whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold;

[0024] Step S1.2.3: If the device communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device.

[0025] As an outlier threshold calculation method, step S1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula:

[0026] F3 = Q3 + 1.5IQR

[0027] Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set;

[0028] Step S1.3: Calculate the ratio of the device's communication frequency to the outlier threshold, and use it as the maximum partition key of the message queue;

[0029] Step S1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

[0030] Preferably, in step S2:

[0031] Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue:

[0032] Step S2.1: Calculate the message lag level of the partition in the message queue;

[0033] Methods for calculating partition message lag include:

[0034] Step S2.1.1: Calculate the current message lag value for the partition. t :

[0035] lag t =offset end -offset current

[0036] Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition.

[0037] Step S2.1.2: Recalculate the current message lag level of the partition based on the current lag value, and calculate the partition message lag level EWMA using the exponentially weighted moving average method. t :

[0038] EWMA t =α×lag t +(1-α)×EWMA t-1

[0039] Among them, lag t EWMA represents the current message lag value for the partition, where α is the weight of the current message lag value, with a value between 0 and 1. t-1 This represents the message lag level from the last calculation.

[0040] Step S2.2: The message queue's consumer group monitors whether the latency of partitioned messages exceeds a threshold;

[0041] Step S2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reassigned.

[0042] Preferably, in step S3:

[0043] The automated data analysis module analyzes equipment data in an automated manner:

[0044] Step S3.1: Preprocess the historical device dataset and store it as a preprocessed device dataset;

[0045] Step S3.1.1: Process the device's historical dataset using a sliding window;

[0046] Step S3.1.2: Trim the noisy data in the equipment's historical dataset;

[0047] Step S3.1.3: Standardize the historical dataset of the processing equipment;

[0048] Step S3.1.4: Perform feature selection on the device historical dataset.

[0049] Step S3.2: Use automated machine learning technology to train a model on the preprocessed device dataset and store the model in the background;

[0050] Step S3.3: Use the model to predict the current data of the device;

[0051] Step S3.3.1: Preprocess the current data of the device using the method of preprocessing historical data sets of the device;

[0052] Step S3.3.2: Use the model stored in the background to predict the current data of the device.

[0053] Preferably, the method for processing the device's historical dataset using a sliding window in step S3.1.1 includes:

[0054] Step S3.1.1.1: Move the sliding window to the first unprocessed row of the device history dataset and take the average value of all rows within the window size range;

[0055] Step S3.1.1.2: Repeat step S3.1.1.1 until all historical data sets of the devices have been processed;

[0056] Step S3.1.2 Trim the noisy data in the equipment's historical dataset, including:

[0057] Step S3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset;

[0058] Step S3.1.2.2: Delete a fixed percentage of rows starting from the last row of the device history dataset;

[0059] The calculation method for the historical data z of the standardized processing equipment in step S3.1.3 is as follows:

[0060]

[0061] Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset;

[0062] The method for feature selection of the device historical dataset in step S3.1.4 includes:

[0063] Step S3.1.4.1: Calculate the mutual information between each feature of the device's historical dataset and the predicted target feature;

[0064] Step S3.1.4.2: Sort the device historical dataset according to the features of mutual information;

[0065] Step S3.1.4.3: Delete the features with the lowest mutual information at a fixed ratio.

[0066] According to the present invention, an IoT data processing system based on load adaptation includes:

[0067] Module M1: Acquires device lifecycle information of IoT devices and classifies IoT devices, receives device data uploaded by IoT devices, and publishes the device data to the message queue according to the IoT device classification;

[0068] Module M2: Processes device data based on the level of data blocking in the message queue;

[0069] Module M3: Analyzes equipment data in an automated manner.

[0070] Preferably, in module M1:

[0071] The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification.

[0072] The message producer load adaptive strategy method specifically includes:

[0073] Module M1.1: Calculates the communication frequency of IoT devices based on their periodic information and stores the communication frequencies of all IoT devices in the backend.

[0074] Module M1.2: Detects outliers in the communication frequency cluster of devices, classifies the devices, and stores the device classification;

[0075] Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including:

[0076] Module M1.2.1: Calculates the outlier threshold of the device communication frequency set for the device frequency;

[0077] Module M1.2.2: Checks whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold;

[0078] Module M1.2.3: If the device's communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device's communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device.

[0079] As an outlier threshold calculation method, module M1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula:

[0080] F3 = Q3 + 1.5IQR

[0081] Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set;

[0082] Module M1.3: Calculates the ratio of the device's communication frequency to the outlier threshold, which serves as the maximum partition key for the message queue;

[0083] Module M1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

[0084] Preferably, in module M2:

[0085] Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue:

[0086] Module M2.1: Calculates the message lag level of partitions in the message queue;

[0087] Methods for calculating partition message lag include:

[0088] Module M2.1.1: Calculates the current message lag value for the partition. t :

[0089] lag t =offset end -offset current

[0090] Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition.

[0091] Module M2.1.2: Recalculates the current message lag level of the partition based on the current lag value, using the exponentially weighted moving average (EWMA) method to calculate the partition message lag level. t :

[0092] EWMA t =α×lag t +(1-α)×EWMA t-1

[0093] Among them, lag t EWMA represents the current message lag value for the partition, where α is the weight of the current message lag value, with a value between 0 and 1. t-1 This represents the message lag level from the last calculation.

[0094] Module M2.2: The message queue consumer group monitors whether the lag of partitioned messages exceeds a threshold;

[0095] Module M2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reallocated.

[0096] Preferably, in module M3:

[0097] The automated data analysis module analyzes equipment data in an automated manner:

[0098] Module M3.1: Preprocesses the historical data set of the device and stores it as the preprocessed device dataset;

[0099] Module M3.1.1: Uses a sliding window to process historical device datasets;

[0100] Module M3.1.2: Noise data from the historical dataset of the trimming equipment;

[0101] Module M3.1.3: Standardizes historical datasets for processing equipment;

[0102] Module M3.1.4: Feature selection for historical device datasets.

[0103] Module M3.2: Uses automated machine learning techniques to train models on preprocessed device datasets and stores the models in the background;

[0104] Module M3.3: Uses a model to predict current device data;

[0105] Module M3.3.1: Preprocesses the current data of the device using a method that preprocesses historical data sets of the device;

[0106] Module M3.3.2: Uses models stored in the background to predict current device data.

[0107] Preferably, the method for processing historical data sets of the device using a sliding window in module M3.1.1 includes:

[0108] Module M3.1.1.1: The sliding window moves to the first unprocessed row of the device's historical dataset and takes the average value of all rows within the window size range;

[0109] Module M3.1.1.2: Repeat module M3.1.1.1 until all historical data sets of the device have been processed;

[0110] The noise data in the historical dataset of the pruning equipment in module M3.1.2 includes:

[0111] Module M3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset;

[0112] Module M3.1.2.2: Deletes a fixed percentage of rows starting from the last row of the device history dataset;

[0113] The calculation method for the historical data z of the standardized processing equipment in module M3.1.3 is as follows:

[0114]

[0115] Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset;

[0116] The methods for feature selection on historical device datasets in module M3.1.4 include:

[0117] Module M3.1.4.1: Calculates the mutual information between the features of the historical dataset of the computing device and the features of the prediction target;

[0118] Module M3.1.4.2: Sort the device historical dataset according to the features of mutual information;

[0119] Module M3.1.4.3: Removes features with the lowest mutual information at a fixed percentage.

[0120] Compared with the prior art, the present invention has the following beneficial effects:

[0121] 1. The data analysis and prediction results of this invention have high accuracy, are easy to use, have fast data processing response, high throughput, and are not easily affected by internal system data blocking;

[0122] 2. This invention proposes an automated data analysis method for the Internet of Things (IoT) that includes data preprocessing, automated model training, and predictive analysis, providing users with accuracy while ensuring greater convenience, and solving the problems of IoT data analysis;

[0123] 3. This invention proposes a method that includes a producer load adaptation strategy and a consumer load adaptation strategy for an IoT message passing system, which respectively address the periodic data traffic and bursty data traffic of IoT devices, thus solving the technical problem of IoT data congestion. Attached Figure Description

[0124] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0125] Figure 1 A detailed flowchart of a load-adaptive IoT data processing system provided by the present invention;

[0126] Figure 2 This is a schematic diagram of the workflow of the producer load adaptive strategy of the Internet of Things messaging system in an embodiment of the present invention.

[0127] Figure 3 This is a schematic diagram illustrating the workflow of the consumer load adaptive strategy of the Internet of Things messaging system in an embodiment of the present invention.

[0128] Figure 4 This is a schematic diagram of the workflow for automated data analysis via the Internet of Things in an embodiment of the present invention. Detailed Implementation

[0129] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0130] Example 1:

[0131] This invention discloses an IoT data processing system and method based on load adaptation. First, in terms of data analysis, the method combines IoT and automated machine learning technologies, and adds a data preprocessing module to make up for the shortcomings of automated machine learning, providing users with accurate and convenient data prediction and analysis capabilities. Second, for the data congestion problem in IoT, the method incorporates two different load adaptation strategies based on the periodicity and burstiness of IoT device data flow.

[0132] This invention provides an IoT data processing system and method based on load adaptation. Firstly, it offers an automated data analysis service that is universally applicable, convenient, and high-performance for IoT users with data analysis needs through an automated data analysis method. Then, it solves the data congestion problem caused by differences and changes in device load within the IoT by employing two IoT load adaptation strategies. The automated data analysis method of this invention provides users with a highly accurate and convenient option for big data analysis. The producer load adaptation strategy and the consumer load adaptation strategy enable the IoT data transmission system to have faster response speeds and higher throughput.

[0133] According to the present invention, a load-adaptive IoT data processing method is provided, such as... Figures 1-4 As shown, it includes:

[0134] Step S1: Obtain the device lifecycle information of IoT devices and classify IoT devices, receive device data uploaded by IoT devices, and publish the device data to the message queue according to the IoT device classification;

[0135] Specifically, in step S1:

[0136] The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification.

[0137] The message producer load adaptive strategy method specifically includes:

[0138] Step S1.1: Calculate the device communication frequency based on the IoT device cycle information, and store the communication frequency set of all IoT devices in the background;

[0139] Step S1.2: Detect outliers in the communication frequency cluster of the devices, classify the devices, and store the device classification;

[0140] Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including:

[0141] Step S1.2.1: Calculate the outlier threshold of the device communication frequency concentration device frequency;

[0142] Step S1.2.2: Check whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold;

[0143] Step S1.2.3: If the device communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device.

[0144] As an outlier threshold calculation method, step S1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula:

[0145] F3 = Q3 + 1.5IQR

[0146] Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set;

[0147] Step S1.3: Calculate the ratio of the device's communication frequency to the outlier threshold, and use it as the maximum partition key of the message queue;

[0148] Step S1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

[0149] Step S2: Process device data according to the data blocking level of the message queue;

[0150] Specifically, in step S2:

[0151] Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue:

[0152] Step S2.1: Calculate the message lag level of the partition in the message queue;

[0153] Methods for calculating partition message lag include:

[0154] Step S2.1.1: Calculate the current message lag value for the partition. t :

[0155] lag t =offset end -offset current

[0156] Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition.

[0157] Step S2.1.2: Recalculate the current message lag level of the partition based on the current lag value, and calculate the partition message lag level EWMA using the exponentially weighted moving average method. t :

[0158] EWMA t =α×lag t +(1-α)×EWMA t-1

[0159] Among them, lag t EWMA represents the current message lag value for the partition, where α is the weight of the current message lag value, with a value between 0 and 1. t-1 This represents the message lag level from the last calculation.

[0160] Step S2.2: The message queue's consumer group monitors whether the latency of partitioned messages exceeds a threshold;

[0161] Step S2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reassigned.

[0162] Step S3: Analyze equipment data using automated methods.

[0163] Specifically, in step S3:

[0164] The automated data analysis module analyzes equipment data in an automated manner:

[0165] Step S3.1: Preprocess the historical device dataset and store it as a preprocessed device dataset;

[0166] Step S3.1.1: Process the device's historical dataset using a sliding window;

[0167] Step S3.1.2: Trim the noisy data in the equipment's historical dataset;

[0168] Step S3.1.3: Standardize the historical dataset of the processing equipment;

[0169] Step S3.1.4: Perform feature selection on the device historical dataset.

[0170] Step S3.2: Use automated machine learning technology to train a model on the preprocessed device dataset and store the model in the background;

[0171] Step S3.3: Use the model to predict the current data of the device;

[0172] Step S3.3.1: Preprocess the current data of the device using the method of preprocessing historical data sets of the device;

[0173] Step S3.3.2: Use the model stored in the background to predict the current data of the device.

[0174] Specifically, the method for processing the device's historical dataset using a sliding window in step S3.1.1 includes:

[0175] Step S3.1.1.1: Move the sliding window to the first unprocessed row of the device history dataset and take the average value of all rows within the window size range;

[0176] Step S3.1.1.2: Repeat step S3.1.1.1 until all historical data sets of the devices have been processed;

[0177] Step S3.1.2 Trim the noisy data in the equipment's historical dataset, including:

[0178] Step S3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset;

[0179] Step S3.1.2.2: Delete a fixed percentage of rows starting from the last row of the device history dataset;

[0180] The calculation method for the historical data z of the standardized processing equipment in step S3.1.3 is as follows:

[0181]

[0182] Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset;

[0183] The method for feature selection of the device historical dataset in step S3.1.4 includes:

[0184] Step S3.1.4.1: Calculate the mutual information between each feature of the device's historical dataset and the predicted target feature;

[0185] Step S3.1.4.2: Sort the device historical dataset according to the features of mutual information;

[0186] Step S3.1.4.3: Delete the features with the lowest mutual information at a fixed ratio.

[0187] Example 2:

[0188] Example 2 is a preferred embodiment of Example 1, and is used to illustrate the present invention in more detail.

[0189] The present invention also provides a load-adaptive IoT data processing system, which can be implemented by executing the process steps of the load-adaptive IoT data processing method. That is, those skilled in the art can understand the load-adaptive IoT data processing method as a preferred embodiment of the load-adaptive IoT data processing system.

[0190] According to the present invention, an IoT data processing system based on load adaptation includes:

[0191] Module M1: Acquires device lifecycle information of IoT devices and classifies IoT devices, receives device data uploaded by IoT devices, and publishes the device data to the message queue according to the IoT device classification;

[0192] Specifically, in module M1:

[0193] The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification.

[0194] The message producer load adaptive strategy method specifically includes:

[0195] Module M1.1: Calculates the communication frequency of IoT devices based on their periodic information and stores the communication frequencies of all IoT devices in the backend.

[0196] Module M1.2: Detects outliers in the communication frequency cluster of devices, classifies the devices, and stores the device classification;

[0197] Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including:

[0198] Module M1.2.1: Calculates the outlier threshold of the device communication frequency set for the device frequency;

[0199] Module M1.2.2: Checks whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold;

[0200] Module M1.2.3: If the device's communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device's communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device.

[0201] As an outlier threshold calculation method, module M1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula:

[0202] F3 = Q3 + 1.5IQR

[0203] Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set;

[0204] Module M1.3: Calculates the ratio of the device's communication frequency to the outlier threshold, which serves as the maximum partition key for the message queue;

[0205] Module M1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

[0206] Module M2: Processes device data based on the level of data blocking in the message queue;

[0207] Specifically, in module M2:

[0208] Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue:

[0209] Module M2.1: Calculates the message lag level of partitions in the message queue;

[0210] Methods for calculating partition message lag include:

[0211] Module M2.1.1: Calculates the current message lag value for the partition. t :

[0212] lag t =offset end -offset current

[0213] Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition.

[0214] Module M2.1.2: Recalculates the current message lag level of the partition based on the current lag value, using the exponentially weighted moving average (EWMA) method to calculate the partition message lag level. t :

[0215] EWMA t =α×lag t +(1-α)×EWMA t-1

[0216] Among them, lag t θ represents the current message lag value for the partition, and θ is the weight of the current message lag value, with a value between 0 and 1. (EWMA) t-1 This represents the message lag level from the last calculation.

[0217] Module M2.2: The message queue consumer group monitors whether the lag of partitioned messages exceeds a threshold;

[0218] Module M2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reallocated.

[0219] Module M3: Analyzes equipment data in an automated manner.

[0220] Specifically, in module M3:

[0221] The automated data analysis module analyzes equipment data in an automated manner:

[0222] Module M3.1: Preprocesses the historical data set of the device and stores it as the preprocessed device dataset;

[0223] Module M3.1.1: Uses a sliding window to process historical device datasets;

[0224] Module M3.1.2: Noise data from the historical dataset of the trimming equipment;

[0225] Module M3.1.3: Standardizes historical datasets for processing equipment;

[0226] Module M3.1.4: Feature selection for historical device datasets.

[0227] Module M3.2: Uses automated machine learning techniques to train models on preprocessed device datasets and stores the models in the background;

[0228] Module M3.3: Uses a model to predict current device data;

[0229] Module M3.3.1: Preprocesses the current data of the device using a method that preprocesses historical data sets of the device;

[0230] Module M3.3.2: Uses models stored in the background to predict current device data.

[0231] Specifically, the method for processing device historical datasets using a sliding window in module M3.1.1 includes:

[0232] Module M3.1.1.1: The sliding window moves to the first unprocessed row of the device's historical dataset and takes the average value of all rows within the window size range;

[0233] Module M3.1.1.2: Repeat module M3.1.1.1 until all historical data sets of the device have been processed;

[0234] The noise data in the historical dataset of the pruning equipment in module M3.1.2 includes:

[0235] Module M3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset;

[0236] Module M3.1.2.2: Deletes a fixed percentage of rows starting from the last row of the device history dataset;

[0237] The calculation method for the historical data z of the standardized processing equipment in module M3.1.3 is as follows:

[0238]

[0239] Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset;

[0240] The methods for feature selection on historical device datasets in module M3.1.4 include:

[0241] Module M3.1.4.1: Calculates the mutual information between the features of the historical dataset of the computing device and the features of the prediction target;

[0242] Module M3.1.4.2: Sort the device historical dataset according to the features of mutual information;

[0243] Module M3.1.4.3: Removes features with the lowest mutual information at a fixed percentage.

[0244] Example 3:

[0245] Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0246] This invention proposes a load-adaptive IoT data processing system and method to solve, or at least partially solve, the problems of IoT data analysis and IoT data congestion. By integrating automated machine learning technology and improving various data preprocessing schemes, it realizes a universal service applicable to data analysis in various IoT fields, providing greater convenience for IoT users. Through optimization of the message passing system in the IoT data processing system, it provides two load-adaptive strategies: producer and consumer. These load-adaptive strategies collectively improve the overall performance of the system.

[0247] This invention provides a load-adaptive IoT data processing system and method. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating the specific process of a load-adaptive IoT data processing system and method. The solution includes:

[0248] First, the IoT data processing system includes multiple data processing modules; the basic steps are as follows:

[0249] S1: The message producer load adaptive strategy obtains the device cycle information of IoT devices and classifies the IoT devices according to the device cycle information; receives device data uploaded by the IoT devices and publishes the device data to the message queue according to the classification of the devices;

[0250] S2: The message consumer load adaptive strategy processes the device data according to the data blocking level of the message queue;

[0251] S3: The automated data analysis module analyzes the device data in an automated manner.

[0252] As a producer load adaptive strategy, S1 optimizes the messaging system of the IoT data processing system by allocating the number of available partitions to devices based on their communication cycles, thereby alleviating data congestion caused by periodic traffic.

[0253] The S1 message producer load adaptive strategy method specifically includes:

[0254] S1.1: Calculate the device communication frequency based on the IoT device periodic information, and store the communication frequency set of all IoT devices in the background;

[0255] S1.2: Detect outliers in the communication frequency set of the devices, classify the devices, and store the device classification;

[0256] S1.3: Calculate the ratio of the communication frequency of the device to the outlier threshold, and use it as the maximum partition key of the message queue;

[0257] S1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key, and sends the device data to the partition of the message queue.

[0258] As an outlier detection method, S1.2 identifies devices with excessively high periodic data traffic from the device frequency dataset, including:

[0259] S1.2.1: Calculate the outlier threshold of the device communication frequency set for the device frequencies;

[0260] S1.2.2: Check whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold;

[0261] S1.2.3: If yes, the device is marked as an outlier device; if no, the device is marked as a normal device. As an outlier threshold calculation method, S1.2.1 uses a box plot method to calculate the outlier threshold, with the following formula:

[0262] F3 = Q3 + 1.5IQR

[0263] Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set.

[0264] As a consumer load adaptive strategy, S2 optimizes the message passing system of the IoT data processing system by allocating consumers according to the partition lag of the message queue, thereby alleviating the data congestion problem caused by sudden traffic surges.

[0265] Specifically, please see Figure 2 This is a schematic diagram illustrating the workflow of the producer load adaptive strategy in an IoT messaging system.

[0266] S2's method for processing device data based on the data blocking level of the message queue specifically includes:

[0267] S2.1: Calculate the message lag level of the partition in the message queue;

[0268] S2.2: The consumer group of the message queue monitors whether the lag of partition messages exceeds a threshold;

[0269] S2.3: If applicable, the message lag levels are sorted from high to low, and consumers are sorted from low to high according to the number of consumption partitions, and consumers of the message queue partitions are reallocated.

[0270] As a method for calculating the degree of lag in partitioned messages, the method for calculating the degree of lag in partitioned messages in S2.1 includes:

[0271] S2.1.1: Calculate the current message lag value for the partition;

[0272] S2.1.2: Recalculate the current message lag level of the partition based on the current lag value.

[0273] As a method for calculating the lag value of partitioned messages, the formula for S2.1.1 is:

[0274] lag t =offset end -offset current

[0275] offset end The offset is the current message end offset of the partition. current The offset of messages consumed in the current partition.

[0276] As a method for calculating the lag of partitioned messages that takes into account the time effect, S2.1.2 uses the exponentially weighted moving average method to calculate the lag of partitioned messages, and the formula is:

[0277] EWMA t =α×lag t +(1-α)×EWMA t-1

[0278] Among them lag t The current message lag value for the partition is α, where α is the weight of the current message lag value, ranging from 0 to 1. (EWMA) t-1 This represents the message lag level from the last calculation.

[0279] Specifically, please see Figure 3 This is a schematic diagram illustrating the workflow of the consumer load adaptive strategy in an IoT messaging system.

[0280] As an automated data analysis technology, S3's method for automatically analyzing the device data specifically includes:

[0281] S3.1: Preprocess the device historical dataset and store it as a preprocessed device dataset;

[0282] S3.2: Use automated machine learning technology to train a model on the preprocessed device dataset and store the model in the background;

[0283] S3.3: Use the model to predict the current data of the device.

[0284] As a method for preprocessing historical device datasets, the methods for preprocessing historical device datasets in S3.1 include:

[0285] S3.1.1: Process the device's historical dataset using a sliding window;

[0286] S3.1.2: Trim the noisy data in the device's historical dataset;

[0287] S3.1.3: Standardize the historical dataset of the device;

[0288] S3.1.4: Perform feature selection on the device's historical dataset.

[0289] As a method for preprocessing device history datasets using a sliding window, the sliding window method for processing the device history dataset in S3.1.1 includes:

[0290] S3.1.1.1: Move the sliding window to the first unprocessed row of the device's historical dataset and take the average value of all rows within the window size range.

[0291] S3.1.1.2: Repeat S3.1.1.1 until all the device historical datasets have been processed.

[0292] As a method for trimming preprocessed device history datasets, S3.1.2 trimming noisy data in the device history dataset includes:

[0293] S3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset;

[0294] S3.1.2.2: Delete a fixed percentage of rows starting from the last row of the device's historical dataset.

[0295] As a method for standardizing and preprocessing historical equipment datasets, the calculation method for standardizing the historical equipment data in S3.1.3 is as follows:

[0296]

[0297] Where x is the data in the device's historical dataset, μ is the mean of the device's historical dataset, and s is the standard deviation of the device's historical dataset.

[0298] As a method for feature selection preprocessing of device historical datasets, the method for feature selection of the device historical dataset in S3.1.4 includes:

[0299] S3.1.4.1: Calculate the mutual information between each feature of the device's historical dataset and the predicted target feature;

[0300] S3.1.4.2: Sort the features of the device historical dataset according to the mutual information;

[0301] S3.1.4.3: Delete a fixed proportion of the features with the lowest mutual information.

[0302] As a method for real-time prediction of datasets, the method for predicting the current data of the device in S3.3 includes:

[0303] S3.3.1: Preprocess the current data of the device using the method described above for preprocessing historical data sets of the device;

[0304] S3.3.2: Use the model stored in the background to predict the current data of the device.

[0305] Specifically, please see Figure 4 This is a schematic diagram illustrating the workflow of automated data analysis in the Internet of Things (IoT).

[0306] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0307] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A load-adaptive IoT data processing method, characterized in that, include: Step S1: Obtain the device lifecycle information of IoT devices and classify IoT devices, receive device data uploaded by IoT devices, and publish the device data to the message queue according to the IoT device classification; Step S2: Process device data according to the data blocking level of the message queue; Step S3: Analyze equipment data using automated methods; In step S1: The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification. The message producer load adaptive strategy method specifically includes: Step S1.1: Calculate the device communication frequency based on the IoT device cycle information, and store the communication frequency set of all IoT devices in the background; Step S1.2: Detect outliers in the communication frequency cluster of the devices, classify the devices, and store the device classification; Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including: Step S1.2.1: Calculate the outlier threshold of the device communication frequency concentration device frequency; Step S1.2.2: Check whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold; Step S1.2.3: If the device communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device. As an outlier threshold calculation method, step S1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula: F3 = Q3 + 1.5IQR Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set; Step S1.3: Calculate the ratio of the device's communication frequency to the outlier threshold, and use it as the maximum partition key of the message queue; Step S1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

2. The IoT data processing method based on load adaptation according to claim 1, characterized in that, In step S2: Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue: Step S2.1: Calculate the message lag level of the partition in the message queue; Methods for calculating partition message lag include: Step S2.1.1: Calculate the current message lag value for the partition. t : law t =offset end -offset current Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition. Step S2.1.2: Recalculate the current message lag level of the partition based on the current lag value, and calculate the partition message lag level EWMA using the exponentially weighted moving average method. t : EWMA t =α×lag t +(1-a)×EWMA t-1 Among them, lag t EWMA represents the current message lag value for the partition, where α is the weight of the current message lag value, with a value between 0 and 1. t-1 This represents the message lag level from the last calculation. Step S2.2: The message queue's consumer group monitors whether the latency of partitioned messages exceeds a threshold; Step S2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reassigned.

3. The IoT data processing method based on load adaptation according to claim 1, characterized in that, In step S3: The automated data analysis module analyzes equipment data in an automated manner: Step S3.1: Preprocess the historical device dataset and store it as a preprocessed device dataset; Step S3.1.1: Process the device's historical dataset using a sliding window; Step S3.1.2: Trim the noisy data in the equipment's historical dataset; Step S3.1.3: Standardize the historical dataset of the processing equipment; Step S3.1.4: Perform feature selection on the device historical dataset; Step S3.2: Use automated machine learning technology to train a model on the preprocessed device dataset and store the model in the background; Step S3.3: Use the model to predict the current data of the device; Step S3.3.1: Preprocess the current data of the device using the method of preprocessing historical data sets of the device; Step S3.3.2: Use the model stored in the background to predict the current data of the device.

4. The IoT data processing method based on load adaptation according to claim 3, characterized in that: The method for processing the device history dataset using a sliding window in step S3.1.1 includes: Step S3.1.1.1: Move the sliding window to the first unprocessed row of the device history dataset and take the average value of all rows within the window size range; Step S3.1.1.2: Repeat step S3.1.1.1 until all historical data sets of the devices have been processed; Step S3.1.2 Trim the noisy data in the equipment's historical dataset, including: Step S3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset; Step S3.1.2.2: Delete a fixed percentage of rows starting from the last row of the device history dataset; The calculation method for the historical data z of the standardized processing equipment in step S3.1.3 is as follows: Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset; The method for feature selection of the device historical dataset in step S3.1.4 includes: Step S3.1.4.1: Calculate the mutual information between each feature of the device's historical dataset and the predicted target feature; Step S3.1.4.2: Sort the device historical dataset according to the features of mutual information; Step S3.1.4.3: Delete the features with the lowest mutual information at a fixed ratio.

5. A load-adaptive Internet of Things (IoT) data processing system, characterized in that, include: Module M1: Acquires device lifecycle information of IoT devices and classifies IoT devices, receives device data uploaded by IoT devices, and publishes the device data to the message queue according to the IoT device classification; Module M2: Processes device data based on the level of data blocking in the message queue; Module M3: Analyzes equipment data in an automated manner; In module M1: The message producer load adaptive strategy obtains device lifecycle information of IoT devices and classifies IoT devices according to the device lifecycle information; it receives device data uploaded by IoT devices and publishes the device data to the message queue according to the IoT device classification. The message producer load adaptive strategy method specifically includes: Module M1.1: Calculates the communication frequency of IoT devices based on their periodic information and stores the communication frequencies of all IoT devices in the backend. Module M1.2: Detects outliers in the communication frequency cluster of devices, classifies the devices, and stores the device classification; Identify devices from the device frequency dataset whose periodic data traffic exceeds a preset standard, including: Module M1.2.1: Calculates the outlier threshold of the device communication frequency set for the device frequency; Module M1.2.2: Checks whether the communication frequency of each device in the device communication frequency set is greater than the outlier threshold; Module M1.2.3: If the device's communication frequency is greater than the outlier threshold, the device is marked as an outlier device; if the device's communication frequency is less than or equal to the outlier threshold, the device is marked as a normal device. As an outlier threshold calculation method, module M1.2.1 uses the box plot method to calculate the outlier threshold F3, with the following formula: F3 = Q3 + 1.5IQR Where Q3 is the third quartile of the device communication frequency set, and IQR is the interquartile range of the device communication frequency set; Module M1.3: Calculates the ratio of the device's communication frequency to the outlier threshold, which serves as the maximum partition key for the message queue; Module M1.4: The message producer of the message queue randomly selects a partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.

6. The load-adaptive IoT data processing system according to claim 5, characterized in that, In module M2: Optimize the message passing system of the IoT data processing system by allocating consumers based on the partition lag of the message queue: Module M2.1: Calculates the message lag level of partitions in the message queue; Methods for calculating partition message lag include: Module M2.1.1: Calculates the current message lag value for the partition. t : law t =offset end -offset current Among them, offset end The offset is the end offset of the current message in the partition. current This represents the consumed offset of the current message in the partition. Module M2.1.2: Recalculates the current message lag level of the partition based on the current lag value, using the exponentially weighted moving average (EWMA) method to calculate the partition message lag level. t : EWMA t =α×lag t +(1-ɑ)×EWMA t-1 Among them, lag t EWMA represents the current message lag value for the partition, where α is the weight of the current message lag value, with a value between 0 and 1. t-1 This represents the message lag level from the last calculation. Module M2.2: The message queue consumer group monitors whether the lag of partitioned messages exceeds a threshold; Module M2.3: If the lag exceeds the threshold, the message lag is sorted from high to low, and the consumers are sorted from low to high according to the number of consumption partitions, and the consumers of the message queue partitions are reallocated.

7. The load-adaptive IoT data processing system according to claim 5, characterized in that, In module M3: The automated data analysis module analyzes equipment data in an automated manner: Module M3.1: Preprocesses the historical data set of the device and stores it as the preprocessed device dataset; Module M3.1.1: Uses a sliding window to process historical device datasets; Module M3.1.2: Noise data from the historical dataset of the trimming equipment; Module M3.1.3: Standardizes historical datasets for processing equipment; Module M3.1.4: Feature selection for historical device datasets; Module M3.2: Uses automated machine learning techniques to train models on preprocessed device datasets and stores the models in the background; Module M3.3: Uses a model to predict current device data; Module M3.3.1: Preprocesses the current data of the device using a method that preprocesses historical data sets of the device; Module M3.3.2: Uses models stored in the background to predict current device data.

8. The IoT data processing system based on load adaptation according to claim 7, characterized in that: The method for handling device history datasets using a sliding window in module M3.1.1 includes: Module M3.1.1.1: The sliding window moves to the first unprocessed row of the device's historical dataset and takes the average value of all rows within the window size range; Module M3.1.1.2: Repeat module M3.1.1.1 until all historical data sets of the device have been processed; The noise data in the historical dataset of the pruning equipment in module M3.1.2 includes: Module M3.1.2.1: Delete rows with consecutive values ​​of 0, starting from the first row of the device history dataset; Module M3.1.2.2: Deletes a fixed percentage of rows starting from the last row of the device history dataset; The calculation method for the historical data z of the standardized processing equipment in module M3.1.3 is as follows: Where x is the data in the device history dataset, μ is the mean of the device history dataset, and s is the standard deviation of the device history dataset; The methods for feature selection on historical device datasets in module M3.1.4 include: Module M3.1.4.1: Calculates the mutual information between the features of the historical dataset of the computing device and the features of the prediction target; Module M3.1.4.2: Sort the device historical dataset according to the features of mutual information; Module M3.1.4.3: Removes features with the lowest mutual information at a fixed percentage.

Images