Real-time data processing system and method based on industrial Internet of Things platform

Abstract

The invention belongs to the field of industrial Internet of Things data processing, and particularly relates to a real-time data processing system and method based on an industrial Internet of Things platform, and the system comprises a data stream segmentation module and a parallel processing module. The data stream segmentation module is used for segmenting the real-time data stream into a plurality of data stream sub-streams according to a time window or a device ID (Identity); the parallel processing module comprises a plurality of processing units which run at the same time, and each processing unit independently processes the distributed data stream substreams in parallel; and the processing unit carries out data operation, screening and cleaning on the distributed data stream substreams, or the processing unit carries out operation on the distributed data stream substreams according to an alarm triggering formula to judge whether a triggering condition is achieved or not. The method has the beneficial effects that the real-time and efficient processing of the data of the Internet of Things equipment is realized by means of data stream segmentation, parallel processing and the like.

Description

Technical Field

[0001] The present invention belongs to the field of industrial Internet of Things data processing, and specifically relates to a real-time data processing system and method based on an industrial Internet of Things platform. Background Art

[0002] With the rapid development of the Industrial Internet of Things, more and more devices are connected to the Internet, generating massive amounts of real-time data. At present, there are some Industrial Internet of Things platforms on the market that can process and analyze the data of IoT devices to a certain extent. However, the existing platforms often have delays when processing real-time data, and the data processing functions are relatively simple, which cannot meet the complex data computing needs.

[0003] Existing industrial IoT platforms often have the problem of slow processing speed when processing real-time data. Many IoT devices send real-time data to the industrial IoT platform. The data processing module of the industrial IoT platform is under high load, a large amount of real-time data is in the queue waiting for data processing, the memory space is not fully and effectively utilized, and the processing speed is slow. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time data processing system and method based on an industrial Internet of Things platform for improving data processing speed.

[0005] In order to achieve the above objectives, the technical solution adopted by the present invention is as follows: a real-time data processing system based on an industrial Internet of Things platform, comprising a data stream segmentation module and a parallel processing module; The data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is allocated to the processing unit with the least number of current connections; data with trigger conditions of the same device ID are allocated to the same processing unit; The parallel processing module includes several processing units running simultaneously, each processing unit independently and in parallel processes the assigned data stream sub-stream; the processing unit performs data calculations, screening and cleaning on the assigned data stream sub-stream, or the processing unit calculates the assigned data stream sub-stream according to the alarm trigger formula and determines whether the trigger condition is met.

[0006] In a preferred embodiment, the real-time data processing system further comprises a storage management module; the storage management module comprises a memory management module; The memory management module calculates the memory space required for data processing of the data substreams according to the data volume of each data substream and the preset memory coefficient of the data processing rule after the data stream segmentation module determines to allocate the data stream substreams to the corresponding processing units, and dynamically allocates the memory space to the corresponding processing units, and releases the memory space allocated to the single data stream substream after the corresponding processing unit completes the data processing of the single data stream substream; In a preferred embodiment, the real-time data processing system further includes a data processing rule module for storing data operations, screening and cleaning rules, and storing trigger conditions and alarm trigger formulas.

[0007] In a preferred embodiment, the data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID. The specific steps are as follows: the data stream segmentation module identifies the data with trigger conditions in the real-time data stream, identifies the device ID corresponding to the data with trigger conditions, and assigns the data with trigger conditions corresponding to the same device ID as a single data stream sub-stream to the same processing unit; when the device ID corresponding to the data with trigger conditions in the real-time data stream appears for the first time, the data is assigned to the processing unit with the least current connections; for the data without trigger conditions in the real-time data stream, it is segmented into different data stream sub-streams according to the time window, and each data stream sub-stream is assigned to the processing unit with the least current connections.

[0008] Furthermore, when a single data flow sub-flow is allocated to the processing unit with the least current number of connections, if there is more than one processing unit with the least current number of connections, the data flow sub-flow is randomly allocated to one of the processing units with the least current number of connections.

[0009] In a preferred embodiment, the storage management module also includes a cache management module. For data stream sub-streams divided according to time windows, the cache management module is used to cache in real time the intermediate results obtained by each processing unit during data processing. The intermediate results are data calculation results and data screening results. When the processing unit completes data processing of a single data stream sub-stream, the corresponding cache is automatically released; for data stream sub-streams divided according to device IDs, the cache management module is used to cache in real time the calculation results of the alarm trigger formula. When the corresponding processing unit completes the determination of whether the trigger condition is met, the corresponding cache is automatically released.

[0010] Furthermore, in the parallel processing module, for the data stream sub-streams divided according to the time window, data operations are performed according to predefined operation rules; data screening is performed according to predefined screening rules; data cleaning is performed according to missing value processing rules, outlier processing rules, and data type conversion rules.

[0011] In a preferred embodiment, the memory management module presets a memory coefficient according to data processing rules of data stream substreams divided according to time windows that is greater than the memory coefficient preset in data processing rules of data stream substreams divided according to device IDs; for data stream substreams divided according to time windows, after a single data stream substream is processed by a corresponding processing unit, the memory space allocated for the single data stream substream is automatically released; for data stream substreams divided according to device IDs, after the determination of whether the trigger condition is met is completed, the memory space allocated for the single data stream substream is automatically released.

[0012] Another object of the present invention is to provide an industrial Internet of Things platform including the aforementioned real-time data processing system.

[0013] Another object of the present invention is to disclose a real-time data processing method based on an industrial Internet of Things platform using the aforementioned real-time data processing system.

[0014] In a preferred embodiment, the aforementioned real-time data processing method comprises the following steps: The real-time data stream is divided into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is assigned to the processing unit with the least number of current connections; the data with trigger conditions of the same device ID is assigned to the same processing unit, so that the load of each processing unit is relatively balanced; Each processing unit processes the assigned data stream sub-stream independently and in parallel; the processing unit performs data calculation, screening and cleaning on the assigned data stream sub-stream; or performs calculation on the assigned data stream sub-stream according to the alarm trigger formula to determine whether the trigger condition is met; The memory space is dynamically allocated to the corresponding processing unit according to the preset memory coefficient based on the data volume and data processing rules of each data stream sub-stream; for the data stream sub-streams divided according to the time window, the memory space allocated for the single data stream sub-stream is automatically released after the corresponding processing unit completes data processing of the single data stream sub-stream; for the data stream sub-streams divided according to the device ID, the memory space allocated for the single data stream sub-stream is automatically released after the determination of whether the trigger condition is met is completed.

[0015] The benefits of the present invention are that by adopting means such as data stream segmentation, parallel processing, and memory management optimization, real-time and efficient processing of IoT device data is achieved. At the same time, the rich data processing functions and flexible configuration options enable the present invention to be widely used in various industrial IoT scenarios, improve data quality and analysis efficiency, and bring better user experience. Specifically, the technical effects of the present invention include increasing data processing speed, enriching data processing functions, and supporting complex data operations. These advantages make the present invention have broad competitiveness and application prospects in the market. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the real-time data processing system of the present invention; Figure 2 The figure is a schematic diagram of the real-time data processing process of the data stream sub-streams divided according to the time windows of the present invention. DETAILED DESCRIPTION

[0017] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The present invention discloses a real-time data processing system based on an industrial Internet of Things platform, comprising a data stream segmentation module and a parallel processing module; The data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is allocated to the processing unit with the least number of current connections; data with trigger conditions of the same device ID are allocated to the same processing unit; The parallel processing module includes several processing units running simultaneously, each processing unit independently and in parallel processes the assigned data stream sub-stream; the processing unit performs data calculations, screening and cleaning on the assigned data stream sub-stream, or the processing unit calculates the assigned data stream sub-stream according to the alarm trigger formula and determines whether the trigger condition is met.

[0019] The real-time data processing system further comprises a storage management module; the storage management module comprises a memory management module; The memory management module calculates the memory space required for data processing of the data stream substreams according to the data volume of each data stream substream and the data processing rule preset memory coefficient after the data stream segmentation module determines to allocate the data stream substreams to the corresponding processing units, and dynamically allocates the memory space to the corresponding processing units. After the corresponding processing unit completes the data processing of a single data stream substream, the memory space allocated for the single data stream substream is released.

[0020] The real-time data processing system also includes a data processing rule module for storing data operations, screening and cleaning rules, as well as storing trigger conditions and alarm trigger formulas.

[0021] The real-time data processing method of the present invention comprises the following steps: The real-time data stream is divided into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is assigned to the processing unit with the least number of current connections; the data with trigger conditions of the same device ID is assigned to the same processing unit, so that the load of each processing unit is relatively balanced; Each processing unit processes the assigned data stream sub-stream independently and in parallel; the processing unit performs data calculation, screening and cleaning on the assigned data stream sub-stream; or performs calculation on the assigned data stream sub-stream according to the alarm trigger formula to determine whether the trigger condition is met; According to the data volume and data processing rules of each data stream sub-stream, the memory coefficient is preset to dynamically allocate memory space to the corresponding processing unit; for the data stream sub-streams divided according to the time window, after the single data stream sub-stream is processed by the corresponding processing unit, the memory space allocated for the single data stream sub-stream is automatically released; for the data stream sub-streams divided according to the device ID, after the determination of whether the trigger condition is met is completed, the memory space allocated for the single data stream sub-stream is automatically released ( Figure 1 ).

[0022] Specifically, the data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID. The specific steps are as follows: the data stream segmentation module identifies the data with trigger conditions in the real-time data stream, identifies the device ID corresponding to the data with trigger conditions, and allocates the data with trigger conditions corresponding to the same device ID as a single data stream sub-stream to the same processing unit, and allocates the data with trigger conditions corresponding to different device IDs as different data stream sub-streams; when the device ID corresponding to the data with trigger conditions in the real-time data stream appears for the first time, the data is allocated to the processing unit with the least current connections; for the data without trigger conditions in the real-time data stream, it is divided into different data stream sub-streams according to the time window, and each data stream sub-stream is allocated to the processing unit with the least current connections.

[0023] The data with trigger conditions refers to the data (variable) belonging to the variables in the alarm trigger formula stored in the data processing rule module.

[0024] That is, for real-time data streams, first select a real-time data stream within a time window, identify the data with trigger conditions in the real-time data stream of the time window, and divide it into data stream sub-streams according to the device ID. The primary principle of dividing the data stream sub-streams according to the device ID is that the data with trigger conditions corresponding to the same device ID are assigned to the same processing unit, wherein when the device ID corresponding to the data with trigger conditions in the real-time data stream appears for the first time, the data is assigned to the processing unit with the least current connections, and the parallel processing module records the processing unit to which the data with trigger conditions of the device ID is first assigned; when the device ID corresponding to the data with trigger conditions in the real-time data stream does not appear for the first time, the data is directly assigned to the processing unit to which the data with trigger conditions of the device ID is first assigned as recorded by the parallel processing module; since the real-time data stream within a time window is selected, the remaining data without trigger conditions is directly used as a data stream sub-stream. Then, the real-time data stream within the next time window is selected, and so on.

[0025] When a single data flow sub-flow is allocated to the processing unit with the least current connection number, if there is more than one processing unit with the least current connection number, the data flow sub-flow is randomly allocated to one of the processing units with the least current connection number.

[0026] The parallel processing module includes 2-5 processing units. In the parallel processing module, the upper limit of the number of data flow sub-streams that are processed simultaneously is the number of CPU cores of the industrial Internet of Things platform server + 1. When the number of data flow sub-streams that need to be processed exceeds the upper limit, the extra data flow sub-streams are in a waiting state. When a data flow sub-stream in the parallel processing module completes data processing and disconnects from the corresponding processing unit, the waiting data flow sub-stream is connected to the idle processing unit. After being assigned to a data flow sub-stream, each processing unit in the parallel processing module calls the corresponding data processing rules in the data processing rule module (i.e., completes rule loading) according to the device ID and variables corresponding to each data in the data flow sub-stream to perform data processing.

[0027] When the number of data stream sub-streams that need to be processed does not exceed the upper limit, whether a data stream sub-stream can successfully connect to the parallel processing module depends on whether the total memory allocated to all processing units after the memory management module calculates the memory space required for the data stream sub-stream is not greater than the total memory that can be allocated by the memory management module. If not, it is successfully connected; if greater, there is insufficient memory, causing the data stream sub-stream to be in a waiting state.

[0028] A data structure (such as a hash table or dictionary) is established that contains all processing units and their current connection numbers (when a data stream sub-flow is assigned to a processing unit for processing, a connection is formed, and when the data stream sub-flow completes data processing, the connection will be disconnected). This data structure is located in the parallel processing module. Whenever a data stream substream allocation request arrives, the parallel processing module checks this data structure and finds the processing unit with the least number of connections (where the data stream substreams divided according to the time window need to be allocated to the processing unit with the least current connections; the data stream substreams divided according to the device ID need to be allocated to the same processing unit, and the device ID is also allocated to the processing unit with the least current connections when it appears for the first time). The memory management module then calculates the memory space required for data processing by the data stream substream to be allocated. When the required memory space plus the total memory allocated to all processing units is not greater than the total memory that can be allocated by the memory management module, the data stream substream allocation request is allocated to the processing unit and the processing unit performs data processing. At this time, the data stream substream is successfully accessed (connected); otherwise, the data stream substream to be allocated is in a waiting state, that is, it waits for the memory to be released before determining whether the required memory space plus the total memory allocated to all processing units is not greater than the total memory that can be allocated by the memory management module.

[0029] Data operation, screening and cleaning rules can be preset rules, and can also be customized; trigger conditions and alarm condition formulas are preset rules. Each rule corresponds to a specific device ID and variable. Therefore, when a data flow sub-flow is connected to the processing unit for data processing, the processing unit can call the rules of the data processing rule module according to the device ID and specific variables corresponding to the data in the data flow sub-flow, and then complete the corresponding data processing.

[0030] The real-time data processing system of the present invention is a part of the Internet of Things middleware, and the Internet of Things middleware is installed in the industrial Internet of Things platform server.

[0031] In the present invention, the total memory that can be allocated by the memory management module is 80% of the total memory M of the CPU of the industrial Internet of Things platform server.

[0032] Specifically, the memory space allocated to a single data stream substream can be 80% of the total memory M of the CPU of the industrial Internet of Things platform server divided by the upper limit of the number of data stream substreams; or, the memory space allocated to a single data stream substream = data volume * memory space required for a single data * preset memory coefficient a. Since the memory must store all the data of the data stream substream and reserve a certain amount of memory space for data processing, a is 1-2. For example, the preset memory coefficient a for the data stream substream divided by time window can be 1.2-2, and the preset memory coefficient a for the data stream substream divided by device ID can be 1.05-1.3.

[0033] When a data stream sub-flow within a time window is assigned to a processing unit, it is regarded as one connection of the processing unit (that is, the number of connections of the processing unit + 1); when the data with trigger conditions corresponding to the same device ID is assigned to a processing unit as a data stream sub-flow, it is also regarded as one connection of the processing unit (that is, the number of connections of the processing unit + 1).

[0034] When a single data stream sub-flow is assigned to a processing unit, the number of connections corresponding to the processing unit is increased by 1. The order in which different data stream sub-flows are assigned to processing units is determined according to the earliest timestamp of the data in different data stream sub-flows. The data stream sub-flow with the earliest timestamp is assigned to the processing unit first.

[0035] A time window can be set to 2-8 seconds (preferably 5 seconds). A time window refers to the difference between the latest and earliest timestamps of the data in a single data stream sub-stream. Allocating a data stream sub-stream without trigger conditions within a time window to a processing unit is suitable for scenarios such as equipment automatic control and data screening and cleaning. After data processing, it is stored in the database (for example, in a time series database) or pushed to the application end of the industrial Internet of Things platform.

[0036] Data with trigger conditions are assigned to the same processing unit based on the device ID, which is suitable for scenarios where it is necessary to understand and analyze device behavior. For example, data with trigger conditions can be alarm data. Through multiple data with trigger conditions of the device ID, it can be understood whether there is a fault and whether the fault has been resolved; the amount of data is small, and the data processing process is relatively simple. The memory management module allocates a smaller memory to such a single data stream sub-stream. By setting the trigger conditions, the alarm trigger formula is set for one or more device variables (such as: variable 1>60&&variable 2>30, (variable 3*1.2+100)>200, etc.). The trigger condition refers to whether the calculation result of the alarm trigger formula reaches (i.e., is equal to or greater than) the preset trigger threshold. If the trigger threshold is reached, the corresponding processing unit determines that the trigger condition is met and sends an alarm to the application end or the corresponding device (to prompt the fault). When the alarm is sent, the corresponding device variables (i.e., data) can be sent to the application end or the corresponding device, and the user can be notified instantly by email, SMS, WeChat, etc.; after the alarm is issued, when at a certain time, the processing unit processes the new data with the trigger condition of the device, and after using the same alarm trigger formula for calculation, the calculation result of the alarm trigger formula is less than the trigger threshold, the processing unit determines that the alarm is lifted, and prompts the application end or the corresponding device that the fault is lifted.

[0037] Specifically, the preset memory coefficient of the data processing rules of the data stream sub-streams divided according to the time window is greater than the preset memory coefficient of the data processing rules of the data stream sub-streams divided according to the device ID; for the data stream sub-streams divided according to the time window, after the corresponding processing unit completes data processing and data warehousing / pushing for a single data stream sub-stream, the number of connections of the corresponding processing unit is reduced by 1 (i.e., the connection between the data stream sub-stream and the corresponding processing unit is disconnected), and the memory management module automatically releases the memory space allocated for the single data stream sub-stream; for the data stream sub-streams divided according to the device ID, after the determination of whether the trigger condition is met is completed and the alarm is issued or the obstacle is prompted to be removed, the number of connections of the corresponding processing unit is reduced by 1 (i.e., the connection between the data stream sub-stream and the corresponding processing unit is disconnected), and the memory management module automatically releases the memory space allocated for the single data stream sub-stream.

[0038] As mentioned above, since the data with trigger conditions corresponding to the same device ID may enter the real-time data processing system in different time windows, for example, when the device ID corresponding to the data with trigger conditions in a certain time window appears for the first time, it is allocated to the processing unit with the least number of current connections, and the processing unit completes data processing and alarm / alarm release, and immediately releases the memory space allocated for the data stream sub-stream; when the data with trigger conditions in another time window also corresponds to the device ID, the parallel processing module also allocates the data with trigger conditions in another time window to the processing unit to which the device ID is first allocated, and the processing unit completes data processing and alarm / alarm release, and immediately releases the memory space allocated for the data stream sub-stream. That is, in order to prevent the waste of memory space, the data stream sub-stream allocated according to the device ID disconnects from the corresponding processing unit and releases the corresponding memory after completing the data processing with trigger conditions in a single time window. The purpose of allocating the data with trigger conditions corresponding to the same device ID to the same processing unit is to facilitate the processing and statistical analysis of the data of the same device ID.

[0039] The storage management module also includes a cache management module. For data stream sub-streams divided according to time windows, the cache management module is used to cache in real time the intermediate results obtained by each processing unit during data processing. The intermediate results are data calculation results and data screening results. When the processing unit completes the data processing of a single data stream sub-stream, the corresponding cache is automatically released; for data stream sub-streams divided according to device IDs, the cache management module is used to cache in real time the calculation results of the alarm trigger formula. When the corresponding processing unit completes the determination of whether the trigger condition is met, the corresponding cache is automatically released.

[0040] In the parallel processing module, for the data flow sub-flows divided according to the time window, data operations are performed according to predefined operation rules; data screening is performed according to predefined screening rules; data cleaning is performed according to missing value processing rules, outlier processing rules, and data type conversion rules.

[0041] The present invention constructs a data stream segmentation module and a parallel processing module, and adopts a streaming processing technology to quickly receive and process the real-time data uploaded by the device access layer, thereby ensuring the real-time nature of the data.

[0042] Conventional stream processing technology is usually based on a general processing framework, while the present invention designs a high-performance stream data processing module based on the characteristics of industrial Internet of Things data, such as large data volume, high speed and diverse types.

[0043] In order to further improve the processing speed, the present invention adopts parallel processing technology, which significantly improves the overall processing capacity by allocating a single data stream sub-flow to the processing unit with the least current connection number and processing the allocated data stream sub-flow in parallel on multiple processing units.

[0044] Considering the large amount of real-time data generated by IoT devices, the present invention implements an efficient memory optimization management strategy. By reasonably allocating and using memory space, memory leakage and waste are avoided, ensuring the stable operation of the system under high load conditions.

[0045] Memory allocation strategy: Adopt dynamic memory allocation strategy to dynamically apply for and release memory space according to actual needs. Use memory pool technology to reduce the overhead of memory allocation and release. That is, preset the total memory available to all processing units - memory pool (that is, the total memory that can be allocated by the memory management module). If the remaining memory in the memory pool is not enough to provide for the processing unit to be connected to the new data flow sub-flow, the new data flow sub-flow will be placed in the waiting queue, and memory space will be allocated to the corresponding processing unit after the memory pool releases the memory space.

[0046] Cache management: Cache frequently used data (such as intermediate results) to reduce repeated calculations and memory accesses. Set the cache invalidation policy to ensure the validity and consistency of cached data. Frequently used data in cache management refers to data that is frequently accessed or processed. Intermediate results of the data processing process, including calculation results and screening results (i.e., filtering results), are considered frequently used data and cached so that they can be accessed faster during subsequent processing. Cleaned data is generally no longer required for subsequent processing and will not be stored in the cache. The cache invalidation policy includes putting frequently used data into the cache during data processing and clearing these cache items after the data flow completes data processing.

[0047] The present invention designs a flexible data processing flow for the data stream sub-streams divided according to the time window, including data calculation, data screening, data cleaning ( Figure 2 ). Among them, data cleaning can automatically identify and correct outliers and noise in the data; data screening can filter data according to predefined rules; data operations support complex operations such as four arithmetic operations and mathematical function operations for single or multiple data.

[0048] All cleaning rules, screening rules, and operation rules can be customized by users, that is, a user-customizable interface is provided for users to define relevant rules, which has strong flexibility. Alternatively, the relevant cleaning, screening, and operation rules can be predefined by developers.

[0049] Examples of operation rules: Arithmetic operations: perform the four basic operations of addition, subtraction, multiplication and division on data.

[0050] Relational operations: equal to, not equal to, greater than, less than, etc.

[0051] Mathematical function operation: Apply mathematical functions such as exponential, logarithmic, and trigonometric functions to process data.

[0052] Logical operations: AND, OR, NOT.

[0053] The data results obtained through calculation can be more conveniently displayed in subsequent visualization.

[0054] Examples of screening rules: ① Time filtering: filter according to the time range of data generation, such as selecting data within the last week or day.

[0055] ② Equipment screening: Screening based on the equipment from which the data comes, such as selecting equipment data for a specific production line or area.

[0056] ③Threshold screening: Set thresholds for specific parameters, such as temperature, humidity, pressure, etc., and select data that exceeds or is below the threshold.

[0057] ④Logical combination screening: logically combine multiple screening conditions to meet more complex screening needs.

[0058] Data cleaning targets problems in the original data, such as missing values, outliers, data type mismatches, etc., and performs specific processing and repair according to predefined cleaning rules. For example, for missing values, you can fill in missing values ​​or delete data rows containing missing values; for outliers, you can ensure data accuracy by identifying and processing them. The purpose of cleaning is to improve data quality, making it more accurate, complete, and consistent to meet the needs of subsequent data analysis. Through cleaning, noise and errors in the data can be eliminated, thereby improving the accuracy of data analysis and prediction.

[0059] Examples of cleaning rules: ①Missing value processing rules: fill missing values ​​with mean, median, mode, specific value, etc., or delete data rows containing missing values.

[0060] ②Outlier processing rules: Identify outliers based on statistical methods such as Z-score, IQR, etc., delete outliers, replace outliers with mean or median, etc.

[0061] ③Data type conversion rules: for example, converting a string to a numeric type, or converting a date and time string to a date and time format.

[0062] Specific steps and methods to identify missing values, outliers, and data type mismatches (as part of the cleaning rules): ① Missing value identification: Steps: Iterate through the dataset and check if the value of each field is empty or a specific marker (such as "NA", "NULL", etc.).

[0063] ②Outlier Identification: Steps: First calculate the statistics of the data (such as mean, standard deviation, median, quartiles, etc.), and then set thresholds based on these statistics to identify outliers.

[0064] Method: Z-score (standard score) and IQR (interquartile range) were used for judgment.

[0065] ③Data type mismatch identification: Steps: Check whether the data type of each field in the dataset is consistent with the predefined data type.

[0066] By using parallel processing, data streams are distributed and processed simultaneously in each processing unit. This can significantly increase data throughput and reduce processing time; when processing large amounts of real-time data, the load balancing algorithm can ensure that data is distributed approximately evenly among the processing units, thereby avoiding the situation where some processing units are overloaded while others are idle, and improving overall processing efficiency. Effective memory management is crucial for large-scale data processing in the Internet of Things to ensure efficient storage and access of data in memory, reduce memory fragmentation and cache failure, and improve processing performance.

[0067] The load balancing algorithm is to ensure that data or requests can be evenly distributed to multiple processing units to improve the overall processing efficiency and stability of the system. The present invention adopts the least connection method (Least Connections) to dynamically select the processing unit with the least current connection number to process new requests. It is necessary to maintain a data structure (such as a hash table or dictionary) containing all processing units and their current connection numbers. Whenever a request comes, check this data structure, find the processing unit with the least number of connections, and assign the request to it.

[0068] The industrial Internet of Things platform is mainly composed of data acquisition layer, device access layer, real-time data processing layer and application layer.

[0069] (1) Data collection layer: responsible for collecting real-time data from various IoT devices. This data may include sensor readings, device status information, etc. The collected data is aggregated to the industrial IoT gateway; (2) Device access layer: used to access data uploaded by the IIoT gateway in the IIoT platform; this layer is located in the IoT middleware; (3) Real-time data processing layer (i.e., the real-time data processing system of the present invention): This is the core part of the technical solution of the present invention. In this layer, the data undergoes a series of processing, including data calculation, data screening, and data cleaning; calculation is performed according to the alarm trigger formula to determine whether the trigger condition is met. These steps are performed in real time to ensure the timeliness and accuracy of the processing results;

[0070] (4) Application layer: The processed data is pushed to the application layer (i.e., the application end) for use by various industrial IoT applications, including monitoring, analysis, and prediction.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the protection scope of the present invention.

Claims

1. A real-time data processing system based on an industrial Internet of Things platform, characterized in that: It includes a data stream segmentation module and a parallel processing module; The data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is allocated to the processing unit with the least number of current connections; data with trigger conditions of the same device ID are allocated to the same processing unit; The parallel processing module includes several processing units running simultaneously, each processing unit independently and in parallel processes the assigned data stream sub-stream; the processing unit performs data calculations, screening and cleaning on the assigned data stream sub-stream, or the processing unit calculates the assigned data stream sub-stream according to the alarm trigger formula and determines whether the trigger condition is met.

2. The real-time data processing system based on the industrial Internet of Things platform according to claim 1 is characterized in that: It also includes a storage management module; the storage management module includes a memory management module; The memory management module calculates the memory space required for data processing of the data substreams according to the data volume of each data substream and the preset memory coefficient of the data processing rule after the data stream segmentation module determines to allocate the data stream substreams to the corresponding processing units, and dynamically allocates the memory space to the corresponding processing units, and releases the memory space allocated to the single data stream substream after the corresponding processing unit completes the data processing of the single data stream substream; It also includes a data processing rule module for storing data operations, screening and cleaning rules, as well as storing trigger conditions and alarm trigger formulas.

3. The real-time data processing system based on the industrial Internet of Things platform according to claim 1 is characterized in that: The data stream segmentation module divides the real-time data stream into several data stream sub-streams according to the time window or device ID in the specific steps as follows: the data stream segmentation module identifies the data with trigger conditions in the real-time data stream, identifies the device ID corresponding to the data with trigger conditions, and allocates the data with trigger conditions corresponding to the same device ID as a single data stream sub-stream to the same processing unit; When the device ID corresponding to the data with trigger conditions in the real-time data stream appears for the first time, the data is assigned to the processing unit with the least number of current connections; For data without trigger conditions in the real-time data stream, it is divided into different data stream sub-streams according to the time window, and each data stream sub-stream is assigned to the processing unit with the least current connection number.

4. The real-time data processing system based on the industrial Internet of Things platform according to claim 2 is characterized in that: When a single data flow sub-flow is allocated to the processing unit with the least current number of connections, if there is more than one processing unit with the least current number of connections, the data flow sub-flow is randomly allocated to one of the processing units with the least current number of connections.

5. The real-time data processing system based on the industrial Internet of Things platform according to claim 2 is characterized in that: The storage management module also includes a cache management module. For data stream sub-streams divided according to time windows, the cache management module is used to cache in real time the intermediate results obtained by each processing unit during data processing. The intermediate results are data calculation results and data screening results. When the processing unit completes the data processing of a single data stream sub-stream, the corresponding cache is automatically released; for data stream sub-streams divided according to device IDs, the cache management module is used to cache in real time the calculation results of the alarm trigger formula. When the corresponding processing unit completes the determination of whether the trigger condition is met, the corresponding cache is automatically released.

6. The real-time data processing system based on the industrial Internet of Things platform according to claim 1 is characterized in that: In the parallel processing module, for the data flow sub-flows divided according to the time window, data operations are performed according to predefined operation rules; data screening is performed according to predefined screening rules; data cleaning is performed according to missing value processing rules, outlier processing rules, and data type conversion rules.

7. The real-time data processing system based on the industrial Internet of Things platform according to claim 2 is characterized in that: The memory management module presets a memory coefficient according to data processing rules of data stream substreams divided according to time windows that is greater than a memory coefficient preset in data processing rules of data stream substreams divided according to device IDs; for data stream substreams divided according to time windows, after a single data stream substream is processed by a corresponding processing unit, the memory space allocated to the single data stream substream is automatically released; for data stream substreams divided according to device IDs, after a determination is made as to whether a trigger condition is met, the memory space allocated to the single data stream substream is automatically released.

8. An industrial Internet of Things platform comprising the real-time data processing system according to any one of claims 1 to 7.

9. A real-time data processing method based on an industrial Internet of Things platform using the real-time data processing system described in any one of claims 1 to 7.

10. The real-time data processing method according to claim 9, characterized in that: The steps include: The real-time data stream is divided into several data stream sub-streams according to the time window or device ID. Each data stream sub-stream divided according to the time window is assigned to the processing unit with the least number of current connections; the data with trigger conditions of the same device ID is assigned to the same processing unit, so that the load of each processing unit is relatively balanced; Each processing unit processes the assigned data stream sub-stream independently and in parallel; the processing unit performs data calculation, screening and cleaning on the assigned data stream sub-stream; or performs calculation on the assigned data stream sub-stream according to the alarm trigger formula to determine whether the trigger condition is met; According to the data volume and data processing rules of each data stream sub-stream, a memory coefficient is preset to dynamically allocate memory space to the corresponding processing unit; For data stream sub-streams divided according to time windows, the memory space allocated for a single data stream sub-stream will be automatically released after the corresponding processing unit completes data processing; for data stream sub-streams divided according to device IDs, the memory space allocated for a single data stream sub-stream will be automatically released after the determination of whether the trigger condition is met is completed.

Images