A processing method and system for industrial internet of things data

Abstract

The application discloses to the technical field of data processing, and particularly relates to a processing method and system for industrial internet of things data. First, multi-dimensional flow data of the industrial internet of things is collected, and after noise reduction, feature extraction and self-adaptive calibration, future flow change trend is predicted, and flow prediction result is output. Then, based on the flow prediction result, data priority is divided and flow load is matched, and meanwhile, preprocessing sequence is planned, differentiated preprocessing is executed, and standardized preprocessing data set is output. Based on this, the application can adjust data collection strategy through flow sensing prediction, can not only adapt to flow fluctuation and optimize collection frequency, avoid transmission congestion, but also can realize stable transmission of key data and effectively reduce data loss.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for processing data in the Industrial Internet of Things (IIoT). Background Technology

[0002] As a core supporting technology of Industry 4.0, the Industrial Internet of Things (IIoT) enables data collection, transmission and analysis throughout the entire industrial production process through a massive number of sensors deployed on production equipment, transmission links and terminal nodes, providing data support for production optimization, fault early warning and energy efficiency management.

[0003] Currently, industrial IoT data exhibits typical characteristics of "massive volume, heterogeneity, suddenness, and strong real-time performance." In production scenarios, data ranging from equipment operating parameters (such as machine tool speed and pipeline pressure), environmental parameters (such as workshop temperature and humidity and dust concentration) to energy consumption data can generate GB or even TB levels per second. Furthermore, the data types cover structured data (such as equipment number and runtime), semi-structured data (such as log files), and unstructured data (such as equipment monitoring images).

[0004] Data processing is a crucial element for realizing the value of the Industrial Internet of Things (IIoT), and its efficiency, accuracy, and real-time performance directly determine the level of intelligence in industrial production. However, existing IIoT data processing methods suffer from the following problems: (i) Lack of traffic prediction during data collection leads to transmission congestion and data loss. In existing methods, data acquisition nodes collect data at a fixed frequency without adjusting the acquisition strategy based on the current network traffic conditions. This can easily lead to transmission congestion during traffic peaks in industrial scenarios, resulting in the loss of critical data. For example, in a welding workshop of an automotive parts processing plant, 500 temperature sensors are deployed to monitor the welding gun temperature in real time (critical data; loss of this data would lead to substandard welding quality). Existing methods set the sensors to collect data every 100ms and transmit it immediately. When 10 welding machines are started simultaneously in the workshop, the sensor data collection volume surges, and network traffic instantly reaches its peak (soaring from the normal 100Mbps to 800Mbps), causing transmission link congestion and resulting in the loss of nearly 30% of the temperature data. Subsequent tracking of temperature anomalies during the welding process becomes impossible, ultimately resulting in over 100 defective products.

[0005] (ii) Data preprocessing did not take into account traffic priority, resulting in low processing efficiency. In the Industrial Internet of Things (IIoT), different types of data have significantly different priorities (e.g., equipment fault data has a higher priority than environmental monitoring data). However, existing preprocessing methods do not prioritize data based on flow characteristics and apply the same preprocessing workflow (e.g., denoising and normalization) to all data. This leads to delays in the preprocessing of critical data, while non-critical data consumes a large amount of processing resources, resulting in low overall processing efficiency. For example, a chemical plant deployed an IIoT system that simultaneously collects equipment fault signals (high priority, requiring millisecond-level preprocessing) and workshop temperature and humidity data (low priority, allowing for second-level preprocessing). Existing methods use a fixed workflow of "denoising-normalization-format conversion" for both types of data, without optimizing the processing order based on the proportion of the two types of data in the flow (fault signals account for 5%, temperature and humidity data account for 95%). This results in a fault signal preprocessing delay of 500ms, exceeding the preset 300ms threshold, failing to trigger timely fault warnings, and ultimately causing a reactor to shut down for maintenance, resulting in hundreds of thousands of yuan in economic losses. Summary of the Invention

[0006] To address the technical problems mentioned above, such as the lack of traffic prediction in data collection and the failure to incorporate traffic priority into data preprocessing, this invention provides the following technical solution: A method for processing data for the Industrial Internet of Things (IIoT) includes the following specific steps: S1, Traffic Sensing and Prediction: Collects multi-dimensional traffic data from the Industrial Internet of Things, performs noise reduction, feature extraction, and adaptive calibration, predicts future traffic change trends, and outputs traffic prediction results; S2, Data preprocessing based on traffic priority: Based on the traffic prediction results output by S1, data priorities are divided and traffic load is matched. At the same time, the preprocessing order is planned, differentiated preprocessing is performed, and a standardized preprocessed dataset is output. S3. Dynamically Adaptive Layered Storage: Based on the traffic prediction results output by S1 and the standardized preprocessed dataset output by S2, the storage medium and structure are matched, and dynamic layered storage is performed and verified and updated. A verification report and the updated layered storage dataset are output. S4. Feedback Optimization and Strategy Adjustment: Based on the traffic prediction results output by S1, the standardized preprocessed dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, evaluate the data processing effect, adjust relevant strategies and implement iterations to achieve closed-loop management of the entire process.

[0007] As a preferred embodiment of the method for processing industrial IoT data according to the present invention, the specific steps of S1 are as follows: S11, Multi-dimensional traffic data acquisition: Deploy traffic acquisition modules on the transmission links, acquisition nodes, and gateway devices of the Industrial Internet of Things to collect multi-dimensional traffic data, and organize the collected multi-dimensional traffic data into a standardized traffic dataset for output; S12, Traffic feature extraction and noise reduction: Based on the standardized traffic dataset output in S11, the traffic data is denoised using a wavelet transform algorithm; then, the core traffic features are extracted and output. S13, Adaptive calibration of abnormal flow: Based on the core flow characteristics output in S12, combined with the flow benchmark threshold preset in the industrial scenario, the flow characteristics are adaptively calibrated, and the calibrated core flow characteristics are output. S14, Traffic Change Trend Prediction: Based on the calibrated core traffic features output in S13, a traffic prediction model is constructed using an LSTM neural network to predict future traffic change trends after inputting historical traffic feature data and current traffic feature data, and output the traffic prediction result.

[0008] As a preferred embodiment of the method for processing industrial IoT data according to the present invention, the specific steps of S2 are as follows: S21, Data priority division based on flow characteristics: Based on the flow prediction results output by S1 and combined with industrial production needs, data priorities are divided; at the same time, a unique priority identifier is assigned to data of different priorities, and the data priority division results are output. S22, Priority-Traffic Load Dynamic Matching: Based on the data priority division result output by S21, reuse the traffic prediction result output by S1, dynamically match the preprocessed load threshold of each priority data, and output the matched priority-load correspondence. S23, Priority-oriented preprocessing sequence planning: Based on the priority-load correspondence output in S22 and combined with the traffic prediction results output in S1, the preprocessing sequence is planned, and the preprocessing sequence planning scheme and resource allocation ratio are output. S24, Differentiated preprocessing execution: Based on the preprocessing sequence planning scheme and resource allocation ratio output in S23, differentiated preprocessing is performed on the raw data collected by the Industrial Internet of Things; and after the preprocessing is completed, a standardized preprocessed dataset is output.

[0009] As a preferred embodiment of the method for processing industrial IoT data according to the present invention, the specific steps of S3 are as follows: S31, Storage medium and storage structure matching: Based on the traffic prediction result output by S1 and the standardized preprocessed dataset output by S2, match the corresponding storage medium and storage structure; at the same time, according to the traffic prediction result, reserve a corresponding proportion of storage space and output the storage matching scheme; S32, Dynamic traffic adaptation storage execution: Based on the storage matching scheme output in S31, combined with the real-time traffic status, dynamic hierarchical storage is executed; and after storage is completed, storage logs are generated and the storage logs and hierarchical storage datasets are output. S33, Data verification and update: Based on the storage log and hierarchical storage dataset output in S32, the stored data is verified and dynamically updated; and after the verification and update are completed, a verification report and the updated hierarchical storage dataset are output.

[0010] As a preferred embodiment of the method for processing industrial IoT data according to the present invention, the specific steps of S4 are as follows: S41, Multi-dimensional processing effect evaluation: Based on the traffic prediction results output by S1, the standardized preprocessed dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, a multi-dimensional processing effect evaluation index system is constructed; at the same time, the actual value of each index is calculated and compared with the preset target value to identify abnormal indicators that do not meet the target, and a processing effect evaluation report is output. S42, Adjustment of traffic perception and processing strategy: Based on the processing effect evaluation report output in S41, the traffic perception and data processing strategy is dynamically adjusted, and the adjusted strategy parameters are output. S43, Strategy Implementation and Closed-Loop Iteration: The adjusted strategy parameters output in S42 are synchronously updated to S1, S2, and S3 to replace the original parameters and implement the strategy. At the same time, iterative monitoring is started to continuously collect and process data to continuously optimize the strategy.

[0011] A data processing system for the Industrial Internet of Things (IIoT) includes: The traffic sensing and prediction module collects multi-dimensional traffic data from the Industrial Internet of Things. After noise reduction, feature extraction, and adaptive calibration, it predicts future traffic change trends and outputs the traffic prediction results. The data preprocessing module based on traffic priority divides data priorities and matches traffic loads based on the traffic prediction results output by the traffic perception and prediction module. At the same time, it plans the preprocessing order, performs differentiated preprocessing, and outputs a standardized preprocessed dataset. The traffic-dynamically adapted hierarchical storage module matches the storage medium and structure based on the traffic prediction results output by the traffic perception and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module. At the same time, it performs dynamic hierarchical storage and verification updates, and outputs a verification report and the updated hierarchical storage dataset. The feedback optimization and strategy control module evaluates the data processing effect based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. At the same time, it adjusts relevant strategies and implements iteratively to achieve closed-loop management of the entire process.

[0012] As a preferred embodiment of the industrial IoT data processing system described in this invention, the traffic sensing and prediction module includes: The multi-dimensional traffic data acquisition unit deploys traffic acquisition modules on the transmission links, acquisition nodes, and gateway devices of the Industrial Internet of Things to collect multi-dimensional traffic data and organize the collected multi-dimensional traffic data into a standardized traffic dataset for output. The traffic feature extraction and denoising unit, based on the standardized traffic dataset output by the multi-dimensional traffic data acquisition unit, uses a wavelet transform algorithm to denoise the traffic data; then it extracts the core traffic features and outputs them. The flow anomaly adaptive calibration unit, based on the core flow features output by the flow feature extraction and noise reduction unit, combined with the flow benchmark threshold preset in the industrial scenario, adaptively calibrates the flow features and outputs the calibrated core flow features. The traffic change trend prediction unit, based on the calibrated core traffic features output by the traffic anomaly adaptive calibration unit, combines an LSTM neural network to construct a traffic prediction model. After inputting historical traffic feature data and current traffic feature data, it predicts future traffic change trends and outputs traffic prediction results.

[0013] As a preferred embodiment of the industrial IoT data processing system described in this invention, the data preprocessing module based on traffic priority includes: The data priority division unit based on flow characteristics divides data priorities based on the flow prediction results output by the flow perception and prediction module and in combination with industrial production needs; at the same time, it assigns a unique priority identifier to data of different priorities and outputs the data priority division results. The priority-load dynamic matching unit, based on the data priority division result output by the data priority division unit based on traffic characteristics, reuses the traffic prediction result output by the traffic perception and prediction module, dynamically matches the preprocessed load threshold of each priority data, and outputs the matched priority-load correspondence. The priority-oriented preprocessing sequence planning unit plans the preprocessing sequence based on the priority-load correspondence output by the priority-load dynamic matching unit and the traffic prediction result output by the traffic perception and prediction module, and outputs the preprocessing sequence planning scheme and resource allocation ratio. The differentiated preprocessing execution unit performs differentiated preprocessing on the raw data collected by the Industrial Internet of Things based on the preprocessing sequence planning scheme and resource allocation ratio output by the priority-oriented preprocessing sequence planning unit; and outputs a standardized preprocessed dataset after the preprocessing is completed.

[0014] As a preferred embodiment of the industrial IoT data processing system described in this invention, the hierarchical storage module with dynamic traffic adaptation includes: The storage medium and storage structure matching unit matches the corresponding storage medium and storage structure based on the traffic prediction result output by the traffic sensing and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module; at the same time, it reserves a corresponding proportion of storage space according to the traffic prediction result and outputs a storage matching scheme. The traffic-dynamically adapted storage execution unit performs dynamic tiered storage based on the storage matching scheme output by the storage medium and storage structure matching unit, combined with real-time traffic status; and after storage is completed, it generates storage logs and outputs the storage logs and tiered storage dataset. The storage data verification and update unit verifies and dynamically updates the stored data based on the storage logs and hierarchical storage datasets output by the traffic-dynamically adapted storage execution unit; and after verification and update are completed, it outputs a verification report and the updated hierarchical storage dataset.

[0015] As a preferred embodiment of the industrial IoT data processing system described in this invention, the feedback optimization and strategy control module includes: The multi-dimensional processing effect evaluation unit constructs a multi-dimensional processing effect evaluation index system based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. At the same time, it calculates the actual value of each index and compares it with the preset target value to identify abnormal indicators that have not met the target, and outputs a processing effect evaluation report. The traffic perception and processing strategy adjustment unit dynamically adjusts the traffic perception and data processing strategy based on the processing effect evaluation report output by the multi-dimensional processing effect evaluation unit, and outputs the adjusted strategy parameters. The strategy implementation and closed-loop iteration unit synchronously updates the adjusted strategy parameters output by the traffic perception and processing strategy adjustment unit to the traffic perception and prediction module, the traffic priority-based data preprocessing module, and the traffic dynamic adaptation hierarchical storage module to replace the original parameters and realize strategy implementation; at the same time, iterative monitoring is started to continuously collect and process data, thereby continuously optimizing the strategy.

[0016] Compared with existing technologies: 1. By adjusting the data collection strategy through traffic perception and prediction, it is possible not only to adapt to traffic fluctuations and optimize the collection frequency to avoid transmission congestion, but also to achieve the advantages of stable transmission of key data and effectively reduce data loss. 2. By combining traffic prediction results to prioritize data, we can not only plan differentiated preprocessing processes and allocate resources reasonably, but also improve preprocessing efficiency and ensure timely processing of critical data. 3. By dynamically adapting storage media and structure based on traffic changes, it is possible not only to optimize storage resource allocation and avoid resource waste, but also to achieve the advantages of reasonable and efficient storage and reduced data read latency; 4. By linking data processing effect feedback with traffic perception, it is possible not only to dynamically adjust processing strategies and break the limitations of one-way processes, but also to achieve the advantages of flexible strategy adaptation and continuous optimization of processing effects. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall system framework of the present invention; Figure 2 This is a schematic diagram of the traffic sensing and prediction module framework of the present invention; Figure 3 This is a schematic diagram of the data preprocessing module framework based on traffic priority of the present invention; Figure 4 This is a schematic diagram of the hierarchical storage module framework for dynamic traffic adaptation in this invention; Figure 5 This is a schematic diagram of the feedback optimization and strategy control module framework of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0019] This invention provides a method for processing data in the Industrial Internet of Things (IIoT), comprising the following specific steps: S1, Traffic Sensing and Prediction: Collects multi-dimensional traffic data from the Industrial Internet of Things, performs noise reduction, feature extraction, and adaptive calibration, predicts future traffic change trends, and outputs traffic prediction results; The specific steps of S1 are as follows: S11, Multi-dimensional Traffic Data Acquisition: Deploy traffic acquisition modules on transmission links, acquisition nodes, and gateway devices in the Industrial Internet of Things to collect multi-dimensional traffic data, including: link bandwidth utilization, data transmission rate, data packet loss rate, data latency, and the traffic proportion of each type of data (equipment operation data, environmental data, and fault data). The acquisition frequency is dynamically adjusted according to the industrial scenario (default 10ms / time), and the collected multi-dimensional traffic data is organized into a standardized traffic dataset for output. S12, Traffic Feature Extraction and Noise Reduction: Based on the standardized traffic dataset output in S11, wavelet transform algorithm is used to denoise the traffic data, removing abnormal traffic data caused by electromagnetic interference and equipment vibration in industrial scenarios (such as anomalies where the instantaneous bandwidth occupancy rate suddenly rises to 100% but there is no corresponding data transmission); then, core traffic features are extracted, including: traffic peak features (historical traffic peak, the difference between the current traffic and the peak), traffic fluctuation features (the change in traffic per unit time), and traffic type features (the proportion of traffic for each priority data), and output; S13, Adaptive calibration for abnormal traffic flow: Based on the core traffic flow characteristics output in S12, combined with the traffic benchmark threshold preset in the industrial scenario (associated with historical normal traffic flow data collected), the traffic flow characteristics are adaptively calibrated to identify and correct feature deviations caused by errors in the acquisition module and link fluctuations, and the calibrated core traffic flow characteristics are output. S14, Traffic Change Trend Prediction: Based on the calibrated core traffic characteristics output in S13, a traffic prediction model is constructed using an LSTM neural network. After inputting historical traffic characteristic data (traffic characteristics in the past hour) and current traffic characteristic data, the model predicts the traffic change trend in the next 5-10 minutes, including whether a traffic peak will occur, the expected value of the traffic peak, and the change in the traffic proportion of each type of data. The model then outputs the traffic prediction result.

[0020] S2, Data preprocessing based on traffic priority: Based on the traffic prediction results output by S1, data priorities are divided and traffic load is matched. At the same time, the preprocessing order is planned, differentiated preprocessing is performed, and a standardized preprocessed dataset is output. The specific steps of S2 are as follows: S21, Data Priority Classification Based on Flow Characteristics: Based on the flow prediction results output in S1, and combined with industrial production needs, data priorities are classified as follows: 1) High-priority data: Equipment fault data, key operating parameter data (such as machine tool speed, pipeline pressure), corresponding to a flow percentage prediction value ≥ 5%, and should be transmitted and processed first during flow peaks; 2) Medium-priority data: Production process data, energy consumption data, corresponding to a flow percentage prediction value of 2%-5%, and can be appropriately compressed during flow peaks; 3) Low-priority data: Environmental monitoring data, equipment inspection logs, corresponding to a flow percentage prediction value < 2%, and can be delayed during flow peaks; At the same time, a unique priority identifier is assigned to different priority data, and the data priority classification results are output; S22, Priority-Traffic Load Dynamic Matching: Based on the data priority division result output by S21, the traffic prediction result output by S1 is reused to dynamically match the preprocessing load threshold of each priority data, so as to avoid high priority data being squeezed out of resources due to excessive traffic load, and output the matched priority-load correspondence. S23, Priority-oriented preprocessing sequence planning: Based on the priority-load correspondence output in S22 and the traffic prediction result output in S1, the preprocessing sequence is planned: If a traffic peak is predicted (warning signal is "yes"), preprocessing is started in the order of "high priority → medium priority → low priority"; if no traffic peak is predicted (warning signal is "no"), a "parallel preprocessing" mode is adopted, and high, medium and low priority data are preprocessed simultaneously, but 60% of the preprocessing resources are allocated to high priority data, 30% to medium priority data, and 10% to low priority data, and the preprocessing sequence planning scheme and resource allocation ratio are output; S24, Differentiated Preprocessing Execution: Based on the preprocessing sequence planning scheme and resource allocation ratio output in S23, differentiated preprocessing is performed on the raw data collected by the Industrial Internet of Things: 1) High-priority data: A fast denoising algorithm (such as adaptive threshold denoising) is used to skip unnecessary format conversion steps, retaining only the core data fields, and the preprocessing delay is controlled within 100ms; 2) Medium-priority data: Conventional denoising and normalization processing is used to retain complete data fields, and the preprocessing delay is controlled within 300ms; 3) Low-priority data: Conventional preprocessing process is used, which can appropriately compress the data volume, and the preprocessing delay is controlled within 1s; and after the preprocessing is completed, a standardized preprocessed dataset is output.

[0021] S3. Dynamically Adaptive Layered Storage: Based on the traffic prediction results output by S1 and the standardized preprocessed dataset output by S2, the storage medium and structure are matched, and dynamic layered storage is performed and verified and updated. A verification report and the updated layered storage dataset are output. The specific steps of S3 are as follows: S31, Storage Media and Storage Structure Matching: Based on the traffic prediction results output by S1 and the standardized preprocessed dataset output by S2, match the corresponding storage media and storage structure: 1) High-priority data: Store in solid-state drives (SSDs) using a hash index storage structure to ensure fast data reading (read latency ≤ 100ms), adapting to the high-frequency reading needs during traffic slumps; 2) Medium-priority data: Store in a hybrid storage array (SSD + HDD) using a B+ tree index storage structure, balancing storage cost and reading efficiency (read latency ≤ 300ms); 3) Low-priority data: Store in HDDs using a sequential storage structure to reduce storage cost and adapt to the large-volume write needs during traffic peaks; At the same time, according to the traffic prediction results, reserve a corresponding proportion of storage space (20% storage space during traffic peaks and 10% storage space during traffic slumps), and output the storage matching scheme; S32, Dynamic Traffic-Adaptive Storage Execution: Based on the storage matching scheme output in S31, and combined with real-time traffic status, dynamic tiered storage is executed: 1) Peak traffic periods (e.g., peak production periods): Prioritize the writing of high and medium priority data, while low priority data can be cached first and then written to avoid storage congestion; simultaneously, a data compression algorithm is activated to compress and store low priority data, reducing storage usage; 2) Low traffic periods (e.g., equipment inspection periods): Prioritize the reading of high and medium priority data, while filtering stored low priority data, deleting redundant data (e.g., duplicate environmental monitoring data), and releasing storage resources; 3) Stable traffic periods: Execute according to the conventional storage scheme to ensure normal writing and reading of various types of data; and after storage is completed, generate storage logs (including the storage location, storage time, and data size of each data), and output the storage logs and tiered storage dataset; S33, Data Verification and Update: Based on the storage logs and tiered storage datasets output in S32, the stored data is verified and dynamically updated: 1) Data Verification: The CRC check algorithm is used to verify the high and medium priority data in real time. If data loss or corruption is found, the preprocessed dataset is immediately called for replenishment. Low priority data is verified periodically (once every 24 hours) to reduce the consumption of verification resources. 2) Data Update: The storage structure is dynamically updated based on real-time traffic changes (reusing traffic prediction results). For example, when the normal traffic value increases, the storage ratio of solid-state drives is increased. Data exceeding the storage period (1 year for high priority data, 6 months for medium priority data, and 3 months for low priority data) is automatically migrated to archive storage devices to release active storage resources. After verification and update are completed, a verification report and the updated tiered storage dataset are output.

[0022] S4. Feedback Optimization and Strategy Adjustment: Based on the traffic prediction results output by S1, the standardized preprocessed dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, evaluate the data processing effect, adjust relevant strategies and implement iterations to achieve closed-loop management of the entire process. The specific steps of S4 are as follows: S41, Multi-dimensional Processing Effect Evaluation: Based on the traffic prediction results output by S1, the standardized preprocessing dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, a multi-dimensional processing effect evaluation index system is constructed. The evaluation indexes include: data transmission success rate (target ≥ 99.5%), preprocessing latency (high priority ≤ 100ms), data read latency (high priority ≤ 100ms), storage resource utilization rate (target ≥ 80%), and critical data loss rate (target ≤ 0.5%). At the same time, the actual values ​​of each index are calculated and compared with the preset target values ​​to identify abnormal indicators that have not met the standards, and a processing effect evaluation report is output. S42, Traffic Awareness and Processing Strategy Adjustment: Based on the processing effect evaluation report output in S41, dynamically adjust the traffic awareness and data processing strategy: 1) If the data transmission success rate does not meet the standard (e.g., due to traffic peak congestion), adjust the traffic prediction model parameters to improve prediction accuracy, and optimize the preprocessing order to prioritize high-priority data transmission; 2) If the preprocessing latency does not meet the standard, adjust the preprocessing algorithm to allocate more processing resources for high-priority data; 3) If the storage resource utilization rate does not meet the standard, adjust the storage matching scheme to optimize the storage cycle and compression ratio of low-priority data; 4) If the critical data loss rate does not meet the standard, adjust the traffic collection frequency, reduce the collection frequency of low-priority data before traffic peaks, reserve more link resources, and output the adjusted strategy parameters; S43, Strategy Implementation and Closed-Loop Iteration: The adjusted strategy parameters output in S42 are synchronously updated to S1, S2, and S3 to replace the original parameters, thus implementing the strategy. At the same time, iterative monitoring is started to continuously collect and process data, thereby continuously optimizing the strategy and forming a closed-loop control of "traffic perception - data processing - effect evaluation - strategy optimization". This ensures that the data processing effect continuously meets the needs of industrial production, and finally outputs an optimized industrial IoT data processing closed-loop system based on traffic perception.

[0023] Please refer to a data processing system for the Industrial Internet of Things (IIoT). Figure 1 ,include: The traffic sensing and prediction module collects multi-dimensional traffic data from the Industrial Internet of Things. After noise reduction, feature extraction, and adaptive calibration, it predicts future traffic change trends and outputs the traffic prediction results. The data preprocessing module based on traffic priority divides data priorities and matches traffic loads based on the traffic prediction results output by the traffic perception and prediction module. At the same time, it plans the preprocessing order, performs differentiated preprocessing, and outputs a standardized preprocessed dataset. The traffic-dynamically adapted hierarchical storage module matches the storage medium and structure based on the traffic prediction results output by the traffic perception and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module. At the same time, it performs dynamic hierarchical storage and verification updates, and outputs a verification report and the updated hierarchical storage dataset. The feedback optimization and strategy control module evaluates the data processing effect based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. At the same time, it adjusts relevant strategies and implements iteratively to achieve closed-loop management of the entire process.

[0024] Please see Figure 2 The traffic sensing and prediction module includes: The multi-dimensional traffic data acquisition unit deploys traffic acquisition modules on the transmission links, acquisition nodes, and gateway devices of the Industrial Internet of Things to collect multi-dimensional traffic data, including: link bandwidth utilization, data transmission rate, data packet loss rate, data latency, and the traffic proportion of each type of data (equipment operation data, environmental data, and fault data). The acquisition frequency is dynamically adjusted according to the industrial scenario (default 10ms / time), and the collected multi-dimensional traffic data is organized into a standardized traffic dataset for output. The traffic feature extraction and denoising unit, based on the standardized traffic dataset output by the multi-dimensional traffic data acquisition unit, uses a wavelet transform algorithm to denoise the traffic data, removing abnormal traffic data caused by electromagnetic interference and equipment vibration in industrial scenarios (such as anomalies where the instantaneous bandwidth occupancy rate suddenly rises to 100% but there is no corresponding data transmission); then it extracts core traffic features, including: traffic peak features (historical traffic peak, the difference between the current traffic and the peak), traffic fluctuation features (the change in traffic per unit time), and traffic type features (the proportion of traffic for each priority data), and outputs them; The traffic anomaly adaptive calibration unit, based on the core traffic features output by the traffic feature extraction and noise reduction unit, combined with the traffic benchmark threshold preset in the industrial scenario (associated with historical normal traffic data collected), adaptively calibrates the traffic features, identifies and corrects feature deviations caused by acquisition module errors and link fluctuations, and outputs the calibrated core traffic features. The traffic change trend prediction unit, based on the calibrated core traffic features output by the traffic anomaly adaptive calibration unit, combines an LSTM neural network to construct a traffic prediction model. After inputting historical traffic feature data (traffic features in the past hour) and current traffic feature data, it predicts the traffic change trend in the next 5-10 minutes, including: whether a traffic peak will occur, the expected value of the traffic peak, the change in the traffic proportion of each type of data, and outputs the traffic prediction result.

[0025] Please see Figure 3 The data preprocessing module based on traffic priority includes: The data priority division unit based on flow characteristics, based on the flow prediction results output by the flow perception and prediction module and combined with industrial production needs, divides data priorities as follows: 1) High-priority data: equipment fault data, key operating parameter data (such as machine tool speed, pipeline pressure), corresponding to a flow percentage prediction value ≥ 5%, and should be transmitted and processed first during flow peaks; 2) Medium-priority data: production process data, energy consumption data, corresponding to a flow percentage prediction value of 2%-5%, and can be appropriately compressed during flow peaks; 3) Low-priority data: environmental monitoring data, equipment inspection logs, corresponding to a flow percentage prediction value < 2%, and can be delayed during flow peaks; Simultaneously, a unique priority identifier is assigned to different priority data, and the data priority division results are output. The priority-load dynamic matching unit, based on the data priority division result output by the data priority division unit based on traffic characteristics, reuses the traffic prediction result output by the traffic perception and prediction module, dynamically matches the preprocessing load threshold of each priority data, avoids high priority data from being squeezed out of resources due to excessive traffic load, and outputs the matched priority-load correspondence. The priority-oriented preprocessing sequence planning unit, based on the priority-load correspondence output by the priority-load dynamic matching unit and the traffic prediction result output by the traffic perception and prediction module, plans the preprocessing sequence: if a traffic peak is predicted (warning signal is "yes"), preprocessing is started in the order of "high priority → medium priority → low priority"; if no traffic peak is predicted (warning signal is "no"), a "parallel preprocessing" mode is adopted, preprocessing high, medium and low priority data simultaneously, but 60% of the preprocessing resources are allocated to high priority data, 30% to medium priority data, and 10% to low priority data, and the preprocessing sequence planning scheme and resource allocation ratio are output. The differentiated preprocessing execution unit performs differentiated preprocessing on the raw data collected by the Industrial Internet of Things based on the preprocessing sequence planning scheme and resource allocation ratio output by the priority-oriented preprocessing sequence planning unit: 1) High-priority data: adopts a fast denoising algorithm (such as adaptive threshold denoising), skips unnecessary format conversion steps, retains only the core data fields, and controls the preprocessing delay within 100ms; 2) Medium-priority data: adopts conventional denoising and normalization processing, retains complete data fields, and controls the preprocessing delay within 300ms; 3) Low-priority data: adopts a conventional preprocessing process, which can appropriately compress the data volume and control the preprocessing delay within 1s; and after the preprocessing is completed, a standardized preprocessed dataset is output.

[0026] Please see Figure 4 The hierarchical storage module with dynamic traffic adaptation includes: The storage medium and storage structure matching unit matches the corresponding storage medium and storage structure based on the traffic prediction results output by the traffic perception and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module: 1) High-priority data: stored in solid-state drives (SSDs) using a hash index storage structure to ensure fast data reading (read latency ≤ 100ms), adapting to the high-frequency reading needs during traffic slack periods; 2) Medium-priority data: stored in a hybrid storage array (SSD + HDD) using a B+ tree index storage structure, balancing storage cost and read efficiency (read latency ≤ 300ms); 3) Low-priority data: stored in hard disk drives (HDDs) using a sequential storage structure to reduce storage costs and adapt to the large-volume write needs during traffic peaks; Simultaneously, based on the traffic prediction results, a corresponding proportion of storage space is reserved (20% storage space is reserved during traffic peaks, and 10% storage space is reserved during traffic slack periods), and a storage matching scheme is output; The dynamic traffic-adaptive storage execution unit, based on the storage matching scheme output by the storage medium and storage structure matching unit, and combined with real-time traffic status, executes dynamic tiered storage: 1) During peak traffic periods (e.g., peak production periods): Prioritize the writing of high and medium priority data, while low priority data can be cached first and then written to avoid storage congestion; simultaneously, a data compression algorithm is activated to compress and store low priority data, reducing storage usage; 2) During off-peak traffic periods (e.g., during equipment inspection periods): Prioritize the reading of high and medium priority data, while filtering stored low priority data, deleting redundant data (e.g., duplicate environmental monitoring data), and releasing storage resources; 3) During periods of stable traffic: Execute according to the conventional storage scheme to ensure normal writing and reading of various types of data; and after storage is completed, generate a storage log (including the storage location, storage time, and data size of each data), and output the storage log and tiered storage dataset; The storage data verification and update unit verifies and dynamically updates the stored data based on the storage logs and tiered storage datasets output by the traffic-dynamically adapted storage execution unit: 1) Data verification: Using the CRC check algorithm, high- and medium-priority data are verified in real time. If data loss or corruption is found, the preprocessed dataset is immediately called for replenishment. Low-priority data is verified periodically (once every 24 hours) to reduce verification resource consumption; 2) Data update: Combining real-time traffic changes (reusing traffic prediction results), the storage structure is dynamically updated. For example, when the normal traffic value increases, the storage ratio of solid-state drives is increased. Data exceeding the storage period (1 year for high-priority data, 6 months for medium-priority data, and 3 months for low-priority data) is automatically migrated to archive storage devices to release active storage resources. After verification and update are completed, a verification report and the updated tiered storage dataset are output.

[0027] Please see Figure 5 The feedback optimization and strategy control module includes: The multi-dimensional processing effect evaluation unit constructs a multi-dimensional processing effect evaluation index system based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. The evaluation indicators include: data transmission success rate (target ≥ 99.5%), preprocessing latency (high priority ≤ 100ms), data read latency (high priority ≤ 100ms), storage resource utilization rate (target ≥ 80%), and critical data loss rate (target ≤ 0.5%). Simultaneously, it calculates the actual values ​​of each indicator and compares them with preset target values ​​to identify non-compliant abnormal indicators and outputs a processing effect evaluation report. The traffic perception and processing strategy adjustment unit dynamically adjusts the traffic perception and data processing strategy based on the processing effect evaluation report output by the multi-dimensional processing effect evaluation unit: 1) If the data transmission success rate does not meet the standard (e.g., due to traffic peak congestion), adjust the traffic prediction model parameters to improve prediction accuracy and optimize the preprocessing order to prioritize high-priority data transmission; 2) If the preprocessing latency does not meet the standard, adjust the preprocessing algorithm to allocate more processing resources for high-priority data; 3) If the storage resource utilization rate does not meet the standard, adjust the storage matching scheme to optimize the storage cycle and compression ratio of low-priority data; 4) If the critical data loss rate does not meet the standard, adjust the traffic collection frequency, reduce the collection frequency of low-priority data before traffic peaks, reserve more link resources, and output the adjusted strategy parameters. The strategy implementation and closed-loop iteration unit synchronously updates the adjusted strategy parameters output by the traffic perception and processing strategy adjustment unit to the traffic perception and prediction module, the traffic priority-based data preprocessing module, and the traffic dynamic adaptation hierarchical storage module to replace the original parameters and realize strategy implementation. At the same time, iterative monitoring is started to continuously collect and process data. Thus, the strategy is continuously optimized to form a closed-loop control of "traffic perception - data processing - effect evaluation - strategy optimization", ensuring that the data processing effect continuously meets the needs of industrial production, and finally outputting an optimized industrial IoT data processing closed-loop system based on traffic perception.

[0028] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for processing data in the Industrial Internet of Things (IIoT), characterized in that, The specific steps are as follows: S1, Traffic Sensing and Prediction: Collects multi-dimensional traffic data from the Industrial Internet of Things, performs noise reduction, feature extraction, and adaptive calibration, predicts future traffic change trends, and outputs traffic prediction results; S2, Data preprocessing based on traffic priority: Based on the traffic prediction results output by S1, data priorities are divided and traffic load is matched. At the same time, the preprocessing order is planned, differentiated preprocessing is performed, and a standardized preprocessed dataset is output. S3. Dynamically Adaptive Layered Storage: Based on the traffic prediction results output by S1 and the standardized preprocessed dataset output by S2, the storage medium and structure are matched, and dynamic layered storage is performed and verified and updated. A verification report and the updated layered storage dataset are output. S4. Feedback Optimization and Strategy Adjustment: Based on the traffic prediction results output by S1, the standardized preprocessed dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, evaluate the data processing effect, adjust relevant strategies and implement iterations to achieve closed-loop management of the entire process.

2. The method for processing industrial IoT data according to claim 1, characterized in that, The specific steps of S1 are as follows: S11, Multi-dimensional traffic data acquisition: Deploy traffic acquisition modules on the transmission links, acquisition nodes, and gateway devices of the Industrial Internet of Things to collect multi-dimensional traffic data, and organize the collected multi-dimensional traffic data into a standardized traffic dataset for output; S12, Traffic feature extraction and noise reduction: Based on the standardized traffic dataset output in S11, wavelet transform algorithm is used to perform noise reduction on the traffic data; Then, the core traffic features are extracted and output; S13, Adaptive calibration of abnormal flow: Based on the core flow characteristics output in S12, combined with the flow benchmark threshold preset in the industrial scenario, the flow characteristics are adaptively calibrated, and the calibrated core flow characteristics are output. S14, Traffic Change Trend Prediction: Based on the calibrated core traffic features output in S13, a traffic prediction model is constructed using an LSTM neural network to predict future traffic change trends after inputting historical traffic feature data and current traffic feature data, and output the traffic prediction result.

3. The method for processing industrial IoT data according to claim 1, characterized in that, The specific steps of S2 are as follows: S21, Data priority division based on flow characteristics: Based on the flow prediction results output by S1 and combined with industrial production needs, data priorities are divided; at the same time, a unique priority identifier is assigned to data of different priorities, and the data priority division results are output. S22, Priority-Traffic Load Dynamic Matching: Based on the data priority division result output by S21, reuse the traffic prediction result output by S1, dynamically match the preprocessed load threshold of each priority data, and output the matched priority-load correspondence. S23, Priority-oriented preprocessing sequence planning: Based on the priority-load correspondence output in S22 and combined with the traffic prediction results output in S1, the preprocessing sequence is planned, and the preprocessing sequence planning scheme and resource allocation ratio are output. S24, Differentiated preprocessing execution: Based on the preprocessing sequence planning scheme and resource allocation ratio output in S23, differentiated preprocessing is performed on the raw data collected by the Industrial Internet of Things. Furthermore, after preprocessing is completed, a standardized preprocessed dataset is output.

4. The method for processing industrial IoT data according to claim 1, characterized in that, The specific steps of S3 are as follows: S31, Storage medium and storage structure matching: Based on the traffic prediction result output by S1 and the standardized preprocessed dataset output by S2, match the corresponding storage medium and storage structure; at the same time, according to the traffic prediction result, reserve a corresponding proportion of storage space and output the storage matching scheme; S32, Dynamic traffic adaptation storage execution: Based on the storage matching scheme output in S31, combined with the real-time traffic status, dynamic hierarchical storage is executed; and after storage is completed, storage logs are generated and the storage logs and hierarchical storage datasets are output. S33, Data verification and update: Based on the storage log and hierarchical storage dataset output in S32, the stored data is verified and dynamically updated; and after the verification and update are completed, a verification report and the updated hierarchical storage dataset are output.

5. The method for processing industrial IoT data according to claim 1, characterized in that, The specific steps of S4 are as follows: S41, Multi-dimensional processing effect evaluation: Based on the traffic prediction results output by S1, the standardized preprocessed dataset output by S2, and the verification report and updated hierarchical storage dataset output by S3, a multi-dimensional processing effect evaluation index system is constructed; at the same time, the actual value of each index is calculated and compared with the preset target value to identify abnormal indicators that do not meet the target, and a processing effect evaluation report is output. S42, Adjustment of traffic perception and processing strategy: Based on the processing effect evaluation report output in S41, the traffic perception and data processing strategy is dynamically adjusted, and the adjusted strategy parameters are output. S43, Strategy Implementation and Closed-Loop Iteration: The adjusted strategy parameters output in S42 are synchronously updated to S1, S2, and S3 to replace the original parameters and implement the strategy. At the same time, iterative monitoring is started to continuously collect and process data to continuously optimize the strategy.

6. A data processing system for the Industrial Internet of Things (IIoT), characterized in that, include: The traffic sensing and prediction module collects multi-dimensional traffic data from the Industrial Internet of Things. After noise reduction, feature extraction, and adaptive calibration, it predicts future traffic change trends and outputs traffic prediction results. The data preprocessing module based on traffic priority divides data priorities and matches traffic loads based on the traffic prediction results output by the traffic perception and prediction module. At the same time, it plans the preprocessing order, performs differentiated preprocessing, and outputs a standardized preprocessed dataset. The traffic-dynamically adapted hierarchical storage module matches the storage medium and structure based on the traffic prediction results output by the traffic perception and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module. At the same time, it performs dynamic hierarchical storage and verification updates, and outputs a verification report and the updated hierarchical storage dataset. The feedback optimization and strategy control module evaluates the data processing effect based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. At the same time, it adjusts relevant strategies and implements iteratively to achieve closed-loop management of the entire process.

7. A data processing system for industrial Internet of Things (IoT) according to claim 6, characterized in that, The traffic sensing and prediction module includes: The multi-dimensional traffic data acquisition unit deploys traffic acquisition modules on the transmission links, acquisition nodes, and gateway devices of the Industrial Internet of Things to collect multi-dimensional traffic data and organize the collected multi-dimensional traffic data into a standardized traffic dataset for output. The traffic feature extraction and denoising unit, based on the standardized traffic dataset output by the multi-dimensional traffic data acquisition unit, uses a wavelet transform algorithm to denoise the traffic data; then it extracts the core traffic features and outputs them. The flow anomaly adaptive calibration unit, based on the core flow features output by the flow feature extraction and noise reduction unit, combined with the flow benchmark threshold preset in the industrial scenario, adaptively calibrates the flow features and outputs the calibrated core flow features. The traffic change trend prediction unit, based on the calibrated core traffic features output by the traffic anomaly adaptive calibration unit, combines an LSTM neural network to construct a traffic prediction model. After inputting historical traffic feature data and current traffic feature data, it predicts future traffic change trends and outputs traffic prediction results.

8. A data processing system for industrial Internet of Things (IoT) according to claim 6, characterized in that, The data preprocessing module based on traffic priority includes: The data priority division unit based on flow characteristics divides data priorities based on the flow prediction results output by the flow perception and prediction module and in combination with industrial production needs; at the same time, it assigns a unique priority identifier to data of different priorities and outputs the data priority division results. The priority-load dynamic matching unit, based on the data priority division result output by the data priority division unit based on traffic characteristics, reuses the traffic prediction result output by the traffic perception and prediction module, dynamically matches the preprocessed load threshold of each priority data, and outputs the matched priority-load correspondence. The priority-oriented preprocessing sequence planning unit plans the preprocessing sequence based on the priority-load correspondence output by the priority-load dynamic matching unit and the traffic prediction result output by the traffic perception and prediction module, and outputs the preprocessing sequence planning scheme and resource allocation ratio. The differentiated preprocessing execution unit performs differentiated preprocessing on the raw data collected by the Industrial Internet of Things based on the preprocessing sequence planning scheme and resource allocation ratio output by the priority-oriented preprocessing sequence planning unit; and outputs a standardized preprocessed dataset after the preprocessing is completed.

9. A data processing system for industrial Internet of Things (IoT) according to claim 6, characterized in that, The hierarchical storage module with dynamic traffic adaptation includes: The storage medium and storage structure matching unit matches the corresponding storage medium and storage structure based on the traffic prediction result output by the traffic sensing and prediction module and the standardized preprocessed dataset output by the traffic priority-based data preprocessing module; at the same time, it reserves a corresponding proportion of storage space according to the traffic prediction result and outputs a storage matching scheme. The traffic-dynamically adapted storage execution unit performs dynamic tiered storage based on the storage matching scheme output by the storage medium and storage structure matching unit, combined with real-time traffic status; and after storage is completed, it generates storage logs and outputs the storage logs and tiered storage dataset. The storage data verification and update unit verifies and dynamically updates the stored data based on the storage logs and hierarchical storage datasets output by the traffic-dynamically adapted storage execution unit; and after verification and update are completed, it outputs a verification report and the updated hierarchical storage dataset.

10. A data processing system for industrial Internet of Things (IoT) according to claim 6, characterized in that, The feedback optimization and strategy control module includes: The multi-dimensional processing effect evaluation unit constructs a multi-dimensional processing effect evaluation index system based on the traffic prediction results output by the traffic perception and prediction module, the standardized preprocessed dataset output by the traffic priority-based data preprocessing module, and the verification report and updated hierarchical storage dataset output by the traffic dynamic adaptation hierarchical storage module. At the same time, it calculates the actual value of each index and compares it with the preset target value to identify abnormal indicators that have not met the target, and outputs a processing effect evaluation report. The traffic perception and processing strategy adjustment unit dynamically adjusts the traffic perception and data processing strategy based on the processing effect evaluation report output by the multi-dimensional processing effect evaluation unit, and outputs the adjusted strategy parameters. The strategy implementation and closed-loop iteration unit synchronously updates the adjusted strategy parameters output by the traffic perception and processing strategy adjustment unit to the traffic perception and prediction module, the traffic priority-based data preprocessing module, and the traffic dynamic adaptation hierarchical storage module to replace the original parameters and realize strategy implementation; at the same time, iterative monitoring is started to continuously collect and process data, thereby continuously optimizing the strategy.

Images