An internet of things edge computing dynamic load prediction data processing method and system

Abstract

The application discloses a kind of Internet of Things edge computing dynamic load prediction data processing method and system, belong to edge computing technical field.The application is by collecting edge node operation, network link quality and Internet of Things task characteristic parameter, uses time series prediction model to complete edge node load prediction, integrates relevant parameters and constructs task dynamic decision state space;Based on state space, complete computing task routing distribution, establish heterogeneous resource scheduling multi-objective optimization model and solve to obtain scheduling scheme;According to the scheme, the original data of Internet of Things are mixed compression, combined with compressed data access statistics and space-time correlation attribute constructs pre-fetching rule, completes data pre-fetching and cache management.The application can realize computing task dynamic adaptation and resource rational utilization, reduce data transmission bandwidth occupation, improve edge node cache hit rate and system operation stability, adapt to the real-time processing demand of Internet of Things edge computing scene.

Description

Technical Field

[0001] This invention relates to the field of edge computing technology, and in particular to a method and system for dynamic load prediction data processing in IoT edge computing. Background Technology

[0002] With the large-scale deployment and deepening industrial applications of IoT technology, the deployment scale of IoT terminal nodes is constantly expanding. The massive amounts of real-time data generated in various scenarios place higher demands on the real-time performance of data processing, the efficiency of transmission, and the rationality of computing resource scheduling. Edge computing, as a distributed computing architecture deployed at the network edge close to the data generation end, can effectively compensate for the transmission latency shortcomings of traditional centralized cloud computing architectures in IoT scenarios. It has become a core technology direction in the field of IoT data processing and has been widely applied in multiple scenarios such as industrial IoT, smart device collaboration, and urban sensing networks. Currently, technological research and development and industrial practice in the field of edge computing are continuously advancing. Around core technical aspects such as edge computing task scheduling, heterogeneous computing resource management, IoT data compression and transmission, and distributed cache optimization, a variety of mature technical implementation paths and engineering application solutions have been formed. Related technologies such as time series prediction algorithms, reinforcement learning decision algorithms, wavelet transform signal processing technology, and intelligent cache management technology have also gradually achieved deep integration with edge computing architecture, providing rich technical support for edge data processing in IoT scenarios. At the same time, the industry's demand for optimization of the operational stability, resource utilization efficiency, and real-time data processing of edge computing systems is also continuously increasing, driving the continuous iteration and innovative development of related technologies.

[0003] Currently, there are still multiple technical limitations in the practical application of IoT edge computing data processing. Existing solutions struggle to predict the timing of load changes at edge nodes and cannot effectively integrate multi-dimensional information such as edge node operating status, network link quality, and IoT task attributes to construct a complete and dynamic decision-making framework. This results in task allocation schemes failing to adapt to the dynamic characteristics of the edge computing environment. Furthermore, existing solutions struggle to dynamically adapt tasks and computing nodes during task routing and allocation. For the scheduling of heterogeneous computing resources, they cannot achieve balanced optimization across multiple related optimization objectives, making it difficult to formulate resource scheduling schemes that adapt to task requirements and node states, easily leading to insufficient utilization of computing resources. Furthermore, existing IoT data processing solutions struggle to balance processing efficiency and data fidelity during data compression, and cannot effectively reduce data redundancy through progressive processing steps, increasing network bandwidth consumption during data transmission. Existing distributed cache management solutions also struggle to build adaptable prefetching rules based on data access patterns and inherent correlation attributes, making it impossible to dynamically update and rationally evict cache copies, effectively reduce end-to-end response latency for data access, and fully adapt to the operational requirements of IoT edge computing scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for dynamic load prediction data processing in IoT edge computing.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for dynamic load prediction data processing in IoT edge computing is provided, which includes the following steps:

[0007] S1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Use a long short-term memory network model to perform time-series prediction of edge node load, obtain load prediction results, integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results, and construct the state space for dynamic task decision-making.

[0008] S2. Input the state space into the deep Q-network algorithm to complete the routing allocation of computing tasks, establish a multi-objective optimization model for heterogeneous computing resource scheduling, and use a non-dominated sorting genetic algorithm with elitist strategy to solve the multi-objective optimization model and obtain a heterogeneous computing resource scheduling scheme.

[0009] S3. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by wavelet decomposition, the data after wavelet decomposition is processed by adaptive threshold quantization, and the data after adaptive threshold quantization is processed by dynamic Huffman coding to obtain hybrid compressed data.

[0010] S4. Perform access frequency statistics on the hybrid compressed data to obtain access frequency statistics. Combine the access frequency statistics with the spatiotemporal correlation attributes of the hybrid compressed data to construct data prefetching rules. According to the data prefetching rules and the heterogeneous computing resource scheduling scheme, complete the data prefetching operation and cache copy update operation.

[0011] Furthermore, step S1 includes the following sub-steps:

[0012] S1.1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Edge node operating parameters include edge node CPU utilization parameters and edge node GPU utilization parameters. Network link quality parameters include a network link service quality parameter matrix. IoT task characteristic parameters include IoT task feature vectors.

[0013] S1.2. A long short-term memory network model is used to predict the load of edge nodes in a time series. The long short-term memory network model includes an input layer, a hidden layer and an output layer. The input layer receives the historical load time series data of edge nodes. The hidden layer contains a forget gate, an input gate and an output gate. The hidden layer performs feature extraction and time series fitting on the input time series data. The output layer outputs the load prediction results of edge nodes.

[0014] S1.3. Integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results to construct a state space for dynamic task decision-making.

[0015] Furthermore, step S2 includes the following sub-steps:

[0016] S2.1. Input the state space into the deep Q network algorithm. The deep Q network algorithm includes a state input layer, a convolutional layer, a fully connected layer, and an action output layer. The state input layer receives the state space data, the convolutional layer and the fully connected layer extract features from the state space data, and the action output layer outputs the routing and allocation actions of the computing tasks, completing the allocation of computing tasks between local edge nodes, adjacent edge clusters and cloud centers.

[0017] S2.2. Taking the processing time and power consumption of computing tasks as optimization objectives, a multi-objective optimization model for heterogeneous computing resource scheduling is established, and the constraint condition of the multi-objective optimization model is the upper limit threshold of the load of edge nodes.

[0018] S2.3. A non-dominated sorting genetic algorithm with an elitist strategy is used to solve the multi-objective optimization model to obtain the Pareto front. Based on the Pareto front, a heterogeneous computing resource scheduling scheme is determined.

[0019] Furthermore, step S3 includes the following sub-steps:

[0020] S3.1. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by multi-level wavelet decomposition using a biorthogonal wavelet basis to obtain the low-frequency coefficients and high-frequency coefficients after wavelet decomposition.

[0021] S3.2. Use Donoho adaptive thresholding to quantize the high-frequency coefficients after wavelet decomposition, retain the high-frequency coefficients that meet the threshold requirements, and complete the adaptive threshold quantization process.

[0022] S3.3. Dynamic Huffman coding is used to entropy code the low-frequency coefficients after adaptive threshold quantization and the high-frequency coefficients after adaptive threshold quantization to generate a compressed bitstream, resulting in hybrid compressed data.

[0023] Furthermore, step S4 includes the following sub-steps:

[0024] S4.1. Statistically analyze the historical access records of the mixed compressed data to obtain the access frequency statistics of the mixed compressed data. The access frequency statistics include the mean access frequency and the standard deviation of access frequency.

[0025] S4.2. Based on the spatial correlation attributes and temporal correlation attributes of the hybrid compressed data, and combined with access frequency statistics, construct spatiotemporal correlation data prefetching rules;

[0026] S4.3. According to the data prefetching rules and heterogeneous computing resource scheduling scheme, complete the prefetching operation of the target data, and complete the update operation of the cache copy and the eviction operation of the cache copy based on the access frequency statistics.

[0027] Furthermore, in step S1, the edge node operating parameters are subjected to a second exponential smoothing process. First, the time series of edge node operating parameters is subjected to a first exponential smoothing calculation to obtain a first smoothing value. Then, the first smoothing value is subjected to a second exponential smoothing calculation to obtain a second smoothing value. The time series of edge node operating parameters is corrected based on the second smoothing value. The corrected time series of edge node operating parameters is normalized. A long short-term memory network model is used to predict the edge node load time series of the normalized time series to obtain the load prediction result.

[0028] Furthermore, in step S2, the heterogeneous computing resources include a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). The optimization objectives of the multi-objective optimization model include the total processing time of the computing task and the total power consumption of the computing task. The total processing time is the weighted sum of the CPU processing time, GPU processing time, and FPGA processing time. The total power consumption is the sum of the CPU power consumption, GPU power consumption, and FPGA power consumption. The constraint of the multi-objective optimization model is the upper limit threshold of the load of the edge node, and the real-time load value of the edge node does not exceed the upper limit threshold.

[0029] Furthermore, in step S3, when performing wavelet decomposition on the original IoT data, the original IoT data is first subjected to symmetric boundary extension processing to supplement the boundary sequence of the original IoT data. Then, a biorthogonal wavelet basis is used to perform multi-level wavelet decomposition on the boundary-extended original IoT data to obtain low-frequency coefficients and multiple sets of high-frequency coefficients at the corresponding level. The high-frequency coefficients obtained by decomposition are then divided into blocks, and each set of high-frequency coefficients after block division is quantized using Donoho adaptive thresholding to filter out high-frequency coefficients that do not meet the threshold requirements.

[0030] Furthermore, in step S4, when constructing the spatiotemporally correlated data prefetching rules, the spatial correlation degree of the mixed compressed data is first calculated to calculate the spatial location correlation weight between different mixed compressed data. Then, the temporal correlation degree of the mixed compressed data is calculated to calculate the temporal access sequence correlation weight between different mixed compressed data. Combining the spatial location correlation weight, the temporal access sequence correlation weight, and the access frequency statistics, the spatiotemporally correlated data prefetching rules are constructed. Based on the data prefetching rules, the target data to be prefetched is determined, and the target data is cached in advance in the storage unit of the corresponding edge node. At the same time, the update operation of the cache copy and the eviction operation of the cache copy are completed based on the access frequency statistics.

[0031] A dynamic load prediction data processing system for IoT edge computing is provided. The system includes a load prediction and state space construction module, a task routing and resource scheduling module, a data hybrid compression module, and a distributed cache management module.

[0032] The load prediction and state space construction module completes the collection of edge node operating parameters, network link quality parameters, IoT task characteristic parameters, edge node load time-series prediction, and task dynamic decision-making state space construction; the task routing and resource scheduling module completes the construction and solution of multi-objective optimization models for computing task routing and allocation, and heterogeneous computing resource scheduling; the data hybrid compression module completes the boundary extension processing, wavelet decomposition processing, adaptive threshold quantization processing, and dynamic Huffman coding processing of IoT raw data; the distributed cache management module completes the access frequency statistics of hybrid compressed data, the construction of spatiotemporally correlated data prefetching rules, data prefetching operations, and cache copy update operations.

[0033] The beneficial effects of this invention are:

[0034] (1) By predicting the load time sequence of edge nodes and constructing the state space of task decision, combined with dynamic task routing allocation and multi-objective optimization of resource scheduling, dynamic adaptation of computing tasks and rational utilization of computing resources are realized, thereby improving the overall operational stability of the edge computing system.

[0035] (2) Through multi-level time-frequency domain decomposition, adaptive quantization and dynamic entropy coding progressive hybrid compression processing, the information redundancy of IoT data is effectively reduced, the network bandwidth occupation during data transmission is reduced, and the fidelity of data processing is guaranteed at the same time.

[0036] (3) By analyzing the data access frequency statistics and spatiotemporal correlation attributes, we can construct adaptive data prefetching rules, complete the dynamic update and eviction of cached copies, improve the cache hit rate of edge nodes, and reduce the end-to-end response latency of data access. Attached Figure Description

[0037] Figure 1 A flowchart illustrating the steps of a dynamic load prediction data processing method for IoT edge computing;

[0038] Figure 2 The flowchart illustrates the specific steps of a dynamic load prediction data processing method for IoT edge computing, as provided in this embodiment. Detailed Implementation

[0039] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1

[0041] See Figure 1This paper provides a method for dynamic load prediction data processing in IoT edge computing, which includes the following steps:

[0042] S1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Use a long short-term memory network model to perform time-series prediction of edge node load, obtain load prediction results, integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results, and construct the state space for dynamic task decision-making.

[0043] S2. Input the state space into the deep Q-network algorithm to complete the routing allocation of computing tasks, establish a multi-objective optimization model for heterogeneous computing resource scheduling, and use a non-dominated sorting genetic algorithm with elitist strategy to solve the multi-objective optimization model and obtain a heterogeneous computing resource scheduling scheme.

[0044] S3. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by wavelet decomposition, the data after wavelet decomposition is processed by adaptive threshold quantization, and the data after adaptive threshold quantization is processed by dynamic Huffman coding to obtain hybrid compressed data.

[0045] S4. Perform access frequency statistics on the hybrid compressed data to obtain access frequency statistics. Combine the access frequency statistics with the spatiotemporal correlation attributes of the hybrid compressed data to construct data prefetching rules. According to the data prefetching rules and the heterogeneous computing resource scheduling scheme, complete the data prefetching operation and cache copy update operation.

[0046] In some embodiments, step S1 includes the following sub-steps:

[0047] S1.1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Edge node operating parameters include edge node CPU utilization parameters and edge node GPU utilization parameters. Network link quality parameters include a network link service quality parameter matrix. IoT task characteristic parameters include IoT task feature vectors.

[0048] S1.2. A long short-term memory network model is used to predict the load of edge nodes in a time series. The long short-term memory network model includes an input layer, a hidden layer and an output layer. The input layer receives the historical load time series data of edge nodes. The hidden layer contains a forget gate, an input gate and an output gate. The hidden layer performs feature extraction and time series fitting on the input time series data. The output layer outputs the load prediction results of edge nodes.

[0049] S1.3. Integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results to construct a state space for dynamic task decision-making.

[0050] In some embodiments, step S2 includes the following sub-steps:

[0051] S2.1. Input the state space into the deep Q network algorithm. The deep Q network algorithm includes a state input layer, a convolutional layer, a fully connected layer, and an action output layer. The state input layer receives the state space data, the convolutional layer and the fully connected layer extract features from the state space data, and the action output layer outputs the routing and allocation actions of the computing tasks, completing the allocation of computing tasks between local edge nodes, adjacent edge clusters and cloud centers.

[0052] S2.2. Taking the processing time and power consumption of computing tasks as optimization objectives, a multi-objective optimization model for heterogeneous computing resource scheduling is established, and the constraint condition of the multi-objective optimization model is the upper limit threshold of the load of edge nodes.

[0053] S2.3. A non-dominated sorting genetic algorithm with an elitist strategy is used to solve the multi-objective optimization model to obtain the Pareto front. Based on the Pareto front, a heterogeneous computing resource scheduling scheme is determined.

[0054] In some embodiments, step S3 includes the following sub-steps:

[0055] S3.1. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by multi-level wavelet decomposition using a biorthogonal wavelet basis to obtain the low-frequency coefficients and high-frequency coefficients after wavelet decomposition.

[0056] S3.2. Use Donoho adaptive thresholding to quantize the high-frequency coefficients after wavelet decomposition, retain the high-frequency coefficients that meet the threshold requirements, and complete the adaptive threshold quantization process.

[0057] S3.3. Dynamic Huffman coding is used to entropy code the low-frequency coefficients after adaptive threshold quantization and the high-frequency coefficients after adaptive threshold quantization to generate a compressed bitstream, resulting in hybrid compressed data.

[0058] In some embodiments, step S4 includes the following sub-steps:

[0059] S4.1. Statistically analyze the historical access records of the mixed compressed data to obtain the access frequency statistics of the mixed compressed data. The access frequency statistics include the mean access frequency and the standard deviation of access frequency.

[0060] S4.2. Based on the spatial correlation attributes and temporal correlation attributes of the hybrid compressed data, and combined with access frequency statistics, construct spatiotemporal correlation data prefetching rules;

[0061] S4.3. According to the data prefetching rules and heterogeneous computing resource scheduling scheme, complete the prefetching operation of the target data, and complete the update operation of the cache copy and the eviction operation of the cache copy based on the access frequency statistics.

[0062] In some embodiments, in step S1, the edge node operating parameters are subjected to a second exponential smoothing process. First, the time series of edge node operating parameters is subjected to a first exponential smoothing calculation to obtain a first smoothing value. Then, the first smoothing value is subjected to a second exponential smoothing calculation to obtain a second smoothing value. The time series of edge node operating parameters is corrected based on the second smoothing value. The corrected time series of edge node operating parameters is normalized. A long short-term memory network model is used to predict the load time series of edge nodes on the normalized time series to obtain the load prediction result.

[0063] In some embodiments, in step S2, the heterogeneous computing resources include a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). The optimization objectives of the multi-objective optimization model include the total processing time of the computing task and the total power consumption of the computing task. The total processing time is the weighted sum of the CPU processing time, GPU processing time, and FPGA processing time. The total power consumption is the cumulative sum of the CPU power consumption, GPU power consumption, and FPGA power consumption. The constraint of the multi-objective optimization model is the upper limit threshold of the load of the edge node, and the real-time load value of the edge node does not exceed the upper limit threshold.

[0064] In some embodiments, in step S3, when performing wavelet decomposition on the original IoT data, the original IoT data is first subjected to symmetric boundary extension processing to supplement the boundary sequence of the original IoT data. Then, a biorthogonal wavelet basis is used to perform multi-level wavelet decomposition on the boundary-extended original IoT data to obtain low-frequency coefficients and multiple sets of high-frequency coefficients at the corresponding level. The high-frequency coefficients obtained by decomposition are then divided into blocks, and each set of high-frequency coefficients after block division is quantized using a Donoho adaptive threshold to filter out high-frequency coefficients that do not meet the threshold requirements.

[0065] In some embodiments, when constructing the spatiotemporally correlated data prefetching rules in step S4, the spatial correlation degree of the mixed compressed data is first calculated to calculate the spatial location correlation weight between different mixed compressed data. Then, the temporal correlation degree of the mixed compressed data is calculated to calculate the temporal access sequence correlation weight between different mixed compressed data. Combining the spatial location correlation weight, the temporal access sequence correlation weight, and the access frequency statistics, the spatiotemporally correlated data prefetching rules are constructed. Based on the data prefetching rules, the target data to be prefetched is determined, and the target data is cached in advance in the storage unit of the corresponding edge node. At the same time, the update operation of the cache copy and the eviction operation of the cache copy are completed based on the access frequency statistics.

[0066] A dynamic load prediction data processing system for IoT edge computing is provided. The system includes a load prediction and state space construction module, a task routing and resource scheduling module, a data hybrid compression module, and a distributed cache management module.

[0067] The load prediction and state space construction module completes the collection of edge node operating parameters, network link quality parameters, IoT task characteristic parameters, edge node load time-series prediction, and task dynamic decision-making state space construction; the task routing and resource scheduling module completes the construction and solution of multi-objective optimization models for computing task routing and allocation, and heterogeneous computing resource scheduling; the data hybrid compression module completes the boundary extension processing, wavelet decomposition processing, adaptive threshold quantization processing, and dynamic Huffman coding processing of IoT raw data; the distributed cache management module completes the access frequency statistics of hybrid compressed data, the construction of spatiotemporally correlated data prefetching rules, data prefetching operations, and cache copy update operations.

[0068] Example 2

[0069] This embodiment provides a specific implementation process for a dynamic load prediction data processing method for IoT edge computing. The method is used for data processing and resource scheduling in IoT edge computing scenarios, achieving dynamic allocation of computing tasks, reasonable scheduling of heterogeneous resources, efficient data compression, and cache management. The specific implementation process is as follows: Figure 2 As shown:

[0070] S1. Parameter Acquisition and State Space Construction:

[0071] S1.1. Basic Operation and Task Parameter Collection:

[0072] Edge nodes are distributed computing units deployed between IoT terminal devices and the cloud center, possessing computing, storage, and network transmission capabilities. They are key execution units in the edge computing architecture. In this solution, edge nodes are used to undertake computing tasks and data processing demands generated by IoT terminals, serving as the primary carriers for load acquisition, data processing, and task execution. This step collects edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Edge node operating parameters include edge node CPU utilization parameters and edge node GPU utilization parameters. These two types of parameters characterize the current computing resource occupancy and operating load of the edge nodes.

[0073] The network link quality of service parameter matrix is ​​a two-dimensional dataset characterizing the transmission capacity of a network link. It integrates key state parameters during link transmission and is a standardized tool for network performance evaluation. In this scheme, the network link quality parameters include the network link quality of service parameter matrix, which contains link state parameters such as link bandwidth, transmission latency, jitter rate, and packet loss rate. This matrix is ​​used to evaluate the feasibility of data transmission and task allocation, providing network state information for task routing decisions.

[0074] The IoT task feature vector is a multi-dimensional vector data obtained after standardized feature extraction of IoT computing tasks. It comprehensively represents the key attributes of the computing tasks and serves as the foundational input data for task scheduling decisions. In this scheme, the IoT task feature parameters include the IoT task feature vector, which contains task attribute features such as computational load, data volume, latency sensitivity, and priority. This provides a basis for task routing and allocation, enabling the matching of tasks with computing resources. After parameter acquisition, all acquired parameters undergo data format alignment and outlier filtering to remove abnormal data exceeding reasonable value ranges, providing standardized input data for subsequent processing.

[0075] S1.2. Edge node load timing prediction processing:

[0076] Long Short-Term Memory (LSTM) network models are time-series data processing models based on recurrent neural network (RNN) architectures. They possess the ability to capture long-term time-dependent features and effectively solve the gradient vanishing problem of traditional RNNs, making them a mature time-series prediction technique. In this scheme, the LSM network model is used to predict the load changes of edge nodes in time series, anticipating the load change trends of edge nodes in advance. The LSM network model consists of an input layer, hidden layers, and an output layer, each fully connected sequentially. The output data of the input layer serves as the input data of the hidden layer, and vice versa. The hidden layer contains a forget gate, an input gate, and an output gate connected sequentially. The forget gate receives the time-series data from the input layer and the output data of the hidden layer from the previous time step, processes it through an activation function, and outputs forget weights to filter out invalid features in the time-series data. The input gate receives the same input data as the forget gate, processes it through an activation function, and outputs input weights and candidate state data, updating the cell state based on the output of the forget gate. The output gate receives the updated cell state and input data, processes it through an activation function, and outputs the hidden layer state data at the current time step, completing feature extraction and time-series fitting of the time-series data.

[0077] The training process of the Long Short-Term Memory (LSTM) network model includes: first, collecting historical load time-series data of edge nodes to construct a training dataset; normalizing the training dataset to eliminate differences in data units; dividing the processed training dataset into a training set and a validation set according to a preset ratio; inputting the training set data into the model; using the error between the predicted load value and the actual load value as the loss function; and iteratively updating the model's weight parameters using the backpropagation algorithm; and validating the accuracy of the iterated model using the validation set data. When the validation accuracy reaches the preset requirements, the model training is completed, resulting in a LSM network model that can be used for load time-series prediction.

[0078] Quadratic exponential smoothing is a time-series data smoothing method based on exponential smoothing. By performing a second smoothing correction on the result of the first exponential smoothing, it can effectively fit the trend changes of time-series data and is a mature existing time-series data preprocessing technique. In this scheme, quadratic exponential smoothing is used to smooth and correct the time-series sequence of edge node operating parameters, eliminating random fluctuations in the time-series data and improving the accuracy of subsequent load prediction results. This step performs quadratic exponential smoothing on the time-series sequence of edge node operating parameters. First, a first exponential smoothing calculation is performed on the time-series sequence of edge node operating parameters to obtain a first smoothed value; then, a second exponential smoothing calculation is performed on the first smoothed value to obtain a second smoothed value; the time-series sequence of edge node operating parameters is corrected based on the second smoothed value to eliminate random fluctuations in the time-series sequence; the corrected time-series sequence of edge node operating parameters is normalized to map the time-series data to a uniform numerical range and eliminate differences in data units; the normalized time-series sequence is input into the trained Long Short-Term Memory (LSTM) network model, and after hierarchical processing by the model, the load prediction results of the edge nodes are output.

[0079] In some embodiments, an autoregressive integral moving average model can be used to replace the long short-term memory network model to complete the time series prediction of edge node load. The autoregressive integral moving average model is an existing mature time series prediction model. After performing differential stabilization on the time series data, it combines the autoregressive term and the moving average term to complete the time series prediction. It is suitable for edge node deployment scenarios with limited hardware resources.

[0080] S1.3. Construction of the task decision state space:

[0081] The state space is a multidimensional dataset used in reinforcement learning algorithms to fully represent the current operating state of the decision-making environment. It contains all the environmental feature parameters required for the decision-making process and serves as the key input for reinforcement learning algorithms. In this scheme, the state space provides the decision input for the deep Q-network algorithm, fully covering all environmental features required for task decision-making. This step aligns and integrates the collected edge node operating parameters, network link quality parameters, and IoT task feature parameters with the load prediction results obtained from load time series prediction to construct the state space for dynamic task decision-making. Each set of data in the state space corresponds to a current decision state of the edge computing environment, providing a complete environmental state basis for subsequent task routing and allocation.

[0082] S2. Generation of Task Routing and Resource Scheduling Schemes:

[0083] S2.1. Task routing allocation based on deep Q-network algorithm:

[0084] The Deep Q-Network algorithm is a deep reinforcement learning algorithm that combines deep neural networks and Q-learning. It possesses sequential decision-making capabilities in a high-dimensional state space and can iteratively optimize decision strategies through interaction with the environment, making it a mature reinforcement learning decision-making technique. In this scheme, the Deep Q-Network algorithm is used to dynamically allocate computational tasks, achieving dynamic matching between tasks and computational nodes. The Deep Q-Network algorithm consists of a state input layer, convolutional layers, fully connected layers, and an action output layer. Each layer is fully connected sequentially. The output data of the state input layer serves as the input data of the convolutional layers. The output data of the convolutional layers, after flattening, serves as the input data of the fully connected layers. The output data of the fully connected layers serves as the input data of the action output layer. The state input layer receives state space data and converts it into tensor data that meets the network input requirements. The convolutional layer extracts local features from the input tensor data using multiple convolutional kernels and outputs a high-dimensional feature map. The fully connected layer integrates global features and performs nonlinear mapping on the high-dimensional features extracted by the convolutional layer, outputting state action values ​​corresponding to different actions. Based on the state action values ​​output by the fully connected layer, the action output layer selects the optimal route to allocate actions using a greedy strategy, completing the allocation of computing tasks among local edge nodes, adjacent edge clusters, and the cloud center.

[0085] The training process of the Deep Q-Network algorithm includes: first, constructing a simulation environment for edge computing task allocation, setting up a state space, action space, and reward function, with task processing latency and node load balancing as the main evaluation indicators; inputting the state data collected in the simulation environment into the network, and outputting the corresponding allocation action through the network, calculating the reward value based on the environmental feedback after the action is executed; using the maximization of the cumulative reward value as the optimization objective, iteratively updating the network's weight parameters using the gradient descent method; after multiple rounds of iterative training, when the network's decision-making performance reaches the preset requirements, the model training is completed, resulting in a Deep Q-Network algorithm that can be used for task routing allocation.

[0086] Local edge nodes are the edge nodes to which the IoT terminal devices generating computing tasks belong, and they are the primary units receiving IoT terminal data. In this solution, local edge nodes are used to handle low-latency tasks generated by local terminals and are the preferred execution unit for task routing and allocation. Adjacent edge clusters are distributed computing clusters composed of multiple edge nodes at the same network layer as the local edge nodes and possessing low-latency network interconnection capabilities. In this solution, adjacent edge clusters are used to offload tasks when local edge nodes are overloaded, enabling resource sharing among edge nodes at the same level. Cloud centers are centralized computing centers deployed in the core network, possessing massive computing, storage, and network resources, and capable of handling large-scale, non-real-time computing tasks. In this solution, cloud centers are used to handle non-real-time, computationally intensive offloaded tasks and serve as a supplementary computing power unit to the edge computing architecture. This step inputs the completed state space into the trained deep Q-network algorithm. After feature extraction and mapping at each layer of the network, it outputs the optimal computing task routing action in the current state. According to the allocation action, the computing task is allocated among the local edge node, adjacent edge clusters and cloud center, realizing dynamic routing decision of computing tasks.

[0087] In some embodiments, the policy gradient algorithm can be used to replace the deep Q-network algorithm to complete task routing and allocation. The policy gradient algorithm is an existing mature reinforcement learning algorithm that achieves sequential decision-making by directly optimizing the policy function and is suitable for task allocation scenarios in continuous action spaces.

[0088] S2.2. Construction of a multi-objective optimization model for heterogeneous resource scheduling:

[0089] Heterogeneous computing resources are collections of computing units with different computing architectures, characteristics, and applicable scenarios. They can adapt to different types of computing task requirements through the collaboration of these units. In this solution, heterogeneous computing resources include three types of computing units: CPUs, GPUs, and FPGAs. CPUs are key computing units with general-purpose computing capabilities, adept at handling logically complex, serially executed computing tasks, and are the primary execution unit for general-purpose computing scenarios. In this solution, the CPU is used to handle logic control-type, serially executed IoT data processing tasks. GPUs are computing units with massively parallel computing capabilities, adept at handling large-scale data and high-parallelism computing tasks, and are the primary execution unit for high-throughput computing scenarios. In this solution, the GPU is used to handle IoT data processing tasks with high parallelism and large data volumes. FPGAs are computing units with hardware programmability, capable of accelerating fixed computing flows through hardware configuration. They are adept at handling low-latency, fixed-flow computing tasks and are the primary execution unit for real-time computing scenarios. In this solution, the field-programmable gate array (FPGA) is used to undertake IoT data compression and preprocessing tasks with low latency requirements and fixed processes.

[0090] Multi-objective optimization models are mathematical models used to find the optimal solution for multiple conflicting optimization objectives. They can achieve balanced optimization among multiple objectives and are a mature existing resource scheduling optimization technique. In this scheme, the multi-objective optimization model uses the processing time and power consumption of computing tasks as optimization objectives to achieve balanced scheduling of heterogeneous computing resources, taking into account both task processing efficiency and operating energy consumption. This step establishes a multi-objective optimization model for heterogeneous computing resource scheduling, using the processing time and power consumption of computing tasks as optimization objectives. The total processing time of a computing task is the weighted sum of the processing time of the CPU, GPU, and FPGA. The total power consumption of a computing task is the cumulative sum of the power consumption of the CPU, GPU, and FPGA. The constraint condition for the multi-objective optimization model is a load upper limit threshold for edge nodes. The real-time load value of the edge nodes does not exceed the load upper limit threshold, ensuring that the edge nodes do not experience overload.

[0091] S2.3. Solving the multi-objective optimization model and determining the scheduling scheme:

[0092] The non-dominated sorting genetic algorithm with elitist strategy is an improved multi-objective optimization algorithm based on genetic algorithms. Through non-dominated sorting and elitist retention strategies, it can quickly converge to the optimal solution set of multi-objective optimization problems and is a mature existing multi-objective optimization technique. In this scheme, the algorithm is used to solve a multi-objective optimization model for heterogeneous resource scheduling, obtaining a resource scheduling scheme that meets the optimization objectives. The Pareto front is the set of solutions in a multi-objective optimization problem that cannot improve any other optimization objectives without compromising at least one objective; it represents the optimal equilibrium solution set of the multi-objective optimization problem. In this scheme, based on the solutions in the Pareto front, the final heterogeneous computing resource scheduling scheme is determined, achieving a balanced optimization of processing time and power consumption.

[0093] This step employs a non-dominated sorting genetic algorithm with an elitist strategy to solve the constructed multi-objective optimization model. First, the population is initialized, generating multiple candidate solutions for heterogeneous resource scheduling. The candidate solutions are then non-dominated sorted, and the crowding degree of each solution is calculated. A new population is generated through selection, crossover, and mutation operations. Elite retention is performed by combining the parent and offspring populations to select the best candidate solutions. After multiple iterations, a convergent Pareto front is obtained. Based on the optimal solution in the Pareto front, a scheduling scheme for heterogeneous computing resources is determined to guide subsequent data processing.

[0094] In some embodiments, particle swarm optimization can be used to replace non-dominated sorting genetic algorithm with elitist strategy to solve multi-objective optimization models. Particle swarm optimization is an existing mature swarm intelligence optimization algorithm that solves the optimal solution by simulating the swarm optimization behavior of particles and is suitable for solving multi-objective optimization models with low dimensionality.

[0095] S3. Data Hybrid Compression Processing:

[0096] S3.1. Wavelet decomposition based on bioorthogonal wavelet basis:

[0097] Biorthogonal wavelet bases are a class of wavelet basis functions with linear phase characteristics used in wavelet transform. They effectively avoid phase distortion during signal decomposition and reconstruction and are currently mature wavelet transform basis functions. In this scheme, this wavelet base is used to perform wavelet decomposition processing on the raw IoT data, achieving time-frequency domain separation of the data and providing a foundation for subsequent data compression. Wavelet decomposition is a time-frequency domain signal analysis method that decomposes a time-domain signal into components at different frequency levels using wavelet basis functions, achieving separation of the low-frequency main components and high-frequency detail components. It is a mature signal processing technique. In this scheme, this processing is used to perform time-frequency domain decomposition on the raw IoT data, separating the main features and detail features of the data, providing a foundation for subsequent quantization and encoding processing. Symmetrical boundary extension processing is a signal boundary preprocessing method that eliminates the boundary effects caused by signal boundary truncation during wavelet decomposition by symmetrically supplementing the boundary sequence of the signal. It is a mature signal preprocessing technique. In this scheme, the processing is used to supplement the boundaries of the original IoT data, avoid boundary distortion in the wavelet decomposition process, and improve the accuracy of data decomposition and reconstruction.

[0098] This step, following the generated heterogeneous computing resource scheduling scheme, performs symmetric boundary extension processing on the original IoT data to supplement the boundary sequence of the original IoT data and eliminate the boundary effects in the wavelet decomposition process. A biorthogonal wavelet basis is then used to perform multi-level wavelet decomposition processing on the boundary-extended original IoT data to obtain low-frequency coefficients and multiple sets of high-frequency coefficients at the corresponding levels. The low-frequency coefficients contain the main feature information of the original IoT data, while the high-frequency coefficients contain the detailed feature information of the original IoT data.

[0099] S3.2. High-frequency coefficient quantization processing based on Donoho adaptive threshold:

[0100] Donoho adaptive thresholding is a wavelet coefficient thresholding method based on signal statistical characteristics. It adaptively adjusts the threshold size according to the signal's noise level, filtering out noise components while preserving effective signal features. It is a mature existing wavelet coefficient quantization technique. In this scheme, this threshold is used to quantize the high-frequency coefficients obtained from wavelet decomposition, filtering out redundant detail components and preserving the effective features of the data, thus achieving initial data compression. Quantization is the process of mapping continuously valued wavelet coefficients to a finite number of discrete values, effectively reducing the number of bits in the data representation and lowering data redundancy. It is a mature existing data compression preprocessing technique. In this scheme, this processing is used to discretize and map the high-frequency coefficients after wavelet decomposition, reducing redundant information while preserving effective data features, providing a foundation for subsequent entropy coding.

[0101] This step divides the high-frequency coefficients obtained from wavelet decomposition into multiple non-overlapping coefficient blocks. Then, each block of high-frequency coefficients is quantized using a Donoho adaptive threshold, filtering out high-frequency coefficients that do not meet the threshold requirements and retaining those that do. This completes the adaptive threshold quantization process, reducing redundant information while preserving the core characteristics of the data.

[0102] S3.3. Entropy coding processing based on dynamic Huffman coding:

[0103] Dynamic Huffman coding is a lossless entropy coding technique based on an improvement of Huffman coding. It dynamically updates the coding table according to the real-time statistical characteristics of the data, without requiring pre-calculation of the probability distribution of all data. It is suitable for streaming data compression and is a mature lossless coding technique. In this scheme, this coding is used to entropy code the quantized wavelet coefficients to achieve lossless data compression and reduce bandwidth consumption. Entropy coding is a lossless data compression method based on the principle of information entropy. By assigning short codes to symbols with high occurrence probabilities and long codes to symbols with low occurrence probabilities, it reduces the average code length of the data, achieving lossless data compression. It is a key mature data compression technology. In this scheme, this processing is the final stage of hybrid data compression, used to losslessly encode the quantized wavelet coefficients to generate the final hybrid compressed data.

[0104] This step uses dynamic Huffman coding to entropy encode both the low-frequency coefficients and high-frequency coefficients after adaptive threshold quantization. First, the Huffman coding tree is initialized, and the quantized coefficient data is read sequentially. The Huffman coding tree is dynamically updated according to the frequency of occurrence of the coefficient data, and a corresponding code is assigned to each coefficient. After the coding of all coefficients is completed, a compressed bitstream is generated, resulting in hybrid compressed data.

[0105] In some embodiments, the Zstd entropy coding algorithm can be used to replace dynamic Huffman coding to complete the entropy coding process. The Zstd entropy coding algorithm is an existing mature lossless compression algorithm. It combines dictionary coding and entropy coding technology and has the characteristics of fast compression speed and high compression ratio. It is suitable for compression processing scenarios of large volume data.

[0106] S4. Data Cache Management and Prefetching:

[0107] S4.1. Statistics on the frequency of access to hybrid compressed data:

[0108] Access frequency statistics are statistical data that characterize the frequency of data access within a statistical period. They are used to quantify the access popularity and patterns of data and are a key basis for cache management and data prefetching. In this scheme, access frequency statistics include the mean access frequency and the standard deviation of access frequency. The mean access frequency is the average number of times the data is accessed within a preset statistical period, used to characterize the overall access popularity of the data. The standard deviation of access frequency is the degree of dispersion of the number of data accesses relative to the mean access frequency, used to characterize the fluctuation of data access popularity.

[0109] This step involves statistically analyzing the historical access records of the mixed compressed data, recording the access time and number of accesses for each group of mixed compressed data within a preset statistical period. Based on the statistically obtained access data, the mean access frequency and standard deviation of the access frequency for each group of mixed compressed data are calculated to obtain the access frequency statistics of the mixed compressed data, providing a statistical basis for subsequent data prefetching rule construction and cache copy management.

[0110] S4.2. Construction of spatiotemporal correlation data prefetching rules:

[0111] Spatiotemporal correlation attributes are the spatial location associations and temporal access sequence associations between data. They characterize the spatial and temporal relationships between different data and are a key basis for constructing data prefetching rules. In this scheme, spatiotemporal correlation attributes include spatial correlation attributes and temporal correlation attributes. Spatial correlation attributes represent the association between the spatial locations of the corresponding IoT terminals; data generated by terminals with similar spatial locations have a higher spatial correlation. Temporal correlation attributes represent the association between the time sequences of data access; data with a sequential access order have a higher temporal correlation.

[0112] This step calculates the spatial correlation of the mixed compressed data, determining the spatial location correlation weights between different mixed compressed data sets. The higher the spatial location correlation, the greater the corresponding spatial location correlation weight. Next, it calculates the temporal correlation of the mixed compressed data, determining the temporal access sequence correlation weights between different mixed compressed data sets. The higher the temporal access sequence correlation, the greater the corresponding temporal access sequence correlation weight. Combining the spatial location correlation weights, temporal access sequence correlation weights, and access frequency statistics, a spatiotemporal correlation data prefetching rule is constructed. This data prefetching rule is used to determine the target data that needs to be cached in advance.

Claims

1. A method for processing dynamic load prediction data in IoT edge computing, characterized in that, Includes the following steps: S1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Use a long short-term memory network model to perform time-series prediction of edge node load, obtain load prediction results, integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results, and construct the state space for dynamic task decision-making. S2. Input the state space into the deep Q-network algorithm to complete the routing allocation of computing tasks, establish a multi-objective optimization model for heterogeneous computing resource scheduling, and use a non-dominated sorting genetic algorithm with elitist strategy to solve the multi-objective optimization model and obtain a heterogeneous computing resource scheduling scheme. S3. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by wavelet decomposition, the data after wavelet decomposition is processed by adaptive threshold quantization, and the data after adaptive threshold quantization is processed by dynamic Huffman coding to obtain hybrid compressed data. S4. Perform access frequency statistics on the hybrid compressed data to obtain access frequency statistics. Combine the access frequency statistics with the spatiotemporal correlation attributes of the hybrid compressed data to construct data prefetching rules. According to the data prefetching rules and the heterogeneous computing resource scheduling scheme, complete the data prefetching operation and cache copy update operation. Step S3 includes the following sub-steps: S3.

1. According to the heterogeneous computing resource scheduling scheme, the original IoT data is processed by multi-level wavelet decomposition using a biorthogonal wavelet basis to obtain the low-frequency coefficients and high-frequency coefficients after wavelet decomposition. S3.

2. Use Donoho adaptive thresholding to quantize the high-frequency coefficients after wavelet decomposition, retain the high-frequency coefficients that meet the threshold requirements, and complete the adaptive threshold quantization process. S3.

3. Dynamic Huffman coding is used to entropy code the low-frequency coefficients and high-frequency coefficients after adaptive threshold quantization to generate a compressed bitstream, resulting in hybrid compressed data. Step S4 includes the following sub-steps: S4.

1. Statistically analyze the historical access records of the mixed compressed data to obtain the access frequency statistics of the mixed compressed data. The access frequency statistics include the mean access frequency and the standard deviation of access frequency. S4.

2. Based on the spatial correlation attributes and temporal correlation attributes of the hybrid compressed data, and combined with access frequency statistics, construct spatiotemporal correlation data prefetching rules; S4.

3. According to the data prefetching rules and heterogeneous computing resource scheduling scheme, complete the prefetching operation of the target data, and complete the update operation of the cache copy and the eviction operation of the cache copy based on the access frequency statistics.

2. The method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S1.

1. Collect edge node operating parameters, network link quality parameters, and IoT task characteristic parameters. Edge node operating parameters include edge node CPU utilization parameters and edge node GPU utilization parameters. Network link quality parameters include a network link service quality parameter matrix. IoT task characteristic parameters include IoT task feature vectors. S1.

2. A long short-term memory network model is used to predict the load of edge nodes in a time series. The long short-term memory network model includes an input layer, a hidden layer and an output layer. The input layer receives the historical load time series data of edge nodes. The hidden layer contains a forget gate, an input gate and an output gate. The hidden layer performs feature extraction and time series fitting on the input time series data. The output layer outputs the load prediction results of edge nodes. S1.

3. Integrate edge node operating parameters, network link quality parameters, IoT task characteristic parameters, and load prediction results to construct a state space for dynamic task decision-making.

3. The method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S2.

1. Input the state space into the deep Q network algorithm. The deep Q network algorithm includes a state input layer, a convolutional layer, a fully connected layer, and an action output layer. The state input layer receives the state space data, the convolutional layer and the fully connected layer extract features from the state space data, and the action output layer outputs the routing and allocation actions of the computing tasks, completing the allocation of computing tasks between local edge nodes, adjacent edge clusters and cloud centers. S2.

2. Taking the processing time and power consumption of computing tasks as optimization objectives, a multi-objective optimization model for heterogeneous computing resource scheduling is established, and the constraint condition of the multi-objective optimization model is the upper limit threshold of the load of edge nodes. S2.

3. A non-dominated sorting genetic algorithm with an elitist strategy is used to solve the multi-objective optimization model to obtain the Pareto front. Based on the Pareto front, a heterogeneous computing resource scheduling scheme is determined.

4. The method according to claim 1, characterized in that, In step S1, the edge node operating parameters are subjected to a second exponential smoothing process. First, the time series of edge node operating parameters is subjected to a first exponential smoothing calculation to obtain a first smoothing value. Then, the first smoothing value is subjected to a second exponential smoothing calculation to obtain a second smoothing value. The time series of edge node operating parameters is corrected based on the second smoothing value. The corrected time series of edge node operating parameters is normalized. A long short-term memory network model is used to predict the edge node load time series of the normalized time series to obtain the load prediction result.

5. The method according to claim 1, characterized in that, In step S2, the heterogeneous computing resources include a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). The optimization objectives of the multi-objective optimization model include the total processing time of the computing task and the total power consumption of the computing task. The total processing time is the weighted sum of the CPU processing time, GPU processing time, and FPGA processing time. The total power consumption is the cumulative sum of the CPU power consumption, GPU power consumption, and FPGA power consumption. The constraint of the multi-objective optimization model is the upper limit threshold of the load of the edge node, and the real-time load value of the edge node does not exceed the upper limit threshold.

6. The method according to claim 1, characterized in that, In step S3, when performing wavelet decomposition on the original IoT data, the original IoT data is first subjected to symmetric boundary extension processing to supplement the boundary sequence of the original IoT data. Then, a biorthogonal wavelet basis is used to perform multi-level wavelet decomposition on the boundary-extended original IoT data to obtain the low-frequency coefficients and multiple sets of high-frequency coefficients at the corresponding level. The high-frequency coefficients obtained by decomposition are then divided into blocks, and each set of high-frequency coefficients after block division is quantized using Donoho adaptive thresholding to filter out high-frequency coefficients that do not meet the threshold requirements.

7. The method according to claim 1, characterized in that, In step S4, when constructing the spatiotemporally correlated data prefetching rules, the spatial correlation degree of the mixed compressed data is first calculated, and the spatial location correlation weight between different mixed compressed data is calculated. Then, the temporal correlation degree of the mixed compressed data is calculated, and the temporal access sequence correlation weight between different mixed compressed data is calculated. Combining the spatial location correlation weight, the temporal access sequence correlation weight, and the access frequency statistics, the spatiotemporally correlated data prefetching rules are constructed. Based on the data prefetching rules, the target data to be prefetched is determined, and the target data is cached in advance in the storage unit of the corresponding edge node. At the same time, the update operation of the cache copy and the eviction operation of the cache copy are completed based on the access frequency statistics.

8. An IoT edge computing dynamic load prediction data processing system, used to perform the method as described in any one of claims 1-7, characterized in that, It includes a load prediction and state space construction module, a task routing and resource scheduling module, a data hybrid compression module, and a distributed cache management module; The load prediction and state space construction module completes the collection of edge node operating parameters, network link quality parameters, IoT task characteristic parameters, edge node load time-series prediction, and task dynamic decision-making, and constructs the state space. The task routing and resource scheduling module completes the construction and solution of a multi-objective optimization model for computing task routing and allocation, heterogeneous computing resource scheduling; the data hybrid compression module completes the boundary extension processing, wavelet decomposition processing, adaptive threshold quantization processing, and dynamic Huffman coding processing of IoT raw data. The distributed cache management module completes the statistics of access frequency of mixed compressed data, the construction of spatiotemporal data prefetching rules, data prefetching operations, and cache copy update operations.

Images