Internet of Things data processing method and system based on edge computing

Abstract

The invention discloses an Internet of Things data processing method and system based on edge computing, and relates to the technical field of data processing, the system is composed of a plurality of functional modules, and the system comprises a lightweight preprocessing module for obtaining Internet of Things data and priority factors, determining the priority of the Internet of Things data through the priority factors, and determining the priority of the Internet of Things data based on the priority of the Internet of Things data; the transmission opportunity and the aggregation granularity are autonomously adapted through multi-target Q learning, and terminal data are output; wherein the priority factor comprises a service state, data timeliness and edge load weighting; the prediction scheduling module is used for predicting communication link quality through the attention-enhanced LSTM and switching an optimal path, and allocating bandwidth and shaping rate in combination with the data priority of the Internet of Things and the optimal path; and based on the terminal data, matching a terminal data type and selecting a compression algorithm, converting the terminal data into an ordered transport stream in combination with the bandwidth and the shaping rate, and feeding back scheduling data to the cloud.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an Internet of Things data processing method and system based on edge computing. BACKGROUND

[0002] Internet of Things data processing is a collaborative mechanism that filters invalid data, optimizes transmission paths, and mines data value through terminal-side lightweight processing, edge-side intelligent scheduling, and cloud-side global optimization of the massive raw data collected by Internet of Things terminal devices such as sensors, surveillance cameras, and smart home appliances, ultimately transforming dispersed raw data into ordered, reliable, and usable information resources, solving the core pain points of large data volume, mixed types, high real-time requirements, and privacy sensitivity in Internet of Things scenarios. Internet of Things data often uses edge computing to sink part of the data processing, storage, and decision-making capabilities originally concentrated in the cloud to edge nodes such as edge gateways, edge servers, and local intelligent terminals, allowing data to be processed closest to the source and then uploading key information or results to the cloud on demand, forming a collaborative computing model of terminals, edges, and clouds.

[0003] Traditional terminals only perform simple data collection and do not preprocess abnormal and redundant data, resulting in a large amount of invalid data occupying transmission bandwidth, and fixed transmission strategies that are difficult to balance energy consumption and timeliness. Meanwhile, existing edge gateways mostly use passive response link switching and resource allocation, which cannot anticipate link quality fluctuations, bandwidth allocation does not consider Internet of Things data priority, and is prone to high-priority data latency exceeding standards, high data transmission costs, and global resource optimization lagging behind actual scenario changes. SUMMARY

[0004] To achieve the above purpose, the present application realizes the following technical solutions: An Internet of Things data processing system based on edge computing, comprising: A lightweight preprocessing module that obtains Internet of Things data and priority factors, determines the priority of Internet of Things data based on priority factors, and outputs terminal data based on the priority of Internet of Things data by autonomously adapting transmission opportunities and aggregation granularity using multi-objective Q-learning; Wherein, the priority factors include business status, data timeliness, and edge load weighting; A prediction and scheduling module that predicts communication link quality and switches the optimal path using attention-enhanced LSTM, allocates bandwidth and shaping rate based on Internet of Things data priority and the optimal path, matches terminal data types to select compression algorithms based on terminal data, converts terminal data into ordered transmission streams based on bandwidth and shaping rate, and feeds back scheduling data to the cloud; A global optimization module obtains edge scheduling feedback data, uses a space-time attention Transformer to predict a global resource peak to iteratively reinforce an orderly transmission flow in layers, and sends a feedforward scheduling instruction downward.

[0005] Further, the process of determining the priority of the Internet of Things data by the priority factor is: The priority factor weight proportion is determined by the Internet of Things scene, including a business state factor, a data timeliness factor, and an edge load factor; a comprehensive score of the priority is calculated in combination with the weight proportion of each priority factor, and the comprehensive score is corresponded to the four preset priority levels according to the range of the comprehensive score, if the comprehensive score is between 0 and 1, the low priority is corresponded, if the comprehensive score is between 1 and 3, the medium priority is corresponded, if the comprehensive score is between 3 and 4, the high priority is corresponded, and if the comprehensive score is between 4 and 5, the super-high priority is corresponded.

[0006] Further, the process of outputting the terminal data is: The priority of the current Internet of Things data, the terminal remaining power, and the real-time load of the edge node are obtained, and the transmission opportunity and the aggregation granularity are obtained; based on the priority of different Internet of Things data, the initial inclination of the basic action is set, including super-high priority data, high priority data, medium priority data, and low priority data; based on the transmission opportunity and the aggregation granularity, the result set containing the timeliness result, the energy consumption result, and the edge interaction feedback is arranged, and the result set is converted into the corresponding feedback signal; the feedback signal is used as the adjustment basis of Q learning, the action preference in different states is dynamically adjusted, and the terminal data is output.

[0007] Further, the process of predicting the communication link quality and switching the optimal path by the attention-enhanced LSTM is: The communication link quality parameters are collected, including the current bandwidth of the link, the data packet loss rate, and the transmission delay; at the same time, a historical data window with a length of 5 cycles is constructed to form an original communication link data sequence, and the original communication link data sequence is preprocessed; the preprocessed original communication link data sequence is input into the attention-enhanced LSTM model, the original communication link data sequence is encoded by the LSTM network to generate a hidden state vector at each time; the attention mechanism performs weighted calculation on the hidden state vectors at all times to obtain a link comprehensive feature vector, and the link comprehensive feature vector is used as input to obtain the predicted link quality; in combination with the predicted link quality and the priority of the Internet of Things data, a multi-dimensional path evaluation standard is established, and the optimal path is the one with the highest score.

[0008] Further, the process of allocating bandwidth and shaping rate in combination with the priority of the Internet of Things data and the optimal path is: The final allocation weight is obtained by multiplying the IoT data priority basic weight and the fitness coefficient, and the available bandwidth of the optimal path is divided according to the final allocation weight. The maximum allowed shaping rate is calculated by combining the allocated available bandwidth and the data amount in the current queue.

[0009] Further, the process of converting terminal data into an ordered transmission stream in combination with bandwidth and shaping rate is as follows: The IoT priority data uploaded by the terminal is classified by type, and a compression algorithm is selected according to the type classification and priority matching principle. Based on the allocation bandwidth and shaping rate of the optimal path, the compression parameters are adjusted, and the amount of compressed data is verified. The ordered transmission stream is formed by sorting the IoT data priority and sorting the same IoT data priority according to the generation timestamp.

[0010] Further, the process of feeding back scheduling data to the cloud is as follows: The compression-related, transmission resource-related, and transmission state-related are collected through the ordered transmission stream, and the data is pushed to the cloud through the feedback channel.

[0011] Further, the process of iterating the ordered transmission stream is as follows: After excluding abnormal data in the scheduling feedback data, the different dimension scheduling feedback data is normalized and classified to obtain effective data; Based on the effective data, the spatiotemporal attention Transformer prediction model is initialized to predict the global resource peak. The resource peak is obtained through spatiotemporal attention encoding, and the transmission stream is optimized to a global optimal solution through resource peak constraint. The feedforward scheduling instruction is generated and issued in layers for verification, and the effect is verified in a closed loop and dynamically adjusted.

[0012] The IoT data processing method based on edge computing includes the following steps: Step 1, obtain IoT data and priority factors, determine IoT data priority through priority factors, and based on IoT data priority, adaptively adapt transmission opportunity and aggregation granularity with multi-objective Q learning, and output terminal data; The priority factor includes business state, data timeliness, and edge load weighting. Step 2, predict communication link quality and switch optimal path through attention-enhanced LSTM, combine IoT data priority with optimal path allocation bandwidth and shaping rate; based on terminal data, match terminal data type to select compression algorithm, and convert terminal data into an ordered transmission stream in combination with bandwidth and shaping rate, and feedback scheduling data to the cloud; Step three, obtain edge scheduling feedback data, use space-time attention Transformer to predict global resource peak to layer reinforcement learning iteration ordered transmission flow, and send feedforward scheduling instructions downward.

[0013] The edge computing-based Internet of Things data processing method and system provided by the application have the following beneficial effects: (1) The application overcomes the pain point of high data transmission delay in traditional Internet of Things by the cooperative mechanism of terminal side prediction and edge side prediction: the terminal side lightweight preprocessing module is based on multi-objective Q learning, and the immediate transmission and small granularity aggregation strategy are adapted for ultra-high or high priority data such as device failure and early warning data, so as to avoid data timeout; the edge side prediction scheduling module adopts attention enhanced LSTM to predict link quality and switch the optimal path, and combines the dynamic bandwidth allocation of Internet of Things data priority to improve the delay compliance rate of industrial fault response, intelligent traffic control and other real-time scenes.

[0014] (2) The application solves the problems of bandwidth waste and cloud computing power overload in the traditional mode through three-layer resource optimization of data simplification, bandwidth adaptation and computing power shunting, and eliminates abnormal and redundant data such as invalid sensor values through terminal side preprocessing; the edge side matches the compression algorithm according to the data type, and adjusts the compression parameters dynamically combined with the bandwidth, so as to improve the link bandwidth utilization rate, reduce the cross-node communication volume compared with the traditional centralized training, and decentralize the cloud computing power pressure, thereby reducing the bandwidth cost and cloud computing power cost of the Internet of Things system as a whole, and adapting to large-scale equipment access scenarios.

[0015] (3) The application uses space-time attention Transformer to accurately predict the global resource peak through the global resource prediction and hierarchical cooperative optimization mechanism, obtains the trend of the change of the whole link resource, and adjusts the strategy dynamically according to different resource shortage areas through cloud, edge and terminal reinforcement learning iteration transmission flow scheme, so as to give priority to high-priority data transmission in resource severely short of areas, and compress non-critical data in advance in the bandwidth gap period. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a system schematic diagram of the application; Figure 2 It is a whole method schematic diagram of the application. DETAILED DESCRIPTION

[0017] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0018] Embodiment 1 Please refer to Figure 1 The embodiment 1 of the present application provides an Internet of Things data processing system based on edge computing, which comprises: A lightweight preprocessing module acquires Internet of Things data and priority factors, determines the priority of the Internet of Things data through the priority factors, and outputs terminal data based on the priority of the Internet of Things data, with multi-objective Q learning to autonomously adapt transmission timing and aggregation granularity. The process of acquiring Internet of Things data is as follows: For Internet of Things devices on the terminal side, such as temperature sensors and vibration sensors in industrial scenarios, and environmental temperature and humidity modules in civilian scenarios, the acquisition logic is configured according to the device type. High-frequency dynamic data such as device vibration parameters are collected at a millisecond level, and static state data such as device operating mode are collected by triggering state changes, i.e. only when the device switches from standby to running. The local interface collects raw data: Through the built-in hardware interface or lightweight protocol interface of the terminal, the raw data output by the device is read, including numerical, state, and waveform formats such as 35.2℃, running, and vibration waveform fragments, and a data generation timestamp is automatically attached, and temporarily stored in the terminal local cache. The collected raw data is subjected to preliminary legality verification, and invalid data such as character encoding disorder and field missing, such as missing timestamp, are filtered out, and only raw data in accordance with the format is retained to avoid processing invalid information in subsequent processes. The specific process of obtaining the priority factor is as follows: The priority factor includes three dimensions of business status, data timeliness, and edge load, which are obtained and quantified by different components. When obtaining the business status factor, the device itself state detection module or the associated business system is used; the real-time output business status identifier of the device is read through the terminal local state interface, such as normal operation, temperature warning, and mechanical failure of industrial devices, or the state label issued by the business system is received, such as package sorting exception in logistics scenarios; the business status is mapped to a score of 1-5, wherein normal corresponds to 1, warning corresponds to 3, and failure corresponds to 5, as the initial value of the business status factor. When obtaining the data timeliness factor, the valid time length of the current Internet of Things data is read from the business configuration rule, such as the valid time length of 10 minutes of production process data and 30 minutes of environmental monitoring data; the elapsed time is calculated in combination with the data generation timestamp and the current time, the ratio of the unit 1 minus the elapsed time to the valid time length is obtained, and when the ratio is negative, 0 is taken; the timeliness ratio is multiplied by 5 to obtain a timeliness factor score of 0-5, such as data generated for 8 minutes and valid time length of 10 minutes, the score is 1; When obtaining the edge load factor, the load monitoring module of the edge gateway receives the real-time load data of the edge node, such as CPU utilization and memory occupancy, through the periodic communication interface between the terminal and the edge every 30 seconds; the difference between the load rate and the unit 1 is multiplied by 5 to obtain an edge load factor score of 0-5, such as edge CPU utilization of 80%, the score is 1; The process of determining the priority of Internet of Things data through the priority factor is as follows: First, according to the core needs of the Internet of Things scene, the weight proportions of the business status factor, the data timeliness factor, and the edge load factor are determined, that is, the weights of the three factors are added up as a whole; according to the range of the comprehensive score, it is corresponded to the pre-set four priority levels: if the comprehensive score is between 0 and 1, it corresponds to low priority, such as non-critical environmental data, which allows delayed transmission; the score between 1 and 3 corresponds to medium priority, such as normal state data of the device; the score between 3 and 4 corresponds to high priority, such as monitoring data when the device is warning; the score between 4 and 5 corresponds to super high priority, such as core parameter data when the device fails; such as the comprehensive score calculated in the industrial failure scene, which corresponds to super high priority, meaning that this kind of data needs to be transmitted in priority.

[0019] For example, in the industrial equipment monitoring scene, the transmission of fault data is the most critical, so the weight proportion of business status is set to be the highest, the data timeliness occupies a moderate proportion, and the edge load occupies the lowest proportion; in the civil environmental monitoring scene, the data timeliness interval is large and the edge resource pressure is small, so the weight proportion of the edge load is the highest and the weight proportion of the business status is the lowest; in the general scene, the weights of the business status, the data timeliness, and the edge load are kept in a relatively balanced proportion; the weights are stored in the terminal and can also be adjusted by the edge gateway receiving the cloud adjustment instruction; The process of outputting the terminal data is as follows: Through multi-objective Q learning, the timeliness result, energy consumption result, and edge interaction feedback are balanced, that is, the data timeliness meets the standard, the terminal energy consumption is controllable, and the edge node is not congested, the required states and actions for learning are defined, the transmission time and data aggregation amount are adjusted autonomously, and finally the terminal data that can be transmitted is output; The state includes the priority of the current Internet of Things data, the terminal remaining power, and the real-time load of the edge node; Actions are divided into two categories: transmission opportunity, i.e. immediate transmission of data, delay for 2 seconds, 5 seconds or 10 seconds; aggregation granularity, i.e. single transmission, aggregation of 5, 10 or 20 data for transmission; Ensuring that the data is transmitted within the validity period, the terminal energy consumption is lower than the threshold, and the edge load is not exceeded, all give positive feedback; otherwise, give negative feedback; Combine the initial action selection of the Internet of Things data priority, set the basic action tendency for different priority data; ultra-high priority data, such as device failure data, is transmitted immediately through the initial tendency and aggregated within 5 data to avoid data timeout failure; high priority data, such as device warning data, is delayed within 2 seconds through the initial tendency and aggregated within 10 data to balance timeliness and transmission efficiency; medium priority data, such as normal operation of the device data, is delayed for 5 seconds through the initial tendency and aggregated for 15 data; low priority data, such as non-critical environmental data, is delayed for 10 seconds through the initial tendency and aggregated for 20 data to reduce terminal energy consumption; According to the selected transmission opportunity and aggregation granularity, the result set is arranged, the result set in the terminal cache is transmitted, and the time from data generation to preparation for transmission is recorded in real time, whether it is within the validity period required by the business; whether the terminal energy consumption of this transmission exceeds the preset single round energy consumption threshold; and whether the current load returned by the edge gateway is significantly increased due to this transmission; and the result set is converted into corresponding feedback signals positive or negative as the basis for adjusting Q learning; Update the action selection strategy of Q learning: According to the feedback signal, dynamically adjust the action preference in different states, if low priority data is selected to delay for 10 seconds and aggregate 20 data, but it appears to be out of time, reduce the selection weight of the action in the corresponding state; if high priority data is selected to delay for 2 seconds and aggregate 10 data, while meeting timeliness, controllable energy consumption and edge congestion, then increase the selection weight of the action; through continuous feedback and adjustment, let the action selection gradually adapt to the multi-objective requirements of the current scene; When the action selection of Q learning tends to be stable, call the optimized strategy according to the priority of the current Internet of Things data, ultra-high priority data is fixed to immediate transmission and aggregation of 3 data to ensure timeliness; medium priority data is delayed for 5 seconds and aggregated for 15 data when the edge load is low, and automatically adjusted to delay for 3 seconds and aggregate 10 data when the load is high to avoid edge congestion; according to this strategy, the data is packaged, the priority label is added, and the output is the terminal data to be transmitted, which is sent to the edge gateway side; and after each output data, the feedback and strategy updating process are repeated, and the adaptation ability of transmission opportunity and aggregation granularity is continuously optimized with the change of the scene.

[0020] The prediction scheduling module predicts the communication link quality and switches the optimal path through attention-enhanced LSTM, combines the Internet of Things data priority with the optimal path to allocate bandwidth and shaping rate; based on terminal data, matches the terminal data type to select a compression algorithm, combines the bandwidth and shaping rate to convert the terminal data into an ordered transmission stream, and simultaneously feeds back scheduling data to the cloud; The process of switching the optimal path is as follows: Through the interaction interface with each communication link, the core link quality parameters are collected in real time, including the current bandwidth of the link, the data packet loss rate, and the transmission delay, the collection frequency is set to 1 time / second, the data timeliness is ensured, at the same time, a historical data window with a length of 5 cycles is constructed, the link quality parameter sequence of the last 5 seconds is stored, and the original link data sequence is formed; and the original link data sequence is preprocessed, the abnormal values caused by communication interference are eliminated, such as the bandwidth suddenly dropping to 0 and the delay exceeding 10 times the normal range, and the parameters of different dimensions are normalized to the interval [0, 1] to eliminate the influence of dimension difference on the prediction model; The initialization and training of the attention-enhanced LSTM model, the model architecture combining attention mechanism and LSTM is adopted to solve the problem of insufficient capture of key features in long sequence by traditional LSTM; the hidden layer dimension of the LSTM network and the initial value of the attention weight matrix are set, wherein the LSTM is responsible for extracting the time-dependent features of the original link data sequence, such as bandwidth fluctuation trend and packet loss rate change law, and the attention mechanism is responsible for allocating weights to Internet of Things data at different times within the historical data window; the link quality historical data of the past 7 days, including different time periods and different load scenarios, are used to pre-train the model, so that the model learns the link change law in common scenarios; the edge gateway adjusts the model parameters every hour using the latest collected link data to ensure that the model adapts to the dynamic changes of real-time link state and avoids the decrease of prediction accuracy caused by scenario drift.

[0021] The preprocessed historical link data sequence is input into the model, the sequence is encoded through the LSTM network, and the forget gate, input gate and output gate of the LSTM work cooperatively to filter and retain key time features, such as the trend of continuous decline in bandwidth during peak hours and the feature of stable packet loss rate during low load, to generate a hidden state vector at each time; the attention mechanism performs weighted calculation on the hidden state vectors at all times, calculates the correlation between each hidden state and the current link state, assigns higher attention weights to the times that have greater impact on future link quality, such as the link data of the last 2 seconds and the times of link parameter mutation, and then sums the weighted hidden state vectors to obtain a link comprehensive feature vector that fuses key time features, greatly improving the relevance of feature expression.

[0022] The link comprehensive feature vector is input into the output layer of the model to predict the link quality parameters in the future 1-5 seconds, i.e., specific values of bandwidth, packet loss rate and delay, to obtain a prediction result; at the same time, the reliability score of the prediction result is calculated: by comparing the error between the historical prediction value of the model and the actual link quality value, if the recent prediction error is small, i.e., the average absolute error is less than 10%, the reliability score is higher than 0.8, and it is determined that the prediction result is reliable; if the error is large, i.e., the average absolute error is higher than 20%, the reliability score is lower than 0.5, at this time, the prediction time window is shortened to 3 seconds, and the data collection frequency is increased to 2 times per second, to improve the prediction stability.

[0023] The edge gateway is pre-configured with 3-5 backup communication links, and a multi-dimensional path evaluation standard is established in combination with the predicted link quality and the Internet of Things data priority. High-priority data is preferentially selected to select a link with a predicted delay < 50 ms, a packet loss rate < 2% and a bandwidth ≥ 5 Mbps, to ensure fast and low-loss data transmission. Medium or low priority data selects a link with a predicted delay < 200 ms, a packet loss rate < 5% and a bandwidth ≥ 1 Mbps, to balance transmission efficiency and resource occupation. Each candidate link is scored according to the above standard, the core index is given a high score, and the secondary index is given an auxiliary score. The link with the highest score is the optimal path. If all links do not meet the minimum requirement, such as all links have a predicted packet loss rate > 10%, the link with the relatively highest score is selected, and a bandwidth reservation request is triggered to improve the link transmission capacity.

[0024] When it is predicted that the current link quality will not meet the demand, such as the bandwidth will decrease below the threshold value in the future 3 seconds, or a better path is selected, a seamless switching process is started, a communication connection is established with the optimal path 1 second in advance, identity authentication and parameter negotiation are completed, and data that has not been transmitted in the current link is cached. The subsequent data to be transmitted is seamlessly switched to the optimal path, and the cached incomplete data is continuously transmitted through the new path to avoid data loss or repeated transmission. After the switching is completed, the actual link quality of the new path is monitored in real time, and the difference between the actual value and the predicted value is compared. If the difference is within the allowable range, i.e., the error < 15%, it is determined that the switching is successful. If the actual link quality is much lower than the predicted value, the prediction and path selection process is immediately re-executed to ensure transmission stability.

[0025] The specific process of allocating bandwidth and shaping rate is as follows: The edge gateway first synchronizes the IoT data priority and the optimal path to provide a basis for decision-making; receives the IoT data priority and priority label uploaded by the terminal, and simultaneously obtains the preset transmission constraint of each IoT data priority to clearly define the transmission bottom line requirement of different IoT data; obtains the real-time link capability data of the optimal path, including the current available bandwidth, actual transmission delay, real-time packet loss rate, and hardware limitation parameters such as the maximum carrying bandwidth and minimum stable delay of the link, and simultaneously determines the dynamic available range of the link in combination with the short-term prediction result of the attention-enhanced LSTM, that is, the future 1-3 second link quality change trend; through the bidirectional synchronization of the data side demand and the link side capability, a basic framework of demand and capability matching is constructed to avoid the bandwidth allocation and flow control from being separated from the actual scene.

[0026] S201: Set the IoT data priority basic weight; according to the preset weight proportion of the IoT data priority importance, the super-high priority is 40%, the high priority is 30%, the medium priority is 20%, and the low priority is 10%, the total weight is 100%, and it is ensured that the high priority data obtains higher resource allocation; S202: Calculate the link adaptation coefficient; for each IoT data priority, the real-time capability of the optimal path and the transmission constraint of the IoT data priority are combined to calculate the adaptation coefficient, which ranges from 0 to 1; if the current bandwidth of the link ≥ the required bandwidth of the priority, the delay ≤ the constraint delay, and the packet loss rate ≤ the constraint packet loss rate, the adaptation coefficient is 1; if one of them does not meet the requirement, such as the bandwidth is slightly lower than the requirement, the coefficient is reduced according to the difference, such as the bandwidth only meets 80% of the requirement, the coefficient is reduced to 0.8; if two or more of them do not meet the requirement, the coefficient is less than 0.5, triggering the link resource optimization request, such as bandwidth reservation; S203: Dynamically allocate available bandwidth; the final allocation weight of the basic weight of each IoT data priority is obtained by multiplying the IoT data priority basic weight × the adaptation coefficient, and the available bandwidth of the optimal path is divided according to the weight, for example, the current available bandwidth of the optimal path is 10 Mbps, the super-high priority adaptation coefficient is 1.0, and the basic weight is 40%, then 4 Mbps is allocated; the high priority adaptation coefficient is 0.9, and the basic weight is 30%, then 2.7 Mbps is allocated, and so on, and 10%-15% of dynamic redundant bandwidth is reserved for super-high or high priority to cope with burst traffic; Get each independent queue in the Internet of Things data priority, first statistics current queue data volume, including the data to be transmitted and the data being transmitted, combined with the allocated bandwidth, calculate the maximum allowed shaping rate: if the queue data volume is small, that is, the estimated transmission time is less than or equal to 50% of the delay constraint, the shaping rate is set to 80%-90% of the allocated bandwidth, and the link buffer space is reserved; if the queue data volume is large, that is, the estimated transmission time is close to 80% of the delay constraint, the shaping rate is increased to 100% of the allocated bandwidth, and the data transmission is accelerated; if the queue data volume exceeds the allowed range of the delay constraint, the priority degradation buffer is triggered, the low-priority data is temporarily stored in the local cache, and the high-priority data maintains full-speed transmission to avoid high-priority data timeout; The edge gateway monitors the running state of each priority queue and link feedback in real time, and dynamically optimizes the shaping strategy; the length, data residence time, and transmission success rate of each queue are counted every 50 milliseconds, and the actual transmission effect of the optimal path is monitored; if the high-priority queue data residence time exceeds 90% of the delay constraint, and there is still remaining link actual bandwidth, that is, the maximum carrying capacity has not been reached, temporarily increase the shaping rate of the queue to 110% of the allocated bandwidth, that is, not more than the maximum bandwidth of the link, and reduce the dynamic redundant bandwidth of the low-priority queue to prioritize high-priority data transmission; if the actual packet loss rate of the link exceeds 1.5 times of the priority constraint, such as the high-priority packet loss rate reaching 3%, reduce the shaping rate of the corresponding queue to 70% of the allocated bandwidth to reduce the transmission pressure on the link, and start the link error correction mechanism, such as retransmission strategy optimization; if the length of the low-priority queue continues to grow, and the growth exceeds 50% within 5 seconds, and there is still remaining link bandwidth, appropriately increase its allocated bandwidth proportion, with a maximum increase of 5 percentage points, to avoid long-term accumulation of data leading to expiration and invalidation.

[0027] When different priority data arrives at the same time and the link resource is tight, a conflict coordination mechanism is established, when the high-priority queue and the low-priority queue request token transmission at the same time, and the link bandwidth can only meet the full-speed transmission of a single queue, it is determined that there is a resource conflict; trigger the priority preemption and low-priority buffer mechanism, allow the high-priority queue to temporarily occupy the dynamic redundant bandwidth of the low-priority queue to complete the transmission, the data of the low-priority queue is temporarily stored in the local cache, and after the high-priority transmission is completed, the transmission is supplemented during the link idle period; if the high-priority has burst traffic and needs to occupy the basic bandwidth of the low-priority, the low-priority data is compressed and temporarily stored in advance to ensure that its core transmission rights are not long-term occupied; Set the minimum transmission guarantee threshold for all priorities, i.e. ultra-high or high priority: transmit at least 10% of the queue data every 100 milliseconds; medium or low priority: transmit at least 15% of the queue data every 1 second, to avoid data congestion for a certain priority due to continuous resource preemption; if the link quality suddenly deteriorates, such as a drop of more than 50% in available bandwidth, automatically trigger priority fuse, only keep ultra-high priority data transmission, suspend medium or low priority data transmission and cache, and then retransmit in order of priority after the link is restored.

[0028] The process of converting to an ordered transmission stream is: Sensor value types, such as temperature, vibration parameters; video stream types, such as device monitoring pictures; text or log types, such as running records; select compression algorithms according to the principles of priority matching for Internet of Things, sensor data selects LZ77 algorithm, video stream selects H.264 according to the priority of Internet of Things, balances efficiency and picture quality or H.266, ultra-high or high priority, high compression ratio to save bandwidth, text uses Deflate algorithm; select time synchronization verification algorithm, ensure that the time delay constraint of the corresponding Internet of Things priority is not exceeded.

[0029] Based on the optimal path, the bandwidth and traffic shaping rate are allocated, the compression parameters are adjusted, and the ultra-high or high priority data is configured with high compression speed and adaptive bandwidth, such as video H.266 with high compression rate to ensure picture quality, and the medium or low priority is adjusted with high compression ratio and saved bandwidth, such as sensor data to improve compression level, while the compressed data volume is checked: if a single piece of data is still compressed beyond the shaping rate carrying range, it is compressed in slices according to the bandwidth upper limit to ensure that each piece of data can be transmitted within the specified time delay.

[0030] Ordered transmission stream construction: According to the priority of data, i.e. ultra-high, high, medium, and low, and the timestamp of data generation within the same Internet of Things priority, an ordered transmission stream is formed; combined with the traffic shaping rule, the queue data is continuously pushed to the transmission channel according to the shaping rate to avoid burst traffic exceeding the bandwidth upper limit; the slice data is marked with a serial number to ensure that the receiving end can reassemble, while the data that has not been transmitted is cached to prevent loss due to link fluctuations, and finally an ordered, adaptive bandwidth, and time delay compliant ordered transmission stream is formed.

[0031] Dispatch data collection and cloud feedback: Collect three types of dispatch data in real time, including compression algorithm type, compression ratio, and compression time consumption; transmission resource related bandwidth allocation actual proportion, traffic shaping rate execution, and link utilization rate; transmission state related data transmission time delay, packet loss rate, and queue overflow situation; according to the periodic and abnormal trigger mode, push data to the cloud through a dedicated feedback channel to ensure that the cloud can grasp the edge side dispatch execution effect in real time, and provide basis for global strategy optimization.

[0032] A global optimization module acquires edge scheduling feedback data, predicts global resource peaks using a space-time attention Transformer, iterates an ordered transmission flow using hierarchical reinforcement learning, and sends downfeed scheduling instructions. The process of predicting global resource peaks is as follows: The cloud first receives all scheduling feedback data uploaded by edge gateways through a dedicated feedback channel, ensuring data integrity and timeliness. The data types collected include edge-side compression execution data, resource allocation data, transmission status data, and edge node operation data, along with data collection timestamps and edge node unique identifiers. Abnormal data, such as extreme values outside the normal range and missing fields, is first removed. Then, data of different dimensions is normalized to convert it into uniform standard feature data. Finally, the data is classified and archived by edge node jurisdiction and data collection period, and effective data that meets the model training requirements is selected to provide high-quality input for subsequent parameter aggregation.

[0033] After obtaining the effective data, the space-time attention Transformer prediction model is initialized, and the effective data is embedded into the model network layer. The prediction target is the global resource peak in the next 10-30 seconds, including the peak period of network traffic, the load bottleneck of key regional edge nodes, and the peak bandwidth occupation of core links. Historical global resources, including traffic changes, load fluctuations, and link states, are imported as model warm-up samples to ensure that the prediction model quickly adapts to the network scenario.

[0034] The space-time attention Transformer model extracts key features through a dual-dimensional attention mechanism and predicts resource trends. Spatial features, temporal features, and business features are used as inputs. Spatial features include the distribution of edge nodes, the number of terminals in the jurisdiction, and link topology relationships. Temporal features include current resource occupation data, 5-minute change trends, and historical peak patterns. Business features include the distribution of high-priority data and the status of key business operations. The spatial attention layer weights the resource correlation of different edge nodes and links, such as the mutual influence of adjacent regional edge loads. The temporal attention layer highlights key time nodes of resource changes, such as the precursor period before a sudden increase in traffic. After dual encoding, a global feature vector that integrates space-time correlation features is generated. The model outputs the details of the global resource peak in the next 10-30 seconds, including the specific time window of the peak, the edge node or link corresponding to the peak load, and the bandwidth gap estimation during the peak period. It also labels the resource tension level in each region as mild, moderate, or severe, providing clear targeting for subsequent optimization. The process of iterating an ordered transmission flow is as follows: According to the hierarchical decision framework, the cloud as a high-level agent is responsible for the global resource allocation strategy; the edge as a middle-level agent is responsible for the regional transmission flow scheduling; the terminal as a low-level agent is responsible for the local transmission parameter adjustment, and the three-level agents make collaborative decisions through information interaction; the cloud adjusts the bandwidth allocation quota of each edge node and the priority weight of high-priority data transmission according to the resource peak distribution; the edge optimizes the traffic shaping rate and queue scheduling rule of different priority data in the region based on the cloud quota and local resource state; the terminal adjusts the data aggregation granularity and transmission opportunity according to the edge instruction. The reward function takes the core of no congestion during peak period, high-priority data time limit compliance, and optimal resource utilization, and the three-level agents obtain feedback and adjust the strategy according to the target; through multiple rounds of interaction iteration, the original ordered transmission flow is optimized into a dynamic transmission flow that adapts to the resource peak; for the area with severe resource shortage, the proportion of low-priority data transmission is reduced, and the priority of high-priority data transmission is improved; for the predicted bandwidth gap period, non-critical data is compressed in advance, the transmission rhythm is adjusted, and congestion is avoided; for the low-load area, the data transmission efficiency is moderately improved, and the idle resources are fully utilized, and finally the globally optimal ordered transmission flow scheme is formed.

[0035] The process of sending the feedforward scheduling instruction is as follows: According to the receiving object, it is divided into edge instruction and terminal instruction. The instruction for the edge includes bandwidth allocation adjustment quota and traffic shaping parameter update, such as 20% improvement of high-priority data shaping rate and high-priority data queue priority configuration during peak period. The instruction for the terminal includes data aggregation granularity adjustment, such as doubling of low-priority data aggregation during resource peak period, transmission opportunity window restriction, such as transmission only during non-peak window period, and priority dynamic adjustment suggestion. All instructions are attached with execution validity period, which matches the resource peak prediction window to avoid the influence of expired instructions on scheduling. The instructions are first sent to each edge gateway, and the edge gateway receives and verifies the adaptability of the instructions to the local resource state after receiving the instructions. If there is a conflict, such as the bandwidth quota required by the instruction exceeding the maximum carrying capacity of the link, the cloud is fed back in real time and adjusted. If there is no conflict, it is synchronized to all terminals in the jurisdiction. After receiving the terminal, the local preprocessing and transmission strategy are immediately updated to ensure that the instructions are quickly implemented. After the execution of the instructions, the cloud monitors the transmission flow optimization effect in real time through the edge feedback channel, compares the difference between the actual resource occupation and the predicted peak, the high-priority data transmission compliance rate and other indicators, and if the optimization effect does not meet the expectation, such as the congestion rate during the peak period still exceeding the standard, the prediction and reinforcement learning iteration process is immediately restarted to dynamically adjust the scheduling instruction, and the transmission flow of the whole link is always adapted to the global resource state.

[0036] Embodiment 2 Please refer to Figure 2 Based on embodiment 1, the application embodiment 2 further provides an Internet of Things data processing method based on edge computing, including the following specific steps: Step one, obtain Internet of Things data and priority factor, determine the priority of Internet of Things data through the priority factor, based on the priority of Internet of Things data, adaptively transmit the transmission opportunity and aggregation granularity through multi-objective Q learning, and output terminal data; Wherein, the priority factor includes service state, data timeliness and edge load weighting; Step two, predict the communication link quality through attention enhanced LSTM and switch the optimal path, combine the Internet of Things data priority and the optimal path to allocate bandwidth and shaping rate, based on the terminal data, match the terminal data type to select the compression algorithm, combine the bandwidth and shaping rate to convert the terminal data into an ordered transmission flow, and feedback the scheduling data to the cloud; Step three, obtain edge scheduling feedback data, predict the global resource peak through space-time attention Transformer, and iteratively enhance the ordered transmission flow through hierarchical reinforcement learning, and send the feedforward scheduling instruction downward.

[0037] The above embodiments can be realized wholly or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solutions.

[0038] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, which can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiments according to actual needs.

[0039] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. An edge-computing based Internet of Things data processing system, characterized in that, The system includes: The lightweight preprocessing module acquires IoT data and priority factors, determines the priority of IoT data through the priority factors, and autonomously adapts the transmission timing and aggregation granularity based on the IoT data priority using multi-objective Q-learning to output terminal data. The priority factors include business status, data timeliness, and edge load weighting. The predictive scheduling module uses attention-enhanced LSTM to predict the quality of communication links and switch to the optimal path. It combines the priority of IoT data with the optimal path to allocate bandwidth and shaping rate. Based on terminal data, it matches the terminal data type and selects the compression algorithm. Combining bandwidth and shaping rate, it transforms the terminal data into an ordered transmission stream and feeds back scheduling data to the cloud. The global optimization module acquires edge scheduling feedback data, uses a spatiotemporal attention Transformer to predict global resource peaks, iterates the ordered transmission flow using hierarchical reinforcement learning, and sends feedforward scheduling instructions downwards.

2. The IoT data processing system based on edge computing according to claim 1, characterized in that, The process of determining the priority of IoT data using priority factors is as follows: Priority factors are determined based on IoT scenarios, including business status factors, data timeliness factors, and edge load factors. A comprehensive score is calculated based on the weight ratio of each priority factor. The comprehensive score is then assigned to one of four preset priority levels: a comprehensive score between 0 and 1 corresponds to low priority, a comprehensive score between 1 and 3 corresponds to medium priority, a comprehensive score between 3 and 4 corresponds to high priority, and a comprehensive score between 4 and 5 corresponds to ultra-high priority.

3. The IoT data processing system based on edge computing according to claim 2, characterized in that, The process of outputting terminal data is as follows: The system acquires the priority of current IoT data, the remaining battery power of the terminal, and the real-time load of the edge node, while also acquiring the transmission timing and aggregation granularity. Based on the priorities of different IoT data, the system sets the initial tendency of basic actions, including ultra-high priority data, high priority data, medium priority data, and low priority data. Based on transmission timing and aggregation granularity, the results are organized into a set that includes timeliness results, energy consumption results, and edge interaction feedback. The result set is then transformed into corresponding feedback signals. The feedback signals are used as the basis for Q-learning adjustments to dynamically adjust action preferences under different states and output terminal data.

4. The IoT data processing system based on edge computing according to claim 1, characterized in that, The process of predicting communication link quality and switching the optimal path using attention-enhanced LSTM is as follows: The system collects communication link quality parameters, including current link bandwidth, data packet loss rate, and transmission latency. Simultaneously, it constructs a historical data window with a length of 5 periods to form the original communication link data sequence. This original data sequence is preprocessed and then input into an attention-enhanced LSTM model. The LSTM network encodes the original communication link data sequence, generating a hidden state vector for each time step. An attention mechanism weights the hidden state vectors at all time steps to obtain a comprehensive link feature vector. This comprehensive feature vector is used as input to obtain the predicted link quality. Combining the predicted link quality with IoT data priority, a multi-dimensional path evaluation standard is established, with the highest-scoring path being the optimal path.

5. The IoT data processing system based on edge computing according to claim 4, characterized in that, The process of combining IoT data priority with optimal path allocation of bandwidth and shaping rate is as follows: By setting a basic weight for IoT data priority, calculating an adaptation coefficient based on the basic weight for IoT data priority, multiplying the basic weight for IoT data priority by the adaptation coefficient to obtain the final allocation weight, and allocating the available bandwidth of the optimal path according to the final allocation weight; Obtain the independent queues in each IoT data priority, count the amount of data in the current queue, and calculate the maximum allowable shaping rate based on the allocated available bandwidth.

6. The IoT data processing system based on edge computing according to claim 5, characterized in that, The process of converting terminal data into an ordered transport stream by combining bandwidth and shaping rate is as follows: The IoT priority data uploaded by the terminal is classified by type, and a compression algorithm is selected based on the matching principle of type classification and priority. The compression parameters are adjusted based on the bandwidth allocation and shaping rate of the optimal path. At the same time, the amount of compressed data is verified and sorted according to the priority of IoT data. Within the same priority of IoT data, it is sorted according to the generation timestamp to form an ordered transmission stream.

7. The IoT data processing system based on edge computing according to claim 6, characterized in that, The process of sending scheduling data back to the cloud is as follows: Data is collected in an ordered transmission stream, including compression-related, transmission resource-related, and transmission status-related data, and then pushed to the cloud through a feedback channel.

8. The IoT data processing system based on edge computing according to claim 7, characterized in that, The process of iterative ordered transport stream is as follows: The system receives scheduling feedback data through a feedback channel, removes abnormal data from the scheduling feedback data, normalizes scheduling feedback data of different dimensions, and classifies and archives the data to obtain valid data. Based on valid data, a spatiotemporal attention Transformer prediction model is initialized to predict global resource peaks. The resource peaks are obtained through spatiotemporal attention encoding, and the transport flow is optimized into the global optimal solution through resource peak constraints. Feedforward scheduling instructions are generated and distributed for verification in a hierarchical manner, and the effect is verified in a closed loop and dynamically adjusted.

9. An IoT data processing method based on edge computing, characterized in that, Includes the following steps: Step 1: Acquire IoT data and priority factors. Determine the priority of IoT data through priority factors. Based on the priority of IoT data, use multi-objective Q-learning to autonomously adapt the transmission timing and aggregation granularity, and output terminal data. The priority factors include business status, data timeliness, and edge load weighting. Step 2: Predict the quality of the communication link and switch the optimal path using attention-enhanced LSTM, and allocate bandwidth and shaping rate based on IoT data priority and the optimal path; based on terminal data, select compression algorithm according to the terminal data type, and combine bandwidth and shaping rate to transform terminal data into an ordered transmission stream, while feeding back scheduling data to the cloud. Step 3: Obtain edge scheduling feedback data, use spatiotemporal attention Transformer to predict global resource peaks, use hierarchical reinforcement learning to iterate ordered transmission flow, and send feedforward scheduling instructions downward.

Images