An internet of things system based on hierarchical decoupling and cold and hot data shunting

Abstract

The application relates to the technical field of Internet of Things data transmission and edge storage management, in particular to an Internet of Things system based on hierarchical decoupling and hot and cold data shunting, which comprises a local storage unit, a data acquisition and link detection module, a heat calculation and partition cache module, a state summary and adaptive scheduling module, and a result feedback and strategy adjustment module; bandwidth detection, heat weighted calculation and hot area, warm area and cold area shunting storage are carried out on data frames, and a differentiated scheduling rule is generated according to effective bandwidth, network abnormal state flag, partition occupancy rate and queue length; and the scheduling rule and heat calculation benchmark are adjusted according to cloud confirmation acknowledgement and local storage abnormal state flag; the application can reduce the queuing delay of key alarm data and improve the overall protection rate under the condition of limited cache.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) data transmission and edge storage management technology, specifically to an IoT system based on hierarchical decoupling and hot / cold data splitting. Background Technology

[0002] With the continuous increase in the number of IoT terminals, edge gateways need to receive and process a large amount of monitoring data from sensor nodes over a long period of time. This data includes both critical priority information such as alarms and anomalies, as well as regular data collected periodically. In application scenarios with fluctuating network bandwidth, frequent link anomalies, and limited local storage resources, how to prioritize the retention and transmission of critical data under limited caching and unstable uplink conditions has become an important problem to be solved in the design of IoT systems. Traditional IoT data upload and caching management currently mainly relies on the following methods: caching and uploading data according to a unified queue order, sending data according to a fixed priority, and performing simple overwriting or unified compression when storage space is insufficient; However, unified queue caching, fixed priority transmission, and simple overlay or unified compression all have certain drawbacks. For example, the unified queue method has difficulty distinguishing between alarm data and ordinary data, which can easily lead to queuing delays for critical data when the link is congested. The fixed priority method is not adaptable to changes in bandwidth, cache usage, and abnormal network conditions, and it is difficult to balance real-time performance and overall security. The simple overlay or unified compression method is prone to loss of important context data when the network is interrupted or storage degrades, and it is difficult to meet the requirements of priority transmission, hierarchical caching, and adaptive scheduling of critical data in complex IoT scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide an Internet of Things (IoT) system based on hierarchical decoupling and hot / cold data splitting, which avoids the queuing delay of critical data and the loss of important context data caused by unified queuing and simple overlay in traditional IoT data processing. Moreover, it can prioritize the retention and adaptive scheduling of critical data under complex working conditions such as network fluctuations and limited local storage resources, thereby reducing the queuing delay of critical alarm data and effectively improving the overall preservation rate under limited caching conditions.

[0004] The objective of this invention can be achieved through the following technical solutions: An Internet of Things (IoT) system based on hierarchical decoupling and hot / cold data splitting includes a local storage unit, a data acquisition and link detection module, a heat calculation and partition caching module, a status aggregation and adaptive scheduling module, and a result feedback and strategy adjustment module. Local storage units are divided into hot zones, warm zones, and cold zones; The data acquisition and link detection module is used to acquire data frames sent by the terminal that contain at least a timestamp, a data source identifier, and a data type identifier, determine the effective bandwidth of the current time window, generate a network anomaly flag based on the link detection results, and generate a link observation record containing the detection time, effective bandwidth, and the network anomaly flag. The heat calculation and partition caching module is used to calculate the heat of data based on data value deviation, event frequency and data source priority, and write data frames to hot zone, warm zone or cold zone accordingly, and output the partition occupancy rate and queue length information of each partition. The status summary and adaptive scheduling module is used to generate differentiated scheduling rules containing partition transmission priority, transmission rate and retry restriction policy based on effective bandwidth, network anomaly flag, partition occupancy rate and queue length information, and send data frames of each partition according to the rules; The results feedback and strategy adjustment module is used to receive cloud confirmation responses and calibration parameters, determine the hot data confirmation timeout rate, generate a local storage anomaly flag based on at least one of the storage medium verification failure records and bad block increase ratio, and adjust the differentiated scheduling rules and heat calculation benchmark according to the timeout rate and the local storage anomaly flag.

[0005] In some possible implementations, the heat calculation and partition caching module is used to obtain the deviation factor, time concentration factor and preset priority factor, and to obtain the comprehensive heat value by weighted summation based on the deviation factor, time concentration factor and preset priority factor, wherein the weight of each factor is a pre-stored non-negative coefficient, and the sum of each weight is one. The deviation factor is the value obtained after normalization of the absolute value of the difference between the current data value and the local benchmark determined for the data source; the time concentration factor is the value obtained after normalization of the frequency of occurrence of the same type of event determined based on the preset event rules within the preset time window; and the preset priority factor is the normalized priority value determined according to the data source type mapping table. When the overall heat value is greater than or equal to the preset heat data threshold, the data frame is written to the heat zone; when the overall heat value is greater than or equal to the preset temperature data threshold but less than the heat data threshold, the data frame is written to the temperature zone. When the overall heat value is less than the temperature data threshold, the data frame is written to the cold zone, where the heat data threshold is greater than the temperature data threshold.

[0006] In some possible implementations, the status aggregation and adaptive scheduling module determines the transmission priority based on the queue length information of hot, warm and cold zones and the effective bandwidth, and sends the data frames of each partition according to the determined transmission priority; When the hot zone is not empty, data frames in the hot zone are sent first; when the partition occupancy rate of the temperature zone is greater than or equal to the preset first occupancy rate threshold and the effective bandwidth is greater than or equal to the preset first minimum bandwidth threshold, data frames in the temperature zone are inserted into the transmission according to the differentiated scheduling rules during the transmission of hot zone data frames. When the hot zone is cleared and the queue length of the warm zone is greater than zero, data frames in the warm zone are sent first. When the hot zone is cleared, the partition occupancy rate of the warm zone is less than the first occupancy rate threshold, and the remaining proportion of effective bandwidth is continuously greater than the preset remaining bandwidth proportion threshold within a consecutive preset time window, the rate-limited background backhaul of cold zone data frames is executed. The remaining proportion of effective bandwidth is the ratio of the bandwidth not occupied by the allocated transmission rate within the current time window to the current effective bandwidth. The transmission rate of the rate-limited background backhaul is not higher than the preset proportion of the current effective bandwidth.

[0007] In some possible implementations, the result feedback and strategy adjustment module is used to stop the retry of uploading non-alarm data frames when the number of consecutive probe failures exceeds a preset limit and the network is determined to be in a state of complete interruption. The state of complete interruption is when the number of consecutive probe failures exceeds a preset limit and no effective bandwidth is obtained within the current time window. Hot zones retain only the latest alarm data frame, data frames with the same event identifier as the latest alarm data frame, and related data frames within the time window before and after the time window where the latest alarm data frame is located. Cold zone data is selectively retained based on the magnitude and timeliness of data frame changes. When the number of consecutive detection failures reaches the bandwidth detection failure threshold, the effective bandwidth will be locked to a fixed preset ratio of the device's pre-configured link bandwidth, and the active transmission of temperature zone data frames will be stopped.

[0008] In some possible implementations, when the partition occupancy rate of the cold zone reaches the high occupancy threshold, the heat calculation and partition caching module determines the retention priority by weighted summation based on the deviation of each data frame within the same statistical window, the change amplitude of adjacent data frames, and the timeliness correction value determined based on the timeliness of the data frames. Selective retention is performed according to retention priority, retaining only the data frames with the highest retention priority, which constitute a preset proportion. The variation amplitude of adjacent data frames is the ratio of the absolute value of the difference between the measured values ​​of adjacent data frames to the full range of the corresponding data type. Data frames with a retention priority lower than the preset retention threshold and a variation amplitude of adjacent data frames lower than the preset compression threshold are merged into statistical summary information, and the remaining unretained data frames are discarded. The statistical summary information includes at least the statistical window identifier, data type identifier, number of data frames, maximum value, minimum value, and average value.

[0009] In some possible implementations, the result feedback and strategy adjustment module is used to update the local benchmark and reference period parameters based on the calibration parameters received from the cloud within a set time window. When the time without receiving calibration parameters exceeds the set duration, the local calibration process is activated, and the median of the corresponding data values ​​of historical data frames within the most recent preset time period is selected as the new local benchmark.

[0010] In some possible implementations, the result feedback and strategy adjustment module is used to generate effective capacity update results for each partition based on the available storage space of hot, warm and cold zones under abnormal conditions when the local storage anomaly flag indicates the existence of at least one of storage media verification failure records and bad block increase ratio. The effective capacity update results include at least the effective capacity of hot zone, effective capacity of warm zone and effective capacity of cold zone. When the effective capacity of the hot zone is lower than the minimum safe capacity threshold, on the premise that the reserved capacity of the warm zone is not lower than its minimum reserved capacity, capacity is allocated from the allocable capacity of the warm zone to the hot zone, and the partition occupancy rate is re-determined based on the updated effective capacity. The status summary and adaptive scheduling module adjusts the differentiated scheduling rules based on the re-determined partition occupancy rate.

[0011] The beneficial effects of this invention are: 1. This invention calculates data heat by comprehensively considering data value deviation, event frequency, and data source priority, and accurately distributes data to hot, medium, or cold zones, breaking the limitations of the unified queue caching of traditional IoT gateways. With the status aggregation and adaptive scheduling module, the system can dynamically adjust the sending order and rate based on the effective bandwidth detected in real time and the occupancy rate of each partition. When there is link congestion or bandwidth fluctuation, it can not only prioritize the protection of critical alarm data from queuing delays, but also make full use of network resources by interleaving and rate-limited backend transmission when bandwidth is abundant, effectively balancing the real-time nature of critical data with the overall bandwidth utilization rate. 2. This invention addresses the pain points of simple overlay and uniform compression caused by complete network outages or storage space constraints. It designs a refined degradation processing mechanism: during network outages, invalid retries are stopped, and only the latest alarm frame and its preceding and following context are retained in the hot zone; simultaneously, when the cold zone is heavily occupied, selective retention is performed based on the deviation, variation amplitude, and timeliness of data frames, and the unretained regular stable data is merged into statistical summary information. This mechanism effectively prevents limited local storage from being exhausted by a large amount of repetitive stable data, maximizing the preservation of the complete development process of abnormal events and long-term data change trends under extreme conditions, and significantly improving the effective information retransmission volume after network recovery. 3. To address the issues of sensor baseline drift and local storage media aging in harsh environments, this invention introduces a local calibration and dynamic allocation mechanism for abnormal storage. When cloud calibration parameters are not received for an extended period, the system can autonomously update the local baseline using the median of historical data, preventing misjudgments of thermal activity due to baseline drift. When storage verification failures or an increase in bad blocks are detected, the system can recalculate the effective capacity based on the actual available space and, if necessary, allocate capacity from warm zones to hot zones across partitions. This mechanism ensures that even with underlying hardware degradation, the system can still prioritize maintaining the minimum safe cache limit for critical alarm data, significantly enhancing the reliability and self-healing capabilities of the device during long-term operation. Attached Figure Description

[0012] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This diagram illustrates an Internet of Things (IoT) system block diagram according to an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 An IoT system based on hierarchical decoupling and hot / cold data splitting includes a local storage unit, a data acquisition and link detection module, a heat calculation and partition caching module, a status aggregation and adaptive scheduling module, and a result feedback and strategy adjustment module; the local storage unit is divided into hot zone, warm zone, and cold zone; The data acquisition and link detection module is used to acquire data frames sent by the terminal that contain at least a timestamp, a data source identifier, and a data type identifier, determine the effective bandwidth of the current time window, generate a network anomaly flag based on the link detection results, and generate a link observation record containing the detection time, effective bandwidth, and the network anomaly flag. The heat calculation and partition caching module is used to calculate the heat of data based on data value deviation, event frequency and data source priority, and write data frames to hot zone, warm zone or cold zone accordingly, and output the partition occupancy rate and queue length information of each partition. The status summary and adaptive scheduling module is used to generate differentiated scheduling rules containing partition transmission priority, transmission rate and retry restriction policy based on effective bandwidth, network anomaly flag, partition occupancy rate and queue length information, and send data frames of each partition according to the rules; The result feedback and strategy adjustment module is used to receive cloud confirmation responses and calibration parameters, determine the hot data confirmation timeout rate, generate a local storage anomaly flag based on at least one of the storage medium verification failure records and bad block increase ratio, and adjust the differentiated scheduling rules and heat calculation benchmark according to the timeout rate and the local storage anomaly flag. The heat calculation and partition caching module is used to obtain the deviation factor, time concentration factor, and preset priority factor, and then calculates the comprehensive heat value by weighted summation based on the deviation factor, time concentration factor, and preset priority factor. The weights of each factor are pre-stored non-negative coefficients, and the sum of the weights is 1. The deviation factor is the value obtained after normalization of the absolute value of the difference between the current data value and the local benchmark determined for the data source; the time concentration factor is the value obtained after normalization of the frequency of occurrence of the same type of event determined based on the preset event rules within the preset time window; and the preset priority factor is the normalized priority value determined according to the data source type mapping table. When the overall heat value is greater than or equal to the preset heat data threshold, the data frame is written to the heat zone; when the overall heat value is greater than or equal to the preset temperature data threshold but less than the heat data threshold, the data frame is written to the temperature zone. When the overall heat value is less than the temperature data threshold, the data frame is written to the cold zone, where the heat data threshold is greater than the temperature data threshold. The status summary and adaptive scheduling module determines the transmission priority based on the queue length information and effective bandwidth of hot, warm and cold zones, and sends data frames of each partition according to the determined transmission priority; when the hot zone is not empty, data frames in the hot zone are sent first. When the partition occupancy rate of the temperature zone is greater than or equal to the preset first occupancy rate threshold and the effective bandwidth is greater than or equal to the preset first minimum bandwidth threshold, data frames in the temperature zone are inserted during the transmission of hot zone data frames according to the differentiated scheduling rules. When the hot zone is cleared and the queue length of the warm zone is greater than zero, data frames in the warm zone are sent first. When the hot zone is cleared, the partition occupancy rate of the warm zone is less than the first occupancy rate threshold, and the remaining proportion of effective bandwidth is continuously greater than the preset remaining bandwidth proportion threshold within a consecutive preset time window, the rate-limited background backhaul of cold zone data frames is executed. The remaining proportion of effective bandwidth is the ratio of the bandwidth not occupied by the allocated transmission rate within the current time window to the current effective bandwidth. The transmission rate of the rate-limited background backhaul is not higher than the preset proportion of the current effective bandwidth.

[0015] This embodiment describes a scenario where an edge gateway connects multiple terminal sensing nodes; the terminal sensing layer continuously collects environmental quantities, device quantities, or event quantities, and the terminal processor encapsulates each raw measurement value into a data frame; the data frame carries at least a timestamp, a data source identifier, and a data type identifier; In optional implementations, the frame sequence number, data length, and terminal preset priority can also be carried simultaneously; the data values ​​correspond to objective physical quantities such as temperature, current, vibration, and switch status, and subsequent heat calculation and scheduling decisions directly serve link transmission and local storage management. The gateway pre-stores event rule tables categorized by data type. Each rule includes at least an event category identifier, trigger threshold, recovery threshold, and minimum event interval. For continuous data, an event is recorded as starting when the current data value exceeds the corresponding trigger threshold, or when the absolute deviation of the current data value relative to the local benchmark is not less than the deviation trigger threshold corresponding to the data type. Only when subsequent data values ​​return to within the recovery threshold and the interval between them and the start time of the previous event is not less than the minimum event interval, can they be recorded as a new event of the same type. For switch quantity or discrete state quantity data, the transition from normal state to abnormal state is taken as the start of the event, and subsequent frames during the same abnormal period are not counted repeatedly. The frequency of events in the time concentration factor is counted based on the number of times the event starts, and is limited to the cumulative number of times within a preset time window under the same data type identifier and the same event category identifier; If a data type is not configured with separate event rules, the type of abnormal deviation event corresponding to that data type will be used as the default event category, and the event count will only be performed on frames whose absolute deviation reaches the default deviation trigger threshold. This avoids repeated accumulation when multiple consecutive frames fall in the same abnormal range. The local storage unit consists of non-volatile memory within the edge gateway, which is logically divided into hot zone, warm zone, and cold zone. The hot zone is used to temporarily store high-priority data that needs to be sent up first, the warm zone is used to store medium-priority data that can be sent up at an opportune time, and the cold zone is used to store low-priority data that can be sent back later. The partitions are maintained using an independent indexing method. Each partition maintains at least the currently occupied space and the length of the pending queue. Specifically, an index table structure is built for each partition in the gateway memory. Each index table record contains at least the partition identifier, the starting physical block address, the free block list head pointer, the used block list head pointer, the current read pointer, and the current write pointer. The initial capacity of the hot zone, warm zone, and cold zone can be allocated according to a preset ratio. It is preferable to adopt a structure in which the capacity of the hot zone accounts for 10% to 20% of the total capacity, the warm zone accounts for 30% to 40%, and the cold zone accounts for 40% to 60%, so that the limited storage is used to save more low-priority historical frames, while reserving a stable write channel for high-priority data. The data acquisition and link detection module is run periodically by the gateway processor. Its prerequisite is that the terminal data frame arrives at the gateway input interface, or the gateway reaches the detection cycle. For the received service data frame, the module first completes the frame header parsing, extracts the timestamp, data source identifier and data type identifier, and writes them into the receiving buffer. For link probing, the module sends probe packets to the cloud. The specific message structure of the probe packet includes at least a probe flag, a sender timestamp, a probe frame sequence number, and a padding payload to reach the set length. The pseudo-random byte sequence; and the effective bandwidth within the current time window is estimated based on the receipt length and round-trip delay; To reduce the impact of instantaneous jitter, continuous detection results can be smoothed to obtain a stable effective bandwidth estimate. If no detection response is received within multiple consecutive detection cycles, this situation is counted as a detection failure. When the number of failures reaches a preset limit and no effective bandwidth is obtained within the current time window, the module sets a network abnormal status flag. Preferably, the preset limit is determined by the detection cycle and the allowable disconnection time. Specifically, it can be the number of times the allowable disconnection time is divided by the single detection cycle and then rounded up, so that short-term jitter will not trigger an anomaly, while continuous disconnection can be stably identified. The resulting link observation records include at least the detection time, effective bandwidth, and network anomaly status flags, and are saved to the local status area for subsequent scheduling. To avoid mixing of units in subsequent scheduling, the effective bandwidth, transmission rate, first minimum bandwidth threshold, and upper limit of cold zone backhaul rate shall all use the same bandwidth unit, preferably bytes per second; when the external interface provides link parameters in bits per second, they shall be converted to the unified unit before being compared and allocated. To prevent fixed physical link propagation delay from being identified as a bandwidth bottleneck and causing a mismatch in the dimensions of bandwidth sample calculation, the data acquisition and link detection module avoids the calculation process of dividing the single packet length by the overall round-trip time. During each detection cycle, the modules continuously transmit payloads back-to-back. The first and second probe packets are recorded, and the receiving time of the corresponding first probe receipt returned from the cloud is recorded. Second detection receipt receiving time ; In the underlying communication of IoT gateways, serial port buffering or interrupt aggregation between the communication module and the gateway processor may cause non-physical link layer merging of reception latency, leading to... The bandwidth is less than the physical tolerance of interface hysteresis or even zero, which may lead to bandwidth overestimation or division by zero anomalies. To eliminate such timing deviations, the module determines... Relationship with interface hysteresis physical tolerance: If Greater than and If the bandwidth exceeds the preset physical tolerance for interface hysteresis, then the bandwidth sample for this test is calculated. The specific calculation formula is as follows: In the formula, This is the bandwidth sample for this test; This represents the total length of a single probe packet, where To detect the net load length of the package, The length of the protocol overhead; The time interval for receipt arrival is denoted as , where To reach the recording time, The time of sending the record; This time difference effectively filters out the constant round-trip propagation delay in the link and the fixed processing latency in the cloud, thereby restoring the true queuing service rate. If caused by interface buffer merging Not exceeding the physical tolerance for interface hysteresis, or caused by clock anomalies or missing records. Less than or equal to If the period fails to accurately reflect the air interface service rate, the bandwidth sample is recorded as 0. Smoothing is preferably performed using the most recent The moving average of each detection period, where It is an integer not less than 3; when the number of valid samples is insufficient When averaging, the average is calculated based on the number of samples obtained so far, with zero-value samples retained in the averaging process to suppress the smoothing effect of detection failures; if the most recent If there are only 0-value samples in a period, the smoothed effective bandwidth is recorded as 0. When implementing the program, it is only necessary to maintain a fixed-length queue of detection results. The effective bandwidth estimate that is reproducible and conforms to the physical facts of the fluid model can be obtained from the time difference after eliminating the hysteresis and the data length. The heat calculation and partition caching module is triggered to execute after each frame of business data arrives; its inputs include the current data value, the local benchmark of the data source, the frequency of similar events in the statistics window, and the preset priority corresponding to the data source type. The local benchmark is a local reference value formed by the gateway for the recent historical data of the data source, which is the median of the historical samples; or when the ratio of the range to the arithmetic mean of the historical data samples within a preset time window is less than or equal to the range ratio threshold, the arithmetic mean of the historical data samples within the preset time window is used as the local benchmark. The deviation factor is obtained by normalizing the absolute value of the difference between the current data value and the local benchmark. Specifically, the absolute value is divided by the maximum fluctuation range of the historical samples of the data source, and the deviation factor is set to zero when the maximum fluctuation range is zero to prevent abnormal division by zero calculation. The time concentration factor is obtained by normalizing the number of occurrences of similar events within a preset time window. Specifically, the number of occurrences is divided by a predefined saturation frequency threshold. If the threshold is exceeded, the factor is truncated to 1. The preset priority factor is obtained from the data source type mapping table; for example, safety interlock or over-limit alarm sources can be mapped to the preset first priority, and general periodic collection quantities can be mapped to the preset second priority, where the preset first priority is greater than the preset second priority. Specifically, the data source type mapping table divides data sources into... Each discrete priority level, level number The range of values ​​is to positive integers, where greater than an integer, and the grade number A larger value indicates higher priority; the formula for calculating the preset priority factor for the corresponding data source type is: In the formula, Preset priority factor; The aforementioned deviation factor, time concentration factor, and preset priority factor are all limited to a closed interval between 0 and 1. When the deviation factor is less than 0, it is treated as 0, and when it is greater than 1, it is treated as 1. The time concentration factor and preset priority factor are also truncated in the same way to ensure that indicators of different dimensions are comparable within the same heat space. To avoid duplicate counting due to inconsistency between the event frequency statistics window and the data frame writing rhythm, the module preferably uses a fixed sliding window containing the timestamp as the unique counting window, and maintains the event category identifier and start time of the most recently counted event for each data source; If the current frame is only a subsequent frame in the same event's duration, the occurrence count will not be increased; only the accumulated window count value will be used in the calculation of the time lumped factor for this frame. Under a set of feasible parameters, the weights of the deviation factor, time concentration factor, and preset priority factor are all non-negative, and the sum of the three is one. Preferably, the weight of the deviation factor can be greater than the weight of the time concentration factor, and the weight of the time concentration factor should not be lower than the weight of the preset priority factor, so as to highlight the impact of abnormal deviation values ​​on the heat determination. In engineering implementation, the weights for deviation, time concentration, and preset priority can be 0.5, 0.3, and 0.2 respectively, or an allocation relationship close to this ratio with a total sum of 1 can be chosen; the module calculates the comprehensive heat value using the following formula: In the formula, the pre-stored weights , , satisfy ; When the overall heat value is not lower than the heat data threshold, the module writes the data frame to the heat zone; when the overall heat value is between the temperature data threshold and the heat data threshold, it writes it to the temperature zone. When the overall heat value is lower than the temperature data threshold, it is written to the cold zone; the heat data threshold is always greater than the temperature data threshold; preferably, the heat data threshold is taken in the high range and the temperature data threshold is taken in the middle and low range, so that significantly abnormal frames enter the heat zone, trend frames enter the temperature zone, and normal stable frames enter the cold zone. To ensure that the thresholds can be implemented, both the hot data threshold and the warm data threshold are determined statistically based on labeled samples or trial operation samples before system deployment: first, the distribution of comprehensive heat values ​​is calculated based on historical data, then the lower limit of the heat distribution corresponding to the data frame that can represent alarms or significant anomalies is set as the hot data threshold, and the lower limit of the heat distribution corresponding to the data frame that needs to retain trend information but has not reached the alarm level is set as the warm data threshold. When there are no labeled samples, we can first take the values ​​in the range of 0.7 to 0.9 for hot data and the values ​​in the range of 0.3 to 0.6 for warm data, and make small adjustments based on the results of the response and cache usage during the operation period. After a data frame is written to the corresponding partition, the module immediately updates the number of bytes occupied by the partition, the partition occupancy rate, and the queue length information, and submits these results to the status aggregation and adaptive scheduling module. The status summary and adaptive scheduling module reads link observation records, partition occupancy rates, partition queue lengths, and network anomaly status flags in each scheduling cycle. The differentiated scheduling rules are a set of rules that give each of the three partitions a sending priority, sending rate, and retry restriction policy. The generation process is not directly fixed, but dynamically determined according to the current effective bandwidth and buffer pressure; if there are data frames to be sent in the hot zone, the hot zone is given the highest sending priority and the sending resources are allocated to the hot zone first, while the warm zone and cold zone are in a waiting state. If the hot zone is empty and the temperature zone is close to full load, and the effective bandwidth reaches the first minimum bandwidth threshold, then the temperature zone obtains the interval transmission condition; if the hot zone has been cleared, the temperature zone occupancy rate is lower than the first occupancy rate threshold, and the remaining effective bandwidth ratio is continuously higher than the preset remaining bandwidth ratio threshold in multiple consecutive statistical windows, then the cold zone enters the rate-limited background backhaul state. The effective bandwidth remaining ratio is the ratio of the bandwidth not occupied by the allocated transmission rate within the current time window to the current effective bandwidth; the scheduling module first calculates the reserved transmission rate of the hot zone, and then determines whether the cold zone can be inserted for transmission based on the remaining bandwidth, and allocates backhaul bandwidth to the cold zone only when the remaining capacity continues to exist. The cold zone backhaul rate is not higher than a preset proportion of the current effective bandwidth, thereby avoiding cold data from filling the uplink channel; preferably, the preset proportion is no more than a portion of the effective bandwidth, so that once a frame to be sent reappears in the hot zone, the cold zone backhaul can immediately give up bandwidth. The first occupancy threshold, the first minimum bandwidth threshold, the remaining bandwidth ratio threshold, and the preset ratio are all preset according to the acceptable latency and link capacity on the device side. The first occupancy threshold can be determined according to the buffer margin reserved before the temperature zone reaches full load, preferably 0.6 to 0.85 of the effective capacity of the temperature zone. The first minimum bandwidth threshold can be determined based on the bandwidth requirement after accommodating at least one typical frame of the temperature zone and retaining the hot zone confirmation overhead within a single scheduling cycle; the remaining bandwidth ratio threshold is used to constrain the cold zone to send only when there is bandwidth surplus, preferably 0.2 to 0.5; the preset ratio of the cold zone rate-limited backend transmission is preferably 0.1 to 0.3. The above thresholds can be slightly adjusted after deployment, taking into account the hot zone confirmation timeout rate and the backlog changes in each zone over multiple consecutive scheduling cycles. However, the order of hot zones first, warm zones second, and cold zones remains unchanged after adjustment. To prevent the ratio from being undefined when the current effective bandwidth is 0, when the current effective bandwidth is less than or equal to 0, the remaining effective bandwidth ratio is directly recorded as 0, and it is considered that the conditions for interleaved transmission in the warm zone and back-office transmission in the cold zone are not met; at this time, the minimum necessary transmission or retry channel in the hot zone is reserved only if the network abnormal status flag is not prohibited. To ensure that the differentiated scheduling rules have a definite generation path, the scheduling module executes in the following order in each scheduling cycle: first, read the queue lengths of hot zone, warm zone, and cold zone; if the queue length of hot zone is greater than 0, then set the hot zone sending priority to 1, the warm zone sending priority to 2, and the cold zone sending priority to 3. If the hot zone queue length is 0 and the warm zone queue length is greater than 0, then the warm zone transmission priority is set to 1 and the cold zone transmission priority is set to 2; if both the hot and warm zone queue lengths are 0, then the cold zone transmission priority is set to 1; the module first calculates and allocates a determined reserved transmission rate for the hot zone. The calculation formula is as follows: In the formula, For the current effective bandwidth, The amount of data to be sent corresponds to the length of the hotspot queue. The duration of a single scheduling cycle. Set a preset bandwidth reservation factor (e.g., a value of 100%). ), and when Greater than At that time, take equal ; If the remaining bandwidth after allocation is still not less than the first minimum bandwidth threshold, and the temperature zone occupancy rate is greater than or equal to the first occupancy rate threshold, then after continuously sending a preset number of hot zone data frames, insert and send one temperature zone data frame. The preset number is preferably an integer between 3 and 10. If the hot zone is empty and the cold zone backhaul condition is met, then the cold zone transmission rate is limited to the result of the current effective bandwidth multiplied by a preset ratio. For retry restriction policies, hot zone data frames are allowed to be retried a preset number of times when the network abnormal status flag is not set, the number of retry allowed for warm zone data frames is no higher than that for hot zone data frames, and cold zone data frames do not perform more than one additional retry in background backhaul mode. If the network anomaly flag is set, both the warm and cold zones will stop actively retrying, and only the hot zone's low-speed retry channel will be retained. Following the order of first determining priority, then allocating rate, and finally determining the number of retries, the corresponding control flow program can be written directly. To avoid exceeding the allocated rate by simply inserting gaps based on the number of frames under different frame lengths, the module maintains a transmit byte budget based on cross-cycle accumulation in each scheduling cycle; If a fixed single-cycle independent budget is used, under extremely low transmission rates or short scheduling cycles, the single-cycle budget will be less than the length of a single frame of data, which will lead to the inability to meet the premise of not exceeding the budget in the long term and cause queue starvation and scheduling stagnation. Therefore, the transmit byte budget for the current period is obtained by summing the remaining transmit byte budget of the corresponding partition in the previous period and the allocation increment for this period, which is the transmit rate of the corresponding partition multiplied by the duration of the scheduling period, and an upper limit for the budget accumulation of each partition is set. When inserting a warm zone data frame and performing a cold zone backhaul, the premise is that the remaining transmitted byte budget after the current accumulation is not less than the length of the current data frame to be transmitted, and the corresponding frame length budget is deducted after the data frame is successfully written into the transmission buffer. If the remaining budget is insufficient to send a single frame, the transmission of that partition in the current cycle is suspended, and the remaining budget is safely rolled over to the next cycle; thus, the risk of scheduling stagnation is eliminated by utilizing the physical traffic cross-cycle constraint mechanism. During actual transmission, considering that the speed at which the processor writes data to the communication module's transmission buffer is usually much greater than the physical link's external bandwidth, in order to avoid the queuing delay of the underlying communication buffer causing the application layer's interleaved transmission and timely termination logic to fail due to time axis misalignment, the scheduling module establishes synchronous back pressure control based on water level feedback. The module continuously reads the number of unsent bytes in the current transmission buffer of the communication module. Only when the number of unsent bytes is lower than the reserved minimum water level will it extract the data frame from the head of the corresponding partition queue and write it into the transmission buffer. If the hot zone is not empty, the hot zone data frames are extracted and written continuously first. During the process of filling the hot zone frames into the sending buffer, only when the hot zone occupancy rate reaches the first occupancy rate threshold and the effective bandwidth is not lower than the first minimum bandwidth threshold, the hot zone data frames are interspersed after the hot zone frames are written continuously according to the differentiated scheduling rules. This allows scheduling strategies to be truly implemented at the communication channel level, rather than just at the application memory level, avoiding long-term accumulation of hot zones while ensuring the hot zone priority principle. If the hot zone is cleared and the temperature zone pressure is low, the module will check the remaining bandwidth ratio in the continuous statistics window. After confirming that the conditions are met, the water level will be controlled according to the speed limit rules to transmit the cold zone segment data back. During the cold zone transmission process, since the transmission buffer is always kept at a shallow level, once a new data frame arrives in the hot zone or the effective bandwidth drops below the threshold, the cold zone transmission is immediately stopped, and there will be no large amount of untransmitted cold data blocking the bottom layer of the communication module. The unfinished fragmentation progress can be retained without loss until the next allowed window to continue processing. The results feedback and strategy adjustment module is responsible for receiving the confirmation response and calibration parameters returned from the cloud. For the confirmation response, the module reads at least the frame sequence number, the received result and the return time, and updates the pending confirmation index accordingly. For hot zone frames, if no acknowledgment is received within the preset acknowledgment time limit, the frame is included in the timeout statistics, thus obtaining the hot data acknowledgment timeout rate. An increase in the hot data acknowledgment timeout rate means that the current link is congested or the acknowledgment quality has decreased. Therefore, the module writes it back to the status summary and adaptive scheduling module as a scheduling correction. The scheduling module tightens the temperature zone insertion conditions accordingly, suspends cold zone backhaul when necessary, and prioritizes the transmission and confirmation of hot zones; the preset confirmation time limit is preferably determined based on the statistical value of the round-trip delay obtained from the detection, which can be 2 to 5 times the smooth round-trip delay, and superimposed with the single frame transmission delay and the cloud processing retention delay, so as to avoid the normal receipt being mistakenly recorded as timeout due to the confirmation window being too short. The result feedback and strategy adjustment module also reads the verification failure records and bad block increase ratio of the storage medium; the edge gateway can perform read-only verification for hot, warm and cold zones according to a preset verification cycle; The bad block increase rate is calculated by dividing the difference between the total number of bad blocks detected after the current verification cycle and the total number of bad blocks detected after the previous verification cycle by the total number of physical blocks in the local storage unit. Once a verification failure record is found, or the increase in bad blocks exceeds the preset limit, a local storage abnormal status flag is generated. The local storage abnormal status flag and the hot data confirmation timeout rate work together to participate in scheduling correction; when the former is set, the system can reduce the number of attempts to send non-critical data and reduce frequent read and write operations. When the latter increases, the system can reduce the scheduling ratio of the warm zone and the cold zone; at the same time, the calibration parameters sent from the cloud can also be used to update the local benchmark and the heat calculation benchmark, so that the subsequent calculation of the deviation factor, time concentration factor and comprehensive heat value are closer to the current working conditions. To avoid merely describing the adjustment of timeout rate and local storage abnormal status flags based on hot data, the result feedback and strategy adjustment module preferably outputs scheduling correction results and hot calculation benchmark correction results according to hierarchical rules. Specifically, the hot data confirmation timeout rate is first divided into three intervals: low, medium and high. The low interval is below the first timeout rate threshold, the medium interval is above or equal to the first timeout rate threshold but below the second timeout rate threshold, and the high interval is above or equal to the second timeout rate threshold, where the second timeout rate threshold is greater than the first timeout rate threshold. When the local storage abnormal status flag is not set in the low range, the original differentiated scheduling rules remain unchanged; when the local storage abnormal status flag is not set in the middle range, the temperature zone insertion threshold is increased, that is, temperature zone insertion is only allowed when the effective bandwidth is higher than the original first minimum bandwidth threshold and the temperature zone occupancy rate is higher than the original first occupancy rate threshold. At the same time, the upper limit of the cold zone backhaul rate is lowered to the first configuration coefficient of the original preset ratio, and the first configuration coefficient is less than the original preset ratio. When in the high range, cold zone backhaul is suspended, and the warm zone interleaving transmission is changed from allowing one frame to be inserted per round to allowing one warm zone data frame to be inserted only after more hot zone data frames have been sent consecutively; if the local storage abnormal status flag is set, it will be further tightened on the basis of any of the above ranges, that is, priority will be given to reducing warm zone and cold zone read / write and transmission attempts. For the calculation benchmark of data heat, the module does not change the heat calculation structure, but only corrects at least one of the local benchmark, saturation frequency threshold or hot data threshold and warm data threshold: when the hot data confirmation timeout rate remains in the high range for multiple consecutive scheduling cycles, it is preferable to increase the preset correction step size of hot data threshold and warm data threshold to reduce non-critical frames entering the high priority partition. When cloud calibration parameters indicate a shift in the field baseline, it is preferable to update the local baseline; when similar events become more frequent across multiple windows, it is preferable to update the saturation frequency threshold. The above-mentioned small step size correction is preferably implemented using a fixed step size table: if it enters the middle interval, the selected correction item in the hot data threshold and the warm data threshold is increased by a maximum of 0.01 to 0.03 in each scheduling cycle; if it enters the high interval and remains there for at least two consecutive scheduling cycles, it is increased by a maximum of 0.03 to 0.05 each time. When the timeout rate falls back to the low range and remains there for at least two scheduling cycles, it is gradually restored by a backtracking step size no larger than the upward adjustment step size. For the correction of the saturation frequency threshold, it is preferable to multiply the median of the number of times the same type of event has been confirmed to have started in the most recent statistical windows by a preset amplification factor to obtain a new value, and treat the new value as 1 if it is less than 1. Thus, both the scheduling correction and the heat calculation benchmark correction have clear inputs, clear judgment conditions and clear output results. With the above structure, a clear data flow is formed from the terminal's generation to the gateway partition, then to link scheduling and acknowledgment feedback: The terminal collects physical quantities and forms data frames. The gateway parses the frame header and estimates the bandwidth. The heat calculation module provides the hot zone, warm zone, and cold zone distribution results. The scheduling module determines the specific sending order and rate based on the bandwidth and buffer pressure. The feedback module then corrects the next round of heat calculation and scheduling rules based on cloud confirmation and storage health status. This avoids low-priority periodic data from occupying the sending queue for a long time, thereby reducing the queuing latency of critical alarm data and improving the overall preservation rate under limited buffer conditions.

[0016] In a preferred embodiment of the present invention, the result feedback and strategy adjustment module is used to stop the retry of uploading non-alarm data frames when the number of consecutive probe failures exceeds a preset limit and the network is determined to be in a state of complete interruption. The state of complete interruption is when the number of consecutive probe failures exceeds the preset limit and no effective bandwidth is obtained within the current time window. Hot zones retain only the latest alarm data frame, data frames with the same event identifier as the latest alarm data frame, and related data frames within the time window before and after the time window where the latest alarm data frame is located. Cold zone data is selectively retained based on the magnitude and timeliness of data frame changes. When the number of consecutive detection failures reaches the bandwidth detection failure threshold, the effective bandwidth will be locked to a fixed preset ratio of the device's pre-configured link bandwidth, and the active transmission of temperature zone data frames will be stopped. When the partition occupancy rate of the cold zone reaches the high occupancy threshold, the heat calculation and partition caching module determines the retention priority by weighted summation based on the deviation of each data frame in the same statistical window, the change amplitude of adjacent data frames, and the timeliness correction value determined based on the timeliness of the data frames. Selective retention is performed based on retention priority, retaining only the top-priority items at a predetermined ratio. The data frames are defined as follows: the variation range of adjacent data frames is the ratio of the absolute difference between the measured values ​​of adjacent data frames to the full range of the corresponding data type; data frames whose retention priority is lower than the preset retention threshold and whose variation range of adjacent data frames is lower than the preset compression threshold are merged into statistical summary information, and the remaining unretained data frames are discarded. The statistical summary information includes at least the statistical window identifier, data type identifier, number of data frames, maximum value, minimum value and average value.

[0017] This implementation describes the degradation process when there is a network outage, bandwidth detection failure, or high cold zone occupancy. This process is still executed by the edge gateway and involves the result feedback and policy adjustment module, the heat calculation and partition caching module, and the local storage unit. Its processing objects are still the service data frames sent by the terminal. The measurement values ​​in these data frames correspond to the objective operating parameters of the field equipment. Therefore, the retention and compression strategies directly affect the amount of information that can be recovered later. The result feedback and strategy adjustment module continuously reads the link probe results; when the number of consecutive probe failures exceeds the preset limit and no effective bandwidth is obtained within the current time window, the module determines that the network is in a state of complete interruption. A complete interruption is not a single probe failure, but is constituted by two conditions: first, the number of probe failures continues to accumulate and exceeds the limit; second, the effective bandwidth within the current window is empty. Only when both conditions are met will the system enter the upload retry restriction mode. Once in this mode, the module stops retrying the upload of non-alarm data frames, which means stopping repeated upload attempts of warm and cold zone data frames, thus preventing the communication module from continuously occupying processor and storage read / write resources on invalid links. In a complete interruption state, the hot zone retention rule is switched to the alarm association retention rule; the hot zone no longer maintains the complete history of the ordinary pending queue, but only retains the latest alarm data frame, the data frame with the same event identifier as the latest alarm data frame, and the associated data frames in the adjacent windows before and after the time window where the latest alarm data frame is located. Since alarm information and short-term correlation data before and after the alarm characterize the fault occurrence process, alarm information and short-term correlation data are retained to prevent ordinary data from crowding out hotspot space. During execution, since each node in the hot zone index table explicitly stores the timestamp of the data frame, the data source identifier, the event identifier, and the physical address offset of the corresponding storage unit, the result feedback and policy adjustment module first compares the timestamp field by traversing the hot zone index table to find the latest alarm data frame in time. Then, using the event identifier of the frame as the search key, a matching search is performed in the hot zone index table to filter out the index nodes of related frames of the same event; the statistical window to which the alarm frame belongs is expanded forward and backward by one adjacent window, and the data related to the event in these windows are included in the reserved set. Other old frames in the hot zone that are not in the retention set will have their indexes released or be transferred to subsequent processing; this ensures that critical alarms and necessary contexts can still be saved even in the interruption state. The associated data frame is preferably limited to a data frame that has at least one of the same data source identifier, the same device identifier, or the same event identifier as the latest alarm data frame; when the same frame meets multiple association conditions at the same time, it is only retained once to avoid duplicate counting. Data within the next time window is only included in the retention set after the business data in that next time window arrives at the gateway and is indexed; Before the next time window is formed, the hot zone first retains the latest alarm data frame, the same event data frame, and the related data frame in the previous time window. The corresponding related set is then filled in after the data of the subsequent window arrives, so as to avoid logical pre-retention in the window that has not yet occurred in time. For cold zones, the result feedback and strategy adjustment module does not perform full retention, but instead triggers selective retention based on the magnitude and timeliness of data frame changes; The variation range is the result of normalizing the absolute amount of the difference between measured values ​​between adjacent data frames. Specifically, it is the dimensionless ratio obtained by dividing the absolute amount of the difference between measured values ​​by the full range of the corresponding data type. Timeliness is a value that is quantitatively adjusted based on frame age to influence the weight of subsequent data analysis. Data closer to the current statistical period typically has a higher retention weight. The specific quantification process involves subtracting the ratio of the data frame age to the predefined longest tolerable latency from the baseline maximum weight value of 1. Furthermore, data frames with an age greater than or equal to the maximum tolerable delay are truncated. When the network is completely interrupted, this process is beneficial for prioritizing the retention of cold data that has changed significantly and is relatively new, so as to restore the trend change process as much as possible after the network is restored, without the storage space being occupied by a large number of stable and repetitive values. The above-mentioned change range and timeliness correction value are all limited to the range of 0 to 1; when the time interval between adjacent frames is abnormally long or the corresponding data type lacks full range configuration, the change range is replaced by the median of the change range already calculated in the statistical window. If a valid value still cannot be obtained, the change range is set to 0 and only the other indicators are used for subsequent sorting. To ensure that selective retention processing in a fully interrupted state has a defined control flow even when the cold zone occupancy rate has not yet reached the high occupancy threshold, the result feedback and strategy adjustment module preferably compares the change magnitude threshold and the timeliness threshold: if the change magnitude of a cold zone data frame is greater than or equal to the interruption retention change threshold, or its timeliness correction value is greater than or equal to the interruption retention timeliness threshold, then the data frame is directly retained. If both are below the corresponding threshold, the data frame is merged into the same statistical window and the same data type identifier of the set to be summarized. When the set already has statistical summary information, only the number of data frames, maximum value, minimum value and average value are updated. When the cold zone occupancy rate reaches the high occupancy threshold, the retention priority sorting process is then performed on the original data frames that have already been retained. In this way, the two-level rules of direct threshold filtering and sorting for optimal retention correspond to the completely interrupted state and the cold zone high occupancy state, respectively, and the input conditions and processing branches remain fixed. When the number of consecutive detection failures reaches the bandwidth detection failure threshold, the system enters the conservative bandwidth estimation mode. At this time, the result feedback and strategy adjustment module locks the effective bandwidth to a fixed preset ratio of the device's pre-configured link bandwidth and stops the active transmission of temperature zone data frames. The fixed preset ratio is preferably set to a lower value to make the bandwidth estimation conservative, so as to avoid the hot zone retrieval deterioration caused by organizing the transmission according to the normal bandwidth when the detection fails; after the active transmission of the warm zone is stopped, the warm zone only performs buffering and waiting, and no longer participates in the interleaved transmission. The hot zone maintains low-speed uplink attempts under conservative bandwidth to preserve the basic availability of the alarm channel; the bandwidth detection failure threshold is preferably greater than the preset limit for determining a complete interruption state, specifically set to 1.5 to 3 times the preset limit; the fixed preset ratio is preferably 0.1 to 0.3 of the device's pre-configured link bandwidth to ensure that the bandwidth estimate is always conservatively conservative to the normal link capacity. The handling of high occupancy in cold zones is performed by the heat calculation and partition caching module; its prerequisite is that the occupancy rate of cold zones reaches the high occupancy threshold. The high occupancy threshold is preferably set in the range that is close to full load but still has room for compression operations, so as to complete the selective retention before the storage is completely full; after the threshold is detected, the module no longer writes all new frames directly in sequence, but calculates the retention priority for each data frame in the same window in the form of a statistical window. Preferably, the high occupancy threshold can be 0.75 to 0.9 of the effective capacity of the cold zone; when the cold zone occupancy rate continues to rise to a higher range above the high occupancy threshold, the module can simultaneously lower the preset ratio described below to accelerate the release of space; The retention priority is determined by a weighted summation formula based on the degree of deviation, the magnitude of change between adjacent data frames, and the timeliness correction value; the specific calculation formula is as follows: Among them, the pre-stored non-negative weight coefficients , , satisfy The degree of deviation reflects the extent to which a single frame of data deviates from a local reference, specifically by dividing the absolute value of the difference between it and the local reference by the maximum allowable fluctuation range. During the transition phase when the local benchmark has not been updated due to calibration timeout, the median of the frames retained in the current statistical window can be temporarily used as a substitute benchmark, and the retention priority of subsequent windows can be recalculated according to the local benchmark after the local benchmark is restored. The magnitude of change between adjacent data frames reflects whether the frame is in a significant upward, downward, or abrupt transition process; the timeliness correction value is given according to the frame age, and the closer the frame is to the current time period, the higher the correction value; To avoid being dominated by a single indicator, the deviation degree is selected as the highest weight among the three weights, followed by the change magnitude, and then the timeliness correction value; this way, outliers are preserved while also taking into account trend turning points and more recent representative frames. Preferably, the degree of deviation, the variation amplitude of adjacent data frames, and the timeliness correction value are normalized to the range of 0 to 1 before being included in the weighting; when the default engineering configuration is used, , and The values ​​can be 0.5, 0.3, and 0.2 respectively, or a distribution relationship that is close to this ratio and whose sum is 1 can be chosen. After the module completes the priority calculation for each frame, it sorts the data frames in the same statistical window from high to low priority and retains only the data frames ranked at the top of the preset proportion. The preset ratio can be set to a lower ratio according to the pressure of the cold zone, so as to significantly reduce the occupancy; the data frames that remain after sorting continue to be stored in the cold zone, and the data frames that do not enter the retention set will enter the second judgment. To ensure that the pre-set ratio yields only a unique result in the program, if the number of data frames within a certain group is... The preset ratio is Then the number of frames retained in this group Pick and The product is rounded up to the nearest integer, and... If less than 1, treat it as 1; when Greater than Press at time deal with; If it is located at the th Name and Rank If data frames with the same retention priority are retained, the final retention boundary will be determined by prioritizing timestamp updates and then prioritizing those with higher deviation. During the second determination, if the retention priority of a non-retained data frame is lower than the retention threshold, and the change between it and its adjacent data frames is also lower than the compression threshold, it means that the frame is neither abnormal nor represents a significant change. The module merges it with other similar frames under the same statistical window and the same data type into statistical summary information. The statistical summary information should include at least the statistical window identifier, data type identifier, number of data frames, maximum value, minimum value, and average value. The average value is preferably obtained by arithmetically averaging the data values ​​of the original data frames being merged. In the program implementation, this can be calculated by simultaneously maintaining the cumulative sum and the number of data frames. When new mergeable data frames continue to enter the same statistical window and the same data type of summary set, only the cumulative sum, number of data frames, maximum and minimum values ​​are updated, and then the average value is recalculated; in this way, even if the original detailed frames are merged, the cloud can still see the overall trend range within the window after receiving resumes. The preset ratio, retention threshold, and compression threshold are all determined before equipment deployment, taking into account the cold zone capacity and the target amount of supplementary data. The preset ratio is preferably 0.1 to 0.4, the retention threshold is preferably 0.2 to 0.5, and the compression threshold is preferably 0.01 to 0.05 of the full range of the corresponding data type. When the range of different data types differs significantly, the compression threshold is configured separately for each data type to avoid accidental deletion caused by different types sharing the same change threshold; To facilitate direct reproduction of selective retention processing, cold zone processing is preferably performed in a fixed order: first, group the cold zone data frames according to the statistical window and data type, and then arrange them in ascending order of timestamp within each group; For each sorted frame, calculate the change magnitude between it and the previous frame. For the first frame, use the change magnitude of the next frame as a substitute. If there are no adjacent frames before or after, the change magnitude of that frame is recorded as 0. The frame age is obtained based on the difference between the current time and the timestamp of the frame, and the frame age is converted into a timeliness correction value according to the predefined maximum tolerable delay. The timeliness correction value exceeding the maximum tolerable delay is directly truncated to 0. After calculating the three indicators, the retention priority of each frame in the same group is determined and sorted. If two frames have the same retention priority, the frame with the updated timestamp is retained first. If the timestamps are still the same, the frame with the greater deviation is retained first. Data frames outside the set are retained and processed according to the secondary judgment rule. Among them, data frames that can be merged into statistical summary information must simultaneously meet the requirements of belonging to the same statistical window and having the same data type identifier. When summarizing, the module accumulates the quantity frame by frame, updates the maximum and minimum values, and calculates the average value. Data frames that do not meet the merging criteria are directly deleted from their corresponding indices; through the above steps of grouping, sorting, comparing, summarizing, and deleting, the inputs, outputs, and execution order of selective retention processing remain determined; For the remaining unretained data frames, if they neither enter the retention set nor meet the conditions for generating statistical summary information, they are discarded directly. Data frames with no significant deviation and no effective change are discarded to free up storage space; the entire process is executed by the gateway processor on the local storage index without cloud involvement, making it suitable for independent operation during network anomalies. Through this implementation, when the network is completely interrupted or bandwidth detection fails, the system can prioritize the use of limited storage for alarm data and its associated context, and reduce the risk of storage space exhaustion caused by smooth repetitive data through cold zone selective retention and statistical summary generation mechanisms. This not only maintains the local preservation capability of critical alarms, but also increases the amount of effective retransmission information after the network is restored.

[0018] In a preferred embodiment of the present invention, the result feedback and strategy adjustment module is used to update the local benchmark and reference period parameters based on the calibration parameters when the calibration parameters returned from the cloud are received within a set time window. When the time without receiving calibration parameters exceeds the set duration, the local calibration process is activated, and the median of the corresponding data values ​​of historical data frames within the most recent preset time period is selected as the new local benchmark. The result feedback and strategy adjustment module is used to generate effective capacity update results for each partition based on the available storage space of hot, warm and cold zones under abnormal conditions when the local storage anomaly flag indicates the existence of storage media verification failure records and bad block increase ratio. The effective capacity update results include at least the effective capacity of hot zone, effective capacity of warm zone and effective capacity of cold zone. When the effective capacity of the hot zone is lower than the minimum safe capacity threshold, on the premise that the reserved capacity of the warm zone is not lower than its minimum reserved capacity, capacity is allocated from the allocable capacity of the warm zone to the hot zone, and the partition occupancy rate is re-determined based on the updated effective capacity. The status summary and adaptive scheduling module adjusts the differentiated scheduling rules based on the re-determined partition occupancy rate.

[0019] This implementation describes calibration parameter updates and partition capacity adjustments under storage anomalies. The execution entity is still the result feedback and policy adjustment module in the edge gateway, and the relevant inputs come from cloud calibration parameters, local historical data frames, and storage medium verification results. Its purpose is to keep heat determination and cache scheduling available even when link conditions change, sensor baselines drift, or the health status of storage media deteriorates. The results feedback and strategy adjustment module checks whether the cloud has returned calibration parameters in each parameter receiving cycle; the calibration parameters are used at least to update the local benchmark and reference cycle parameters. The local benchmark is a reference value for a certain data source at the current stage, used for subsequent deviation factor calculation; the reference period parameter is the typical sampling interval of the data source under normal conditions, used for subsequent time lumped factor or frequency correlation calculation. When the preconditions are met, i.e., when the cloud calibration parameters are successfully received within the set time window, the module first parses the parameter package, identifies the corresponding data source identifier, the updated local reference value, and the updated reference period parameter, and writes these parameters into the local configuration table. When the heat calculation and partition caching module processes the data frame of the data source again, it directly reads the updated parameters and recalculates the deviation factor and time concentration factor, so that the diversion results of hot zone, warm zone and cold zone are adjusted according to the changes in the field working conditions. If the duration of no calibration parameters received exceeds the set duration, the result feedback and strategy adjustment module will activate the local calibration process. Activating local calibration does not immediately replace the reference frame by frame, but rather extracts the corresponding data values ​​from historical data frames within the most recent preset time period, sorts them by value, and takes the median as the new local reference. Using the median as the local reference helps suppress offsets caused by occasional sensor fluctuations or short-term sudden changes. This reduces the chance of a large number of normal frames being misclassified as hot data due to baseline drift; the reference period parameter can be updated based on the sampling interval statistics of recent consecutive frames during local calibration, so that the time concentration factor still reflects the current rhythm of the data source. The time window, duration, and most recent preset time period should preferably be determined by combining the normal rate of change of the data source and the frequency of cloud parameter updates; For data sources with a change rate lower than the preset change rate threshold, the set duration is configured as the first set duration. For data sources with a change rate higher than the preset change rate threshold or safety-related data sources, the set duration is configured as the second set duration, and the second set duration is less than the typical operating condition change cycle of the data source. To avoid median distortion due to insufficient historical samples, when the number of valid historical data frames in the most recent preset time period is lower than the minimum number of samples, the module continues to use the previous local benchmark and performs local calibration after supplementing with subsequent samples; the minimum number of samples is preferably no less than 5 frames. To enable the calibration update process to be executed directly, the result feedback and strategy adjustment module preferably maintains independent parameter record items for each data source. The record items should include at least the time of the most recent successful reception of cloud calibration parameters, the current local benchmark, the current reference cycle parameter, and the historical data index within the most recent preset time period. The historical data index is maintained in the underlying software implementation using a fixed-length circular buffer queue data structure. Each node of the buffer queue contains a timestamp and a corresponding data value field, and the valid samples for the most recent preset time period are maintained by using head and tail pointers to chase and overwrite the oldest node. When receiving calibration parameters from the cloud, the corresponding record item is located first according to the data source identifier, and then parameter validity check, writing new local benchmark, writing new reference period parameter and updating timestamp are performed in sequence. The parameter validity check includes at least whether the local reference falls within the allowable range of the data type and whether the reference period parameter is greater than zero; calibration parameters that fail the validity check are not written to the local configuration table, but are retained as valid parameters from the previous round. When calibration parameters are not received for more than the set time, the local calibration process is executed in a fixed order: first, the valid data values ​​of the most recent preset time period are extracted from the historical data index, and frames with missing or duplicate timestamps are removed. Then, the median of the sampling interval of adjacent valid frames is calculated in ascending order of timestamps as the new reference period parameter, and the median is taken as the new local reference after sorting by numerical value. If the number of valid samples is less than the minimum number of samples, the parameters of the previous local benchmark and the previous reference period remain unchanged. In this way, the input conditions, exception handling, and write-back objects for cloud calibration and local calibration are all governed by clear rules. The results feedback and strategy adjustment module monitors local storage anomaly status flags; these flags indicate at least one of the following: the existence of verification failure records and the increase in the proportion of bad blocks in the storage medium. When the flag is set, it means that the original three-part capacity can no longer be used simply according to the nominal capacity. The module needs to generate an effective capacity update result based on the available storage space of the hot zone, warm zone and cold zone under abnormal conditions. The specific approach is as follows: the module reads the number of physical blocks in each partition that can still be read and written normally after verification, deducts the part marked as verification failure or bad block growth, and obtains the effective capacity of the hot zone, the effective capacity of the warm zone, and the effective capacity of the cold zone respectively. The occupancy rate of the three partitions is recalculated based on the effective capacity rather than the original capacity; the resulting occupancy rate better reflects the true storage pressure under abnormal conditions. If any partition has more bytes occupied than the updated effective capacity after the effective capacity is updated, the partition's occupancy rate is recorded as 1 and it is set to an overloaded state. At the same time, new writes or append caches to the partition are suspended until the number of bytes occupied is reduced to no more than the updated effective capacity through sending, transferring, compressing or deleting. This processing method can avoid occupancy rate values ​​greater than 1 after the denominator is reduced, which would cause distortion in subsequent scheduling comparisons. When the effective capacity of the hot zone is lower than the minimum safe capacity threshold, the module initiates capacity allocation; the minimum safe capacity threshold represents the minimum space required for the hot zone to maintain critical alarm buffering functions, which is estimated based on the number of critical frames that the hot zone must retain at least and the typical frame length. Before allocating, the module first checks the reserved capacity of the temperature zone to confirm that the temperature zone will not be lower than its minimum reserved capacity after the allocation. If this condition is met, a portion of the capacity is allocated from the available capacity of the temperature zone to the hot zone. After the allocation is completed, the effective capacity of the hot zone increases, the effective capacity of the temperature zone decreases accordingly, and the effective capacity of the cold zone remains unchanged. The module recalculates the updated occupancy rates of the hot zone, warm zone, and cold zone, and writes the new occupancy rates back to the status summary in the adaptive scheduling module. Preferably, the minimum safe capacity threshold can be buffered at least continuously by hot zone. The requirement for key alarm data in frames has been determined, among which The minimum retention capacity for a temperature zone is determined based on the number of alarm frames that may be generated within the longest permissible period of disconnection; the minimum retention capacity for a temperature zone can be determined based on the requirement to retain trend data for at least one or more statistical windows. If the available capacity of the temperature zone is insufficient to supplement the hot zone to the minimum safe capacity threshold, the module will prioritize allocating all available capacity of the temperature zone to the hot zone, and simultaneously increase the transmission priority of the hot zone and tighten the writing and transmission conditions of the temperature zone and cold zone to form a protection closed loop in the case of insufficient capacity. To avoid the lack of executable boundaries when allocating capacity from the allocable capacity of the temperature zone to the hot zone, capacity allocation is preferably carried out in the following order: First, calculate the capacity gap for the hot zone, which is the minimum safe capacity threshold minus the current effective capacity of the hot zone; then calculate the allocable capacity for the warm zone, which is the current effective capacity of the warm zone minus the minimum reserved capacity of the warm zone. If the allocable capacity of the temperature zone is less than or equal to 0, no allocation will be performed, and the zone will only be set to a state of insufficient capacity. If the allocable capacity of the temperature zone is greater than 0 and not less than the capacity gap, a one-time allocation will be completed according to the capacity gap. If the allocable capacity of the temperature zone is greater than 0 but less than the capacity gap, all the allocable capacity of the temperature zone will be allocated to the hot zone. To avoid triggering large-scale indexing and erasing operations of non-volatile memory that take up to seconds due to the pursuit of physical boundary continuity, and to prevent the real-time writing of front-end alarm frames from being severely blocked under high-voltage system conditions, capacity allocation is implemented by decoupling logical storage units from continuous physical blocks. When allocating, the logical storage unit is used as the smallest unit, and the discrete free block pointer linked list of each partition is directly operated: the module directly removes the storage block pointers that meet the allocation requirements from the free block linked list of the warm zone, and appends them to the available free block linked list of the hot zone. If there are not enough free blocks in the warm zone and some old data space needs to be released, the corresponding index record of the oldest data block in the warm zone is directly cancelled and its block pointer is forcibly transferred to the hot zone. Since no physical data transfer occurs, a single capacity adjustment operation can be reduced to a preset low latency range, avoiding the blockage of control link timing caused by slow input / output reconstruction at the underlying level. After the allocation is completed, the result feedback and strategy adjustment module immediately recalculates the occupancy rate of the three zones and sends the changes in the hot zone occupancy rate and the changes in the temperature zone occupancy rate to the status summary and adaptive scheduling module. The latter will release the hot zone frames to be sent in the next scheduling cycle based on this information. By providing capacity gap calculation, allocable capacity calculation, and pointer-based allocation mechanism, partition capacity adjustment can be efficiently and with low blocking by the program. After the status summary and adaptive scheduling module reads the newly determined partition occupancy rate, it adjusts the differentiated scheduling rules according to the determined parameter mapping logic. For example, when the effective capacity of a hot zone is reduced, causing the hot zone occupancy rate to rise to a preset alarm threshold, the scheduling module increases the allowed retries for the hot zone by a set number based on the original preset number, and tightens the transmission conditions for warm and cold zones. The specific execution rules are as follows: The first minimum bandwidth threshold that allows insertion of data frames in the warm zone is increased by a preset step size, for example, by 20%. The first occupancy threshold is increased to a higher range, for example, 0.9. At the same time, the upper limit of the transmission rate of the cold zone backend is set to half of the current effective bandwidth preset ratio, so as to give absolute priority to releasing hot zone space. Conversely, when the effective capacity of a temperature zone is reduced due to capacity allocation, its occupancy rate may also increase. If the length of the hot zone queue is 0 at this time, the scheduling module will appropriately increase the sending opportunities of the temperature zone. That is, the rule of inserting a temperature zone frame only after sending a preset number of consecutive hot zone data frames will be temporarily changed to prioritize sending the data at the front of the temperature zone queue continuously until its occupancy rate falls back to below the first occupancy rate threshold, so as to avoid the temperature zone from being squeezed for writing due to insufficient remaining space. Therefore, the capacity changes after partition migration directly alter the transmission scheduling behavior through specific control flow and parameter update boundaries, eliminating ambiguous functional adjustments; In an actionable storage anomaly scenario, after the edge gateway periodically performs read-only verification, it finds that some physical blocks in the hot zone cannot pass the verification, and the proportion of bad blocks increases to a preset limit. The result feedback and strategy adjustment module sets the local storage abnormal status flag accordingly and recalculates the effective capacity of the three partitions. If the statistical results show that the effective capacity of the hot zone is lower than the minimum safe capacity threshold, while the warm zone still has spare space higher than its minimum reserved capacity, the module allocates a portion of the allocatable capacity from the warm zone to supplement the hot zone. After the adjustment is completed, the heat calculation and partition caching module continues to maintain the occupancy rate according to the new effective capacity, while the status summary and adaptive scheduling module prioritizes releasing hot zone frames to be sent; this can prioritize maintaining the local caching and uploading capabilities of critical alarm data when storage health declines. By adopting this implementation method, the problem of mispartition caused by baseline drift can be reduced by combining cloud calibration and local median calibration. When a storage medium experiences a verification failure or an increase in bad blocks, the effective capacity can be recalculated and the partition allocation adjusted in a timely manner, ensuring that hot zones still maintain the necessary minimum cache capacity under abnormal operating conditions. This can improve the stability of the system during long-term operation and reduce the risk of loss of critical data due to expired parameters or storage degradation.

Claims

1. An Internet of Things (IoT) system based on hierarchical decoupling and hot / cold data splitting, characterized in that, include: Local storage units are divided into hot zones, warm zones, and cold zones; The data acquisition and link detection module is used to acquire data frames sent by the terminal that contain at least a timestamp, a data source identifier, and a data type identifier, determine the effective bandwidth of the current time window, generate a network anomaly flag based on the link detection results, and generate a link observation record containing the detection time, effective bandwidth, and the network anomaly flag. The heat calculation and partition caching module is used to calculate the heat of data based on data value deviation, event frequency and data source priority, and write data frames to hot zone, warm zone or cold zone accordingly, and output the partition occupancy rate and queue length information of each partition. The status summary and adaptive scheduling module is used to generate differentiated scheduling rules containing partition transmission priority, transmission rate and retry restriction policy based on effective bandwidth, network anomaly flag, partition occupancy rate and queue length information, and send data frames of each partition according to the rules; The result feedback and strategy adjustment module is used to receive cloud confirmation responses and calibration parameters, determine the hot data confirmation timeout rate, generate a local storage anomaly flag based on at least one of the storage medium verification failure records and bad block increase ratio, and adjust the differentiated scheduling rules and heat calculation benchmark according to the timeout rate and the local storage anomaly flag.

2. The Internet of Things system based on hierarchical decoupling and hot / cold data splitting according to claim 1, characterized in that, The heat calculation and partition caching module is used to obtain the deviation factor, the time concentration factor and the preset priority factor, and to obtain the comprehensive heat value H by weighted summation of the deviation factor, the time concentration factor and the preset priority factor, wherein the weight of each factor is a pre-stored non-negative coefficient and the sum of the weights is 1. The deviation factor is a normalized value obtained by normalizing the absolute value of the difference between the current data value and the local benchmark determined for the data source. The time concentration factor is a normalized value obtained by normalizing the frequency of occurrence of similar events determined based on preset event rules within a preset time window. The preset priority factor is a normalized priority value determined according to the data source type mapping table. When the overall heat value is greater than or equal to the preset heat data threshold, the data frame is written to the heat zone; When the comprehensive heat value is greater than or equal to the preset temperature data threshold and less than the heat data threshold, the data frame is written to the temperature zone; When the overall heat value is less than the temperature data threshold, the data frame is written to the cold zone, wherein the heat data threshold is greater than the temperature data threshold.

3. The Internet of Things (IoT) system based on hierarchical decoupling and hot / cold data splitting according to claim 1, characterized in that, The status aggregation and adaptive scheduling module determines the transmission priority based on the queue length information of hot zone, warm zone and cold zone and the effective bandwidth, and sends the data frames of each partition according to the determined transmission priority; When the hot zone is not empty, data frames in the hot zone are sent first. When the partition occupancy rate of the temperature zone is greater than or equal to the preset first occupancy rate threshold and the effective bandwidth is greater than or equal to the preset first minimum bandwidth threshold, data frames in the temperature zone are inserted during the transmission of hot zone data frames according to the differentiated scheduling rules. When the hot zone is cleared and the queue length of the warm zone is greater than zero, data frames in the warm zone are sent first. When the hot zone is cleared, the partition occupancy rate of the warm zone is less than the first occupancy rate threshold, and the remaining proportion of effective bandwidth is continuously greater than the preset remaining bandwidth proportion threshold within a consecutive preset time window, the rate-limited background backhaul of cold zone data frames is executed. The remaining proportion of effective bandwidth is the ratio of the bandwidth not occupied by the allocated transmission rate within the current time window to the current effective bandwidth. The transmission rate of the rate-limited background backhaul is not higher than the preset proportion of the current effective bandwidth.

4. The Internet of Things system based on hierarchical decoupling and hot / cold data splitting according to claim 1, characterized in that, The result feedback and strategy adjustment module is used to stop the retry of uploading non-alarm data frames when the number of consecutive detection failures exceeds a preset limit and the network is determined to be in a state of complete interruption. The state of complete interruption is when the number of consecutive detection failures exceeds a preset limit and no effective bandwidth is obtained within the current time window. The hot zone retains only the latest alarm data frame, data frames with the same event identifier as the latest alarm data frame, and associated data frames within the previous and next time windows relative to the time window where the latest alarm data frame is located. The cold zone data is selectively retained based on the magnitude and timeliness of data frame changes. When the number of consecutive detection failures reaches the bandwidth detection failure threshold, the effective bandwidth will be locked to a fixed preset ratio of the device's pre-configured link bandwidth, and the active transmission of temperature zone data frames will be stopped.

5. The Internet of Things system based on hierarchical decoupling and hot / cold data splitting according to claim 2, characterized in that, When the partition occupancy rate of the cold zone reaches the high occupancy threshold, the heat calculation and partition caching module determines the retention priority by weighted summation based on the deviation of each data frame in the same statistical window, the change amplitude of adjacent data frames, and the timeliness correction value determined based on the timeliness of the data frames. Selective retention is performed according to the retention priority, retaining only the top-ranked proportions of the retention priority. Data frames; Wherein, the variation amplitude of adjacent data frames is the ratio of the absolute amount of the difference between the measured values ​​of adjacent data frames to the full range of the corresponding data type; Data frames whose retention priority is lower than a preset retention threshold and whose adjacent data frame change amplitude is lower than a preset compression threshold are merged into statistical summary information. The remaining unretained data frames are discarded. The statistical summary information includes at least a statistical window identifier, a data type identifier, the number of data frames, the maximum value, the minimum value, and the average value.

6. The Internet of Things system based on hierarchical decoupling and hot / cold data splitting according to claim 1, characterized in that, The result feedback and strategy adjustment module is used to update the local benchmark and reference period parameters based on the calibration parameters returned from the cloud within a set time window. When the calibration parameters are not received for a period of time exceeding the set duration, the local calibration process is activated, and the median of the corresponding data values ​​of historical data frames within the most recent preset time period is selected as the new local benchmark.

7. The Internet of Things system based on hierarchical decoupling and hot / cold data splitting according to claim 1, characterized in that, The result feedback and strategy adjustment module is used to generate effective capacity update results for each partition based on the available storage space of hot zone, warm zone and cold zone under abnormal conditions when the local storage anomaly flag indicates the existence of at least one of storage medium verification failure record and bad block increase ratio. The effective capacity update results include at least the effective capacity of hot zone, effective capacity of warm zone and effective capacity of cold zone. When the effective capacity of the hot zone is lower than the minimum safe capacity threshold, on the premise that the reserved capacity of the warm zone is not lower than its minimum reserved capacity, capacity is allocated from the allocable capacity of the warm zone to the hot zone, and the partition occupancy rate is re-determined according to the updated effective capacity. The status summary and adaptive scheduling module adjusts the differentiated scheduling rules based on the re-determined partition occupancy rate.

Images