Iot edge node big data collaborative processing system

Abstract

The present application relates to the technical field of big data, and particularly to an Internet of Things edge node big data collaborative processing system, which comprises: collecting real-time running state data of each edge node in an Internet of Things edge computing system, wherein the real-time running state data comprises a task queue backlog length, a node residual energy parameter and communication channel state information, and a multi-dimensional queuing system model is constructed based on the real-time running state data.In the present application, a time interval sliding window is used to capture data distribution characteristics, which can identify network jitter and real node failure precursors.A node failure dynamic risk assessment index is calculated based on the data distribution characteristics, which quantifies the node health condition, and task migration is triggered immediately when the risk index is abnormal, a predictive takeover mechanism is used to ensure the continuity and integrity of big data processing tasks, eliminate the risk of data loss caused by sudden downtime of edge nodes, and improve the robustness of the collaborative computing architecture.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a big data collaborative processing system for IoT edge nodes. Background Technology

[0002] Traditional big data processing architectures deployed at the edge often rely on static thresholds or single-dimensional CPU load metrics for task allocation, neglecting the limited battery life of edge devices and the susceptibility of wireless channels to interference. This leads to high-energy-consuming tasks being scheduled to nodes with low battery levels, causing these nodes to prematurely run out of power and go offline. Furthermore, existing technologies generally use fixed timeout mechanisms to determine node liveness, which are prone to misjudgments due to frequent random fluctuations in edge networks and channel congestion, forcibly marking healthy nodes as offline and causing unnecessary task interruptions and resource instability. Therefore, improvements are needed. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing an IoT edge node big data collaborative processing system.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: The IoT edge node big data collaborative processing system includes:

[0005] Real-time operating status data of each edge node in the IoT edge computing system is collected. The real-time operating status data includes task queue backlog length, node remaining energy parameters and communication channel status information. A multi-dimensional queuing system model is constructed based on the real-time operating status data.

[0006] Based on the multidimensional queuing system model, a Lyapunov drift plus penalty function is defined. The Lyapunov drift plus penalty function is used to solve the scheduling decision of the big data tasks to be processed in each edge node, and the target task processing mode of each edge node is determined. The target task processing mode includes local processing mode, neighboring edge node offloading mode and cloud upload mode.

[0007] During the execution of the big data task according to the target task processing method, the heartbeat arrival time data of each edge node is continuously monitored through a non-fixed period heartbeat detection mechanism, and a time interval sliding window is maintained based on the heartbeat arrival time data.

[0008] Calculate the node failure dynamic risk assessment index based on the data distribution characteristics within the time interval sliding window, determine the health status of each edge node based on the node failure dynamic risk assessment index, and trigger the task migration mechanism to take over the pending tasks in each edge node when the health status is determined to be abnormal.

[0009] Preferably, the steps for constructing a multi-dimensional queuing system model based on the real-time operating status data are as follows:

[0010] Obtain the task queue backlog length of each edge node, whereby the task queue backlog length represents the total number of data packets waiting to be processed in the buffers of each edge node in the current decision time slot;

[0011] The remaining energy parameters of each edge node are collected synchronously. These remaining energy parameters reflect the current battery life of each edge node and the duration of high-load computing that it can support.

[0012] Real-time detection of the communication channel status information, which includes the uplink signal-to-noise ratio between the edge node and the cloud server and the quality of the direct communication link between the edge node and its neighboring edge nodes;

[0013] The multidimensional queuing system model is constructed based on the task queue backlog length, the node's remaining energy parameters, and the communication channel state information.

[0014] Preferably, the steps for defining a Lyapunov drift plus penalty function based on the multidimensional queuing system model, and using the Lyapunov drift plus penalty function to solve the scheduling decision for the big data tasks to be processed in each edge node, are as follows:

[0015] Construct a Lyapunov function to characterize half of the sum of squares of the task queue backlog length;

[0016] A penalty term is set, which is the weighted sum of total system energy consumption and task processing latency, wherein the total system energy consumption is calculated based on computing energy consumption and transmission energy consumption, and the task processing latency is calculated based on queuing latency and computing latency;

[0017] The change of the Lyapunov function between two adjacent decision time slots is defined as Lyapunov drift. The Lyapunov drift is combined with the penalty term to obtain the Lyapunov drift plus penalty function.

[0018] Within each decision time slot, the optimal control variable is solved by minimizing the supremum of the Lyapunov drift plus penalty function, and the control variable is the target task processing method for the current big data task.

[0019] Preferably, the steps for determining the target task processing method for each edge node, including local processing mode, neighboring edge node offloading mode, and cloud upload mode, are as follows:

[0020] The task scheduling path is determined based on the minimization result of the Lyapunov drift plus penalty function:

[0021] When the solution result indicates that the local computing overhead is minimal, the local processing mode is selected, and the local CPU resources of each edge node are directly called to process the big data task.

[0022] When the solution results indicate that the local load of each edge node is too high and the neighboring edge nodes have idle resources that can minimize the Lyapunov drift plus penalty value, the neighboring edge node offload mode is selected, and the big data task is divided and transmitted to the neighboring edge nodes for collaborative computing through the device-to-device communication link;

[0023] When the solution result indicates that neither the local processing mode nor the adjacent edge node offloading mode can meet the latency constraints or energy consumption budget, the cloud upload mode is selected, and the big data task is sent to the central cloud server for centralized processing via a wide area network link, and the processing result is sent back to each edge node.

[0024] Preferably, during the execution of the big data task according to the target task processing method, the step of continuously monitoring the heartbeat arrival time data of each edge node through a non-fixed-period heartbeat detection mechanism and maintaining a time interval sliding window based on the heartbeat arrival time data specifically includes:

[0025] A mutual trust detection connection is established between each edge node, and the receiving node records the actual arrival time of each heartbeat packet.

[0026] Calculate the time difference between the heartbeat packet received at the current moment and the heartbeat packet received at the last moment, and store the time difference as the heartbeat arrival time data in the time interval sliding window;

[0027] The time interval sliding window has a fixed sampling length. When new heartbeat arrival time data is generated, the oldest historical data in the time interval sliding window is removed.

[0028] The maximum likelihood estimation method is used to fit all sample data within the time interval sliding window to obtain a normal distribution model, which is described by the mean parameter and the standard deviation parameter.

[0029] Preferably, the step of calculating the node failure dynamic risk assessment index based on the data distribution characteristics within the time interval sliding window, and determining the health status of each edge node based on the node failure dynamic risk assessment index, specifically includes:

[0030] Obtain the actual waiting time since the last heartbeat packet was received at the current time, and calculate the tail probability of not receiving a heartbeat packet after the actual waiting time based on the normal distribution model;

[0031] The dynamic risk assessment index for node failure is constructed by combining the tail probability with historical fluctuation characteristics;

[0032] The formula for calculating the node failure dynamic risk assessment index is as follows: ,in, This represents the dynamic risk assessment index for node failure; The tail probability is the probability value calculated based on the normal distribution model that the heartbeat interval is greater than the actual waiting time. It is a logarithmic function with base 10; Indicates the weighting coefficient of the probability term; This represents the weighting coefficient of the volatility term; The parameter representing the standard deviation of the data within the time interval sliding window; The parameter represents the mean value of the data within the sliding window of the time interval.

[0033] Preferably, the step of triggering a task migration mechanism to take over the pending tasks in each edge node when the health status is determined to be abnormal specifically includes:

[0034] A first risk threshold and a second risk threshold are preset, wherein the value of the second risk threshold is greater than the value of the first risk threshold;

[0035] The calculated node failure dynamic risk assessment index is compared with the first risk threshold. If the node failure dynamic risk assessment index exceeds the first risk threshold but does not exceed the second risk threshold, then each edge node is determined to be in a suspected failure state.

[0036] In the suspected fault state, a pre-migration operation is initiated to asynchronously copy the intermediate computing state data of the tasks in the memory of each edge node to the neighboring edge node as a backup, but without interrupting the current computing process of each edge node.

[0037] If the node failure dynamic risk assessment index continues to rise and exceeds the second risk threshold, then each edge node is determined to be in a confirmed failure state.

[0038] In the confirmed fault state, the connection with each edge node is forcibly disconnected, and the neighboring edge node is activated to directly take over and continue to execute the unfinished computing task using the copied intermediate computing state data of the task.

[0039] Preferably, in the suspected fault state, the step of initiating a pre-migration operation to asynchronously copy the intermediate computation state data of the tasks in the memory of each edge node to the neighboring edge node as a backup specifically includes:

[0040] Identify the key context information of the big data task currently being processed by each edge node, including program counter values, register states, and stack memory data;

[0041] The key context information is incrementally compressed and encoded to generate a lightweight state snapshot package;

[0042] The status snapshot packet is sent to the nearest neighboring edge node with the lowest physical distance and lowest load via a device-to-device communication link;

[0043] Upon receiving the status snapshot packet, the neighboring edge node temporarily stores it in the cache area and continuously updates the status snapshot packet until it receives a formal takeover instruction or the suspected fault state is resolved.

[0044] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0045] This invention collects task queue backlog length, remaining energy parameters, and communication channel status information from each node of an IoT edge computing system to construct a multi-dimensional queuing system model. This model comprehensively quantifies the dynamic load and resource bottlenecks of the edge environment, avoiding scheduling imbalances caused by a single indicator. A Lyapunov drift plus penalty function is introduced to solve the multi-dimensional queuing model, achieving a Pareto optimal balance between maintaining queue stability and optimizing energy consumption and latency. Based on the solution results, local processing, neighbor collaboration, and cloud offloading modes are flexibly switched to maximize resource utilization and minimize computational latency. During execution, a non-fixed-period heartbeat detection mechanism continuously tracks heartbeat arrival times, and a sliding window over time intervals captures data distribution characteristics, enabling the identification of network jitter and early signs of node failure. Based on data distribution characteristics, a dynamic risk assessment index for node failure is calculated to quantify node health status. When the risk index is abnormal, task migration is triggered immediately. A predictive takeover mechanism ensures the continuity and integrity of big data processing tasks, eliminates the risk of data loss due to sudden edge node downtime, and improves the robustness of the collaborative computing architecture. Attached Figure Description

[0046] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] Please see Figure 1 The present invention provides a technical solution: an IoT edge node big data collaborative processing system comprising:

[0049] Real-time operational status data of each edge node in the IoT edge computing system is collected. The real-time operational status data includes task queue backlog length, node remaining energy parameters, and communication channel status information. A multi-dimensional queuing system model is constructed based on the real-time operational status data.

[0050] Based on the multidimensional queuing system model, a Lyapunov drift plus penalty function is defined. The Lyapunov drift plus penalty function is used to solve the scheduling decision of the big data tasks to be processed in each edge node, and to determine the target task processing mode of each edge node. The target task processing mode includes local processing mode, neighboring edge node offloading mode and cloud upload mode.

[0051] During the execution of big data tasks according to the target task processing method, a non-fixed periodic heartbeat detection mechanism is used to continuously monitor the heartbeat arrival time data of each edge node, and a time interval sliding window is maintained based on the heartbeat arrival time data.

[0052] The dynamic risk assessment index for node failure is calculated based on the data distribution characteristics within the time interval sliding window. The health status of each edge node is determined based on the dynamic risk assessment index for node failure. When the health status is determined to be abnormal, the task migration mechanism is triggered to take over the pending tasks in each edge node.

[0053] In this embodiment, the steps for constructing a multi-dimensional queuing system model based on real-time operating status data are as follows: obtaining the task queue backlog length of each edge node, which represents the total number of data packets waiting to be processed in the buffer of each edge node in the current decision time slot; synchronously collecting the node remaining energy parameters of each edge node, which reflect the current battery endurance and the duration of high-load computing that each edge node can support; real-time detection of communication channel status information, which includes the uplink signal-to-noise ratio between the edge node and the cloud server and the quality of the direct communication link between the edge node and its neighboring edge nodes; and constructing a multi-dimensional queuing system model based on the task queue backlog length, node remaining energy parameters, and communication channel status information.

[0054] Specifically, by reading the hardware register values ​​of the network interface card driver in the edge node's network interface controller, or by directly accessing the socket buffer descriptor in the operating system kernel protocol stack, the number of all received but not yet scheduled data packets to be processed by the computing unit at the current moment is counted. This value is used as the task queue backlog length, which reflects the data lag between the network layer and the application layer. By accessing the register interface of the power management integrated circuit (PMIC), the current battery voltage and discharge current values ​​are read at a millisecond sampling frequency. The historical discharge amount is integrated using the coulomb calculation method. Combined with the battery aging characteristic curve and the current temperature environmental factor, the remaining power percentage is estimated. Based on the current node's average power consumption benchmark, the estimated remaining available time under full load operation is calculated, i.e., the node's remaining energy parameter. Using the pilot signal emitted by the physical layer, the Received Signal Strength Indication (RSSI) and Signal-to-Noise Ratio (SNR) are continuously measured within a preset detection period. For the edge... The connection between the node and the cloud server is evaluated by sending Internet Control Message Protocol (ICMP) echo requests and calculating round-trip time and packet loss rate. Combined with the uplink physical bandwidth limitation, the uplink signal-to-noise ratio between the edge node and the cloud server is assessed. At the same time, the device discovery protocol is enabled to scan other edge nodes within the preset communication range, establish temporary point-to-point test links, send test data blocks of a predetermined length, and measure the time required for transmission completion and the bit error rate. This quantifies the quality of the direct communication link between the edge node and its neighboring edge nodes. The task queue backlog length collected above is used as the congestion dimension in the state space, the node's remaining energy parameter is used as the energy constraint dimension, and the communication channel state information is mapped as a variable factor of the service rate. A state transition probability matrix is ​​defined, which describes the probability of transitioning from one queue length state to the next possible state under the current decision time slot. A multi-dimensional queuing system model is constructed based on the task queue backlog length, the node's remaining energy parameter, and the communication channel state information.

[0055] In this embodiment, a Lyapunov drift plus penalty function is defined based on a multidimensional queuing system model. The specific steps for solving the scheduling decision of the big data tasks to be processed in each edge node using the Lyapunov drift plus penalty function are as follows: A Lyapunov function is constructed to represent half of the sum of squares of the task queue backlog length. A penalty term is set, which is the weighted sum of the total system energy consumption and the task processing delay. The total system energy consumption is calculated based on the computation energy consumption and transmission energy consumption, and the task processing delay is calculated based on the queuing delay and computation delay. The change of the Lyapunov function between two adjacent decision time slots is defined as the Lyapunov drift. The Lyapunov drift and penalty term are combined to obtain the Lyapunov drift plus penalty function. In each decision time slot, the optimal control variable is solved by minimizing the supremum of the Lyapunov drift plus penalty function. The control variable is the target task processing method for the current big data task.

[0056] Specifically, define a scalar function Its form is the sum of the squares of the backlog lengths of all edge node task queues at the current moment divided by two. As a Lyapunov function, its physical meaning lies in amplifying the effect of larger queue lengths through the squared term, causing the optimization process to prioritize eliminating longer queue backlogs, thereby ensuring the strong stability of the system. A non-negative control parameter is set. This parameter The weighting coefficient, used as a penalty term, is employed to adjust the system's preference between pursuing queue stability and optimizing energy consumption and latency. For example, it can be used to... Setting it to

[100] means the system prioritizes energy consumption and latency optimization, while having a relatively high tolerance for queue backlog. The total system energy consumption is calculated, consisting of two parts: one is the energy consumption generated by local computation, calculated by multiplying the number of CPU cycles required for the computation task by the CPU's effective capacitance coefficient, then by the square of the operating frequency and the square of the voltage; the other is the energy consumption for data transmission, calculated by dividing the amount of data to be transmitted by the transmission rate and then by the transmission power. The task processing latency is calculated by summing the average waiting time of data packets in the buffer with the actual service time required for computation or transmission. A drift penalty term is constructed. ,in This indicates that the Lyapunov function is in Time slots and The expected difference between time slots Given the weighted sum of energy consumption and latency, at the beginning of each discrete decision slot, the current state observation is obtained. Using a convex optimization algorithm or a variant of gradient descent, within the feasible region satisfying peak power constraints and maximum retransmission constraints, a scheduling decision vector is searched that minimizes the supremum of the drift plus penalty term. This vector contains the proportion of tasks allocated to local computation, the proportion of tasks allocated to neighboring nodes for offloading, and the proportion of tasks allocated to cloud uploads. The change of the Lyapunov function between two adjacent decision slots is defined as Lyapunov drift. Combining the Lyapunov drift with the penalty term yields the Lyapunov drift plus penalty function. Within each decision slot, the optimal control variable is solved by minimizing the supremum of the Lyapunov drift plus penalty function. The control variable is the target task processing method for the current big data task.

[0057] In this embodiment, the steps for determining the target task processing mode for each edge node, including local processing mode, neighboring edge node offloading mode, and cloud upload mode, are as follows: The task scheduling path is determined based on the minimization result of the Lyapunov drift plus penalty function. When the solution result indicates that the local computational overhead is minimized, the local processing mode is selected, directly calling the local CPU resources of each edge node to process the big data task. When the solution result indicates that the local load of each edge node is too high and neighboring edge nodes have idle resources that can minimize the Lyapunov drift plus penalty value, the neighboring edge node offloading mode is selected, dividing the big data task and transmitting it to neighboring edge nodes for collaborative computation via a device-to-device communication link. When the solution result indicates that neither the local processing mode nor the neighboring edge node offloading mode can meet the latency constraints or energy consumption budget, the cloud upload mode is selected, sending the big data task to the central cloud server for centralized processing via a wide area network link, and then sending the processing result back to each edge node.

[0058] Specifically, the control variable values ​​obtained from the previous optimization stage are read, and the objective function gain corresponding to different processing methods is compared. If the calculation results show that the drift plus penalty value generated by executing the task locally is the smallest, and the current utilization of the local CPU is lower than a preset safety threshold, for example, lower than... If the local resources are sufficient to support the current task, the task data packet is moved directly from the waiting queue to the memory space of the local processing process, and the local CPU instruction set is invoked for computation. If the solution results indicate that local processing will cause a sharp increase in queue backlog, causing the penalty value to exceed the preset limit, and feedback signals from nearby edge nodes indicate that they have idle computing resources, such as the CPU utilization of nearby nodes being lower than [a certain value], then [the solution is not implemented]. And the remaining battery power is higher than If the total overhead of transmitting the task via the device-to-device (D2D) link and calculated by the neighboring node is less than the overhead of local processing, then the D2D communication module is activated, an encrypted transmission channel is established, the big data task is divided according to the preset data packet size, header verification information is added, and it is sent to the selected neighboring edge node via wireless direct connection. If the calculation finds that local processing will lead to excessive energy consumption and exhaust the power, and the neighboring node has excessive transmission latency due to excessive distance or poor channel quality, the combined cost of the first two modes exceeds the preset latency constraint, such as the task must be completed within

[200] milliseconds, or exceeds the energy consumption budget, such as the energy consumption of task processing must not exceed Joule initiates a wide area network (WAN) communication interface, establishes a TCP / IP connection with the central cloud server, encapsulates the complete big data task into an HTTP request or MQTT message, and uploads it to the cloud data center via a 4G / 5G network or fiber optic network. The high-performance cluster in the cloud processes the data and sends the results back. The task scheduling path is determined based on the minimization result of the Lyapunov drift plus penalty function: when the solution indicates that local computational overhead is minimized, the local processing mode is selected, directly utilizing the local CPU resources of each edge node to process the big data task; when the solution indicates that the local load of each edge node is too high and there are idle resources on neighboring edge nodes that can minimize the Lyapunov drift plus penalty value, the neighboring edge node offloading mode is selected, dividing the big data task and transmitting it to neighboring edge nodes for collaborative computation via a device-to-device communication link; when the solution indicates that neither the local processing mode nor the neighboring edge node offloading mode can meet latency constraints or energy consumption budgets, the cloud upload mode is selected, sending the big data task to the central cloud server for centralized processing via a WAN link and sending the processing results back to each edge node.

[0059] In this embodiment, during the execution of big data tasks according to the target task processing method, the heartbeat arrival time data of each edge node is continuously monitored through a non-fixed-period heartbeat detection mechanism. The specific steps of maintaining a time interval sliding window based on the heartbeat arrival time data are as follows: a mutual trust detection connection is established between each edge node, the receiving node records the actual arrival time of each heartbeat packet, the time difference between the heartbeat packet received at the current time and the heartbeat packet received at the last time is calculated, and the time difference is stored as the heartbeat arrival time data in the time interval sliding window. The time interval sliding window has a fixed sampling length. When new heartbeat arrival time data is generated, the oldest historical data in the time interval sliding window is removed. The maximum likelihood estimation method is used to fit all sample data in the time interval sliding window to obtain a normal distribution model. The normal distribution model is described by the mean parameter and the standard deviation parameter.

[0060] Specifically, a lightweight heartbeat detection program based on the UDP protocol is deployed at the network layer. Nodes do not strictly adhere to timer-triggered sending; instead, they randomly append a heartbeat identifier packet after processing a batch of data, depending on the workload. The receiving end starts a high-precision system clock monitoring thread. When the network card interrupts and parses the heartbeat packet, it immediately calls the operating system's timestamp function to obtain the nanosecond-level current time value, reads the time value recorded when the heartbeat packet from that node was successfully received, and subtracts the two to obtain the current heartbeat interval. A fixed-capacity circular buffer is allocated in memory as a sliding window, for example, with a set window length. The latest calculated interval duration is written to the head pointer position of the buffer. If the buffer is full, the tail pointer is moved to overwrite the earliest stored interval data, ensuring that the window always stores the most recent interval. For each interval sample in the communication, a statistical analysis subroutine is invoked to iterate through all data samples within the window. The sample mean is calculated as an estimate of the mean parameter, and the sample variance is calculated and its square root is taken to obtain an estimate of the standard deviation parameter. For example, if these interval data follow a Gaussian distribution, the sample mean is used as the expectation of the normal distribution model using the principle of maximum likelihood estimation. Using the sample variance as the variance of the normal distribution model This allows for the construction of a dynamic probability density function describing the current heartbeat fluctuation characteristics of the network. Mutual trust detection connections are established between each edge node. The receiving node records the actual arrival time of each heartbeat packet and calculates the time difference between the heartbeat packet received at the current moment and the heartbeat packet received at the last moment. This time difference is stored as heartbeat arrival time data in a time interval sliding window with a fixed sampling length. When new heartbeat arrival time data is generated, the oldest historical data in the time interval sliding window is removed. The maximum likelihood estimation method is used to fit all sample data in the time interval sliding window to obtain a normal distribution model, which is described by the mean parameter and the standard deviation parameter.

[0061] In this embodiment, the step of calculating the node failure dynamic risk assessment index based on the data distribution characteristics within the time interval sliding window, and judging the health status of each edge node based on the node failure dynamic risk assessment index, specifically involves: obtaining the actual waiting time since the last heartbeat packet received at the current time; calculating the tail probability of not receiving a heartbeat packet after the actual waiting time based on a normal distribution model; and constructing the node failure dynamic risk assessment index using the tail probability combined with historical fluctuation characteristics. The calculation formula for the node failure dynamic risk assessment index is as follows: ,in, This represents the dynamic risk assessment index for node failures. This represents the tail probability, which is the probability value calculated based on the normal distribution model that the heartbeat interval is greater than the actual waiting time. It is a logarithmic function with base 10. This represents the weighting coefficient of the probability term. This represents the weighting coefficient of the volatility term. The parameter representing the standard deviation of data within a sliding window of time intervals. The parameter represents the mean of the data within the sliding window of the time interval.

[0062] Specifically, a monitoring timer is started. At the end of each detection cycle, the current system clock is read, and the time when the target node's heartbeat packet was last received is subtracted to obtain the current actual waiting time. Call the cumulative distribution function (CDF) of the standard normal distribution, denoted as Calculate standardized variables Then use the formula Calculations show that, under the current normal distribution model, the heart rate interval is greater than... The probability, or tail probability, is determined by the value of the probability term. A smaller probability value indicates a lower likelihood that the current waiting time is within the normal range and a higher risk of failure. This involves reading pre-configured weighting coefficients, such as setting the probability term weighting coefficients. Set the weight coefficient for the fluctuation term. These two coefficients were set based on regression analysis of historical fault data. It focuses on reflecting the degree of abnormality in the current delay. Focusing on reflecting the overall instability of the network, the calculated values ​​are substituted into the risk assessment formula for weighted summation. During the calculation process, The node failure dynamic risk assessment index is a dimensionless comprehensive score used to intuitively quantify the downtime risk level of a node. This represents the tail probability, which is the probability value calculated based on the normal distribution model that the heartbeat interval is greater than the actual waiting time. The value ranges from [0] to [1] and is used to measure the rarity of the current delayed event. It is a logarithmic function with base

[10] , used to convert extremely small probability values ​​into positive numbers of a larger order of magnitude, thereby amplifying the warning effect of low probability events. This represents the weighting coefficient of the probability term, which is an empirical constant and typically ranges from 1 to 2. to This is used to adjust the proportion of the tail probability in the total score. The weighting coefficient for the fluctuation term is an empirical constant, typically ranging from [value range missing]. to ,and This is not necessarily true; it depends on the specific design of the scoring system. The standard deviation parameter, representing the data within a sliding window of time intervals, is obtained by calculating the dispersion of the data within the window and reflects the severity of network jitter. This parameter represents the mean value of the data within a sliding window of time intervals. It is obtained by calculating the average value of the data within the window and reflects the average heart rate cycle under normal conditions. The term actually constitutes the coefficient of variation, used to eliminate the influence of the time dimension and simply measure the relative magnitude of the fluctuation. It obtains the actual waiting time since the last heartbeat received, and calculates the tail probability of not receiving a heartbeat after the actual waiting time based on a normal distribution model. Using the tail probability combined with historical fluctuation characteristics, it constructs a dynamic risk assessment index for node failure. The formula for calculating the dynamic risk assessment index for node failure is: ,in, This represents the dynamic risk assessment index for node failures. This represents the tail probability, which is the probability value calculated based on the normal distribution model that the heartbeat interval is greater than the actual waiting time. It is a logarithmic function with base 10. This represents the weighting coefficient of the probability term. This represents the weighting coefficient of the volatility term. The parameter representing the standard deviation of data within a sliding window of time intervals. The parameter represents the mean of the data within the sliding window of the time interval.

[0063] In this embodiment, the specific steps of triggering the task migration mechanism to take over the pending tasks in each edge node when the health status is determined to be abnormal are as follows: a first risk threshold and a second risk threshold are preset, the value of the second risk threshold is greater than the value of the first risk threshold, the calculated node failure dynamic risk assessment index is compared with the first risk threshold, if the node failure dynamic risk assessment index exceeds the first risk threshold but does not exceed the second risk threshold, then each edge node is determined to be in a suspected failure state, in the suspected failure state, a pre-migration operation is started, the task intermediate calculation state data in the memory of each edge node is asynchronously copied to the neighboring edge node as a backup, but the current calculation process of each edge node is not interrupted for the time being, if the node failure dynamic risk assessment index continues to rise and exceeds the second risk threshold, then each edge node is determined to be in a confirmed failure state, in the confirmed failure state, the connection with each edge node is forcibly disconnected, and the neighboring edge node is activated to directly take over and continue to execute the unfinished calculation tasks using the already copied task intermediate calculation state data.

[0064] Specifically, during the system initialization phase, a first risk threshold is set by analyzing the false alarm rate and false negative rate data in historical fault logs. Second risk threshold For example, setting This value corresponds to a heartbeat delay exceeding approximately In the case of confidence intervals, set This value corresponds to an extremely abnormal delay state, ensuring , will be calculated in real time A logical comparison is performed with these two thresholds; if... The value is greater than and less than or equal to This indicates that although the node is slow to respond, it has not completely lost connection, and the edge nodes are judged to be in a suspected fault state. At this time, the background synchronization process is called to trigger the shadow copy operation of data. However, in order to avoid interrupting normal calculation due to misjudgment, the task process of the original node is kept running. If the calculation in the subsequent detection cycle is... The number continued to climb, eventually exceeding If the node fails to recover, it indicates that the possibility of node recovery is extremely low, and each edge node is determined to be in a confirmed failure state. At this point, a disconnect command is immediately sent to the network controller, the node's IP address mapping is cancelled, and an activation signaling message is sent to the neighboring edge nodes acting as backups. This signaling message contains the latest task breakpoint information, instructing the neighboring nodes to load the previously copied data, restore the computing context, and continue executing the subsequent instruction sequence from the breakpoint. A first risk threshold and a second risk threshold are pre-set; the value of the second risk threshold is greater than the value of the first risk threshold. The calculated node failure dynamic risk assessment index is compared with the first risk threshold. If the node failure dynamic risk assessment index exceeds... If the first risk threshold is exceeded but the second risk threshold is not, each edge node is determined to be in a suspected fault state. In the suspected fault state, a pre-migration operation is initiated, which asynchronously copies the intermediate task computation state data in the memory of each edge node to the neighboring edge node as a backup, but does not interrupt the current computation process of each edge node. If the node fault dynamic risk assessment index continues to rise and exceeds the second risk threshold, each edge node is determined to be in a confirmed fault state. In the confirmed fault state, the connection with each edge node is forcibly disconnected, and the neighboring edge node is activated to directly take over and continue to execute the unfinished computation tasks using the copied intermediate task computation state data.

[0065] In this embodiment, under a suspected fault state, the pre-migration operation is initiated to asynchronously copy the intermediate computation state data of the task in the memory of each edge node to the neighboring edge node as a backup. Specifically, this involves: identifying the key context information of the big data task currently being processed by each edge node, including the program counter value, register state, and stack memory data; incrementally compressing and encoding the key context information to generate a lightweight state snapshot package; and sending the state snapshot package to the nearest neighboring edge node with the lowest physical distance and lowest load via a device-to-device communication link. After receiving the state snapshot package, the neighboring edge node temporarily stores it in a cache area and continuously updates the state snapshot package until it receives a formal takeover instruction or the suspected fault state is resolved.

[0066] Specifically, using the debugging interface or process tracing tools provided by the operating system, such as the ptrace mechanism, the process currently processing a large data task is suspended. Key information in the process control block (PCB) is read, the current value of the program counter (PC) is extracted (pointing to the address of the next instruction to be executed), and data from general-purpose registers, status registers, and stack pointer registers is read. This data stores the hierarchy of the function call stack and the current state of local variables. The process's heap memory region is scanned to identify dirty pages that have recently been written to. Using incremental compression algorithms, such as LZ4 or Zstandard, the changed parts of the register states and stack memory data are serialized and encoded to generate a compact state snapshot packet. This snapshot packet contains the minimum dataset required to resume computation. Through a pre-established device-to-device (D2D) high-speed communication link, using the reliable User Datagram Protocol (RUDP), this state snapshot packet is sent to the nearest and least overloaded neighboring device. Upon receiving data, neighboring edge nodes do not immediately load and execute it. Instead, they write it into a dedicated isolated buffer and update the status data in the buffer based on subsequent incremental update packets, maintaining basic synchronization between the data version and the original node. Only when a takeover command confirming a fault is received is the status data loaded into memory and the process execution resumed. Alternatively, the buffer is released when the risk index of the original node drops to a safe range. The critical context information of the big data tasks currently being processed by each edge node is identified. This critical context information includes program counter values, register states, and stack memory data. The critical context information is incrementally compressed and encoded to generate a lightweight status snapshot packet. This status snapshot packet is sent to the nearest neighboring edge node with the lowest physical distance and lowest load via a device-to-device communication link. Upon receiving the status snapshot packet, the neighboring edge node temporarily stores it in its buffer and continuously updates the status snapshot packet until a formal takeover command is received or the suspected fault status is resolved.

Claims

1. An IoT edge node big data collaborative processing system, characterized in that, The system includes: Real-time operating status data of each edge node in the IoT edge computing system is collected. The real-time operating status data includes task queue backlog length, node remaining energy parameters and communication channel status information. A multi-dimensional queuing system model is constructed based on the real-time operating status data. Based on the multidimensional queuing system model, a Lyapunov drift plus penalty function is defined. The Lyapunov drift plus penalty function is used to solve the scheduling decision of the big data tasks to be processed in each edge node, and the target task processing mode of each edge node is determined. The target task processing mode includes local processing mode, neighboring edge node offloading mode and cloud upload mode. During the execution of the big data task according to the target task processing method, the heartbeat arrival time data of each edge node is continuously monitored through a non-fixed period heartbeat detection mechanism, and a time interval sliding window is maintained based on the heartbeat arrival time data. Calculate the node failure dynamic risk assessment index based on the data distribution characteristics within the time interval sliding window, determine the health status of each edge node based on the node failure dynamic risk assessment index, and trigger the task migration mechanism to take over the pending tasks in each edge node when the health status is determined to be abnormal.

2. The IoT edge node big data collaborative processing system according to claim 1, characterized in that, The specific steps for constructing a multidimensional queuing system model based on the real-time operational status data are as follows: Obtain the task queue backlog length of each edge node, whereby the task queue backlog length represents the total number of data packets waiting to be processed in the buffers of each edge node in the current decision time slot; The remaining energy parameters of each edge node are collected synchronously. These remaining energy parameters reflect the current battery life of each edge node and the duration of high-load computing that it can support. Real-time detection of the communication channel status information, which includes the uplink signal-to-noise ratio between the edge node and the cloud server and the quality of the direct communication link between the edge node and its neighboring edge nodes; The multidimensional queuing system model is constructed based on the task queue backlog length, the node's remaining energy parameters, and the communication channel state information.

3. The IoT edge node big data collaborative processing system according to claim 2, characterized in that, Based on the multidimensional queuing system model, a Lyapunov drift plus penalty function is defined. The specific steps for solving the scheduling decision of the big data tasks to be processed in each edge node using the Lyapunov drift plus penalty function are as follows: Construct a Lyapunov function to characterize half of the sum of squares of the task queue backlog length; A penalty term is set, which is the weighted sum of total system energy consumption and task processing latency, wherein the total system energy consumption is calculated based on computing energy consumption and transmission energy consumption, and the task processing latency is calculated based on queuing latency and computing latency; The change of the Lyapunov function between two adjacent decision time slots is defined as Lyapunov drift. The Lyapunov drift is combined with the penalty term to obtain the Lyapunov drift plus penalty function. Within each decision time slot, the optimal control variable is solved by minimizing the supremum of the Lyapunov drift plus penalty function, and the control variable is the target task processing method for the current big data task.

4. The IoT edge node big data collaborative processing system according to claim 3, characterized in that, The steps for determining the target task processing method for each edge node, including local processing mode, adjacent edge node offloading mode, and cloud upload mode, are as follows: The task scheduling path is determined based on the minimization result of the Lyapunov drift plus penalty function: When the solution result indicates that the local computing overhead is minimal, the local processing mode is selected, and the local CPU resources of each edge node are directly called to process the big data task. When the solution results indicate that the local load of each edge node is too high and the neighboring edge nodes have idle resources that can minimize the Lyapunov drift plus penalty value, the neighboring edge node offload mode is selected, and the big data task is divided and transmitted to the neighboring edge nodes for collaborative computing through the device-to-device communication link; When the solution result indicates that neither the local processing mode nor the adjacent edge node offloading mode can meet the latency constraints or energy consumption budget, the cloud upload mode is selected, and the big data task is sent to the central cloud server for centralized processing via a wide area network link, and the processing result is sent back to each edge node.

5. The IoT edge node big data collaborative processing system according to claim 1, characterized in that, During the execution of the big data task according to the target task processing method, the step of continuously monitoring the heartbeat arrival time data of each edge node through a non-fixed-period heartbeat detection mechanism and maintaining a time interval sliding window based on the heartbeat arrival time data is as follows: A mutual trust detection connection is established between each edge node, and the receiving node records the actual arrival time of each heartbeat packet. Calculate the time difference between the heartbeat packet received at the current moment and the heartbeat packet received at the last moment, and store the time difference as the heartbeat arrival time data in the time interval sliding window; The time interval sliding window has a fixed sampling length. When new heartbeat arrival time data is generated, the oldest historical data in the time interval sliding window is removed. The maximum likelihood estimation method is used to fit all sample data within the time interval sliding window to obtain a normal distribution model, which is described by the mean parameter and the standard deviation parameter.

6. The IoT edge node big data collaborative processing system according to claim 5, characterized in that, The steps for calculating the dynamic risk assessment index of node failure based on the data distribution characteristics within the time interval sliding window, and determining the health status of each edge node based on the dynamic risk assessment index of node failure, are as follows: Obtain the actual waiting time since the last heartbeat packet was received at the current time, and calculate the tail probability of not receiving a heartbeat packet after the actual waiting time based on the normal distribution model; The dynamic risk assessment index for node failure is constructed by combining the tail probability with historical fluctuation characteristics; The formula for calculating the node failure dynamic risk assessment index is as follows: in, This represents the dynamic risk assessment index for node failure; The tail probability is the probability value calculated based on the normal distribution model that the heartbeat interval is greater than the actual waiting time. It is a logarithmic function with base 10; Indicates the weighting coefficient of the probability term; This represents the weighting coefficient of the volatility term; The parameter representing the standard deviation of the data within the time interval sliding window; The parameter represents the mean value of the data within the sliding window of the time interval.

7. The IoT edge node big data collaborative processing system according to claim 6, characterized in that, The specific steps for triggering a task migration mechanism to take over pending tasks in each edge node when the health status is determined to be abnormal are as follows: A first risk threshold and a second risk threshold are preset, wherein the value of the second risk threshold is greater than the value of the first risk threshold; The calculated node failure dynamic risk assessment index is compared with the first risk threshold. If the node failure dynamic risk assessment index exceeds the first risk threshold but does not exceed the second risk threshold, then each edge node is determined to be in a suspected failure state. In the suspected fault state, a pre-migration operation is initiated to asynchronously copy the intermediate computing state data of the tasks in the memory of each edge node to the neighboring edge node as a backup, but without interrupting the current computing process of each edge node. If the node failure dynamic risk assessment index continues to rise and exceeds the second risk threshold, then each edge node is determined to be in a confirmed failure state. In the confirmed fault state, the connection with each edge node is forcibly disconnected, and the neighboring edge node is activated to directly take over and continue to execute the unfinished computing task using the copied intermediate computing state data of the task.

8. The IoT edge node big data collaborative processing system according to claim 7, characterized in that, In the suspected fault state, the pre-migration operation is initiated, and the intermediate computation state data of the tasks in the memory of each edge node is asynchronously copied to the neighboring edge nodes that serve as backups. The specific steps are as follows: Identify the key context information of the big data task currently being processed by each edge node, including program counter values, register states, and stack memory data; The key context information is incrementally compressed and encoded to generate a lightweight state snapshot package; The status snapshot packet is sent to the nearest neighboring edge node with the lowest physical distance and lowest load via a device-to-device communication link; Upon receiving the status snapshot packet, the neighboring edge node temporarily stores it in the cache area and continuously updates the status snapshot packet until it receives a formal takeover instruction or the suspected fault state is resolved.

Images