A method for edge cloud cooperative real-time data stream processing

Abstract

This invention belongs to the field of distributed data scheduling technology in the Industrial Internet of Things (IIoT) and relates to a real-time data stream processing method with edge-cloud collaboration. The method includes: establishing a dynamic circular buffer at the edge node, calculating the time interval and slope of arriving data packets, and extracting a sampling matrix to generate feature variance; sending timestamped data packets to the cloud node and receiving acknowledgment packets with network latency parameters, and generating a node state vector based on remaining computing power; calculating a traffic splitting decision value based on the slope, state vector, and feature variance, and accordingly calling a feature dimensionality reduction matrix to output low-dimensional data packets or distributing them to the local core processing. This invention changes the fixed timing of waiting window filling, cuts off the accumulation of the computing queue in the early stages of sudden traffic, solves the allocation mismatch problem caused by parameter lag, and eliminates the occupation of channel bandwidth by active probe packets.

Description

Technical Field

[0001] This invention belongs to the field of distributed data scheduling technology of industrial Internet of Things, and relates to a real-time data stream processing method with edge-cloud collaboration. Background Technology

[0002] Currently, distributed computing and edge-cloud collaborative architectures are widely used to process the massive amounts of data generated by IoT terminals distributed across various locations, and to undertake tasks such as data scheduling, task distribution, and cache management. With the development of the Industrial Internet, smart cities, and intelligent network systems, the amount of data and the operational status at the edge are changing faster and faster. However, existing technologies still have significant shortcomings when simultaneously handling multi-dimensional data characteristics, limited computing and communication resources, and rapid closed-loop adjustments to millisecond-level burst states. First, there are static weight-based policy scheduling and adaptive caching technologies. For example, Chinese invention patent application CN119316426A discloses a cloud-edge collaborative data distribution system. This system sets up a cloud server and edge devices. Based on the amount of data to be processed, the current load, and network latency, it calculates a distribution decision score using a specific formula to dynamically adjust the distribution strategy. The adaptive cache management module determines the caching strategy on the edge side. However, the decision-making process of this scheme is still limited by static parameters. The weight coefficients used in the distribution scheduling mainly rely on manually preset empirical values, which are difficult to accurately reflect the complex relationship between the computing load of heterogeneous edge devices and sudden link jitter over a long operating cycle, and are prone to causing scheduling decision lag. At the same time, its cache capacity management model is relatively simple and lacks sufficient analysis of semantic information and correlation features in the original data stream, which leads to a sudden drop in edge cache hit rate in complex multimodal scenarios.

[0003] Second, there is the feature vector-based traffic splitting technology based on fixed thresholds. For example, Chinese invention patent application CN121037449A discloses a real-time data splitting and intelligent caching method and system based on edge computing. This method collects raw data streams and converts them into structured data, calculates the timeliness weight, correlation weight, and security level of the data, and forms feature vectors accordingly. The feature vectors are then input into a pre-trained splitting decision model, which outputs a splitting signal to manage the edge cache space and synchronize the global cache distribution topology. This technology mainly relies on feature-level hash matching and classification prediction. Its decision-making process is limited by multiple preset thresholds, such as the first timeliness threshold, the second timeliness threshold, the correlation threshold, and the security level threshold. When dealing with sudden and unstructured task streams, fixed threshold judgment can easily cause the classification results to fluctuate repeatedly around the threshold, resulting in the task splitting results remaining in a suboptimal state. In addition, frequent synchronization of the entire cache distribution topology in large-scale distributed nodes will consume a lot of communication bandwidth. When the bandwidth of the edge network is limited, this synchronization method can also easily cause clock drift and network congestion, forming additional signaling overhead and further increasing transmission latency.

[0004] Third, there is the task offloading technology based on cloud-based digital twin feedback and multi-task models. For example, Chinese invention patent application CN120687221A discloses an edge computing task offloading method, device, electronic device, and storage medium. This method utilizes a pre-trained multi-task model that includes a shared feature encoding layer, independent classification branches, and independent threshold regression branches to simultaneously classify the current task and calculate the offloading threshold for the collected multi-dimensional features. It also analyzes the difference between the simulation model and the actual operating state through a digital twin model in the central cloud and adjusts the offloading threshold parameters and training weights accordingly. Such methods usually require extracting multi-dimensional spatiotemporal data matrices from physical control objects and are mostly used in specific heavy industrial scenarios such as power distribution network grid connection. They are heavily dependent on the steady-state operation characteristics and transient response characteristics of this scenario. The constraints and derived features used are difficult to directly apply to other scenarios. At the same time, the difference analysis between the simulation model and the actual operating state, as well as the global optimization model, are all deployed on the central cloud server. The closed-loop link from edge execution, cloud analysis to parameter return is long, making it difficult to make timely local adjustments and recovery for transient abnormal conditions such as local overload and link hardware failures on the edge side, which last only for microseconds or milliseconds.

[0005] Therefore, the technical problem to be solved by this invention is how to construct an edge-cloud collaborative diversion and caching framework that can adapt to different types of multi-dimensional data characteristics, reduce multi-node collaborative overhead, and respond to sudden situations at the edge in a timely manner, so that the data flow and task allocation do not exceed the actual resource tolerance range during the entire processing. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and solve the technical problems of delayed traffic splitting decisions, network congestion, and inability to self-heal in real time when dealing with multi-dimensional feature coupling and resource boundary constraints of highly dynamic edge environments due to parameter lag, rigid threshold interception, and ultra-long spatial scale closed-loop links. The invention provides a real-time data stream processing method for edge-cloud collaboration.

[0007] To achieve the above-mentioned objectives, the present invention provides a real-time data stream processing method for edge-cloud collaboration, comprising the following steps: Step S101: Edge nodes build buffers and calculate slopes to generate feature variance: Edge nodes build dynamic circular buffers containing a fixed number of data packet storage bits within the cache time window. When a data packet arrives at the network interface, the edge nodes read the nanosecond-level system time of the local high-precision timer, calculate the difference between the system time of the data packet and the previous data packet and record it as the arrival time interval. The edge nodes calculate the first-order difference of the continuously accumulated arrival time intervals and record it as the arrival interval slope. At the same time, the edge nodes extract the real-time sampling matrix of multi-dimensional monitoring indicators in the data packet to dynamically generate feature variance. Step S102, edge node merges network latency and computing power to generate state vector: edge node sends data packet with first timestamp tag to cloud node, receives downlink computing confirmation packet carrying network latency parameter returned by cloud node, and combines network latency parameter with remaining computing power data read from state register to generate node state vector. Step S103: Edge nodes perform feature dimensionality reduction or local computation based on decision values: Edge nodes calculate the traffic splitting decision scalar value based on arrival interval slope, node state vector, and feature variance; when the traffic splitting decision scalar value is lower than the set lower threshold, the feature dimensionality reduction matrix is ​​called to truncate multidimensional redundant feature components of data packets within the cache time window and output low-dimensional data packets; when the traffic splitting decision scalar value is higher than the set upper threshold, the network interface is closed and the data packets are distributed to the local computing core to complete local streaming computation.

[0008] The generation of the node state vector in step S102 of the present invention includes: when the edge node sends a data packet to the cloud node, it writes a first timestamp tag in the header of the data packet; when the cloud node receives the data packet, it reads the system reception time as a second timestamp tag, calculates the difference between the second timestamp tag and the first timestamp tag, and determines it as a network delay parameter; the cloud node attaches the network delay parameter to the downlink calculation confirmation packet and returns it to the edge node, which together with the remaining computing power data read by the edge node from the status register to generate the node state vector.

[0009] Step S103 of the present invention includes: the edge node calculates the traffic splitting decision scalar value based on the node state vector and feature variance; when the traffic splitting decision scalar value is lower than the lower threshold of 0.4, the feature matrix of the data packets within the cache time window is reduced, and the feature reduction matrix is ​​called to truncate the multi-dimensional redundant feature components of the data packets, so that the dimension of the feature matrix is ​​reduced from 64 dimensions to 8 dimensions; when the traffic splitting decision scalar value is higher than the upper threshold of 0.8, the network interface is closed and the data packets are distributed to the local computing core to complete the local streaming computing.

[0010] The step S103 of this invention, which enables dimensionality reduction of the feature matrix of data packets within the cache time window, includes the following sub-steps: Step S1031, the edge node constructs the original high-dimensional feature matrix of the data packets within the cache time window, and transforms the original high-dimensional feature matrix into a low-dimensional feature space through the feature dimensionality reduction matrix and outputs the low-dimensional feature matrix; Step S1032, the edge node writes the feature dimension and spatial sparsity into the header of the dimensionality-reduced data packet corresponding to the low-dimensional feature matrix, and sends the dimensionality-reduced data packet to the target processing node through the network interface.

[0011] After receiving the dimensionality-reduced data packet, the target processing node of the present invention reads the feature dimension and spatial sparsity carried in the header and compares the spatial sparsity with the set sparsity critical threshold. When the spatial sparsity exceeds the sparsity critical threshold, the target processing node calls the inverse projection transformation matrix to complete zero-element filling and feature reconstruction of the low-dimensional feature matrix.

[0012] The present invention further includes the following steps: Step S104, the edge node continuously records the time series of the diversion decision scalar value within the historical control cycle and calculates the slip variance of the time series; Step S105, when the slip variance continuously exceeds the set stability threshold, the edge node lowers the lower limit threshold according to the set adjustment ratio and simultaneously increases the sampling cutoff frequency of the buffer time window.

[0013] The method described in this invention is applied to the processing of large-scale time-series status monitoring data streams. The fixed number of dynamic ring buffers is 64 to 512, the resolution of the network interface for acquiring nanosecond-level system time is not less than 1ns, and the data reading bus bandwidth between the edge node and the status register is not less than 10Gbps.

[0014] The dynamic generation of feature variance in step S101 of the present invention includes the following sub-steps: Step S1011, after receiving the data packet within the buffer time window, the edge node periodically extracts the real-time sampling matrix of the multi-dimensional monitoring indicators in the data packet and calculates the sampling mean of each dimension in the real-time sampling matrix; Step S1012, the edge node calculates the squared difference between the real-time sampling value of each dimension and the sampling mean, and accumulates the sliding weighted sum on the time axis and outputs the feature variance.

[0015] The present invention also includes the following data packet retransmission steps: Step S106, when the diversion path is blocked or the channel transmission quality is lower than the set communication threshold, the edge node temporarily stores the data packets that are not sent in time in the retry queue of the local memory, and after detecting that the network delay parameter has recovered to the set threshold range, the network interface is enabled to send the data packets in the retry queue in the order of timestamps.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Edge nodes can end the current buffer time window in advance when a burst of traffic just begins, based on the arrival time interval and slope of adjacent data packets, and transfer subsequent data packets to the next buffer time window. Compared with the fixed method of waiting for the window to be filled before processing, this process can interrupt the continuous accumulation of data packets in the current calculation queue in a timely manner, and avoid the mismatch between the traffic distribution result and the actual load caused by the lag in the transmission of status parameters.

[0017] 2. Edge nodes obtain network latency parameters using normally sent data packets and downlink computing confirmation packets returned by cloud nodes, and combine the network latency parameters with the remaining computing power data into a node state vector. There is no need to send additional active probe packets, which can reduce the bandwidth occupation of network state measurement, avoid additional probe traffic from further increasing the network burden, and reduce bandwidth conflicts between control signaling and service data.

[0018] 3. Edge nodes calculate the scalar value for traffic splitting decisions based on the node state vector and feature variance, and select the appropriate data processing method according to the interval in which the value is located. When it is necessary to reduce the network transmission burden, multi-dimensional redundant feature components are truncated through the feature dimensionality reduction matrix to reduce the amount of feature data that needs to be transmitted in the data packet. When it is not suitable to continue sending data to the cloud, the data packet is transferred to the local computing core to complete local streaming computing, avoiding the additional computing power consumption caused by continuously adjusting the traffic splitting path, and helping to balance the computing queue load of edge nodes. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the process of generating state vectors and executing decisions for edge nodes in this invention. Figure 2 This is a schematic diagram of the control flow and state transition of the data packet processing system of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings.

[0021] Example 1: This embodiment discloses a real-time data stream processing method for edge-cloud collaboration, including the following steps: Step S101: Edge nodes build buffers and calculate slopes to generate feature variance: Edge nodes build dynamic circular buffers containing a fixed number of data packet storage bits within the cache time window. When a data packet arrives at the network interface, the edge nodes read the nanosecond-level system time of the local high-precision timer, calculate the difference between the system time of the data packet and the previous data packet and record it as the arrival time interval. The edge nodes calculate the first-order difference of the continuously accumulated arrival time intervals and record it as the arrival interval slope. At the same time, the edge nodes extract the real-time sampling matrix of multi-dimensional monitoring indicators in the data packet to dynamically generate feature variance. Step S102, edge node merges network latency and computing power to generate state vector: edge node sends data packet with first timestamp tag to cloud node, receives downlink computing confirmation packet carrying network latency parameter returned by cloud node, and combines network latency parameter with remaining computing power data read from state register to generate node state vector. Step S103: Edge nodes perform feature dimensionality reduction or local computation based on decision values: Edge nodes calculate the traffic splitting decision scalar value based on arrival interval slope, node state vector, and feature variance; when the traffic splitting decision scalar value is lower than the set lower threshold, the feature dimensionality reduction matrix is ​​called to truncate multidimensional redundant feature components of data packets within the cache time window and output low-dimensional data packets; when the traffic splitting decision scalar value is higher than the set upper threshold, the network interface is closed and the data packets are distributed to the local computing core to complete local streaming computation.

[0022] In this embodiment, step S102, generating the node state vector, includes: when the edge node sends a data packet to the cloud node, it writes a first timestamp tag in the header of the data packet; when the cloud node receives the data packet, it reads the system reception time as a second timestamp tag, calculates the difference between the second timestamp tag and the first timestamp tag, and determines it as a network delay parameter; the cloud node attaches the network delay parameter to the downlink calculation confirmation packet and returns it to the edge node, which together with the remaining computing power data read by the edge node from the status register, generates the node state vector.

[0023] Step S103 in this embodiment includes: the edge node calculates the traffic splitting decision scalar value based on the node state vector and feature variance; when the traffic splitting decision scalar value is lower than the lower threshold of 0.4, the feature matrix of the data packets within the cache time window is reduced, and the feature reduction matrix is ​​called to truncate the multi-dimensional redundant feature components of the data packets, so that the dimension of the feature matrix is ​​reduced from 64 dimensions to 8 dimensions; when the traffic splitting decision scalar value is higher than the upper threshold of 0.8, the network interface is closed and the data packets are distributed to the local computing core to complete the local streaming computing.

[0024] In this embodiment, step S103, which enables dimensionality reduction of the feature matrix of data packets within the cache time window, includes the following sub-steps: Step S1031, the edge node constructs the original high-dimensional feature matrix of the data packets within the cache time window, and transforms the original high-dimensional feature matrix into a low-dimensional feature space through the feature dimensionality reduction matrix and outputs the low-dimensional feature matrix; Step S1032, the edge node writes the feature dimension and spatial sparsity into the header of the dimensionality-reduced data packet corresponding to the low-dimensional feature matrix, and sends the dimensionality-reduced data packet to the target processing node through the network interface.

[0025] After receiving the dimensionality-reduced data packet, the target processing node in this embodiment reads the feature dimension and spatial sparsity carried in the header and compares the spatial sparsity with the set sparsity critical threshold. When the spatial sparsity exceeds the sparsity critical threshold, the target processing node calls the inverse projection transformation matrix to complete zero-element filling and feature reconstruction of the low-dimensional feature matrix.

[0026] The embodiment also includes the following steps: Step S104, the edge node continuously records the time series of the diversion decision scalar value within the historical control cycle and calculates the slip variance of the time series; Step S105, when the slip variance continuously exceeds the set stability threshold, the edge node lowers the lower limit threshold according to the set adjustment ratio and simultaneously increases the sampling cutoff frequency of the buffer time window.

[0027] The method described in this embodiment is applied to the processing of large-scale time-series status monitoring data streams. The fixed number of dynamic ring buffers is 64 to 512, the resolution of the network interface for acquiring nanosecond-level system time is not less than 1ns, and the data reading bus bandwidth between the edge node and the status register is not less than 10Gbps.

[0028] In this embodiment, step S101, which dynamically generates feature variance, includes the following sub-steps: Step S1011, after receiving the data packet within the cache time window, the edge node periodically extracts the real-time sampling matrix of the multi-dimensional monitoring indicators in the data packet and calculates the sampling mean of each dimension in the real-time sampling matrix; Step S1012, the edge node calculates the squared difference between the real-time sampling value of each dimension and the sampling mean, and accumulates the sliding weighted sum on the time axis and outputs the feature variance.

[0029] The embodiment also includes the following data packet retransmission steps: Step S106, when the diversion path is blocked or the channel transmission quality is lower than the set communication threshold, the edge node temporarily stores the data packets that are not sent in time in the retry queue of the local memory, and after detecting that the network delay parameter has recovered to the set threshold range, the network interface is enabled to send the data packets in the retry queue in the order of timestamps.

[0030] Example 2: In this embodiment, multi-dimensional high-frequency sensing components in a distributed industrial field continuously generate large-scale time-series status monitoring data streams. Edge nodes and cloud nodes constitute a distributed electronic digital data processing system (EDDMS) to perform cross-network node scheduling and streaming computation on the time-series status monitoring data streams. Under conditions of strong electromagnetic interference and dynamic congestion of concurrent network tasks, network latency fluctuates over time, the remaining computing power of heterogeneous edge nodes decreases temporarily, and the continuously flowing data processing queue is prone to congestion. Data packets containing high-dimensional redundancy accumulate in the channel, leading to buffer saturation and overflow, monitoring data packet loss, and increased processing latency. The DEDMS inputs the time-series status monitoring data streams into the data processing queues of the edge nodes, and the data transformation units within the edge nodes buffer the data within the buffer time window. An internal dynamic circular buffer containing a fixed number of data packet storage bits is established. When a data packet arrives at the network interface, the system time at the nanosecond level is read by a local high-precision timer with a resolution of not less than 1ns. The difference between the system time of the current data packet and the previous data packet is calculated and recorded as the arrival time interval. The data transformation unit calculates the first-order difference of the continuously accumulated arrival time intervals in the dynamic circular buffer to obtain the arrival interval slope. When the absolute value of the arrival interval slope exceeds the preset critical threshold and the number of data packets contained in the buffer time window reaches 45% of the initial window span, the edge node triggers a discrete truncation event, closes the current buffer time window in advance, and writes the subsequent data packets that have not yet been loaded into the current window into the next protection time window with a higher dimensionality reduction priority.

[0031] When an edge node sends a data packet to a cloud node, a first timestamp tag is written into the header of the data packet. After receiving the data packet, the cloud node reads the system reception time as the second timestamp tag, calculates the difference between the second timestamp tag and the first timestamp tag, determines it as the network latency parameter, and attaches the network latency parameter to the downlink calculation confirmation packet and returns it to the edge node. After the buffer time window is closed, the data transformation unit periodically extracts the real-time sampling matrix of multi-dimensional monitoring indicators from each data packet within the window, calculates the sampling mean of each dimension in the real-time sampling matrix, calculates the squared difference between the real-time sampling value of each dimension and the corresponding sampling mean, and performs cumulative sliding weighted summation along the time axis to obtain the feature variance. The edge node reads the remaining computing power data from the status register through a data reading bus with a bandwidth of not less than 10Gbps, and combines the network latency parameter and the remaining computing power data into a node status vector. This node status vector adopts a 2-dimensional column vector, where the first element is the network latency parameter in milliseconds, and the second element is a scalar of the remaining computing power percentage normalized to between 0 and 1.

[0032] When calculating the scalar value of the traffic splitting decision, edge nodes input the arrival interval slope, node state vector, and feature variance into a preset load optimization relationship. The processor first constructs a 1x2 weight transformation row vector, whose two elements are the network latency weight constant and the computing power distribution weight constant, respectively. Then, it performs an inner product operation with the 2D node state vector to obtain the comprehensive operating cost scalar value. This scalar value is equal to the product of the network latency weight constant and the network latency parameter, plus the product of the computing power distribution weight constant and the remaining computing power data, and then divided by the feature variance. The processor then combines the data packet arrival trend represented by the arrival interval slope to output the traffic splitting decision scalar value corresponding to the current cache time window. Before the system goes online, through network physical... The test platform calibrated the network latency weight constant and the computing power distribution weight constant. The network loss generator controlled the network latency to fluctuate between 10 milliseconds and 500 milliseconds, and the computing load generator controlled the remaining computing power of the edge processor to vary between 20% and 90%, forming a total of 5,000 heterogeneous combination working conditions. The high-precision network analyzer continuously recorded the full-link flow computing throughput. The calibration target was set with a global average flow computing response latency of no more than 65 milliseconds and a dynamic ring buffer overflow count of zero. The least squares method was used to perform regression fitting on the 5,000 samples. The network latency weight constant corresponding to the fitting result was 0.6, and the computing power distribution weight constant was 0.4. The two constants were written into the static register of the data transformation unit.

[0033] Edge nodes perform traffic control based on the traffic splitting decision scalar value. When the traffic splitting decision scalar value is lower than the lower limit threshold of 0.4, the edge node calls the feature dimensionality reduction matrix to truncate the multidimensional redundant feature components of the data packets within the cache time window, reducing the original high-dimensional feature matrix from 64 dimensions to 8 dimensions, and outputting a low-dimensional feature matrix. The feature dimensionality reduction matrix uses a preset orthogonal projection matrix. The truncated 56-dimensional feature components correspond to redundant features or low-variance noise in the industrial monitoring data. The edge node calculates the spatial sparsity based on the ratio of the number of zero elements in the low-dimensional feature matrix to the total number of matrix elements, writes the feature dimension and spatial sparsity into the header of the low-dimensional data packet, and then sends the low-dimensional data packet to the target processing node through the network interface. In this embodiment, all the dimensionality-reduced low-dimensional data packets are distributed to the corresponding cloud nodes through the network interface.

[0034] When the traffic splitting decision scalar value exceeds the upper limit threshold of 0.8, the edge node logically shuts down the network interface and distributes the data packets within the cached time window to the local computing core to complete local streaming computation. Logically shutting down the network interface means suspending the uplink queue in the cloud or the sending thread of the corresponding network socket. The physical network card and the local high-precision timer remain operational. When subsequent data packets arrive at the network interface, the processor continues to read its nanosecond-level system time, calculates the arrival time interval and the arrival interval slope, and writes the data packets that have not been sent in time into the retry queue in the local memory according to the timestamp. This retry queue uses a temporary circular overflow buffer to store the data packets to be sent. During the streaming computation performed by the local computing core, the self-healing monitoring thread built into the edge node reads the network latency parameters carried by the downlink computation confirmation packet every 50 milliseconds. When the network latency parameters recover to the set threshold range, the processor reactivates the uplink sending thread and the network interface, and sends the data packets in the retry queue in the order of the timestamps, so that the edge-cloud collaborative scheduling resumes operation.

[0035] After receiving a low-dimensional data packet, the target processing node reads the feature dimension and spatial sparsity from the header and compares the spatial sparsity with a pre-set sparsity threshold. The sparsity threshold is pre-calibrated based on the sparsity of the low-dimensional feature matrix that the target processing node can directly process. When the spatial sparsity exceeds the sparsity threshold, the target processing node calls the inverse projection transformation matrix to perform zero-element filling and feature reconstruction on the low-dimensional feature matrix. The target processing node internally stores a feature space basis encoding table consistent with that of the edge nodes, and the inverse projection transformation matrix uses the transpose matrix corresponding to the feature dimension reduction matrix. When performing feature reconstruction, the target processing node allocates a 64-dimensional continuous data block in memory, writes the received 8-dimensional low-dimensional feature matrix into the corresponding position according to the channel index in the basis encoding table, and fills the remaining 56 empty positions with zero elements. Then, it calls the inverse projection transformation matrix to perform matrix operations, remapping the low-dimensional feature matrix back to the original feature space to obtain a 64-dimensional reconstructed feature matrix.

[0036] When the spatial sparsity does not exceed the sparsity critical threshold, the target processing node adopts a distributed block parallel aggregation rule to perform streaming aggregation calculations on the received data packets to reflect the state changes of external entities in the distributed industrial site. When the data transformation unit adjusts the filling timing of the buffer time window and performs diversion control in combination with network latency parameters, the global average streaming calculation response latency of the distributed electronic digital data processing system is maintained within 65ms when affected by sudden data influx and network connection deterioration. The overall streaming calculation throughput is maintained at 98.5%, and the dynamic ring buffer of the edge node does not experience saturation overflow.

[0037] Example 3: This embodiment verifies the operation of the real-time data stream processing method under network fluctuations and computational load impacts on a network physical test platform. The platform includes edge nodes with a multi-core processor architecture and cloud nodes composed of a high-specification distributed server cluster, which are connected by a logical network. The large-scale time-series state monitoring data stream used in the test is taken from a standardized public industrial monitoring dataset. Gaussian white noise with a signal-to-noise ratio of 20dB and power frequency interference harmonics with a frequency of 50Hz are superimposed at the input of the data transformation unit to simulate electromagnetic interference in the industrial field. The judgment threshold for closing the buffer time window in advance is set according to the effective spectral bandwidth of the monitored signal. When the effective spectral bandwidth exceeds 500kHz, in order to satisfy the Nyquist sampling theorem and reduce signal feature aliasing, the sampling period is adjusted to the lower limit of its value range. In this embodiment, it is set to 20 milliseconds. The lower limit threshold of the diversion decision scalar value is set to 0.4, and the upper limit threshold is set to 0.8. The interval between the two thresholds corresponds to the normal edge-cloud collaborative streaming computing state.

[0038] During platform operation, the network loss generator injects discrete packet loss and time-varying delay jitter into the channel. First, second, and third experimental groups, a first control group, and a second control group are set up. The initial signal-to-noise ratio of the unprocessed raw input data is 32dB, which drops to 14dB after electromagnetic interference, resulting in significant waveform distortion. In each experimental group, the edge node calculates the diversion decision scalar value based on the arrival interval slope, node state vector, and feature variance, and then performs corresponding diversion processing according to the lower and upper thresholds. The first experimental group uses 0.4 as the lower threshold. When the real-time calculated diversion decision scalar value is lower than 0.4, the edge node calls the feature dimensionality reduction matrix to project the original high-dimensional feature matrix of the data packets within the buffer time window, reducing the 64-dimensional feature matrix to 8-dimensional. The feature variance obtained during this process stabilizes at 12.6. The dimensionality-reduced low-dimensional data packets are sent to the target processing node via the network interface. The target processing node reads the data packet header. The first experimental group used 0.8 as the upper limit threshold. When the spatial sparsity exceeded the sparsity critical threshold, the low-dimensional feature matrix was filled with zero elements, and the inverse projection transformation matrix was called to complete the feature reconstruction. The second experimental group had a split decision scalar value of 0.6, which was between the lower limit threshold of 0.4 and the upper limit threshold of 0.8. The edge nodes kept their network interfaces open and distributed data packets to the cloud nodes to complete edge-cloud collaborative streaming computing. The feature matrix dimensionality reduction was not initiated, and the data packets were not switched to the local computing core. The third experimental group used 0.8 as the upper limit threshold. When the split decision scalar value obtained in real time was higher than 0.8, the edge node logic closed the network interface and distributed the data packets in the current cache time window to the local computing core for local streaming computing. The first control group canceled the early truncation of the cache time window and continued to receive data packets according to a fixed period. The second control group adjusted the split decision scalar value to 0.95, so that the value exceeded the upper limit threshold and continuously triggered local streaming computing.

[0039] A high-precision network analyzer and system monitoring registers were used to record the data of each group throughout the entire process. Under multiple data influxes and high noise impacts, the global average flow computing response latency of the first, second, and third experimental groups was 58 ms, 62 ms, and 64 ms, respectively, and the overall flow computing throughput was 98.8%, 98.6%, and 98.2%, respectively. After zero-element filling and feature reconstruction, the waveform signal-to-noise ratio of the low-dimensional feature matrix of the first experimental group was restored to 28.5 dB. The overflow count of the dynamic ring buffer was zero in all three experimental groups. In the first control group, the buffer time window was not closed in advance, and data packets were continuously written to the dynamic ring buffer during the burst data input, increasing the average flow computing response latency to 284 ms. The packet loss rate caused by buffer saturation overflow was 15.4%. In the second control group, the flow splitting decision scalar value was 0.95, the network interface was continuously in a logically closed state, and data packets were concentratedly distributed to the local computing core. The local computing load increased with the increase of concurrent data volume, the flow computing response latency increased to 342 ms, and the computing convergence accuracy decreased.

[0040] Example 4: This embodiment combines Figures 1 to 2 This describes a real-time data stream processing method for edge-cloud collaboration, such as... Figure 1 As shown, in step S101, the method involves edge nodes building buffers and calculating slopes to produce feature variances; in step S102, edge nodes fuse network latency and computing power to generate state vectors; and in step S103, edge nodes perform feature dimensionality reduction or local computation based on decision values.

[0041] like Figure 2As shown, the control flow during system operation is as follows: Data packet arrival triggers the system to enter a dynamic circular buffer state. In this state, edge nodes perform operations such as establishing a dynamic circular buffer and calculating the arrival time interval and slope. The state then transitions to a node state evaluation state by generating feature variance. In this state, edge nodes send timestamped data packets and fuse network latency and remaining computing power, thereby generating a node state vector and transitioning to a traffic splitting decision control state. In this state, edge nodes calculate the traffic splitting decision scalar value and evaluate the system load optimization relationship. When the historical decision slip variance exceeds the stability value, the state transitions to an adaptive parameter adjustment state. In this state, edge nodes calculate and compare the slip variance and adaptively lower the lower limit threshold. This lowering of the lower limit and increasing the sampling frequency further optimizes the system's performance. The system re-enters the dynamic ring buffer state. In the traffic splitting decision control state, when the decision value is below the lower threshold, the state transitions to the feature dimensionality reduction sending state. In this state, edge nodes execute the feature dimensionality reduction matrix call and truncate multi-dimensional redundant feature components. If the channel transmission quality is below the communication threshold at this time, the state transitions to the blocking temporary retry state. In this state, edge nodes temporarily store data in the local retry queue and retransmit in timestamp order. After the network latency recovers to the set range, the system re-enters the dynamic ring buffer state. Furthermore, in the traffic splitting decision control state, when the decision value is above the upper threshold, the state transitions to the local streaming computing state. In this state, edge nodes execute logic to close the network interface and distribute data to the local computing core. If the traffic splitting path is blocked in this state, the state will also transition to the blocking temporary retry state.

[0042] Example 5: In this embodiment, multi-dimensional high-frequency sensing components in a distributed industrial field continuously generate large-scale time-series status monitoring data streams. Edge nodes and cloud nodes form a distributed electronic digital data processing system to perform cross-network node scheduling and streaming computation on this data stream. When concurrent network tasks dynamically occupy the channel, the latency of the logical network changes over time, causing blockage in the data processing queue of the edge node's data transformation unit. Data packets containing high-dimensional redundancy features continuously accumulate in the channel, and the dynamic circular buffer subsequently enters a saturation overflow state. The edge node establishes a dynamic circular buffer with a fixed number of data packet storage bits within the buffer time window through the data transformation unit. When a data packet arrives at the network interface, the nanosecond-level system time of the local high-precision timer is read, the difference between the system time of the current data packet and the previous data packet is calculated, and this difference is recorded as the arrival time interval. The first-order difference is calculated on the continuously accumulated arrival time intervals within the current buffer time window to obtain the arrival interval slope.

[0043] In this embodiment, the ratio of the number of data packets accumulated in the dynamic ring buffer to the initial window span is used as the stacking density index. When data packets containing high-dimensional redundancy features continuously flow into the input, the processor periodically reads the number of data packets currently contained from the status register of the dynamic ring buffer. When this number reaches 500 and the initial window span is 1000, the stacking density index is 50%. If the arrival interval slope of three consecutive sampling points is negative, it is determined that the current data processing queue is blocked. The edge node then triggers a discrete truncation event, closes the current cache time window in advance, and writes the subsequent data packets that have not yet been loaded into the window into the next protection time window with a higher dimensionality reduction priority, so as to interrupt the continued accumulation of data packets in the current queue. The data transformation unit periodically extracts the real-time sampling matrix of multi-dimensional monitoring indicators in the data packets within the cache time window, dynamically generates feature variance, and the edge node reads the remaining computing power data from the status register through the data reading bus, and combines the data with the network delay parameters carried by the downlink calculation confirmation packet to generate a node state vector.

[0044] Edge nodes calculate a traffic splitting decision scalar value based on the arrival interval slope, node state vector, and feature variance. The network latency parameter and remaining computing power data contained in the node state vector are incorporated into a pre-defined load optimization relationship. The product of the network latency weight constant and the network latency parameter, plus the product of the computing power distribution weight constant and the remaining computing power data, is then divided by the feature variance. The calculation result, combined with the data packet arrival trend reflected by the arrival interval slope, yields the traffic splitting decision scalar value for the current buffer time window. Both the network latency weight constant and the computing power distribution weight constant are determined through offline multi-condition calibration tests. When the traffic splitting decision scalar value is low... When the lower threshold is 0.4, the edge node calls the built-in feature dimensionality reduction matrix to truncate the multidimensional redundant feature components of the data packets within the cache time window, reducing the original high-dimensional feature matrix from 64 dimensions to 8 dimensions. The edge node writes the feature dimension and spatial sparsity into the header of the low-dimensional data packet and then sends the low-dimensional data packet to the cloud node, which is the target processing node, through the network interface. When the traffic splitting decision scalar value is higher than the upper threshold of 0.8, the edge node logically closes the network interface and distributes the data packets within the cache time window to the local computing core, which then performs local streaming computing and data aggregation processing.

[0045] During continuous control, edge nodes record the scalar values ​​of diversion decisions according to the control cycle, forming a time series within the historical control cycle. Each time a new scalar value of a diversion decision is obtained, it is written into the time series and the oldest record is removed. Then, the slip variance is calculated based on the deviation of each value in the current sequence from the sequence mean. When the continuously updated slip variance exceeds the set stability threshold, the edge nodes lower the lower limit threshold according to the adjustment ratio determined by offline calibration, and simultaneously shorten the sampling truncation period of the buffer time window to increase the sampling truncation frequency. After receiving the low-dimensional data packet, the cloud node reads the feature dimension and spatial sparsity from the header and compares the spatial sparsity with the sparsity critical threshold. When the spatial sparsity... When the sparsity exceeds the sparsity critical threshold, the cloud node calls the inverse projection transformation matrix and performs zero-element filling and feature reconstruction on the low-dimensional feature matrix according to the lightweight operator sequence of sparse matrix algebraic multiplication. After the cloud node or local computing core completes the corresponding streaming computation, it outputs digital instructions to represent the changes in the state of external entities in the high-frequency streaming data monitoring scenario of industrial IoT. When the edge node closes the cache time window in advance and performs feature dimensionality reduction or local streaming computation according to the scalar value of the split decision, the global average streaming computation response latency is maintained within 65 milliseconds when the logical network is affected by the deterioration of the connection state, the overall streaming computation throughput is maintained at 98.5%, and the dynamic circular buffer does not saturate and overflow.

[0046] Example 6: In this embodiment, in a high-frequency data stream monitoring scenario of the Industrial Internet of Things (IIoT), the distributed electronic digital data processing system continuously receives large-scale time-series status monitoring data streams. Edge nodes perform feature verification and anomaly filtering on data packets flowing to the cloud node according to preset logical mapping rules, while simultaneously performing local real-time status monitoring. When field devices generate sudden data streams, the data packet inflow rate exceeds the processing rate of the edge nodes, causing a backlog in the data processing queue within the dynamic circular buffer. When the distribution path is blocked or the channel transmission quality is lower than the communication threshold, some data packets cannot be sent in time, and data packets containing abnormal status information are at risk of being lost due to buffer overflow. The communication threshold is preset according to the minimum channel transmission quality that can maintain normal data packet transmission between the edge node and the cloud node. The edge node establishes a dynamic circular buffer containing a fixed number of data packet storage bits within the buffer time window and sets up a retry queue in the local memory. The processor reads the remaining space of the dynamic circular buffer and the percentage of local memory usage in each sampling cycle. When the distribution path is blocked or the channel transmission quality is lower than the communication threshold, the data packets that were not sent in time, along with their timestamps, are temporarily stored in the retry queue, and the local status monitoring task continues to run while the data packets are waiting to be retransmitted.

[0047] To identify data packets that need to be prioritized, edge nodes compare the monitoring features within the data packets with an event type feature library item by item. The event type feature library is pre-stored in the static memory of the edge nodes and uses key-value pairs to store typical voltage drop waveforms generated by key power sensing components in industrial sites during short circuits or transient overloads, as well as abnormal oscillation bands with frequencies exceeding 50Hz. When the overlap between the monitoring features carried by a data packet and any abnormal feature entry exceeds 85%, the data packet is marked as a negative event record. After a negative event record is detected, the processor calls an insertion instruction to write it to the front of the retry queue and retains the arrival timestamp of the data packet. The processor compares the arrival timestamp with the current system time to obtain the timing deviation value. When the timing deviation value exceeds the preset time delay warning threshold, the storage space of the dynamic circular buffer is reallocated, and the already stored ordinary monitoring data packets are moved backward to make room for the negative event record, allowing it to enter the next computing power scheduling cycle.

[0048] During the continuous writing of data packets to the retry queue, the processor synchronously monitors the local memory usage percentage. When this percentage exceeds 90%, time-domain downsampling is performed on ordinary monitoring data packets stored in the dynamic circular buffer for more than 10 seconds. Within a preset sliding time window, only the maximum, minimum, and arithmetic mean values ​​of the data sequence are retained, and the corresponding original high-frequency sampling points are deleted, reducing the storage space occupied by a single ordinary monitoring data packet by more than 80%. Negative event records do not participate in this time-domain downsampling process, and the freed-up contiguous storage space is used to store their complete data. Edge nodes continuously read the network latency parameters carried by downlink calculation confirmation packets and compare them with a preset threshold range. In comparison, this threshold range is pre-calibrated according to the network latency parameters corresponding to when the network interface can resume normal transmission. When the network latency parameters recover to this threshold range, the processor enables the network interface and rearranges the retry queue according to the timestamps of each data packet, sending the data packets to the cloud node in the order of their timestamps. After adopting the above processing method, even if the burst data flow load reaches the system design limit, negative event records can still enter the next computing power scheduling cycle first and be sent with the retry queue after the channel is restored. System monitoring results show that the average response latency of negative event records remains within 30 milliseconds; when the local memory usage percentage reaches 90%, no negative event records are lost.

[0049] Example 7: In this embodiment, the edge node records the scalar values ​​of traffic splitting decisions within each historical control cycle during continuous operation, forming a time series in chronological order. Upon completion of each control cycle, the processor writes the current scalar value of the traffic splitting decision into the time series and removes the earliest record, ensuring that the data used in the calculation always corresponds to the current historical control cycle. The processor calculates the slip variance based on the deviation between each scalar value of the traffic splitting decision and the sampled mean in the time series, and updates the calculation result after each new scalar value of the traffic splitting decision is written. The stability threshold is pre-determined based on the slip variance corresponding to the edge node when the network latency parameters and remaining computing power data are in a stable state, used to determine whether the scalar value of the traffic splitting decision continues to fluctuate. When the slip variance exceeds the stability threshold in consecutive control cycles, the edge node lowers the lower limit threshold according to a pre-set adjustment ratio and simultaneously increases the sampling cutoff frequency of the buffer time window. The adjustment ratio is calibrated based on the data packet transmission volume, the dynamic ring buffer occupancy status, and the remaining computing power of the edge node before and after the lower limit threshold adjustment, ensuring that a single adjustment of the lower limit threshold does not exceed the effective judgment range of the traffic splitting decision scalar value.

[0050] By increasing the sampling truncation frequency, edge nodes shorten the time interval between two adjacent sampling truncations, enabling the data transformation unit to end the current buffer time window earlier when the arrival status of data packets changes continuously. Subsequent data packets are then written into the next buffer time window, and data packets within the current window continue to undergo feature matrix dimensionality reduction, edge-cloud collaborative streaming computation, or local streaming computation according to the traffic splitting decision scalar value. After the sliding variance recovers to within the stability threshold, the edge nodes stop further lowering the lower limit threshold and maintain the current sampling truncation frequency. Thus, the edge nodes adjust the lower limit threshold and the sampling truncation frequency of the buffer time window based on the fluctuation status of the traffic splitting decision scalar value within the historical control period, ensuring continuous data packet processing in the dynamic circular buffer as network latency parameters and remaining computing power data continue to change.

Claims

1. A real-time data stream processing method for edge-cloud collaboration, characterized in that, Includes the following steps: Step S101: The edge node establishes a dynamic circular buffer containing a fixed number of data packet storage bits within the cache time window. When the data packet arrives at the network interface, it reads the nanosecond-level system time of the local high-precision timer, calculates the difference between the system time of the data packet and the previous data packet and records it as the arrival time interval. It calculates the first-order difference of the continuously accumulated arrival time intervals and records it as the arrival interval slope. At the same time, it extracts the real-time sampling matrix of the multi-dimensional monitoring indicators in the data packet to dynamically generate the feature variance. In step S102, the edge node sends a data packet with a first timestamp tag to the cloud node, receives a downlink calculation confirmation packet carrying network latency parameters returned by the cloud node, and combines the network latency parameters with the remaining computing power data read from the status register to generate a node state vector. In step S103, the edge node calculates the traffic splitting decision scalar value based on the arrival interval slope, node state vector, and feature variance. When the traffic splitting decision scalar value is lower than the set lower threshold, the feature dimensionality reduction matrix is ​​called to truncate the multidimensional redundant feature components of the data packets within the cache time window and output low-dimensional data packets. When the traffic splitting decision scalar value is higher than the set upper threshold, the network interface is closed and the data packets are distributed to the local computing core to complete the local streaming computing.

2. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, The generation of the node state vector in step S102 includes: when the edge node sends a data packet to the cloud node, it writes a first timestamp tag in the header of the data packet; when the cloud node receives the data packet, it reads the system reception time as a second timestamp tag, calculates the difference between the second timestamp tag and the first timestamp tag, and determines it as a network delay parameter; the cloud node attaches the network delay parameter to the downlink calculation confirmation packet and returns it to the edge node, which together with the remaining computing power data read by the edge node from the status register to generate the node state vector.

3. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, Step S103 includes: the edge node calculates the traffic splitting decision scalar value based on the node state vector and feature variance; when the traffic splitting decision scalar value is lower than the lower threshold of 0.4, the feature matrix of the data packets within the cache time window is reduced, and the feature reduction matrix is ​​called to truncate the multi-dimensional redundant feature components of the data packets, so that the dimension of the feature matrix is ​​reduced from 64 dimensions to 8 dimensions; when the traffic splitting decision scalar value is higher than the upper threshold of 0.8, the network interface is closed and the data packets are distributed to the local computing core to complete the local streaming computing.

4. The real-time data stream processing method for edge-cloud collaboration according to claim 3, characterized in that, Step S103, which enables dimensionality reduction of the feature matrix of data packets within the cache time window, includes the following sub-steps: Step S1031, the edge node constructs the original high-dimensional feature matrix of the data packets within the cache time window, and transforms the original high-dimensional feature matrix into a low-dimensional feature space through the feature dimensionality reduction matrix and outputs the low-dimensional feature matrix; Step S1032, the edge node writes the feature dimension and spatial sparsity into the header of the dimensionality-reduced data packet corresponding to the low-dimensional feature matrix, and sends the dimensionality-reduced data packet to the target processing node through the network interface.

5. The real-time data stream processing method for edge-cloud collaboration according to claim 4, characterized in that, After receiving the dimensionality-reduced data packet, the target processing node reads the feature dimension and spatial sparsity carried in the header and compares the spatial sparsity with the set sparsity critical threshold. When the spatial sparsity exceeds the sparsity critical threshold, the target processing node calls the inverse projection transformation matrix to complete zero-element filling and feature reconstruction of the low-dimensional feature matrix.

6. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, It also includes the following steps: Step S104: The edge node continuously records the time series of the diversion decision scalar values ​​within the historical control cycle and calculates the sliding variance of the time series; Step S105: When the sliding variance continuously exceeds the set stability threshold, the edge node lowers the lower limit threshold according to the set adjustment ratio and simultaneously increases the sampling cutoff frequency of the buffer time window.

7. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, The method is applied to the processing of large-scale time-series status monitoring data streams. The fixed number of dynamic ring buffers is 64 to 512. The resolution of the network interface for acquiring nanosecond-level system time is not less than 1ns, and the data reading bus bandwidth between the edge node and the status register is not less than 10Gbps.

8. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, The dynamic generation of feature variance in step S101 includes the following sub-steps: Step S1011: After receiving the data packet within the buffer time window, the edge node periodically extracts the real-time sampling matrix of the multi-dimensional monitoring indicators in the data packet and calculates the sampling mean of each dimension in the real-time sampling matrix; Step S1012: The edge node calculates the squared difference between the real-time sampling value and the sampling mean of each dimension, accumulates and performs a sliding weighted summation on the time axis, and outputs the feature variance.

9. The real-time data stream processing method for edge-cloud collaboration according to claim 1, characterized in that, It also includes the following data packet retransmission steps: Step S106, when the diversion path is blocked or the channel transmission quality is lower than the set communication threshold, the edge node temporarily stores the data packets that are not sent in time in the retry queue of the local memory, and after detecting that the network delay parameter has recovered to the set threshold range, it enables the network interface to send the data packets in the retry queue in the order of timestamps.

Images