Multi-node oriented cloud data intelligent processing system

Abstract

This invention relates to the field of electronic digital data processing and digital information transmission technology, specifically to a cloud-based intelligent data processing system for multiple nodes. The system includes a data preprocessing unit, a topology-aware scheduling engine, a graph neural network device, and a task execution unit. The data preprocessing unit extracts the temporal dependencies and logical associations of the input data stream to generate a directed acyclic graph (DAG). The topology-aware scheduling engine maps the graph to the physical network topology and calculates link bandwidth margin and congestion status. The graph neural network device jointly encodes node computational load and edge link status to output a task allocation strategy. The task execution unit schedules strongly dependent tasks to adjacent nodes with sufficient link margin and idle computing power according to the strategy, and dynamically migrates tasks to alternative path nodes based on link packet loss rate during execution. This invention reduces data transmission latency across switches in multi-node cascading and avoids node computing power idleness caused by local link congestion.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing and digital information transmission technology, specifically to a cloud-based intelligent data processing system for multiple nodes. Background Technology

[0002] Cloud-based intelligent data processing systems for multiple nodes typically distribute data processing tasks to individual computing nodes within the cluster using static hashing algorithms or round-robin strategies after receiving data streams from the cloud. During intelligent processing, each computing node independently retrieves the required data blocks from the underlying storage. Data forwarding in the underlying network follows standard routing protocols, and the data transmission path between nodes is fixed during system initialization. When a computing node needs to process a data stream with a sequential logical relationship, the data blocks processed by the preceding node are directly pushed to the server where the subsequent node resides. This process is completely unaware of the data block's generation sequence characteristics, the logical association between the source and destination addresses carried in the data packet header, or the real-time data accumulation in the switch port queues along the data transmission path.

[0003] In the above conventional schemes, the task allocation strategy and the actual data transmission capacity of the underlying physical network are disconnected. When the static allocation or polling mechanism allocates data processing tasks with strong logical correlation to different computing nodes, it does not take into account the real-time bandwidth margin of the link between the source node and the target node and the queue depth of the switch port for scheduling. This results in local link congestion of highly correlated data streams on fixed paths, which in turn increases the latency of cross-node data transmission and idles the computing power of nodes. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud-based intelligent data processing system for multiple nodes, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A cloud-based intelligent data processing system for multiple nodes includes a data preprocessing unit, a topology-aware scheduling engine, a graph neural network unit, and a task execution unit. The data preprocessing unit performs feature analysis on the input cloud data stream, extracting temporal dependency features and logical association features between data blocks, and generates a directed acyclic graph (DAG) based on these features. The topology-aware scheduling engine maps the DAG to the cloud-based multi-node physical network topology, calculating the link bandwidth margin and congestion status between the source node and the target computing node in the DAG. The graph neural network unit jointly encodes the node computing load and edge link status in the DAG, outputting a task allocation strategy. The task execution unit schedules data processing tasks with strong dependencies to adjacent nodes with sufficient link margin and idle computing resources according to the task allocation strategy, and dynamically migrates the data processing tasks to alternative path nodes based on the link packet loss rate during task execution.

[0007] Preferably, the data preprocessing device includes a sliding window parsing component and a graph structure construction component. The sliding window parsing component intercepts the input cloud data stream according to a preset time window, extracts the arrival time interval of data packets within the preset time window to construct a time sequence, and parses the source Internet Protocol address and destination Internet Protocol address in the data packet header field to construct a logical association matrix. The graph structure construction component uses the time sequence as node features and the logical association matrix as edge features, and inputs the node features and edge features into a preset graph structure construction algorithm to generate the directed acyclic graph containing node attributes and edge weights.

[0008] Preferably, the topology-aware scheduling engine includes a physical topology mapping component and a link status monitoring component. The physical topology mapping component acquires the connection relationships between switches and computing nodes in the cloud multi-node physical network topology and establishes a mapping table from virtual nodes in the directed acyclic graph to physical nodes in the physical network topology. The link status monitoring component periodically collects the real-time throughput and switch port queue depth of each physical link in the physical network topology, subtracts the real-time throughput from the rated maximum bandwidth of the physical link to obtain the link bandwidth margin, and calculates the congestion status based on the ratio of the switch port queue depth to the total capacity of the switch port queue.

[0009] Preferably, the graph neural network device includes a graph convolutional network layer and a fully connected layer. The graph convolutional network layer takes the adjacency matrix of the directed acyclic graph, the node computational load vector, and the edge link state vector as input. It updates the hidden layer vector of the current node by aggregating the features of neighboring nodes. After propagation through multiple graph convolutional network layers, the global graph feature vector output by the last graph convolutional network layer is input to the fully connected layer. The fully connected layer outputs the allocation probability distribution of each node in the directed acyclic graph for the data processing task, and uses the allocation probability distribution as the task allocation strategy.

[0010] Preferably, the task execution device includes a candidate node screening component and a task distribution component. The candidate node screening component traverses the allocation probability distribution, selects the node with the highest probability value as a candidate node, queries the physical distance between the candidate node and the source node in the physical network topology, and when the physical distance is one hop and the link bandwidth margin corresponding to the candidate node is greater than the preset threshold, and the CPU utilization of the candidate node is lower than the preset utilization lower limit, the task distribution component encapsulates the data processing task with strong dependency relationship into an execution instruction and distributes it to the local execution queue of the candidate node.

[0011] Preferably, the task execution device further includes a packet loss rate statistics component and a state transition component. During the execution of the data processing task, the packet loss rate statistics component counts in real time the number of Transmission Control Protocol (TCP) retransmission messages sent by the source node to the candidate node and the total number of messages sent, and calculates the link packet loss rate based on the ratio of the number of retransmission messages to the total number of messages sent. When the link packet loss rate exceeds a preset packet loss threshold, the state transition component suspends the data processing task on the candidate node, extracts the intermediate computing state in the memory of the candidate node, re-establishes the transmission channel based on the real-time resource state of the candidate path node, and forwards the intermediate computing state to the candidate path node.

[0012] Preferably, the graph structure construction component is also configured with a dynamic update mechanism. When it is detected that the time sequence corresponding to the newly arrived data packet has a time overlap interval with the existing time sequence in the directed acyclic graph, and the source Internet Protocol address and destination Internet Protocol address in the logical association matrix are swapped compared with the address relationship recorded in the directed acyclic graph, the edge weight update process is triggered. The graph structure construction component extracts the data size of the newly arrived data packet and the data size of the data packet corresponding to the existing time sequence, and adjusts the absolute value of the edge weight connecting the corresponding node in the directed acyclic graph according to the difference between the two.

[0013] Preferably, the link status monitoring component acquires data by deploying a lightweight data probe in the switch operating system. The lightweight data probe uses an extended Berkeley packet filter to directly read the byte counter and queue length register in the network interface card register in kernel mode, bypassing the context switching between user mode and kernel mode. The link status monitoring component uses the difference between the byte counter in two adjacent acquisition cycles as the real-time throughput and the direct read value of the queue length register as the switch port queue depth.

[0014] Preferably, the graph convolutional network layer embeds a multi-head attention mechanism. The multi-head attention mechanism assigns independent attention heads to the node computational load vector and the edge link state vector, respectively. It calculates the first attention weight of the current node and its neighboring nodes in the dimension of the node computational load vector and the second attention weight in the dimension of the edge link state vector through vector dot product operation. The first attention weight and the second attention weight are multiplied by the features of the neighboring nodes respectively, and then the feature dimensions are concatenated to obtain the hidden layer vector of the current node that fuses the node computational load vector and the edge link state vector.

[0015] Preferably, the state transition component creates a checkpoint thread on the candidate node that is independent of the data processing task execution thread. The checkpoint thread serializes the memory pointer address of the intermediate computation state and the data block content by writing to a memory-mapped file. It then uses remote direct memory access technology to directly write the serialized data block to a preset receiving memory address of the candidate path node. The candidate path node performs reverse serialization of the data block according to the memory-mapped file to recover the memory pointer address, and sends a resume execution instruction to the task execution device after the reverse serialization is completed.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. A time-series sequence composed of data packet arrival time intervals is extracted by a data preprocessing device, along with a logical association matrix composed of source and destination Internet Protocol (IP) addresses. The time-series sequence is used as node features, and the logical association matrix is ​​used as edge features to construct a directed acyclic graph (DAG). Combined with the link bandwidth margin obtained by the topology-aware scheduling engine and the congestion status calculated by the switch port queue depth, a graph neural network device is used to jointly encode the node computing load and edge link status to output an allocation probability distribution. This allows the task execution device to schedule data processing tasks with strong dependencies to adjacent nodes that are one hop away from each other, have a link bandwidth margin that meets a preset threshold, and have a CPU utilization rate that is below a preset lower limit. This eliminates the disconnect between task allocation and the underlying physical network transmission capacity, reduces the cross-switch cascading latency of highly correlated data streams during transmission, and avoids node computing power idleness caused by local link congestion.

[0018] 2. During the execution of data processing tasks, a lightweight data probe is deployed in the switch operating system. The extended Berkeley packet filter is used to directly read registers in kernel mode to obtain real-time throughput and queue depth. When the link packet loss rate exceeds the preset packet loss threshold, a checkpoint thread independent of the execution thread is used to serialize the intermediate calculation state through a memory-mapped file. The data is then directly written to the preset receiving memory address of the alternative path node via remote direct memory access technology for reverse serialization recovery. This shortens the context switching overhead in the link status monitoring process, reduces the data recalculation cost in the case of single-point link deterioration, and ensures the continuous execution of intelligent processing tasks for long-series data streams. Attached Figure Description

[0019] Figure 1 This is a flowchart of the cloud data stream feature parsing and directed acyclic graph generation process of the present invention;

[0020] Figure 2 This is a flowchart of the physical network topology mapping and link status monitoring process of the present invention;

[0021] Figure 3 A flowchart for generating the graph neural network feature joint encoding and task allocation strategy of the present invention;

[0022] Figure 4 This is a flowchart illustrating the candidate node selection and strong dependency task assignment and execution process of the present invention.

[0023] Figure 5 This is a flowchart of the real-time packet loss rate statistics and task migration triggering process of the present invention;

[0024] Figure 6 This is a flowchart illustrating the intermediate computation state serialization and remote node recovery execution process of the present invention. Detailed Implementation

[0025] This section provides a detailed description of the technical solution. The described implementation methods are feasible and exemplary solutions. Those skilled in the art can reproduce the technical solution and obtain the corresponding technical effects based on the content disclosed in this section without any creative effort.

[0026] In one embodiment, a multi-node cloud-based intelligent data processing system is deployed on a cloud-based distributed computing cluster. The cluster includes multiple physical computing nodes and at least one switching device. The physical computing nodes and the switching device establish physical connections through network interfaces, forming a cloud-based multi-node physical network topology. The system includes a data preprocessing unit, a topology-aware scheduling engine, a graph neural network unit, and a task execution unit. Each unit completes data interaction and command transmission through the cluster's internal network.

[0027] refer to Figure 1 The data preprocessing unit receives the input cloud data stream, performs feature analysis on the cloud data stream, extracts temporal dependency features and logical association features between data blocks, and generates a directed acyclic graph based on the temporal dependency features and logical association features. Specifically, the input cloud data stream is a sequence of continuously arriving data packets. The data preprocessing unit parses the packet header and payload information of the data packets, extracting the arrival timestamp, source address, destination address, and data size information of each data packet. For the extraction of temporal dependency features, the data preprocessing unit calculates the arrival time interval of adjacent data packets, constructs a temporal sequence based on the arrival time interval, and the temporal sequence reflects the time distribution pattern of data packet arrivals and the temporal dependency relationship between consecutive data blocks. For the extraction of logical association features, the data preprocessing unit counts the number of data packet interactions between different address pairs based on the source address and destination address of the data packets, and constructs a feature matrix reflecting the logical association relationship between addresses. The data preprocessing unit uses temporal dependency features as node attributes of the directed acyclic graph (DAG) and logical association features as edge weights. It determines the topological sorting of nodes according to the arrival time of data packets, removes reverse connection edges that may form loops, and generates a DAG containing node attributes and edge weights.

[0028] Here, a formula is introduced to define the data packet sequence as follows: ,in For the first One incoming data packet, For data packets The corresponding arrival timestamp, the arrival time interval between adjacent data packets The calculation method is as follows:

[0029]

[0030] A sequence consisting of all arrival time intervals That is, the time series sequence corresponding to the time-dependent features.

[0031] Define the set of independent network addresses that appear within the preset statistical range as Logical association matrix , where matrix elements The calculation method is as follows:

[0032]

[0033] in This is an indicator function; the function takes the value 1 when the logical condition within the parentheses is true, and 0 otherwise. For data packets The source network address, For data packets The destination network address. Logical association matrix. This is a quantitative expression of logical association characteristics, where the numerical values ​​of matrix elements reflect the interaction strength between corresponding address pairs.

[0034] refer to Figure 2 The topology-aware scheduling engine receives the directed acyclic graph (DAG) generated by the data preprocessing unit and maps it to the cloud-based multi-node physical network topology. It then calculates the link bandwidth margin and congestion status between the source node and target computing node containing the data block in the DAG. Specifically, the topology-aware scheduling engine obtains the connection relationships between switching devices and computing nodes in the cloud-based multi-node physical network topology, constructs an adjacency matrix of the physical network topology, and establishes a mapping table from virtual nodes in the DAG to physical nodes in the physical network topology. This mapping table stores the correspondence between virtual node identifiers and their corresponding physical node identifiers and network addresses. The topology-aware scheduling engine periodically collects the real-time throughput and switching device port queue depth of each physical link in the physical network topology, and calculates the link bandwidth margin and congestion status based on the collected values.

[0035] Here, a formula is introduced to define the physical link. The rated maximum bandwidth is Real-time throughput is Link bandwidth margin The calculation method is as follows:

[0036]

[0037] Define physical link The corresponding switching device port queue depth is The total queue capacity of this port is The congestion status value corresponding to the link The calculation method is as follows:

[0038]

[0039] in The value range is [0,1], and its magnitude is positively correlated with the degree of congestion of the link.

[0040] refer to Figure 3 The graph neural network device receives structural data and node attribute data of a directed acyclic graph (DAG), as well as node computational load data and edge link state data output by a topology-aware scheduling engine. It jointly encodes the node computational load and edge link state in the DAG and outputs a task allocation strategy. Specifically, the node computational load data is a feature vector composed of the computational resource occupancy status of physical nodes, and the edge link state data is a feature vector composed of the bandwidth margin and congestion status of physical links. The graph neural network device takes the adjacency matrix, node computational load vector, and edge link state vector of the DAG as input. Through multi-layer feature aggregation and nonlinear transformation, it generates a feature vector containing global topology information and node and edge state information. Based on this feature vector, it outputs the allocation probability distribution of data processing tasks corresponding to each node in the DAG on different physical nodes, and uses this allocation probability distribution as the task allocation strategy.

[0041] Here, we introduce a formula to define the adjacency matrix of a directed acyclic graph as follows: The adjacency matrix after adding self-loops is ,in It is the identity matrix. for The corresponding degree matrix, where the diagonal elements of the degree matrix represent the degree of the corresponding node. The node feature matrix of a layer in a graph convolutional network is The trainable weight matrix of this layer is The nonlinear activation function is Then the node features are updated as follows:

[0042]

[0043] After feature propagation through multiple layers of graph convolutional networks, the node feature matrix output by the final layer is subjected to global pooling to obtain the global graph feature vector. The calculation method is as follows:

[0044]

[0045] in Let be the total number of nodes in the directed acyclic graph. For the first The output of the first layer of the graph convolutional network Hidden layer feature vectors of each node, This represents the total number of layers in the graph convolutional network.

[0046] The task execution device receives the task allocation strategy output by the graph neural network device. Based on this strategy, it schedules data processing tasks with strong dependencies to adjacent nodes where the link margin meets a preset threshold and computing resources are available. During task execution, it dynamically migrates the data processing tasks to alternative path nodes based on the link packet loss rate. Specifically, a data processing task with strong dependencies corresponds to a task in a directed acyclic graph where two nodes are directly connected by a directed edge. The output data of the preceding node becomes the input data of the following node, and the completion status of the preceding node becomes a prerequisite for the following node to start processing. Based on the allocation probability distribution in the task allocation strategy, the task execution device selects physical nodes that meet the link and computing resource conditions as target nodes and schedules the strongly dependent tasks to these target nodes for execution. During task execution, the device monitors the link packet loss rate between the source and target nodes in real time. When the link packet loss rate exceeds a preset threshold, it triggers a dynamic task migration process, suspends task execution on the current target node, extracts the intermediate computation state of the task, and forwards it to a pre-selected alternative path node, which then resumes task execution.

[0047] Here, a formula is introduced to define the number of Transmission Control Protocol (TCP) retransmission messages sent from the source node to the target node within a statistical period as follows: The total number of messages sent is Packet loss rate The calculation method is as follows:

[0048]

[0049] In this embodiment, to clearly illustrate the mapping relationship between virtual nodes and physical network nodes in a directed acyclic graph and their corresponding state parameters, the following table is constructed:

[0050] Table 1. Mapping parameters between virtual nodes and physical network nodes in a directed acyclic graph.

[0051] Virtual node identifier Temporal feature vector Related source address Associated destination address Mapping physical node identifier Physical node computing resource utilization Link bandwidth margin Congestion status value [0.21,0.34,0.18] 0.28 0.62 0.15 [0.19,0.27,0.22] 0.32 0.58 0.18 [0.33,0.21,0.29] 0.25 0.71 0.12 [0.26,0.31,0.24] 0.30 0.65 0.16

[0052] Table 1 records the mapping relationship between virtual nodes and physical nodes and the corresponding real-time status parameters, providing basic data support for the generation of task allocation strategies and task scheduling.

[0053] In this embodiment, the feature parsing of cloud data streams and the construction of directed acyclic graphs are completed by a data preprocessing device, the mapping of virtual graph structure to physical network and link status monitoring are completed by a topology-aware scheduling engine, the joint encoding of node and edge features and the generation of task allocation strategies are completed by a graph neural network device, and the scheduling of strongly dependent tasks and dynamic migration when links are abnormal are completed by a task execution device, thereby realizing the coordinated matching of task allocation strategies and the transmission capabilities of the underlying physical network.

[0054] In a preferred embodiment, the data preprocessing device includes a sliding window parsing component and a graph structure construction component, and the topology-aware scheduling engine includes a physical topology mapping component and a link status monitoring component.

[0055] The sliding window parsing component extracts the input cloud data stream according to a preset time window, constructs a time series sequence by extracting the arrival time intervals of data packets within the preset time window, and parses the source and destination network addresses in the data packet header fields to construct a logical association matrix. Specifically, the sliding window parsing component is configured with a preset window length and sliding step size, and continuously moves the time window according to the sliding step size, performing independent feature parsing on the data packets within each time window. For the data packets within each time window, the sliding window parsing component sorts the data packets according to the order of their arrival timestamps, calculates the arrival time intervals of adjacent data packets, and constructs the time series sequence corresponding to that window from the arrival time intervals. The sliding window parsing component extracts the source and destination network addresses of each data packet within the window, counts the number of data packet interactions between each address pair, and constructs the logical association matrix corresponding to that window.

[0056] Here, a formula is introduced to define the start timestamp of data stream acquisition as: The length of the sliding window is The sliding step size is , No. The time interval of a sliding window The calculation method is as follows:

[0057]

[0058] in It is a non-negative integer.

[0059] The sliding window parsing component normalizes the time series, eliminating the impact of differences in numerical magnitudes on subsequent graph structure construction. The normalized arrival time interval... The calculation method is as follows:

[0060]

[0061] in This is the mean of all arrival time intervals within the sliding window. This represents the standard deviation of all arrival time intervals within the sliding window.

[0062] The sliding window parsing component performs row normalization on the logical association matrix, and the normalized matrix elements... The calculation method is as follows:

[0063]

[0064] in This represents the total number of unique network addresses appearing within the sliding window.

[0065] The graph structure construction component uses normalized time-series sequences as node features and normalized logical association matrices as edge features. These node and edge features are input into a pre-defined graph structure construction algorithm to generate a directed acyclic graph (DAG) containing node attributes and edge weights. Specifically, the algorithm first clusters the time-series sequences, aggregating data packets with time-series feature similarity exceeding a preset threshold into the same virtual node, using the mean of the corresponding time-series sequence as the node feature. The algorithm then determines the directed edges between virtual nodes based on the logical association matrix, with edge weights being the corresponding normalized element values ​​in the matrix. Finally, the algorithm determines the topological sorting of the virtual nodes according to the arrival time of the data packets, traverses all directed edges, and removes reverse edges (those opposite to the topological sorting direction) to ensure the generated graph structure is free of cycles, ultimately resulting in a directed acyclic graph.

[0066] The graph structure construction component also features a dynamic update mechanism. When a newly arrived data packet's time sequence overlaps with an existing time sequence in the directed acyclic graph (DAG), and the source and destination network addresses in the logical association matrix are swapped compared to the recorded address relationships in the DAG, an edge weight update process is triggered. Specifically, the graph structure construction component continuously monitors newly arrived data packets, determining if their arrival time falls within an existing sliding window time interval. If so, it identifies a time overlap. The component extracts the source and destination network addresses of the data packet and checks if any recorded address pairs exist where the source and destination addresses are completely opposite to those of the current data packet. If so, an edge weight update process is triggered. The component then compares the data size of the newly arrived data packet with the average data size of corresponding data packets in the existing time sequence, adjusting the absolute values ​​of the edge weights connecting the corresponding nodes in the DAG based on the difference.

[0067] Here, we introduce a formula, defining the weight of the corresponding edge before the update as follows: The updated edge weights are The preset update weight coefficient is The size of the newly arrived data packet is The average data size of the data packets corresponding to the existing time series is Then the edge weights are updated as follows:

[0068]

[0069] in The value range is (0,1).

[0070] The physical topology mapping component acquires the connection relationships between switching devices and computing nodes in the cloud-based multi-node physical network topology, establishing a mapping table from virtual nodes in the directed acyclic graph to physical nodes in the physical network topology. Specifically, the component reads the address forwarding table of the switching devices through network management protocols, obtains the correspondence between the switching device ports and the addresses of connected computing nodes, constructs an adjacency matrix of the physical network topology, and the elements of the adjacency matrix reflect whether there is a direct physical connection between two physical nodes. The component assigns a corresponding physical node to each virtual node, establishes a correspondence between the virtual node identifier and the physical node identifier and address information, and stores this information in the mapping table.

[0071] The link status monitoring component periodically collects the real-time throughput and switch port queue depth of each physical link in the physical network topology. It calculates the link bandwidth margin by subtracting the real-time throughput from the rated maximum bandwidth of the physical link, and calculates the congestion status based on the ratio of the switch port queue depth to the total capacity of the switch port queue. The link status monitoring component achieves data collection by deploying a lightweight data probe in the switching device's operating system. This lightweight data probe utilizes an extended Berkeley packet filter to directly read the byte counter and queue length register in the network interface card register in kernel mode, bypassing the context switch between user mode and kernel mode. The link status monitoring component uses the difference in the byte counter values ​​between two adjacent collection periods as the basis for calculating the real-time throughput, and the directly read value of the queue length register as the switch port queue depth.

[0072] Here, a formula is introduced and defined. and The timestamps are for two adjacent data collection periods, and , for The real-time throughput of the physical link is determined by the byte counter value in the network interface card register at any given time. The calculation method is as follows:

[0073]

[0074] In this embodiment, to clearly display the operating parameters and output results of the sliding window parsing component, the following table is constructed:

[0075] Table 2. Parameter Configuration and Output Characteristics of the Sliding Window Parsing Component

[0076] Sliding window number Window time interval Number of data packets in the window Time series mean Standard deviation of time series Number of non-zero elements in the logical association matrix Number of virtual nodes generated Number of directed edges generated [0,T] 1246 0.26 0.11 18 6 14 [ , +T] 1312 0.24 0.13 21 7 16 [2 ,2 +T] 1187 0.28 0.10 16 5 12 [3 ,3 +T] 1279 0.25 0.12 19 6 15

[0077] Table 2 records the analytical parameters and graph structure construction results under different sliding windows, providing data basis for the dynamic updating of the graph structure.

[0078] In this embodiment, a sliding window parsing component is used to realize segmented feature extraction and normalization of cloud data streams. A graph structure construction component is used to generate and dynamically update a directed acyclic graph with node and edge features. A physical topology mapping component is used to establish a precise mapping between the virtual graph structure and the physical network. A lightweight data probe based on the extended Berkeley packet filter is used to realize low-overhead real-time acquisition of link status, ensuring the timeliness and accuracy of graph structure data and physical network status.

[0079] In a preferred embodiment, the graph neural network device includes a graph convolutional network layer and a fully connected layer, wherein a multi-head attention mechanism is embedded within the graph convolutional network layer.

[0080] The graph convolutional network layer takes the adjacency matrix, node computational load vector, and edge link state vector of the directed acyclic graph (DAG) as input. It updates the hidden layer vector of the current node by aggregating neighbor node features. After propagation through multiple layers of graph convolutional network layers, the global graph feature vector output by the final layer is input to the fully connected layer. The fully connected layer outputs the allocation probability distribution of each node in the DAG for the data processing task, which is used as the task allocation strategy. Specifically, the node computational load vector is a multi-dimensional feature vector composed of the computational resource occupancy state of each physical node, and the edge link state vector is a multi-dimensional feature vector composed of the transmission state of each physical link. The number of graph convolutional network layers is set to a preset value. Each layer performs neighbor aggregation and nonlinear transformation on the input node features. As the number of layers increases, the node features can fuse topological and state information from multiple levels of neighbors. The final graph convolutional network layer outputs the hidden feature vectors of all nodes. Global pooling is then applied to these hidden feature vectors to obtain the global graph feature vector. This global graph feature vector contains global topological information of the directed acyclic graph, computational load information for all nodes, and link state information for all edges. The fully connected layer performs linear transformations and non-linear mappings on the global graph feature vector, outputting a vector with the same dimension as the number of physical nodes. A normalization function maps this vector to a probability distribution, where each element corresponds to a task allocation probability for a physical node. Higher probability values ​​indicate that the physical node is more suitable for executing the corresponding data processing task.

[0081] Here, we introduce a formula defining a fully connected layer as consisting of two linear transformation layers. The weight matrix of the first linear transformation layer is: The bias vector is The weight matrix of the second linear transformation layer is The bias vector is The nonlinear activation function is If the normalization function is the Softmax function, then the task assignment probability distribution is... The calculation method is as follows:

[0082]

[0083] The multi-head attention mechanism embedded within the graph convolutional network layer assigns independent attention heads to the node computational load vector and the edge link state vector, respectively. It calculates the first attention weight between the current node and its neighbors in the node computational load vector dimension and the second attention weight in the edge link state vector dimension through vector dot product operations. These first and second attention weights are then multiplied by the features of the neighboring nodes and concatenated along the feature dimension to obtain the hidden layer vector of the current node, which fuses the node computational load vector and the edge link state vector. Specifically, the multi-head attention mechanism sets up two independent attention heads with non-shared parameters, each weighting the node computational load features and the edge link state features separately. Each attention head contains three independent linear transformation layers: query, key, and value, mapping the input features to the corresponding feature space. Attention weights are calculated through the dot product of the query and key vectors, and the value vectors are weighted and summed based on these attention weights to obtain the output features for that attention head. The output features of the two attention heads are concatenated along the feature dimension to obtain the final hidden layer feature vector of the current node.

[0084] Specifically, for each node, the attention head corresponding to the load vector is calculated, and the current node is defined. The node computes the load feature vector as follows Neighbor nodes The node computes the load feature vector as follows The query, key, and value linear transformation weight matrices of this attention head are respectively , , The corresponding query, key, and value vectors are calculated as follows:

[0085]

[0086] superscript This indicates the attention head corresponding to the node's computational load.

[0087] Current node For neighboring nodes First attention weight The calculation method is as follows:

[0088]

[0089] in is the dimension of the key vector.

[0090] Define nodes for the attention heads corresponding to the edge link state vectors. with neighboring nodes The edge link state feature vector between them is The query, key, and value linear transformation weight matrices of this attention head are respectively , , The corresponding query, key, and value vectors are calculated as follows:

[0091]

[0092] superscript This represents the attention head corresponding to the edge link state.

[0093] Current node For neighboring nodes Second attention weight The calculation method is as follows:

[0094]

[0095] in is the dimension of the edge feature key vector.

[0096] Define nodes The set of neighbor nodes is If the feature concatenation operation is Concat, then the current node Hidden layer vectors The calculation method is as follows:

[0097]

[0098] The graph neural network device undergoes offline training before deployment. The training dataset consists of directed acyclic graph samples corresponding to historical cloud data streams, physical network topology samples, link state samples, node computational load samples, and corresponding task assignment labels. Task assignment labels are generated using task completion latency, link congestion rate, and node computational utilization data from historical task execution. When task completion latency is below a preset latency threshold, link congestion rate is below a preset congestion threshold, and node computational utilization is within a preset range, the corresponding task assignment scheme is labeled as a positive sample. The training process uses the cross-entropy loss function as the optimization objective and updates the network layer weight parameters through an adaptive moment estimation optimizer. A preset number of iterations is set during training. Training stops when the loss function value on the validation set no longer decreases after several consecutive iterations, and the trained model weight parameters are saved for subsequent online inference.

[0099] In this embodiment, to clearly illustrate the layer structure and parameter configuration of the graph neural network device, the following table is constructed:

[0100] Table 3 shows the layer structure and parameter configuration of the neural network device.

[0101] Network layer name Input feature dimension Output feature dimension Weight matrix size Activation function type Attention to the number of heads Input layer 8 8 - - - First image: Convolutional layer 8 16 [8,16] ReLU 2 The second image shows a convolutional layer. 16 32 [16,32] ReLU 2 Global pooling layer 32 32 - - - First fully connected layer 32 16 [32,16] ReLU - Second fully connected layer 16 10 [16,10] - - Output layer (Softmax) 10 10 - Softmax -

[0102] Table 3 records the structural parameters and configuration information of each layer of the graph neural network device, providing standardized parameter basis for model deployment and inference.

[0103] In this embodiment, multi-level aggregation of node and edge features in a directed acyclic graph is achieved through multi-layer graph convolutional network layers. Differentiated weighted fusion of node computational load and edge link state is achieved by embedding a multi-head attention mechanism. A standardized task allocation probability distribution is generated through fully connected layers and normalization functions, ensuring the matching degree between the task allocation strategy and physical node resources and link state.

[0104] In a preferred embodiment, the task execution device includes a candidate node screening component and a task distribution component, and the task execution device also includes a packet loss rate statistics component and a state transition component.

[0105] refer to Figure 4The candidate node selection component iterates through the allocation probability distribution in the task allocation strategy, selecting the node with the highest probability value as a candidate node. It then queries the physical distance between the candidate node and the source node in the physical network topology. When the physical distance is one hop, the link bandwidth margin corresponding to the candidate node is greater than a preset threshold, and the processor utilization of the candidate node is lower than a preset utilization lower limit, the task distribution component encapsulates the data processing tasks with strong dependencies into execution instructions and sends them to the local execution queue of the candidate node. Specifically, a one-hop physical distance means that the source node and the candidate node are connected to different ports of the same switching device, and data transmission between the two nodes does not require forwarding through other switching devices. The candidate node selection component sequentially verifies three constraints: the first constraint is that the physical distance between the source node and the candidate node is one hop; the second constraint is that the link bandwidth margin between the source node and the candidate node is greater than a preset threshold; and the third constraint is that the processor utilization of the candidate node is lower than a preset utilization lower limit. When all three constraints are met simultaneously, the candidate node is determined to be a qualified target node. The task distribution component encapsulates the processing logic, input data address, execution parameters, and priority information of data processing tasks with strong dependencies into standardized execution instructions, which are then distributed to the local execution queues of candidate nodes via the cluster's internal network. The candidate nodes execute the received tasks according to the priority order of their local execution queues.

[0106] refer to Figure 5 During the execution of data processing tasks, the packet loss rate statistics component continuously counts the number of Transmission Control Protocol (TCP) retransmission messages sent from the source node to the candidate nodes, compared to the total number of messages sent. The link packet loss rate is calculated based on the ratio of retransmission messages to the total number of messages sent. Specifically, the packet loss rate statistics component is configured with a preset statistical period. Within each statistical period, retransmission markers in TCP messages are captured using kernel-mode probes, and the number of retransmission messages and the total number of messages sent within that period are counted to calculate the link packet loss rate. When the link packet loss rate exceeds a preset packet loss threshold, a state transition process is triggered.

[0107] When the link packet loss rate exceeds a preset packet loss threshold, the state transition component suspends the data processing tasks on the candidate nodes, extracts the intermediate computing state from the candidate nodes' memory, re-establishes the transmission channel based on the real-time resource status of the alternative path nodes, and forwards the intermediate computing state to the alternative path nodes. Specifically, the alternative path nodes are nodes whose probability value is second only to the candidate nodes in the task allocation probability distribution, and whose physical link with the source node does not pass through the currently congested link. They also meet the constraints of having a link bandwidth margin greater than a preset threshold and a processor utilization rate lower than a preset utilization lower limit. The state transition component sends a task pause command to the candidate nodes. Upon receiving the command, the candidate nodes pause the execution thread of the data processing task and save the current instruction execution position and register state.

[0108] refer to Figure 6 The state transition component creates a checkpoint thread on the candidate node, independent of the data processing task execution thread. The checkpoint thread serializes the memory pointer addresses and data block contents of intermediate computation states by writing to a memory-mapped file. Using remote direct memory access (RDM) technology, it directly writes the serialized data blocks to a preset receiving memory address of the candidate path node. The candidate path node deserializes the data blocks according to the memory-mapped file to recover the memory pointer addresses and sends a resumption execution command to the task execution device after deserialization. Specifically, the checkpoint thread has a lower scheduling priority than the task execution thread to avoid interfering with the normal execution of the task. The checkpoint thread maps the virtual memory address space of the task process to a disk file via a memory-mapped system call, allowing operations on the memory space to be directly synchronized to the disk file without additional system call overhead. The checkpoint thread traverses the virtual memory space of the task process, organizing the non-contiguous memory data blocks corresponding to the intermediate computation states into a continuous byte stream according to the memory pointer relationships, while recording the virtual memory base address and offset of each data block, thus completing the serialization process.

[0109] The state transition component establishes a transmission channel between the source node and the alternative path node using Remote Direct Memory Access (RDM). RDM directly accesses the node's memory space via the network interface card, bypassing the operating system kernel and reducing data transmission latency and processor overhead by bypassing the kernel protocol stack. The source node's network interface card directly reads serialized data blocks from the memory-mapped file and sends them directly to the alternative path node's network interface card via the physical network. The alternative path node's network interface card then directly writes the received data blocks to a preset receiving memory address, without processor intervention.

[0110] After receiving the complete serialized data block, the candidate path node initiates the deserialization process. Based on the memory base address and offset recorded in the memory-mapped file, it remaps the continuous byte stream to the virtual memory address space of the candidate path node, restoring a memory layout and pointer addresses completely consistent with the original candidate node. Here, a formula is introduced, defining the base address of the memory-mapped file in the original candidate node as... The original virtual memory address is The preset receiving memory base address in the alternative path nodes is The restored virtual memory address The calculation method is as follows:

[0111]

[0112] After the deserialization process is completed, the alternative path node restores the instruction execution location and register state of the task, and sends a resume execution instruction to the task execution device. After receiving the instruction, the task execution device issues an instruction to the alternative path node to continue the task execution, and the task continues execution from the interrupted position.

[0113] In this embodiment, to clearly illustrate the various configuration parameters during task execution and state transition, the following table is constructed:

[0114] Table 4 Parameter Configuration Table for Task Execution and State Transition

[0115] Parameter name Parameter meaning Preset value Adjustment range Bandwidth margin preset threshold Minimum constraint value for link bandwidth margin 0.3 [0.2,0.5] Processor utilization lower limit Maximum constraint value for candidate node processor utilization 0.4 [0.3,0.6] Preset threshold for packet loss rate The critical value of link packet loss rate that triggers state transition 0.05 [0.02,0.1] Sliding window length Sliding window length for data stream parsing T [0.5T, 2T] Sliding window step Sliding window movement step [0.2T, 0.8T] Link status collection cycle Link status data acquisition interval 100ms [50ms, 500ms] Packet loss rate statistics period Statistical time interval of link packet loss rate 100ms [50ms, 200ms] Edge weight update coefficients The edge weights of the graph structure are dynamically updated. 0.8 [0.1,1.0]

[0116] Table 4 is used to record various configuration parameters during task execution and state transition, providing a standardized parameter basis for system debugging and optimization.

[0117] In this embodiment, a candidate node screening component is used to screen target nodes that meet the link and resource conditions; a task distribution component is used to schedule strongly dependent tasks nearby; a packet loss rate statistics component is used to monitor the link status in real time; an independent checkpoint thread and memory-mapped file are used to achieve low-overhead serialization of intermediate computing states; and remote direct memory access technology is used to achieve high-speed transmission and reverse serialization recovery of intermediate states. This ensures the continuous execution of data processing tasks when the link is abnormal and reduces the cost of task recalculation.

Claims

1. A cloud-based intelligent data processing system for multiple nodes, characterized in that: It includes a data preprocessing device, a topology-aware scheduling engine, a graph neural network device, and a task execution device. The data preprocessing device performs feature analysis on the input cloud data stream, extracts the temporal dependency features and logical association features between data blocks, and generates a directed acyclic graph based on the temporal dependency features and the logical association features. The topology-aware scheduling engine maps the directed acyclic graph to the cloud multi-node physical network topology and calculates the link bandwidth margin and congestion status between the source node and the target computing node where the data block is located in the directed acyclic graph. The graph neural network device jointly encodes the computational load of nodes and the state of edge links in the directed acyclic graph, and outputs a task allocation strategy. The task execution device schedules data processing tasks with strong dependencies to adjacent nodes with sufficient link margin and idle computing resources according to the task allocation strategy, and dynamically migrates the data processing tasks to alternative path nodes according to the link packet loss rate during the execution of the data processing tasks.

2. The cloud-based intelligent data processing system for multiple nodes according to claim 1, characterized in that, The data preprocessing device includes a sliding window parsing component and a graph structure construction component. The sliding window parsing component intercepts the input cloud data stream according to a preset time window, extracts the data packet arrival time interval within the preset time window to construct a time sequence, and parses the source Internet Protocol address and destination Internet Protocol address in the data packet header field to construct a logical association matrix. The graph structure construction component uses the time sequence as node features and the logical association matrix as edge features. It inputs the node features and edge features into a preset graph structure construction algorithm to generate the directed acyclic graph containing node attributes and edge weights.

3. The cloud-based intelligent data processing system for multiple nodes according to claim 1, characterized in that, The topology-aware scheduling engine includes a physical topology mapping component and a link status monitoring component. The physical topology mapping component obtains the connection relationship between switches and computing nodes in the cloud multi-node physical network topology and establishes a mapping table from virtual nodes in the directed acyclic graph to physical nodes in the physical network topology. The link status monitoring component periodically collects the real-time throughput and switch port queue depth of each physical link in the physical network topology, obtains the link bandwidth margin by subtracting the real-time throughput from the rated maximum bandwidth of the physical link, and calculates the congestion status based on the ratio of the switch port queue depth to the total capacity of the switch port queue.

4. The cloud-based intelligent data processing system for multiple nodes according to claim 3, characterized in that, The graph neural network device includes a graph convolutional network layer and a fully connected layer. The graph convolutional network layer takes the adjacency matrix of the directed acyclic graph, the node computational load vector, and the edge link state vector as input. It updates the hidden layer vector of the current node by aggregating the features of neighboring nodes. After propagation through multiple graph convolutional network layers, the global graph feature vector output by the last graph convolutional network layer is input to the fully connected layer. The fully connected layer outputs the allocation probability distribution of each node in the directed acyclic graph for the data processing task, and uses the allocation probability distribution as the task allocation strategy.

5. The cloud-based intelligent data processing system for multiple nodes according to claim 4, characterized in that, The task execution device includes a candidate node screening component and a task distribution component. The candidate node screening component traverses the allocation probability distribution and selects the node with the highest probability value as a candidate node. It queries the physical distance between the candidate node and the source node in the physical network topology. When the physical distance is one hop and the link bandwidth margin corresponding to the candidate node is greater than the preset threshold, and the CPU utilization of the candidate node is lower than the preset utilization lower limit, the task distribution component encapsulates the data processing task with strong dependency relationship into an execution instruction and distributes it to the local execution queue of the candidate node.

6. The cloud-based intelligent data processing system for multiple nodes according to claim 5, characterized in that, The task execution device further includes a packet loss rate statistics component and a state transition component. During the execution of the data processing task, the packet loss rate statistics component counts in real time the number of Transmission Control Protocol retransmission messages sent by the source node to the candidate node and the total number of messages sent, and calculates the link packet loss rate based on the ratio of the number of retransmission messages to the total number of messages sent. When the link packet loss rate exceeds a preset packet loss threshold, the state transition component suspends the data processing task on the candidate node, extracts the intermediate computing state in the memory of the candidate node, re-establishes the transmission channel according to the real-time resource status of the alternative path node, and forwards the intermediate computing state to the alternative path node.

7. The cloud-based intelligent data processing system for multiple nodes according to claim 2, characterized in that, The graph structure construction component is also configured with a dynamic update mechanism. When it is detected that the time sequence corresponding to a newly arrived data packet has a time overlap interval with the existing time sequence in the directed acyclic graph, and the source Internet Protocol address and destination Internet Protocol address in the logical association matrix are swapped compared with the address relationship recorded in the directed acyclic graph, the edge weight update process is triggered. The graph structure construction component extracts the data size of the newly arrived data packet and the data size of the data packet corresponding to the existing time sequence, and adjusts the absolute value of the edge weight connecting the corresponding node in the directed acyclic graph according to the difference between the two.

8. The cloud-based intelligent data processing system for multiple nodes according to claim 3, characterized in that, The link status monitoring component acquires data by deploying a lightweight data probe in the switch operating system. The lightweight data probe uses an extended Berkeley packet filter to directly read the byte counter and queue length register in the network interface card register in kernel mode, bypassing the context switching between user mode and kernel mode. The link status monitoring component uses the difference between the byte counter in two adjacent acquisition cycles as the real-time throughput and the direct read value of the queue length register as the switch port queue depth.

9. The cloud-based intelligent data processing system for multiple nodes according to claim 4, characterized in that, The graph convolutional network layer embeds a multi-head attention mechanism. This mechanism assigns independent attention heads to the node computational load vector and the edge link state vector, respectively. It calculates the first attention weight of the current node and its neighboring nodes in the dimension of the node computational load vector and the second attention weight in the dimension of the edge link state vector through vector dot product operation. The first attention weight and the second attention weight are multiplied by the features of the neighboring nodes, and then the feature dimensions are concatenated to obtain the hidden layer vector of the current node that fuses the node computational load vector and the edge link state vector.

10. The cloud-based intelligent data processing system for multiple nodes according to claim 6, characterized in that, The state transition component creates a checkpoint thread on the candidate node that is independent of the data processing task execution thread. The checkpoint thread serializes the memory pointer address of the intermediate computation state and the data block content by writing to a memory-mapped file. It then uses remote direct memory access technology to directly write the serialized data block to a preset receiving memory address of the candidate path node. The candidate path node performs reverse serialization of the data block according to the memory-mapped file to recover the memory pointer address, and sends a resume execution instruction to the task execution device after the reverse serialization is completed.

Images