AI server data processing optimization system and method based on distributed heterogeneous computing

Abstract

The invention discloses an AI server data processing optimization system and method based on distributed heterogeneous computing, and relates to the technical field of artificial intelligence and distributed heterogeneous computing, the AI server data processing optimization system comprises a distributed heterogeneous computing cluster, a dynamic collaborative scheduling subsystem and a double-layer closed loop verification subsystem; the distributed heterogeneous computing cluster is characterized in that the distributed heterogeneous computing cluster comprises a plurality of computing nodes which are interconnected, and each node integrates at least two hardware units. According to the AI server data processing optimization system and method based on distributed heterogeneous computing, deep collaborative optimization of computing resource scheduling and data stream transmission is achieved by constructing a hardware-data-task ternary graph model and a space-time collaborative propagation algorithm, so that end-to-end processing delay is shortened, and the heterogeneous resource utilization rate is maximized.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and distributed heterogeneous computing, and specifically to an AI server data processing optimization system and method based on distributed heterogeneous computing. Background Art

[0002] With the explosive growth of artificial intelligence applications in cloud inference, training, and edge computing, AI server clusters need to process massive, dynamic, and computationally intensive data loads. To improve processing efficiency, modern AI servers widely adopt a distributed heterogeneous computing architecture, that is, integrating multiple computing units including CPUs, GPUs, FPGAs, and various dedicated AI acceleration chips, and connecting multiple server nodes through a high-speed network to work together. This architecture aims to utilize the expertise of different hardware units and can theoretically significantly improve system throughput and energy efficiency ratio. However, existing technical solutions face a fundamental challenge in achieving efficient data processing optimization: in a distributed heterogeneous environment, there is a lack of deep coordination between the dynamic scheduling of computing tasks and the data flow across devices and nodes, resulting in serious resource utilization imbalance and end-to-end processing latency. Specifically, existing scheduling strategies often focus on local load balancing of computing resources while ignoring the global characteristics and real-time requirements of the data flow. AI data processing tasks usually contain complex forward and backward dependencies, forming a data flow graph, and the task characteristics change dynamically with the input data. Existing methods fail to fully perceive these dynamic task characteristics and the resulting data dependencies, leading to rigid scheduling decisions. The consequences are as follows: powerful heterogeneous units may be idle waiting for data generated by their dependent pre-tasks; or, tasks are assigned to nodes far from the data they need to execute, causing high-latency data migration; or, communication-intensive subtasks are wrongly scheduled to locations with limited network bandwidth. This disconnection between computing resources and data flow makes it impossible to efficiently coordinate and utilize valuable heterogeneous computing power, and the overall system performance is limited by communication bottlenecks and resource idleness, severely restricting the potential of the distributed heterogeneous architecture and unable to meet the growing low-latency, high-throughput AI data processing requirements. Summary of the Invention

[0003] Aiming at the deficiencies of the prior art, the present invention provides an AI server data processing optimization system and method based on distributed heterogeneous computing, which solves the problem of how to achieve deep collaborative optimization of dynamic scheduling of computing resources and real-time transmission of data flow to reduce end-to-end processing latency and maximize heterogeneous resource utilization.

[0004] To achieve the above objectives, the present invention is realized through the following technical solutions: An AI server data processing optimization system based on distributed heterogeneous computing, comprising: a distributed heterogeneous computing cluster, a dynamic collaborative scheduling subsystem, and a double-layer closed-loop verification subsystem; The distributed heterogeneous computing cluster includes multiple interconnected computing nodes, and each node integrates at least two types of hardware units; The dynamic cooperative scheduling subsystem is connected to the heterogeneous resource monitoring module and the task dependency parsing module, and is used to construct a hardware-data-task tripartite graph model and execute cooperative scheduling; The double-layer closed-loop verification subsystem is connected to the dynamic cooperative scheduling subsystem, and includes a real-time verification layer and a feedback verification layer.

[0005] It should be further noted that in the specific implementation process, when the system starts, the computing nodes in the distributed heterogeneous computing cluster are interconnected through a high-speed network. The heterogeneous resource monitoring module built in each node continuously collects real-time computing power utilization rate, memory bandwidth occupancy rate, and cross-device communication delay data of the CPU, GPU, FPGA, and AI acceleration chips; at the same time, the task dependency parsing module receives the AI data stream to be processed, dynamically parses the dependency relationship between tasks and generates a directed acyclic graph. The nodes in the graph represent atomic tasks, and the directed edges represent the task execution order and data transfer path.

[0006] The dynamic cooperative scheduling subsystem constructs a hardware-data-task tripartite graph model based on the above inputs: abstract the physical computing nodes as hardware nodes, and its vector dynamically fuses the current computing power and communication delay metrics; abstract the physical location of the stored data as data nodes, and its vector encodes the data size, access popularity, and the location of the node where it is located; abstract the AI data processing tasks as task nodes, and its vector contains the computing mode characteristics and the dependency depth in the task directed acyclic graph. These three types of nodes are connected by three types of directed edges: the edge weight between hardware nodes reflects the network transmission delay, the edge weight between task nodes reflects the data dependency strength, and the edge weight between task nodes and hardware nodes reflects the expected execution efficiency of the task on this hardware.

[0007] During the scheduling process, the timing cooperative propagation algorithm is run: Forward propagation stage: Starting from the root task node without dependencies, predict the earliest start time of the subtasks layer by layer along the task dependency edges, and generate an initial scheduling plan by combining the execution efficiency of the bound hardware nodes; Backward propagation stage: Trace back from the last-level task node. If it is detected that the hardware node is idle due to unready data, trace back to the data source node along the hardware communication edge, dynamically shorten the data prefetch time window or switch the prefetch path, such as changing the data originally transmitted through Ethernet to be directly connected to the accelerator memory via NVLink. Among them, the condition for switching the data prefetch path is: when the backward propagation detects that the data prefetch delay will cause the hardware idle time to be greater than 50% of the task computing time, automatically enable the standby high-speed link.

[0008] When the hardware resource monitoring module detects that the actual delay of a critical path task exceeds the threshold, it triggers a tearing rescheduling, including: only deconstructing the subgraph associated with this task in the ternary graph model, including its direct predecessor / successor task nodes and the bound hardware nodes, re-executing the spatio-temporal co-propagation algorithm within the subgraph range to generate a local optimization plan, and keeping the non-critical path task scheduling unchanged; among them, the determination of the tearing rescheduling range: only when the delayed task is on the critical path of the current task graph and the delay affects the global progress by more than 3% will the subgraph deconstruction be activated.

[0009] The double-layer closed-loop verification subsystem operates synchronously: the real-time verification layer replicates the production environment topology in an independent sandbox, injects shadow data streams to execute the candidate scheduling plan in parallel. If the actual hardware idle rate deviates from the predicted value by more than 5% or the data arrival delay deviation exceeds 10 milliseconds, this plan will be abandoned; The feedback verification layer captures the optimal execution trajectory of the production environment and injects two types of perturbations through the negative sample adversarial training engine: simulating a sudden drop in network bandwidth by more than 30% or randomly reducing the frequency of hardware units by 20%. Using the performance decay data after perturbation, the vector generation module of the ternary graph model is trained in reverse to improve its tolerance to abnormal working conditions, and the trained parameters are fed back to the scheduling subsystem in real time; during this period, the intensity control of the negative sample perturbation is as follows: the network congestion simulation adopts an exponential decay model, and the duration of the hardware frequency reduction perturbation does not exceed 20% of the task execution cycle. Among them, the vector generation logic is: the hardware node vector is immediately reconstructed when the detected computing power fluctuation exceeds 15%, and the data node vector is updated at the minute level according to the access heat.

[0010] Preferably, the heterogeneous resource monitoring module collects the computing power utilization rate, memory bandwidth occupancy rate, and cross-device communication delay of hardware units in real time; the task dependency parsing module dynamically generates a directed acyclic graph of AI data processing tasks and identifies data dependency relationships.

[0011] Preferably, the hardware-data-task ternary graph model is constructed in the following way: Abstract the computing nodes, storage locations, and data processing tasks into three types of graph nodes; Dynamically generate a vector containing real-time computing power and communication delay for the hardware nodes; Generate a vector containing the storage location and access heat for the data nodes; Generate a vector containing the computing mode and dependency relationship for the task nodes; Map the hardware communication link, task data dependency, and task hardware binding relationship into weighted directed edges.

[0012] Preferably, the collaborative scheduling includes a spatio-temporal co-propagation algorithm: Forward propagation stage: Predict the earliest start time of subtasks along the task dependency edges to generate an initial scheduling plan; Backpropagation stage: Feed back the resource competition status along the hardware communication edge and dynamically adjust the data prefetch path.

[0013] Preferably, during the construction of the hardware-data-task tripartite graph model, a tearing rescheduling mechanism is adopted to handle task delays, including: when the critical path task delay exceeds the threshold, only deconstruct the associated subgraph; re-execute the spatio-temporal co-propagation algorithm within the subgraph range and output a local optimization plan.

[0014] Preferably, the real-time verification layer performs the following operations: Inject shadow data streams in the sandbox environment to parallelly test candidate solutions; Compare the difference between the actual and predicted hardware idle rates; Compare the difference between the actual and predicted data arrival delays.

[0015] Preferably, the feedback verification layer includes a negative sample adversarial training engine: Inject simulated network congestion and hardware downclocking perturbations based on the historical optimal plan; Use the perturbed data to train the tripartite graph model and update the model parameters to the real-time verification layer.

[0016] An AI server data processing optimization method based on distributed heterogeneous computing includes the following steps: S1: Construct a hardware-data-task tripartite graph model; S2: Generate a joint scheduling and data prefetch plan through the spatio-temporal co-propagation algorithm; S3: Perform a two-layer closed-loop verification: real-time sandbox testing and negative sample adversarial training.

[0017] Preferably, for the joint plan, the error between the wake-up time of the computing unit and the data arrival time is less than a predetermined time threshold; the reduction amplitude of the cross-node data migration volume compared to the unoptimized plan exceeds a predetermined ratio threshold.

[0018] Preferably, the parameter update output by the negative sample adversarial training directly drives the iteration of the vector generation strategy of the tripartite graph model.

[0019] The present invention provides an AI server data processing optimization system and method based on distributed heterogeneous computing. It has the following beneficial effects: (1). The AI server data processing optimization system and method based on distributed heterogeneous computing realizes the deep collaborative optimization of computing resource scheduling and data stream transmission by constructing a hardware-data-task ternary graph model and a spatio-temporal collaborative propagation algorithm. The dynamic collaborative scheduling subsystem abstracts computing nodes, data storage, and processing tasks as graph nodes based on real-time resource monitoring and task dependency parsing, maps hardware communication, task dependencies, and binding relationships through weighted directed edges, generates an initial scheduling plan through forward propagation, dynamically adjusts the data prefetch path through backward propagation, and combines the tearing rescheduling mechanism to locally optimize the critical path delay. This mechanism controls the error between the wake-up time of the computing unit and the data arrival time, reduces the amount of cross-node data migration, improves the utilization rate of heterogeneous resources, and avoids hardware idle caused by data dependency waiting or cross-node transmission delay.

[0020] (2). The AI server data processing optimization system and method based on distributed heterogeneous computing forms a self-reinforcing feedback mechanism through real-time sandbox testing and negative sample adversarial training. The real-time verification layer injects shadow data streams in an isolated environment to parallelly test candidate solutions, and ensures the feasibility of the scheduling plan by comparing the hardware idle rate and the predicted data delay value; the feedback verification layer injects perturbations such as simulated network congestion and hardware downclocking based on the historical optimal plan, trains the ternary graph model using the generated negative sample data, and dynamically updates parameters such as hardware vectors and communication edge weights to improve the system's robustness to abnormal working conditions. This "test-train-iterate" closed-loop mechanism enables the system to adapt to dynamic load changes and sudden failures in a distributed heterogeneous environment, continuously optimize the scheduling strategy, and form a complete intelligent optimization link from data perception, collaborative scheduling to verification feedback, providing a good solution for high-throughput and low-latency AI data processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of the overall framework of the present invention; Figure 2 is the control logic timing diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the protection scope of the present invention.

[0023] Please refer to Figure 1 and Figure 2, the present invention provides a technical solution: an AI server data processing optimization system based on distributed heterogeneous computing, including: a distributed heterogeneous computing cluster, a dynamic cooperative scheduling subsystem, and a double-layer closed-loop verification subsystem; The distributed heterogeneous computing cluster includes multiple interconnected computing nodes, and each node integrates at least two types of hardware units; The dynamic cooperative scheduling subsystem is connected to the heterogeneous resource monitoring module and the task dependency parsing module, and is used to construct a hardware-data-task ternary graph model and execute cooperative scheduling; The double-layer closed-loop verification subsystem is connected to the dynamic cooperative scheduling subsystem, and includes a real-time verification layer and a feedback verification layer.

[0024] The heterogeneous resource monitoring module collects the computing power utilization rate, memory bandwidth occupancy rate, and cross-device communication delay of the hardware units in real time; the task dependency parsing module dynamically generates a directed acyclic graph of AI data processing tasks and identifies data dependency relationships.

[0025] It should be further noted that in the specific implementation process, when the heterogeneous resource monitoring module operates continuously, a differential monitoring strategy is executed for different hardware units integrated in each computing node: for the GPU unit, its SM occupancy rate and video memory bandwidth usage ratio are captured in real time. When it is detected that the context switching frequency caused by multi-task queuing exceeds 50 times per second, the sampling frequency is automatically increased to the millisecond level; for the FPGA unit, the utilization rate of the logic unit and the DDR access delay are mainly monitored. If it is found that the queue depth of the memory control unit continues to increase, this hardware is marked as an oversaturated state; among them, the hardware oversaturation response condition is: when the GPU video memory bandwidth occupancy rate > 95% lasts for 200 ms, the monitoring module marks the "resource overload" state, and the task parsing module automatically downgrades the binding weight of the new tasks of this node by 50%.

[0026] The cross-device communication delay monitoring covers three types of channels: PCIe / NVLink / RDMA. When the transmission delay between nodes suddenly increases and the packet retransmission rate exceeds 5%, it is determined that the network link is abnormal and the topology path is recorded. All monitoring data is aggregated to the dynamic cooperative scheduling subsystem in a timestamp-aligned manner. Among them, network anomaly isolation includes: when it is detected that the RDMA path retransmission rate is continuously sampled > 8% three times, the monitoring module amplifies its communication delay value by 3 times and reports it, triggering the parsing module to avoid task binding on this path.

[0027] When the task dependency parsing module processes the input AI data stream, it first constructs the initial dependency relationship based on the input and output declarations in the task metadata, and dynamically expands the dependency graph in the streaming computing scenario: when there is data overlap between the newly arrived task and the executing task, a data dependency edge is immediately inserted; when the task output is referenced by multiple downstream tasks, a virtual aggregation node is automatically created to avoid duplicate data transmission. For task flows with conditional branches, the module maintains probabilistic dependency edges and prunes invalid paths in real time after the branch is determined. When triggered by a hardware failure or priority change event, the module reconstructs the affected task subgraph: if the GPU node fails, the task edges bound to it are released and the execution weights of each hardware node are re-evaluated; if a high-priority task is inserted, its predecessor tasks are forced to adopt preemptive scheduling and the depth of the dependency path is compressed.

[0028] The collaborative operation logic of the two modules: when the resource monitoring module detects a hardware state change, it immediately pushes the event to the task parsing module; the latter then releases the edge weight constraint of the task bound to the hardware and triggers the scheduling subsystem to reallocate tasks. Conversely, when the parsing module identifies a data-intensive task chain, it issues an instruction to the monitoring module to enable cache hit rate monitoring for the associated storage nodes and increase the network bandwidth sampling frequency. In the distributed training scenario, the gradient aggregation task is identified as a critical path node, and the monitoring module performs microsecond-level latency tracking on the AllReduce communication link bound to it. At the same time, the parsing module locks the forward propagation dependency chain of the task to prohibit rescheduling. Among them, the dynamic dependency processing process is as follows: when a new node is added to the streaming task graph, if the matching degree between its input port and the output port of the running task is ≥80%, a strong dependency edge is established; otherwise, a weak dependency edge is created and the data verification mechanism is started. For tasks marked with a priority ≥8, such as real-time inference requests, the parsing module freezes the subgraph structure of its direct predecessor tasks, and the monitoring module performs read-only monitoring on the hardware resources involved in the subgraph.

[0029] The hardware-data-task ternary graph model is constructed in the following ways: Abstract the computing nodes, storage locations, and data processing tasks into three types of graph nodes; Dynamically generate a vector containing real-time computing power and communication latency for the hardware nodes; Generate a vector containing storage location and access popularity for the data nodes; Generate a vector containing computing mode and dependency relationship for the task nodes; Map the hardware communication links, task data dependencies, and task hardware binding relationships into weighted directed edges.

[0030] It should be further noted that in the specific implementation process, when constructing the hardware-data-task ternary graph model, the system first performs an abstract mapping of the physical entities, including the following: Hardware Node Generation Rule: Each physical computing unit is mapped to an independent node, and its real-time vector is dynamically synthesized according to the following logic: When the monitoring module reports that the GPU video memory bandwidth occupancy rate > 90%, the value of the "Computing Availability" dimension in the vector decays by 30%; if the DDR access latency of the FPGA suddenly increases by more than 50%, the "Memory Efficiency" dimension is marked as an abnormal state; the cross-node communication link status directly affects the "Network Affinity" dimension. For example, when the detected RDMA path retransmission rate > 5%, the value of this dimension is reset to zero.

[0031] Data Node Creation Mechanism: Each data shard in the distributed storage corresponds to a data node, and its vector is dynamically updated according to the access pattern: When it is requested by more than 3 tasks within 10 seconds, the value of the "Popularity" dimension linearly increases to the maximum value; if the data shard is stored across nodes, the "Location" dimension records the topological path cost, and when the access latency of the backup copy exceeds 200% of the primary copy, the location identifier is automatically switched. Among them, the trigger conditions for vector dynamic update include the following: Hardware Node Vector: It is immediately reconstructed when the computing power fluctuation > 15% or the communication latency change > 20%; Data Node Vector: It is updated when the access frequency change rate > 50% / minute or the location changes; Task Node Vector: It is recalculated when the task execution fails or the priority is adjusted.

[0032] Task Node Binding Logic, including: The atomic task node vector extracts key attributes from the task directed acyclic graph: Tasks with a recursive depth > 5 levels are marked with the "High Dependence" feature; Tasks that need to cross more than 4 hardware units activate the "Distributed" flag. The binding edge weight between the task node and the hardware node is initialized according to the historical execution data: If the convolution task takes 3 times faster to execute on the GPU than on the FPGA, the GPU binding edge weight is set to 3 times that of the FPGA.

[0033] Dynamic Maintenance Strategies for Three Types of Directed Edges, including the following: 1. Hardware Communication Edge: The initial weight = reference latency × real-time congestion coefficient. When it is monitored that the path packet loss rate for 3 consecutive samples > 2%, the congestion coefficient is increased from 1.0 to 2.5; if the NVLink direct connection channel is idle, a preferred edge with a weight of 0.3 times the reference value is automatically created. Among them, the communication edge optimization path switching includes: When the edge weight > the preset threshold and there is an NVLink path, a new edge is automatically created and the original edge is discarded. For example, when the Ethernet path weight > 1.2 and the weight of the existing NVLink path ≤ 0.5, a new edge is automatically created and the original edge is discarded. If the new path fails to transmit continuously for 2 times, it will fall back to the original path and attach a 50% penalty weight.

[0034] 2. Task dependency edge: Weight = data volume / transmission rate. When the compression rate of the upstream task output data is greater than 60%, the weight value decreases according to the compression ratio. If the downstream task triggers retransmission due to input data verification failure, the dependency edge weight is temporarily doubled.

[0035] 3. Task hardware binding edge: Weight = 1 / expected execution time. When the actual execution time deviates from the predicted value by more than 15% twice in a row, the weight reconstruction is triggered: if the actual execution time is faster, the weight increases by the deviation percentage; otherwise, the weight decreases.

[0036] Abnormal working condition response mechanism includes the following: Hardware node failure: When a GPU node's temperature exceeds the threshold and is forced to reduce its frequency, all its bound edge weights are reset to zero, and the associated tasks are automatically migrated to suboptimal hardware. The edge weight correction rules include: when a task is first executed on a hardware unit, the weight is equal to the inverse of the average duration of similar tasks. After three executions, a weighted moving average is used, that is, the new weight = 0.7 × historical average + 0.3 × the inverse of the current duration.

[0037] Data location mutation: When a storage node disk failure causes data replica migration, the associated data nodes are automatically scanned to update the location vector, while the original dependency edges are released and a new path is rebuilt.

[0038] Task dependency breaking: In streaming computing, if the upstream task times out and the system detects that the downstream task input port times out and does not receive data, the dependency edge is released and the data reconstruction subtask is started.

[0039] Co-scheduling includes spatio-temporal cooperative communication algorithms: Forward propagation phase: predict the earliest start time of subtasks along the task dependency edges and generate an initial scheduling plan; Backward propagation: Feedback on resource contention along hardware communication edges dynamically adjusts data prefetching paths. It should be noted that, in its implementation, when executing the spatiotemporal co-propagation algorithm, the system initializes the scheduling framework with forward propagation: Starting from the root node of the task directed acyclic graph, the system calculates data transfer times based on the task dependency edge weights and uses the target hardware node's execution efficiency vector to predict the earliest subtask start time. For example, if an image preprocessing task requires transferring 10GB of data to a GPU node, and the dependency edge weight indicates a transmission rate of 5GB / s and the GPU's current execution efficiency is 0.8, the predicted transmission time is 2 seconds and the execution time is 1.25 seconds. The earliest start time for this task is the end time of the upstream task + 2 seconds. The initial scheduling plan is generated layer by layer according to this rule, ensuring that tasks without dependency conflicts are executed in parallel to the greatest extent possible. The priority for parameter conflict arbitration is: end-to-end latency optimization > migration reduction > resource utilization improvement.

[0040] During the backpropagation phase, data flow defects are corrected in real time: scanning backward from the last-level task node, if it is detected that the idle rate of the task ready queue of the GPU node is greater than 30%, it is determined that there is a data supply delay. At this time, the hardware communication edge is traced back to the storage node where the data is located: if the data node is located in the NVLink topology of the same cabinet and the communication edge weight is less than 0.4, the high-speed channel is forced to enable pre-fetching of data; if the data needs to be transmitted across cabinets and the communication edge weight is greater than 1.2, data sharding compression or switching to the nearest replica is initiated. When the cumulative idle time of the hardware node due to data waiting exceeds 40% of the task calculation time, the current binding edge is automatically released and the suboptimal hardware is re-evaluated. The conditions for releasing the binding edge are: the hardware node has two consecutive task execution timeouts greater than 30% of the predicted value; the hardware utilization rate is less than 50% for 5 task cycles due to data waiting.

[0041] Dynamic response to network fluctuation strategy: Continuously monitor hardware communication edge status changes during the backpropagation process. When the RDMA path packet loss rate suddenly increases to more than 8%, the data prefetch operation associated with the path is immediately frozen, and the alternative path weight evaluation is started: if the InfiniBand backup path communication edge weight is less than 60% of the original path, the prefetch path is switched and the dependent edge weight is reset; if the weights of all backup paths are degraded, the data reconstruction task is triggered to generate a copy locally on the target node. For time-sensitive tasks, such as real-time inference requests, when backpropagation detects the risk of data prefetch delay, it directly seizes the allocated network bandwidth of low-priority tasks to ensure the completion of critical data stream transmission. Among them, data prefetch optimization cascade response, including the first response, the second response and the third response. The first response is to enable the same-node memory copy, the second response is to switch to the copy in the same NUMA domain, and the third response is to start data compression during cross-domain transmission.

[0042] Abnormal working condition response mechanism includes the following: Data readiness misjudgment handling: When the forward propagation predicted data arrival time deviates by more than 15% from the actual time, the backpropagation module automatically traces the source of the deviation. If the cause is an inaccurate dependent edge weight, such as a 30% drop in the actual transmission rate compared to the predicted value, a 150% weight penalty is applied to that edge. If the cause is hardware node execution delay, such as a sudden GPU thermal throttling, the priority of the bound edge weight at that node is reduced. The forward propagation time prediction rules include: the earliest start time of the root task is equal to the current system timestamp; the earliest start time of the downstream task is equal to MAX(the end time of all upstream tasks) + the transmission time of the dependent edge; and the task end time is equal to the earliest start time + (task computational load / hardware node execution efficiency).

[0043] Hardware resource contention arbitration: When multiple backpropagation instructions simultaneously request high-speed channels, such as NVLink, the system arbitrates based on task priority and latency impact: If the priorities are the same, the task with a smaller data volume is selected to pass first; the impact is calculated as the depth of task dependency multiplied by the latency duration, and the higher the depth and the longer the latency, the higher the priority.

[0044] Cross-node data conflict resolution: If two backpropagation instructions require the same data shard to be migrated to different nodes, such as task A needs to be migrated to Node1 and task B needs to be migrated to Node3, the system checks the data dependency strength: The data copy that is relied on by more downstream tasks obtains the migration priority, and the remaining copies are converted to remote access and the communication edge weights are marked to decay. Among them, the backpropagation trigger conditions include: The idle rate of the hardware node queue is continuously > 25% for 3 seconds; the actual start time of the task is delayed > 200 milliseconds compared to the predicted value.

[0045] Adopt a tearing rescheduling mechanism to handle task latency: When the latency of the critical path task exceeds the threshold, only the associated subgraph is deconstructed; Re-execute the spatio-temporal co-propagation algorithm within the subgraph range and output a local optimization plan.

[0046] It should be further noted that in the specific implementation process, when the hardware resource monitoring module detects that the actual latency of the critical path task exceeds the preset threshold, the system triggers the tearing rescheduling mechanism, including: First, locate the node position of the task in the ternary graph model, trace up all its direct predecessor task nodes until the nearest parallel branch point, and scan down its direct successor task nodes until the convergence point to form a closed subgraph range. The hardware nodes included in this subgraph are limited to the units that have a binding relationship with these tasks, and non-associated hardware is not included in the deconstruction scope.

[0047] Re-run the spatio-temporal co-propagation algorithm within the subgraph range: In the forward propagation stage, only evaluate the task dependency edges within the subgraph. If it is found that the output data of the predecessor task can be compressed and the compression rate > 40%, then update the dependency edge weight and shorten the transmission time window; in the backpropagation stage, focus on optimizing the hardware communication path within the subgraph. When it is detected that the PCIe path weight from FPGA1 to GPU3 > 1.5, automatically create an alternative path bridged by NVLink and migrate the data stream. After the local optimization plan is generated, only replace the scheduling instructions corresponding to the atomic graph part, and the scheduling parameters of the non-critical tasks being executed outside the subgraph remain unchanged.

[0048] Among them, the subgraph range determination rule is: Critical path task: A task whose latency affects the global progress ≥ 3%; Subgraph boundary: Trace up to the first bifurcation point and extend down to the first convergence point; Hardware inclusion principle: only bind physical units of tasks within a subgraph.

[0049] Resource conflict handling during rescheduling: If subgraph optimization requires the use of global shared resources, and the resources have been locked by tasks outside the subgraph, the system starts priority arbitration: compare the dependency depth of the delayed task with the depth of the occupied task, and the priority increases by 2 times for each level of depth increase; if the depth is the same, the task with the shorter remaining execution time is selected to obtain resources first. When a hardware node in a subgraph needs to execute a new scheduling plan but is running a non-critical task, the system calculates the task interruption cost, which is equal to the execution time divided by the total estimated time. If the cost is less than 25%, it is forced to preempt, otherwise it waits until the current task checkpoint before intervening. The resource preemption threshold is calculated as follows: Interruption cost = (computing power resources used by the task / total required computing power) × 100%; Preemption is allowed if the cost is ≤ 25% or the task priority is ≤ 5 (1-10, 10 is the highest).

[0050] Abnormal working condition response mechanism includes the following: Subgraph optimization failure handling: If local rescheduling fails to reduce latency to within a threshold, the system unblocks the subgraph, expands it to the next level to include more parallel branches, and re-executes the optimization process. If failure persists after expansion, the task quality requirements are downgraded to reduce computation time. The conditions for expansion failure include: latency reduction after the initial optimization is less than 30%; and the critical path impact remains greater than 2.5%.

[0051] Hardware resources are unavailable: If the optimization plan specifies migration to the FPGA2 node but the node suddenly fails, historical execution records are automatically enabled: the second most efficient available hardware for the task in the past 72 hours is selected, and interruptible low-priority tasks on the hardware are frozen.

[0052] Data consistency conflict: When rescheduling requires modifying the location of the distributed data replica, but the Node1 replica is being read by other tasks, the system starts the dual-replica transition mode: maintaining the original replica for external tasks, while creating a new replica to serve the tasks in the subgraph. After the external tasks are completed, the old replica is synchronously deleted. The path switching priority strategy includes: First priority: same-node memory copy; Second priority: shared memory path within the NUMA domain; Third priority: cross-node RDMA direct connection.

[0053] The real-time verification layer performs the following operations: injecting shadow data streams in a sandbox environment to parallelly test candidate solutions; comparing the difference between the actual and predicted hardware idle rates; comparing the difference between the actual and predicted data arrival delays. It should be further noted that during the specific implementation process, when the real-time verification layer operates, it completely replicates the hardware topology and software configuration of the production cluster in an independent sandbox environment, including heterogeneous computing unit types, network link bandwidths, and storage node distributions. The injected shadow data streams use de-sensitized copies of real business data, and their task dependencies evolve synchronously with the production environment. During the parallel execution of candidate scheduling solutions in the sandbox, the verification engine simultaneously starts three groups of monitoring, including hardware idle rate tracking, data arrival delay monitoring, and cross-node transmission verification; Hardware idle rate tracking: Record the proportion of the idle duration of the task queue for each GPU / FPGA unit. When the deviation between the actual value and the predicted value is > 5% and lasts for 3 task cycles, mark it as a "resource prediction failure" event; Data arrival delay monitoring: Capture the actual ready timestamp of the data at the task input port. If the delay compared to the preset time in the scheduling solution is > 10 milliseconds, trigger a "data stream break" alarm; Cross-node transmission verification: Compare the actual throughput of the RDMA link with the bandwidth quota applied in the solution. When the actual value is less than 70% of the quota and the data shard verification failure rate is > 3%, determine that the "path reliability is insufficient".

[0054] The results of the above three groups of monitoring are dynamically imported into the decision-making engine: If any solution accumulatively triggers ≥ 2 "resource prediction failures" or ≥ 3 "data stream breaks" within a single test cycle, then immediately discard this solution; If all candidate solutions show "path reliability insufficiency" in two consecutive tests, then automatically relax the network bandwidth constraint conditions to regenerate the solution. Among them, the candidate solution elimination rules include: resource prediction failure ≥ 2 times or data stream break ≥ 3 times within a single cycle → immediately discard; path reliability insufficiency in two consecutive cycles → trigger solution regeneration; delay increase > 50% in the anti-interference test → degrade to a backup solution.

[0055] The abnormal condition response mechanism includes sandbox resource contention handling, shadow data loss and distortion compensation, and network jitter simulation; Sandbox resource contention handling includes: When multiple candidate solutions need to be tested concurrently but the physical resources are insufficient, the system sorts by the solution generation timestamp and preferentially verifies the latest solution; At the same time, start the lightweight simulation mode, that is: use an efficiency model to replace the real execution for non-critical hardware nodes, and only the GPU / FPGA units on the critical path retain full physical testing. Among them, the lightweight simulation trigger conditions include: the number of candidate solutions > the number of available GPU units × 2, and the simulation error rate of non-critical nodes < 8%. Among them, the judgment conditions for excellent anti-interference solutions are: the delay increase ≤ 20% under a 5% packet loss rate; the task failure rate ≤ 1 / 1000 under a 10% packet loss rate.

[0056] Shadow data loss true compensation includes: If the desensitized data causes calculation feature deviation due to encryption, the monitoring module dynamically calibrates the predicted value: When it is detected that the actual execution duration of a certain type of task deviates from the historical benchmark by more than 15%, an equal proportion correction is applied to the predicted values of all candidate solutions in the same batch. Among them, the data distortion compensation logic includes: calibration coefficient = (actual task duration / benchmark duration) × predicted value of the solution; it is enabled only when the deviation directions of the solutions in the same batch are the same (such as all actual values > benchmark values).

[0057] Network jitter simulation includes: Actively injecting controllable packet loss into the sandbox InfiniBand switch to test the stability of candidate solutions under perturbations. If the data delay of the solution increases by less than 20% under a 5% packet loss rate compared to the non-perturbed environment, it is marked as an "excellent anti-perturbation" solution and preferentially output.

[0058] The feedback verification layer contains a negative sample adversarial training engine: Based on the historical optimal solution, simulate network congestion and hardware downclocking perturbations; use the perturbed data to train the ternary graph model and update the model parameters to the real-time verification layer. It should be further noted that in the specific implementation process, when the negative sample adversarial training engine is started, the complete execution trajectory of the historical optimal scheduling solution is captured from the production environment, including the hardware resource utilization curve, the data flow transmission sequence, and the task dependency graph. The core operation of the engine is divided into three stages: perturbation factor injection, degradation scenario generation, and model adversarial training; among them, perturbation factor injection includes network congestion simulation and hardware downclocking attack. Network congestion simulation includes: Dynamically injecting controllable packet loss on the data flow transmission path. When the original solution uses the RDMA path, randomly select 30% of the data packets and apply a packet loss rate of 1% - 15%, and at the same time amplify the associated communication edge weight by 2 - 5 times; Hardware downclocking attack includes: Simulating performance attenuation for the GPU / FPGA units bound to the solution. If the average historical execution duration is T seconds, force the hardware to extend the execution time in the next cycle to 1.2T - 2.0T times, and block 20% - 50% of the video memory bandwidth.

[0059] Degradation scenario generation includes: Executing the perturbed solution and collecting performance attenuation data: When it is detected that the key task delay increases by more than 40% or the heterogeneous resource utilization rate decreases by more than 35%, mark this scenario as a valid negative sample. For high-frequency failure modes, automatically strengthen the intensity of the same type of perturbation, that is: In the future, the hardware downclocking ratio of the same model is increased to 1.5 times the benchmark value.

[0060] Among them, the determination of valid negative samples includes two items: must meet and additional value. Among them, must meet includes: delay increase ≥ 40% or resource utilization rate decrease ≥ 35%, and additional value includes: exposing unrecorded system vulnerabilities, such as communication conflicts in specific hardware combinations.

[0061] Model adversarial training includes: mapping the perturbation parameters and performance degradation in negative samples to the training labels of a ternary graph model. If the task failure rate of a GPU node increases by 25% when the frequency is reduced by 20%, then reduce the value of the "computing reliability" dimension of this node in the vector; if the transmission delay of an InfiniBand path surges by 300% under 5% packet loss, then permanently increase the communication edge weight base by 50%. The trained model parameters are synchronized to the real-time verification layer in real time, and candidate solutions generated with weights strengthened by negative samples are preferentially used in sandbox tests.

[0062] Among them, the abnormal working condition response mechanism includes out-of-control perturbation intensity handling, negative sample overfitting prevention, and real-time production environment protection; Out-of-control perturbation intensity handling includes: when a single injection causes the system performance to degrade by > 80%, such as the entire task chain crashing, the engine automatically rolls back to the previous stable state and divides the perturbation factors: network congestion and hardware frequency reduction are applied in batches to locate the specific failure cause. If it is determined to be a specific hardware compatibility problem, such as abnormal driver after AMD GPU frequency reduction, then add this hardware model to the avoidance list. Among them, the perturbation division strategy includes: The first-level division: separate network congestion and hardware frequency reduction; The second-level division: apply frequency reduction according to hardware model classification; The third-level division: test different frequency reduction ratios for the same model hardware in batches.

[0063] Negative sample overfitting prevention includes: when 10 consecutive negative samples all originate from the same type of perturbation, such as NVLink path packet loss, the engine actively introduces orthogonal perturbation types, such as CPU cache pollution simulation, to force the model to learn generalization ability. At the same time, retain 5% of the original model parameter copies, and perform hot switching when the failure rate of the new parameters in the verification layer test > 15%. Among them, the overfitting prevention trigger conditions are as follows: The same type of perturbation continuously generates ≥ 10 valid negative samples; The failure rate of the new model parameters in the first round of verification layer test > 12%.

[0064] Real-time production environment protection includes: isolating physical resources during training, using containerized resource pools to simulate hardware frequency reduction, and prohibiting direct operation of production equipment. The perturbation of data stream transmission is achieved through software-defined networks, and there is no actual performance loss of the hardware. Among them, the model parameter update rules are as follows: The communication edge weight update amount = original weight × (1 + actual delay increase rate / 100), and the hardware node vector attenuation rate = MIN(50%, performance attenuation rate × 0.8).

[0065] An AI server data processing optimization method based on distributed heterogeneous computing includes the following steps: S1: Construct a hardware-data-task ternary graph model; S2: Generate a joint scheduling and data prefetching scheme through a spatio-temporal collaborative propagation algorithm; S3: Perform double-layer closed-loop verification: real-time sandbox testing and negative sample adversarial training.

[0066] It should be further noted that in the specific implementation process, in the system initialization stage, a hardware-data-task ternary graph model is constructed based on the physical topology of the current distributed cluster: scan all online computing nodes, create hardware nodes for each activated GPU / FPGA unit and inject real-time monitoring vectors, and the real-time monitoring vectors include three dimensions: computing power availability, memory efficiency, and network affinity; traverse the distributed storage system, generate data nodes for data shards stored across nodes and label the location cost and access popularity; parse the input AI task flow, map atomic tasks to task nodes and mark the computing mode and dependency depth. The three types of nodes are connected by dynamically weighted directed edges: the communication edge weights are initialized according to the actual network link type between hardware nodes, NVLink = 0.3, RDMA = 0.7, Ethernet = 1.2, the dependency edge weights are set according to the data dependency strength between task nodes, strong dependency = 1.0, weak dependency = 0.3, and the initial weights are bound between tasks and hardware according to the historical execution efficiency.

[0067] When executing the spatio-temporal collaborative propagation algorithm to generate a joint scheme: in the forward propagation stage, it advances from the root node of the task graph, and if it is found that the bound edge weight of the hardware node < 0.4 when predicting the start time of downstream tasks, mark the inefficient device and automatically insert an alternative hardware evaluation branch; in the reverse propagation stage, monitor the ready state of the data stream. When it is detected that the GPU queue idle rate > 25% and the data prefetch path weight > 1.0, start the path switching decision tree: preferentially enable the in-node memory copy, secondly select the shared channel within the NUMA domain, and finally enable the cross-node compressed transmission. If multiple tasks compete for the same high-speed link, perform arbitration according to the formula: priority = task depth × delay impact factor, and the weight is increased by 2 times for each increase of 1 level in depth. Among them, the update rules of the bound edge weight include: For the first execution of a task: weight = 1 / average time-consuming of the same type of tasks; Subsequent updates: new weight = 0.6 × old weight + 0.4 × (1 / current time-consuming).

[0068] The double-layer closed-loop verification operates according to the following logic: In the real-time sandbox testing stage, clone the production environment topology and inject desensitized data streams to execute candidate schemes in parallel. When it is monitored that the deviation of the actual hardware idle rate from the predicted value continuously > 5%, automatically abandon the scheme and record the failure mode; if the data arrival delay continuously exceeds the preset value of 10 milliseconds three times, trigger an emergency correction of the path weight; During the negative sample adversarial training phase, the optimal solution trajectory for grasping is obtained, and two types of perturbations are injected: applying 5% - 15% random packet loss on the RDMA path, or simulating a 20% - 50% frequency reduction for the bound GPU. When the perturbation causes the task delay to increase by more than 40%, the feature is extracted to update the tripartite graph model, that is: the dimension of the "computing reliability" vector of the frequency-reduced hardware node is scaled down geometrically according to the decay rate, and the communication edge weight base of the high-packet-loss path is permanently increased by 30% - 50%. The parameters obtained from the training are synchronized to the sandbox test environment in real time to form a self-reinforcing loop. Among them, the termination conditions for negative sample training include: Successful scenario: The delay increase of the new model solution is ≤ 25% under 10% packet loss / 30% frequency reduction; Failed scenario: The model's anti-interference ability has not been improved after 5 consecutive trainings.

[0069] During the response to abnormal working conditions, if a hardware node suddenly fails during model construction, such as an FPGA temperature alarm, immediately freeze its bound edge and start the discovery of alternative nodes: retrieve the available device with the second highest efficiency for the same type of task in the historical execution record; when backpropagation detects that all paths of the critical data shard deteriorate, that is, the weight > 1.5, and compressed transmission is not feasible, that is: the data entropy > 0.9, trigger local replica regeneration: preempt the idle storage space of the target node to reconstruct the data, and create a temporary data node to join the graph model; If three consecutive sets of solutions fail for the same reason in the sandbox test, such as all failing due to NVLink congestion, the feedback verification layer will specifically strengthen the perturbation intensity of this scenario, forcing the new model to generate an anti-interference solution.

[0070] Among them, the path switching threshold includes the following three conditions: Same-node switching condition: The current path weight > 0.0; NUMA domain switching: The current path weight > 0.4 and the paths within the domain are available; Compressed transmission enabled: The current path weight > 0.8 and the data compressibility rate > 40%.

[0071] The combined solution reduces the error between the wake-up time of the computing unit and the data arrival time to less than 5 milliseconds; the cross-node data migration volume is reduced by more than 40% compared with the unoptimized solution. It should be further noted that during the specific implementation process, the system strictly follows two core performance indicators when executing the combined optimization solution: the error between the wake-up time of the computing unit and the arrival time of the dependent data is controlled within 5 milliseconds, and the total cross-node data migration volume is reduced by more than 40% compared with the baseline solution. To achieve the first indicator, a four-level time synchronization mechanism is adopted: when a task is scheduled to the target hardware unit, the system predicts the data arrival time based on the hardware communication edge weight. If the GPU node needs to receive 10GB of intermediate data from the FPGA node and the RDMA path weight indicates a transmission time of 48 milliseconds, the hardware wake-up instruction is issued 53 milliseconds before the data arrives. When backpropagation detects a risk of prediction deviation caused by network jitter, such as a sudden increase in the packet loss rate resulting in an actual transmission delay exceeding 60 milliseconds, an emergency channel switch is immediately initiated: first, direct memory copy within the same computing node is preferred, followed by shared links within the NUMA domain, and finally parallel transmission of data shards is used. If all paths cannot meet the time window requirements, the data precision is forcibly reduced to reduce the transmission volume by 35% to compress the time consumption.

[0072] The reduction of the data migration volume is ensured by a triple strategy: First, reusable data shards are identified during the task dependency resolution phase. When it is detected that three consecutive convolution tasks all require the same set of weight parameters, a shared copy is created at the execution location of the first task and the storage location is locked, and subsequent tasks are forcibly scheduled to this node for execution to avoid migration; Second, an intelligent compression decision is enabled in the data prefetch path selection. A dynamic lossy compression algorithm is used for floating-point intermediate data, and the compression rate threshold is set to 45%. When backpropagation determines that the entropy value of the feature map data < 0.7, the ZFP compression module is automatically activated; Finally, the replica distribution is optimized at the storage topology level, and the number of replicas is dynamically adjusted according to the data node access heat vector, that is: replicas of data with a heat > 0.8 are distributed in three cabinets, and only a single replica is retained for data with a heat < 0.3. When it is monitored that the cross-node migration volume of the dataset increases by more than 15% year-on-year, the feedback verification layer injects simulated perturbations: the data is marked as a false high-heat state to induce the system to create redundant replicas, and then the heat determination model is corrected through negative sample training.

[0073] A degradation strategy is implemented under abnormal operating conditions to ensure performance. When global network congestion causes the time error to approach the 5-millisecond threshold, a data truncation mechanism is activated for non-critical tasks (i.e., priority ≤ 6), such as discarding 10% of pixels from the edge of an image. If the amount of data migration is forced to increase due to a sudden failure, such as a primary replica node downtime, the system automatically triggers a compensation mechanism after the migration is complete. This mechanism involves merging an equal amount of low-frequency cold data into storage to free up storage space and ensure a balanced total migration volume within the cycle. If the 40% migration reduction target is not achieved for three consecutive task cycles, the dynamic collaborative scheduling subsystem removes local optimal constraints, allowing some tasks to be migrated to more distant nodes with double the computational efficiency. This indirectly reduces the global migration load by reducing the total number of task execution rounds.

[0074] Parameter updates generated by negative adversarial training directly drive the iteration of the ternary graph model's vector generation strategy. It should be noted that, in specific implementations, these updates directly drive the iteration of the ternary graph model's vector generation strategy, forming a closed-loop self-evolutionary mechanism. After the feedback validation layer completes a round of adversarial training, the new parameter package is injected into the model engine through a versioned channel. First, a difference threshold is compared with the current production model. If the average adjustment in the "computing reliability" dimension of the hardware node vectors exceeds 15% or the cardinality of communication edge weights changes by more than 20%, a hot update process is triggered. The hot update is only effective if the latency fluctuation of critical tasks in sandbox testing is ≤18ms and the migration reduction is ≥37%. During the update, a dual-version parallel mode is used: the new parameters undergo three rounds of stress testing in the sandbox environment, injecting 8%-12% packet loss and 25%-35% hardware frequency reduction, while the old version maintains production scheduling. The switchover is completed only if the new parameter model meets the following conditions during testing: critical task latency fluctuation is less than 18ms and the data migration reduction is stabilized within the range of 37%-42%.

[0075] The parameter iteration process incorporates built-in conflict resolution logic: If a GPU node's computational reliability vector is detected to have been downgraded by 30% by negative training, but real-time monitoring shows that the node has actually had zero failures over the past 24 hours, the system automatically initiates a false positive check. Specifically, the system replicates the node's frequency reduction scenario ten times in a sandbox. If the median performance degradation is less than 70% of the training report value, the parameter update is frozen and marked as an overfitting example. If multiple sets of parameter updates are mutually exclusive, such as requiring an increase in communication edge weights while the data node location vector recommends reducing transmission requirements, a decision tree is used for arbitration: the parameter combination that maximizes end-to-end latency reduction is prioritized; if latency impacts are comparable, the solution that reduces cross-node transmission is selected.

[0076] Rollback to the previous version under abnormal conditions: When there are more than two types of failure modes in the production environment after the deployment of new parameters, such as the concurrency of GPU task queuing timeout and RDMA verification error, and the root cause analysis points to parameter updates, immediately switch back to the previous stable version. After the rollback, a fault analysis report is automatically generated: Extract the hardware status snapshot and data flow snapshot at the failure time point, reconstruct the fault scenario in the feedback verification layer, and locate the problem parameters through the method of controlling variables. Three consecutive rollback events trigger a conservative iteration strategy: Subsequent update packages are split into atomic parameter groups, such as only updating the edge weights related to the network. After each group is verified and passed, they are merged and deployed. When the model stagnates in evolution due to long-term conservative iteration, such as the update amplitude is <5% for 20 consecutive rounds, high-risk and high-gain parameters are forcibly injected, such as perturbing the path weights of the core switch by ±40%, to break through the local optimal trap. Among them, the forced breakthrough mechanism is activated: The model evolution stagnation period > 20 rounds or the production environment performance has not improved for three months; The version rollback threshold includes: ≥2 types of correlated failure events occur within two hours after the deployment of new parameters.

[0077] By constructing a hardware-data-task tripartite graph model and a spatio-temporal collaborative propagation algorithm, the deep collaborative optimization of computing resource scheduling and data flow transmission is realized. Through the dynamic collaborative scheduling subsystem based on real-time resource monitoring and task dependency parsing, computing nodes, data storage, and processing tasks are abstracted as graph nodes. The hardware communication, task dependency, and binding relationships are mapped through weighted directed edges. The initial scheduling plan is generated through forward propagation, and the data prefetch path is dynamically adjusted through backward propagation. Combined with the tearing rescheduling mechanism, the delay of the critical path is locally optimized. This mechanism controls the error between the computing unit wake-up time and the data arrival time, reduces the cross-node data migration volume, improves the utilization rate of heterogeneous resources, and avoids hardware idle caused by data dependency waiting or cross-node transmission delay.

[0078] Through real-time sandbox testing and negative sample adversarial training, a self-reinforcing feedback mechanism is formed. The real-time verification layer injects shadow data flows in an isolated environment to parallelly test candidate solutions. By comparing the hardware idle rate and the predicted data delay value, the feasibility of the scheduling plan is ensured; The feedback verification layer injects perturbations such as simulated network congestion and hardware downclocking based on the historical optimal plan, trains the tripartite graph model with the generated negative sample data, and dynamically updates parameters such as hardware vectors and communication edge weights to improve the system's robustness to abnormal conditions. This "test-training-iteration" closed-loop mechanism enables the system to adapt to dynamic load changes and sudden failures in a distributed heterogeneous environment, continuously optimize the scheduling strategy, and form a complete intelligent optimization link from data perception, collaborative scheduling to verification feedback, providing a good solution for high-throughput and low-latency AI data processing requirements.

[0079] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0080] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI server data processing optimization system based on distributed heterogeneous computing, comprising: Distributed heterogeneous computing cluster, dynamic collaborative scheduling subsystem and double-layer closed-loop verification subsystem; characterized in that: The distributed heterogeneous computing cluster includes multiple interconnected computing nodes, and each node integrates at least two types of hardware units; The dynamic collaborative scheduling subsystem is connected to the heterogeneous resource monitoring module and the task dependency parsing module, and is used to construct a hardware-data-task ternary graph model and execute collaborative scheduling; The double-layer closed-loop verification subsystem is connected to the dynamic collaborative scheduling subsystem, and includes a real-time verification layer and a feedback verification layer.

2. The AI server data processing optimization system based on distributed heterogeneous computing according to claim 1, wherein: The heterogeneous resource monitoring module collects the computing power utilization rate, memory bandwidth occupancy rate and cross-device communication delay of the hardware units in real time; the task dependency parsing module dynamically generates a directed acyclic graph of AI data processing tasks and identifies data dependency relationships.

3. An AI server data processing optimization system based on distributed heterogeneous computing according to claim 2, characterized in that: The hardware-data-task ternary graph model is constructed in the following way: abstracting computing nodes, storage locations, and data processing tasks into three types of graph nodes, dynamically generating a vector containing real-time computing power and communication delay for hardware nodes, generating a vector containing storage location and access heat for data nodes, and generating a vector containing computing mode and dependency relationship for task nodes; mapping hardware communication links, task data dependencies, and task-hardware binding relationships into weighted directed edges.

4. An AI server data processing optimization system based on distributed heterogeneous computing according to claim 3, characterized in that: The collaborative scheduling includes a spatio-temporal collaborative propagation algorithm: Forward propagation stage: Predict the earliest start time of subtasks along task dependency edges to generate an initial scheduling plan; Backward propagation stage: Feedback the resource competition status along hardware communication edges to dynamically adjust the data prefetch path.

5. An AI server data processing optimization system based on distributed heterogeneous computing according to claim 4, characterized in that: During the construction of the hardware-data-task ternary graph model, a tearing rescheduling mechanism is adopted to handle task delays, including: when the delay of critical path tasks exceeds the threshold, only deconstruct the associated subgraph; re-execute the spatio-temporal collaborative propagation algorithm within the subgraph range to output a local optimization plan.

6. The AI server data processing optimization system based on distributed heterogeneous computing according to claim 1, characterized in that: The real-time verification layer performs the following operations: Inject shadow data streams in a sandbox environment to parallelly test candidate solutions, compare the difference between the actual and predicted hardware idle rates, and compare the difference between the actual and predicted data arrival delays.

7. An AI server data processing optimization system based on distributed heterogeneous computing according to claim 6, characterized in that: The feedback verification layer includes a negative sample adversarial training engine: Inject simulated network congestion and hardware downclocking perturbations based on the historical optimal solution; Use the perturbed data to train the ternary graph model and update the model parameters to the real-time verification layer.

8. An AI server data processing optimization method based on distributed heterogeneous computing, characterized in that, Including the following steps: S1: Construct a hardware-data-task ternary graph model; S2: Generate a joint scheduling and data prefetch solution through the spatio-temporal collaborative propagation algorithm; S3: Perform double-layer closed-loop verification: real-time sandbox testing and negative sample adversarial training.

9. An AI server data processing optimization method based on distributed heterogeneous computing according to claim 8, characterized in that: The joint solution makes the error between the wake-up time of the computing unit and the data arrival time less than a predetermined time threshold; the cross-node data migration volume is reduced by more than a predetermined ratio threshold compared with the unoptimized solution.

10. An AI server data processing optimization method based on distributed heterogeneous computing according to claim 8, characterized in that: The parameter update output by the negative sample adversarial training directly drives the iteration of the vector generation strategy of the ternary graph model.

Images