Adaptive AI Intelligent Measurement Engine: Elastic Data Stream Orchestration Method and System

By adaptively adjusting the data flow topology and computing resource configuration, the problem of insufficient topology adaptability and untimely resource matching of the AI ​​intelligent measurement engine in streaming measurement tasks with high concurrency, load fluctuations and large differences in task structure has been solved, achieving more efficient resource utilization and fault recovery, and improving operational stability.

CN123114746APending Publication Date: 2026-11-17BEIJING MEIDESONG TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611292955.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing AI intelligent measurement engines suffer from insufficient data flow topology adaptability, untimely matching of computing resources, and low efficiency in recovering from local faults when handling streaming measurement tasks with high concurrency, load fluctuations, and large differences in task structure. They also struggle to meet the processing needs of both low-latency and high-computation tasks.

Method used

By acquiring the data stream's running status, measuring task characteristics, and the resource status of computing nodes, a data stream association status is generated, the load status is predicted, and the topology is adjusted. The topology adjustment requirements are determined, and target computing nodes are selected based on changes in computing requirements and resource status. The bearing relationship of task nodes is updated, the bearing relationship of tasks is adjusted within the scope of the fault's impact, and execution continues from the execution breakpoint. The load prediction parameters are corrected to adapt to continuous changes.

Benefits of technology

It improves the adaptability of data stream orchestration, enhances the utilization rate of computing resources, reduces the overhead of local fault recovery, and improves the operational stability under dynamic load and abnormal operating scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114746A_ABST
    Figure CN123114746A_ABST
Patent Text Reader

Abstract

This application provides a method and system for elastic data stream orchestration of an adaptive AI intelligent computing engine. The method includes: acquiring the data stream running status, computing task characteristics, and computing node resource status to generate a data stream association status; predicting the data stream load status within a time period based on the data stream association status and matching it with the processing capacity of the current data stream topology to determine topology adjustment needs; determining the scope of tasks to be adjusted, structurally adjusting the task nodes and dependencies within the scope of tasks to be adjusted to generate candidate data stream topologies; determining resource adjustment needs, selecting target computing nodes and updating the carrying relationships of task nodes to obtain the target data stream topology; executing computing tasks according to the target data stream topology, and in the event of an anomaly, determining the scope of the fault impact and the execution breakpoint, and continuing execution from the execution breakpoint; and correcting the load prediction parameters and data stream orchestration parameters based on the deviation. This application can improve the adaptability of data stream orchestration, increase the utilization rate of computing resources, and reduce the overhead of local fault recovery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence measurement and distributed data processing technology, and in particular to a flexible data stream orchestration method and system for an adaptive AI intelligent measurement engine. Background Technology

[0002] With the development of large-scale model inference, intelligent analysis, and data iterative deduction, AI intelligent measurement engines need to continuously process streaming measurement tasks with high concurrency, significant load fluctuations, and large differences in task structure. To ensure that measurement tasks are executed according to predetermined dependencies, existing technologies typically use static directed acyclic graphs or pre-set task pipelines to orchestrate data flows, and combine fixed resource configurations or resource scaling mechanisms based on node load to complete task scheduling.

[0003] However, the data throughput, task granularity, and computational requirements of AI measurement data streams continuously change with business status. Static topologies struggle to adjust task nodes, parallel structures, and data dependencies in a timely manner based on load changes. When sudden tasks or localized processing bottlenecks occur, existing solutions typically only allow for increased computing resources or task rescheduling. The lack of coordination between data stream topology adjustments and computing resource allocation easily leads to idle or insufficient computing resources in certain areas. Furthermore, using a unified orchestration method for different measurement tasks makes it difficult to simultaneously address the processing needs of both low-latency and high-computation tasks. In addition, when a task node fails, existing solutions usually rely on restarting the entire data stream or rolling back a large number of tasks, making it difficult to perform localized recovery based on task execution progress and data dependencies, resulting in significant redundant computations. Therefore, existing technologies still suffer from insufficient data stream topology adaptability, untimely matching of data streams with computing resources, and low efficiency in recovering from localized failures. Summary of the Invention

[0004] In view of this, embodiments of this application provide an elastic data stream orchestration method and system for an adaptive AI intelligent computing engine to solve the problems of insufficient data stream topology adaptability, imbalance in computing resources, and low efficiency in local fault recovery in the prior art.

[0005] A first aspect of this application provides a method for elastic data stream orchestration of an adaptive AI intelligent computing engine, comprising: acquiring the data stream running status, computing task characteristics, and computing node resource status; performing time-series correlation according to task execution relationships to generate a data stream correlation status; predicting the data stream load status within a time period based on the data stream correlation status, and matching it with the processing capacity of the current data stream topology to determine topology adjustment requirements; determining the scope of tasks to be adjusted based on the topology adjustment requirements and the data dependencies of the computing tasks; structurally adjusting the task nodes and dependencies within the scope of tasks to be adjusted to generate a candidate data stream topology; and performing data stream... The changes in computational requirements of the topology and the available resource status of computing nodes determine the resource adjustment needs. Target computing nodes are selected and the carrying relationships of task nodes are updated to obtain the target data flow topology. The measurement task is executed according to the target data flow topology. In case of anomalies, the scope of the fault impact and the execution breakpoint are determined based on the task execution progress and data dependencies. The execution state is maintained outside the scope of the fault impact, the task carrying relationships and data dependencies within the scope of the fault impact are adjusted, and execution continues from the execution breakpoint. Based on the deviation between the actual running state and the predicted data flow load state, the load prediction parameters and data flow orchestration parameters are corrected for subsequent data flow orchestration.

[0006] A second aspect of this application provides an elastic data stream orchestration system for an adaptive AI intelligent calculation engine, comprising: an acquisition module, configured to acquire the data stream running status, calculation task characteristics, and computing node resource status, and perform time-series correlation according to task execution relationships to generate a data stream correlation status; a determination module, configured to predict the data stream load status within a time period based on the data stream correlation status, and match it with the processing capacity of the current data stream topology to determine topology adjustment requirements; an adjustment module, configured to determine the scope of tasks to be adjusted based on the topology adjustment requirements and the data dependencies of the calculation tasks, perform structural adjustments on the task nodes and dependencies within the scope of tasks to be adjusted, and generate a candidate data stream topology; and a calculation module, configured to... Based on the changes in computational requirements of candidate data flow topologies and the available resource status of computing nodes, resource adjustment requirements are determined. Target computing nodes are selected and the carrying relationships of task nodes are updated to obtain the target data flow topology. The execution module is used to execute the measurement tasks according to the target data flow topology. In case of anomalies, the scope of the fault impact and the execution breakpoint are determined based on the task execution progress and data dependencies. The execution state is maintained outside the fault impact scope, the task carrying relationships and data dependencies within the fault impact scope are adjusted, and execution continues from the execution breakpoint. The correction module is used to correct the load prediction parameters and data flow orchestration parameters based on the deviation between the actual operating state and the predicted data flow load state for subsequent data flow orchestration.

[0007] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By acquiring the data stream's running status, measuring task characteristics, and computing node resource status, a time-series correlation is performed according to task execution relationships to generate a data stream correlation status. Based on the data stream correlation status, the data stream load status within a given time period is predicted and matched with the processing capacity of the current data stream topology to determine topology adjustment requirements. Based on the topology adjustment requirements and the data dependencies of the measured tasks, the scope of tasks to be adjusted is determined, and the task nodes and dependencies within the scope of tasks to be adjusted are structurally adjusted to generate candidate data stream topologies. Based on changes in the computational requirements of the candidate data stream topologies and the available resource status of computing nodes, resource adjustment requirements are determined, a target computing node is selected, and the carrying relationships of task nodes are updated to obtain the target data stream topology. The measured tasks are executed according to the target data stream topology. In the event of an anomaly, the scope of the fault impact and the execution breakpoint are determined based on the task execution progress and data dependencies. The execution status outside the fault impact scope is maintained, the task carrying relationships and data dependencies within the fault impact scope are adjusted, and execution continues from the execution breakpoint. Based on the deviation between the actual running status and the predicted data stream load status, the load prediction parameters and data stream orchestration parameters are corrected for subsequent data stream orchestration. This application can improve the adaptability of data stream orchestration, enhance the utilization rate of computing resources, and reduce the overhead of local fault recovery. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the elastic data stream orchestration method of the adaptive AI intelligent calculation engine provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the elastic data stream orchestration system of the adaptive AI intelligent calculation engine provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0010] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0011] In existing technologies, AI intelligent computing engines typically organize computing tasks using pre-configured data flow topologies or static directed acyclic graphs, and perform data flow and task scheduling according to preset task dependencies. When resource adjustments are needed, independent resource scaling or task migration is usually performed based on the current load of the computing nodes. This approach can meet the needs of computing scenarios with relatively stable loads, but for dynamic scenarios such as large model inference, batch iterative computing, and multi-task concurrent simulation, data throughput, task computing requirements, and available resources of computing nodes may continuously change, making it difficult for the preset data flow topology to adapt to the actual operating conditions in a timely manner.

[0012] Therefore, existing technologies mainly suffer from insufficient adaptability to data flow topology, imbalance in computing resource matching, and low efficiency in recovering from local faults. On the one hand, when the data flow processing pressure changes, fixed task nodes and dependencies are difficult to adjust their structure according to load changes, easily leading to local processing bottlenecks or resource redundancy. On the other hand, data flow topology adjustments lack correlation with computing resource allocation, making it difficult to match the corresponding computing capacity in a timely manner after topology changes. Furthermore, when local task nodes malfunction, existing solutions often use large-scale rollbacks or re-executions to restore the data flow, which can easily cause repeated processing of completed calculation results and increase fault recovery overhead.

[0013] To address the aforementioned issues, this application acquires the data stream's operational status, assesses task characteristics, and analyzes the resource status of computing nodes. It then performs time-series correlation based on task execution relationships to form a data stream correlation state reflecting data processing needs and computing power capacity. Based on this correlation state, it predicts the data stream load for subsequent periods and matches the predicted load with the processing capacity of the current data stream topology to determine the scope of tasks requiring structural adjustments. Furthermore, it adjusts the corresponding task nodes and dependencies according to the data dependencies of the assessed tasks, generating candidate data stream topologies. Finally, it determines resource adjustment needs based on changes in computational demands caused by the candidate data stream topologies and the available resource status of computing nodes, updating the capacity relationship between task nodes and target computing nodes. This ensures that data stream topology adjustments and computing resource adjustments are linked.

[0014] Furthermore, during the operation of the target data flow topology, this application determines the scope of the fault impact and the execution breakpoint corresponding to the anomaly based on the task execution progress and data dependencies. The execution state outside the fault impact scope remains unchanged; only the task carrying relationships and data dependencies within the fault impact scope are adjusted, and the calculation task continues execution from the execution breakpoint. Simultaneously, based on the deviation between the actual operating state and the predicted data flow load state, the load prediction parameters and data flow orchestration parameters are corrected, enabling subsequent orchestration to continuously adapt to changes in the data flow and computing power state.

[0015] Through the above technical solutions, this application can adaptively adjust the data flow topology according to the predicted load and synchronize the topology changes with the computing resource configuration, thereby improving the adaptability of data flow orchestration and the utilization rate of computing resources. At the same time, by limiting the scope of the fault impact and retaining the execution status of unaffected tasks, it reduces the repetitive calculations in the fault recovery process, reduces the local fault recovery overhead, and improves the operational stability of the AI ​​intelligent calculation engine under dynamic load and abnormal operation scenarios.

[0016] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] Figure 1 This is a flowchart illustrating the elastic data stream orchestration method of the adaptive AI intelligent measurement engine provided in the embodiment. For example... Figure 1 As shown, the method may specifically include: S101: Obtain the data stream running status, calculate task characteristics and computing node resource status, perform time-series association according to task execution relationship, and generate data stream association status; S102, predict the data flow load status within the time period based on the data flow association status, and match it with the processing capacity of the current data flow topology to determine the topology adjustment requirements; S103, Determine the scope of tasks to be adjusted based on the topology adjustment requirements and the data dependencies of the calculation tasks, perform structural adjustments on the task nodes and dependencies within the scope of tasks to be adjusted, and generate candidate data flow topology; S104, Based on the changes in computing requirements of the candidate data flow topology and the available resource status of computing power nodes, determine the resource adjustment requirements, select the target computing power node and update the bearing relationship of the task nodes to obtain the target data flow topology. S105, Execute the measurement task according to the target data flow topology. When an anomaly occurs, determine the scope of the fault impact and the execution breakpoint based on the task execution progress and data dependency relationship. Maintain the execution state outside the scope of the fault impact, adjust the task carrying relationship and data dependency relationship within the scope of the fault impact, and continue execution from the execution breakpoint. S106, based on the deviation between the actual operating state and the predicted data stream load state, correct the load prediction parameters and data stream orchestration parameters for subsequent data stream orchestration.

[0018] In some embodiments, the data stream running status, task characteristics, and computing node resource status are acquired, and time-series correlations are performed according to task execution relationships to generate a data stream correlation status, including: The state data corresponding to the data stream processing process, the calculation task execution process, and the computing node operation process are obtained according to a unified sampling period, and the state data is standardized to form a time-series state sequence. Based on the task nodes and task execution relationships corresponding to the measurement tasks, the time-series state sequence is mapped to a unified time window to establish the association between the data flow state and the corresponding measurement tasks and the computing power nodes. Based on the correlation, extract the data processing pressure, computing requirements and resource availability of each task node, and generate a data flow correlation state that represents the correspondence between data flow processing requirements and computing power carrying capacity.

[0019] Specifically, this embodiment is deployed on an edge-cloud collaborative AI intelligent computing cluster. The cluster includes a cloud computing server and edge inference nodes, continuously carrying out real-time inference of large models, batch data iterative computing, and intelligent data analysis tasks. After the data stream enters the computing engine, the system assigns a task identifier to each computing task and establishes task nodes and data transfer relationships between nodes according to the actual execution chain. For example, a batch iterative computing task can sequentially pass through a data preprocessing node, a model calculation node, and a result aggregation node. Different model calculation parts can also form parallel task nodes. The system also records the computing power nodes currently bound to each task node, so that the data stream status, task execution status, and computing power carrying status can be associated around the same task identifier.

[0020] During the status acquisition phase, the system synchronously acquires status data from the data stream inlet, task executor, and computing power monitoring interface according to a unified sampling period of 1 second. For the data stream processing process, it collects the data input volume, processed data volume, cache backlog, and upstream and downstream waiting status within the current time slice, and calculates the data throughput changes of the corresponding task nodes. For the measurement task execution process, it records the task type, current execution stage, pending data size, completed percentage, expected computation volume, and latency constraints. For the computing power node operation process, it acquires the current schedulable computing power, remaining GPU or RAM, available communication capacity, queued task size, and load saturation level. Data from different sources is re-marked with a unified timestamp, missing sampling points are filled in with adjacent valid states, outliers that significantly exceed the device's acquisition range are removed, and scaling is performed based on the effective change range of each state quantity to form a time-series state sequence arranged according to the sampling time.

[0021] Subsequently, the system uses a 10-second correlation time window and maintains corresponding predecessor nodes, successor nodes, and currently supporting computing power nodes for each task node. When the same computational task is executed across the edge and cloud, the system uses the task scheduling timestamp and data batch identifier to correct for time offsets caused by network transmission, ensuring that the data transmission status generated on the edge side and the reception and computation status of the corresponding task node on the cloud fall within the same correlation window, avoiding state mismatch caused by cross-node transmission delays. Then, the time-series state sequence is mapped to the corresponding task execution link based on the task identifier and task node identifier.

[0022] For the data stream status within the same window, the system first identifies the task node that generated the status. Then, based on the task scheduling records, it determines whether the task node is actually hosted on a cloud computing server or an edge inference node, thus establishing a correspondence between the task node, input data status, execution status, and computing node status. For example, if the input throughput of a model computing node for a batch computing task increases from 180MB / s to 310MB / s within 10 seconds, and the data to be processed increases from 1.2GB to 2.8GB, while the available computing power of its hosting GPU node continuously decreases and the number of queued tasks increases, the system will associate these statuses with the same model computing node, without mixing resource changes from adjacent nodes or other tasks into the node's status.

[0023] Furthermore, after completing the temporal correlation, the system generates a unified state description for each task node. Data processing pressure is determined based on the current data arrival rate, the processed rate, and the cache growth trend. When the data arrival rate consistently exceeds the processing rate, the corresponding pressure state is increased. Computational requirements are determined based on the scale of the data to be processed, the computational complexity of the task, and the remaining execution stages, and are adjusted in conjunction with the latency constraints of real-time inference tasks or batch iterative tasks. Resource availability is determined based on the remaining computing power of the current supporting computing nodes, the task queuing status, and the available data transmission capacity. Taking real-time inference tasks as an example, even if the amount of input data per instance is small, when the task latency constraints are high and the queuing time of the current supporting nodes increases, the system can still generate high data processing pressure and low resource availability. For batch iterative tasks, the continuous computing requirements are determined more by considering the scale of the data backlog and the remaining computational load.

[0024] The system writes the data processing pressure, computing requirements, and resource availability of each task node into the same data flow state structure according to the task execution relationship, and retains the corresponding time window, task node location, and computing power capacity relationship to form a data flow association state. The data flow association state can reflect which task node is currently experiencing processing pressure, how much computing power capacity that task node will subsequently need, and whether the currently bound computing power node has the capacity to continue to handle the load. This provides a unified state basis for predicting short-term load changes, determining whether the data flow topology needs to be adjusted, and determining the scope of computing power resource expansion and contraction.

[0025] By using the above-mentioned correlation method, we can avoid making orchestration decisions based solely on a single throughput or resource utilization rate, ensuring that data stream processing requirements are consistent with the actual computing power capacity, and improving the accuracy and stability of subsequent flexible orchestration decisions.

[0026] In some embodiments, the data flow load status within a predicted time period is determined based on the data flow association status and matched with the processing capacity of the current data flow topology to determine topology adjustment requirements, including: Extract changes in data processing demand and computing power load within a continuous time window from the data stream correlation state to generate load time-series characteristics; Input the load time-series characteristics into the load prediction model to determine the prediction processing pressure of each task node within the prediction period and generate the data stream load status. Based on the execution relationship of task nodes in the current data flow topology, determine the corresponding processing capacity, match the data flow load status with the processing capacity node by node, and determine the target task nodes with insufficient or redundant processing capacity and the corresponding degree of deviation. Based on the dependency position and deviation degree of the target task node in the current data flow topology, a topology adjustment requirement is generated, which represents the scope and direction of the topology adjustment.

[0027] Specifically, in this embodiment, after establishing the data flow association state, the system uses the most recent 120 seconds of continuous running data as the load analysis window, and extracts the data processing pressure, computing requirements, and resource availability of each task node according to a 10-second sliding step. For the same task node, the system compares the data arrival speed, processing completion speed, cache backlog changes, and changes in the available computing power of the supporting computing nodes within adjacent time windows, and jointly encodes the growth direction of data processing requirements with the change direction of computing power capacity.

[0028] For example, when a batch iterative calculation task enters the centralized submission phase, the data arrival rate of its model computing nodes gradually increases from 180MB / s to 320MB / s, and the amount of data to be processed continues to increase. Meanwhile, the available computing power of the GPU nodes decreases from 42% to 18%, and the task queue length increases synchronously. The system combines the incremental data demand, the rate of change in backlog, and the resource capacity decay state formed within a continuous time window into load time-series characteristics. For real-time inference nodes, the system further strengthens the effect of waiting time growth by combining task latency constraints, so that tasks with smaller data volumes but tighter response times can still reflect the actual load pressure.

[0029] The system inputs load time-series characteristics into a lightweight time-series prediction model to perform rolling predictions of the operational status of each task node within the next 30 seconds. The load prediction model can employ a time-series neural network trained based on historical windows or other prediction models capable of handling continuous state sequences. Its input maintains a time granularity consistent with the data stream's associated state, and its output includes data arrival changes, the growth trend of pending tasks, and corresponding processing pressure for each prediction time slice. During the model training phase, historical actual operating results are used as supervisory data, and the training weights are increased for rapidly changing samples formed by concentrated bursts of task submissions to enhance the ability to identify short-cycle load mutations.

[0030] For example, for the aforementioned batch iterative calculation task, if the input throughput of the model calculation node is predicted to continue to increase to around 400MB per second within the next 30 seconds, while cached data continues to accumulate, then a high prediction processing pressure will be generated. For the edge inference node at the same time, if the request volume remains stable and the current resource margin is sufficient, then a relatively stable prediction processing pressure will be generated. The system organizes the prediction processing pressure according to the task node location, forming a data flow load state corresponding to the current data flow topology.

[0031] To determine whether the existing topology can handle the predicted load, the system determines the actual processing capacity of each node based on the execution relationship of task nodes in the current data flow topology. Specifically, the node processing capacity can be determined based on the current number of parallel task instances, the historical stable throughput of a single instance, the availability of the current computing power nodes, and the upstream and downstream data transmission status. The effective processing capacity is then limited by the output capacity of the predecessor node and the receiving capacity of the successor node.

[0032] For example, if a model computation node is currently executed by two parallel instances, and the combined stable processing capacity of these two instances under current resource conditions is 300MB per second, even if the subsequent result aggregation node still has significant capacity, this model computation node remains the processing ceiling of the current link. The system matches the predicted processing demand of 400MB per second with the effective processing capacity of 300MB per second, identifying a processing capacity gap of approximately 100MB per second, and records the duration and growth trend of this gap. Conversely, if the predicted demand of a preprocessing node decreases to 40% of its current processing capacity, and this low-load state persists for multiple prediction time slices, then it is determined that this node has processing capacity redundancy.

[0033] Subsequently, the system marks task nodes with insufficient or redundant processing capacity as target task nodes, and determines the degree of deviation by combining the difference between predicted demand and effective processing capacity, duration, and direction of change. For short-term, minor fluctuations, the current topology can be maintained; for capacity gaps that continue to widen and may propagate downstream, the degree of deviation is increased. The system then tracks the corresponding upstream and downstream relationships based on the dependency position of the target task node in the current data flow topology.

[0034] For example, if the model computation node is located between data preprocessing and result aggregation, and its insufficient processing capacity is mainly caused by the concentrated influx of batch iterative tasks, then the topology adjustment scope should first cover the model computation node and the connections directly related to task splitting and result aggregation. If data backlog also occurs at upstream nodes, the adjustment scope should be expanded upstream. For parallel branches with significantly redundant processing capacity, the adjustment direction should be determined by reducing parallel instances or merging consecutive lightweight tasks.

[0035] Ultimately, the system generates topology adjustment requirements that include target task nodes, the scope of impact, and the direction of expansion or contraction. This provides a basis for subsequently determining the scope of tasks to be adjusted based on task dependencies and generating candidate data flow topologies. By matching the predicted load with the current topology's processing capacity node by node, potential bottlenecks can be identified before data accumulation occurs. This also avoids ineffective expansion and contraction based solely on instantaneous resource utilization, improving the timeliness and accuracy of data flow topology adjustments.

[0036] In some embodiments, the scope of tasks to be adjusted is determined based on topology adjustment requirements and data dependencies of the measurement tasks. The task nodes and dependencies within the scope of tasks to be adjusted are structurally adjusted to generate candidate data flow topologies, including: Based on the topology adjustment requirements, locate the corresponding target task nodes, determine the task nodes that have execution associations with the target task nodes along the data dependency relationships, and form the scope of tasks to be adjusted. Based on the topology adjustment requirements, determine the structural adjustment method corresponding to the scope of tasks to be adjusted, and adjust the execution granularity and connection relationship of task nodes in combination with the separable state and dependency constraints of each task node. Keep the data flow topology outside the scope of the task to be adjusted unchanged, and connect the adjusted task nodes and dependencies to the original data flow topology to generate a candidate data flow topology.

[0037] Specifically, this embodiment continues to use the aforementioned edge-cloud collaborative AI intelligent computing cluster as an example. When the load prediction result indicates that there is a continuous processing capacity gap in the model computing node corresponding to the batch iterative computing task within the next 30 seconds, the system locates the corresponding model computing node in the current data flow topology based on the target task node identifier, deviation degree, and adjustment direction in the topology adjustment requirements, and reads the data dependency relationship between this node and the preceding data preprocessing node and the subsequent result aggregation node.

[0038] The system further tracks related nodes along the task execution direction that have direct data input, result aggregation, or execution sequence constraints with the target task node. Simultaneously, it determines whether related nodes must be adjusted along with the target task node based on task batch identifiers and iteration stages. For example, if the model computation node receives unified batch input from the data preprocessing node, and the computation results need to be consistently merged at the result aggregation node, then the model computation node, along with the data input connections and result aggregation connections directly related to task splitting, are included in the scope of tasks to be adjusted. Real-time inference nodes that only share physical computing power but have no data dependencies are not included in this adjustment scope, thus avoiding unrelated tasks being affected by topology changes.

[0039] Furthermore, after determining the scope of the tasks to be adjusted, the system judges whether to expand or shrink the processing structure based on the topology adjustment requirements. For model computation nodes with insufficient predictive processing capacity, the system first reads the separable states recorded in the task definition. Separable states can be determined based on whether the input data can be processed independently in data blocks, whether there are state dependencies between model computation stages that must be executed sequentially, and whether the results of each subtask can be re-aggregated.

[0040] In some examples, the batch iterative computation task currently processes four independent data partitions. Each data partition can complete inference computation independently under the same round of model parameters. Therefore, the system splits the original model computation node into four parallel task nodes and binds them to the corresponding data partition identifiers. If a certain computation stage must use the output state of the previous stage, the sequential dependency before and after that stage is preserved, and cross-dependency splitting is not performed. For low-load scenarios, when multiple consecutive lightweight task nodes are under low processing pressure for a long time and there is no independent scaling requirement, the system can merge adjacent task nodes into one execution unit and keep the original data semantics of the input and output unchanged to reduce node scheduling and intermediate data transfer overhead.

[0041] Furthermore, after adjusting the execution granularity, the system redetermines the connection relationships based on the original data dependencies. For the four parallel task nodes formed by the split, the output of the original data preprocessing node is sent to the corresponding parallel task nodes according to the data partition identifier, and the outputs of the four parallel task nodes are uniformly connected to the result aggregation node. The result aggregation node determines whether the conditions for subsequent execution are met based on the task batch identifier and the partition completion status. Only when the partition results required for the same batch meet the aggregation conditions will the aggregation result be output to the downstream task nodes.

[0042] For tasks with iterative dependencies, the system retains the state transfer edges between the current iteration and the next iteration to prevent parallel adjustments from disrupting the original execution order. If the topology adjustment requirement indicates redundant processing capacity, some parallel branches are removed according to the task execution relationship, and the corresponding inputs are remapped to the retained task nodes. At the same time, it is verified that the processing capacity of the retained nodes can cover the predicted load.

[0043] Furthermore, when the scope of the task to be adjusted includes task nodes with different execution constraints, the system does not adopt a uniform split ratio. Instead, it determines the intensity of structural adjustment based on the latency requirements and dependency boundaries of each node. For example, for real-time inference nodes that share upstream input with batch computation, only their low-latency direct links are retained, and parallel branches are not expanded with batch computation nodes to avoid increasing the waiting levels of real-time tasks due to local reconstruction.

[0044] Furthermore, after completing the local structural adjustment, the system freezes the data flow topology outside the scope of the task to be adjusted, without changing the execution order, data dependencies, or current execution status of other task nodes. The system records the input and output boundary nodes of the task scope to be adjusted in the original topology, reconnects the adjusted set of task nodes to the two boundaries respectively, and verifies whether the data format, task batch, and execution constraints of each newly added connection are consistent with the original topology.

[0045] For example, the original data preprocessing node remains the input boundary, and the result aggregation node remains the output boundary. Only the intermediate single model computation node is replaced with four parallel task nodes. Therefore, the real-time inference branch, intelligent analysis branch, and other batch task branches all maintain their original operating structure. After the system completes node reachability, dependency closure, and input / output integrity verification, it generates a candidate data flow topology and records the task nodes and dependencies that are added, removed, or changed relative to the current data flow topology.

[0046] By implementing structural adjustments only within the scope of the target task node, the execution granularity and parallel structure of the data stream can be reconstructed in a targeted manner according to the predicted load, while maintaining the stability of the unaffected area. This provides a clear topological basis for matching computing resources with changes in computing requirements based on candidate data stream topologies.

[0047] In some embodiments, after generating the candidate data stream topology, the method further includes: Based on the predicted data flow load status, the candidate data flow topology is executed and deduced to determine the processing status and data flow status of each task node within the prediction period. The processing status is matched with the predicted data stream load status to determine the load capacity of the candidate data stream topology. When the load capacity meets the preset orchestration conditions, the candidate data stream topology is determined as the topology to be switched. Keep the current data flow topology running until the preset switching conditions are met, then migrate the task execution state to the topology to be switched and enable the corresponding task nodes and dependencies.

[0048] Specifically, in this embodiment, after generating a candidate data flow topology, the current data flow topology is not immediately replaced. Instead, the execution of the candidate data flow topology in a non-effective state is performed using the aforementioned data flow load status for the next 30 seconds to determine whether local structural adjustments can cover the upcoming load changes. The system reads the task nodes, node execution granularity, parallel branches, and data dependencies in the candidate data flow topology, and establishes the simulation state based on the current stable processing capabilities of each task node.

[0049] For example, taking the aforementioned batch iterative calculation task as an example, the original model calculation node is split into 4 parallel task nodes. The system divides the future data arrival amount into 4 parallel branches according to the prediction time slice, and simulates the task reception, calculation execution and result output process time slice by time slice based on the data partition size, computational complexity and estimated resource carrying capacity of each branch, thereby obtaining the amount of data to be processed, task completion progress and expected waiting status of each task node in the prediction period.

[0050] During the simulation, the system synchronously simulates upstream and downstream data flow according to the data dependencies in the candidate data flow topology. Data generated by the data preprocessing node only enters the parallel model computing node after meeting the corresponding partition input conditions. After each parallel model computing node completes the current batch calculation, it sends the results to the result aggregation node. When the result aggregation node receives the partition results that meet the requirements of the current batch, it outputs them to the subsequent analysis nodes.

[0051] The system records the data input, output, and buffer changes for each connection relationship within each prediction time slice, and determines whether the upstream output consistently exceeds the downstream receiving capacity. For example, if the data preprocessing node outputs 400MB per second during the peak prediction phase, and the expected total stable processing capacity of the four parallel model computing nodes reaches 460MB per second, and the buffer backlog does not continuously increase during the simulation, it indicates that the split computing structure can cover the prediction input for this phase. If the result aggregation node can only continuously receive 350MB per second, the system will still identify this node as a potential downstream bottleneck in the simulation, rather than solely determining the usability of the candidate topology based on the completion of model computing node expansion.

[0052] The system further matches the derived processing status with the predicted data flow load status node by node. For each task node, the predicted processing pressure is compared with the effective processing capacity that the system can provide, and the node's load status is formed by combining cache growth, task waiting, and dependency blocking. Then, the load status of each node is summarized along the task execution chain to form the load load level of the candidate data flow topology. The preset orchestration conditions can be set according to the application scenario to ensure that there is no continuous processing capacity gap during the prediction period, that critical low-latency tasks do not exceed the allowable waiting range, and that data backlog remains within a recoverable range.

[0053] In some examples, if the four parallel model computing nodes can handle the predicted peak data, the result aggregation node does not experience continuous backlog after adjustment, and the expected waiting time of the real-time inference branch remains within its latency constraints, then the candidate data flow topology meets the preset orchestration conditions and is registered as a topology to be switched. If the simulation still finds that a certain task node has significant insufficient processing capacity, the current data flow topology is retained and continues to run, and the location of the abnormal node and the corresponding capacity gap are returned to the topology adjustment process for re-determining the structural adjustment method.

[0054] Furthermore, once a candidate data stream topology is determined as the topology to be switched, the system does not switch directly at any time. Instead, it continues to process tasks already in the pipeline using the current data stream topology and monitors task batch boundaries, node execution status, and upstream and downstream cache status. This embodiment uses the completion of a complete data batch, the absence of uninterrupted computation at the node to be migrated, and the persistence of corresponding intermediate results as preset switching conditions. For real-time inference tasks, switching can occur after the current request is completed; for batch iterative computation tasks, the switching point is selected when the current iteration round is completed and the next round of data has not yet entered the original model computation node, thus avoiding the simultaneous execution of the same task batch across two topologies.

[0055] When the preset switching conditions are met, the system freezes new task entry within the switching range, records the completed batches, current iteration rounds, unconsumed input data, and generated intermediate results for each relevant task node, and maps the above task execution states to the corresponding task nodes in the switching topology. For the four parallel task nodes formed by splitting the original single model computation node, the corresponding input positions are restored according to the partition identifier of the unprocessed data; completed computation results are directly retained without recalculation. After the state transition is completed, the system enables the newly added task nodes and data dependencies in the switching topology, and removes the subsequent scheduling entry of the replaced task nodes in the original topology. Real-time inference branches and other data flow branches that have not participated in the adjustment continue to run.

[0056] By performing predictive load simulations on candidate data stream topologies before the formal switchover and completing state transitions at stable task boundaries, it is possible to verify in advance whether topology expansion can cover future processing needs, reduce passive adjustments after sudden loads arrive, and reduce task interruptions and redundant calculations caused by topology switching.

[0057] In some embodiments, resource adjustment requirements are determined based on changes in computational demands of candidate data flow topologies and the available resource status of computing power nodes. A target computing power node is selected, and the bearer relationships of task nodes are updated to obtain the target data flow topology, including: Based on the changes in task structure between the candidate data flow topology and the current data flow topology, determine the changes in computational requirements corresponding to the task nodes to be adjusted; Match changes in computing demand with the available resource status of computing nodes to determine the amount of resources to be added or released, and generate resource adjustment requirements. Based on resource adjustment requirements and task node execution constraints, target computing nodes are determined from available computing power nodes, and their corresponding capacity is determined. Update the carrying relationship between the task node to be adjusted and the target computing power node according to the carrying capacity, and associate it with the candidate data flow topology to obtain the target data flow topology.

[0058] Specifically, this embodiment continues to use the aforementioned edge-cloud collaborative AI intelligent computing cluster as an example. After the candidate data flow topology is executed and deduced, the system compares the changes in the number of task nodes, execution granularity, and parallel relationships in the candidate data flow topology and the current data flow topology to determine the changes in computing requirements caused by the topology adjustment. Taking the aforementioned batch iterative computing task as an example, the current data flow topology uses a single model computing node to serially process four data partitions, while the candidate data flow topology splits this model computing node into four parallel task nodes.

[0059] In some examples, the system calculates the computing power required for each parallel task node based on the data partition size, model computation volume, and task arrival volume during the prediction period. This is then subtracted from the existing usable resources already occupied by the original model computing node to obtain the new computing requirement. If the original model computing node stably requires one GPU computing unit, while the four split parallel task nodes require a total of 3.6 equivalent GPU computing units during the peak prediction period, then the increased computing requirement is determined to be 2.6 equivalent GPU computing units. For cases where parallel branches are withdrawn under low load, the remaining computing requirement corresponding to the withdrawn task node is determined as the releaseable computing requirement.

[0060] The system synchronously acquires the available resource status of each cloud computing server and edge inference node, and maps changes in computing demand to a unified resource capacity scale. Resource status not only represents the currently idle resources of a node, but also determines the actual amount of resources that can be allocated within the prediction period by combining queued and reserved tasks. For example, cloud node A can currently provide 1.4 equivalent GPU computing units, node B can provide 1.1, and node C can provide 0.8. However, node C has already reserved 0.4 units for high-priority real-time inference tasks within the prediction period, so node C's actual schedulable resource is 0.4 units. The system matches the actual schedulable resources of each node according to the requirement of adding 2.6 equivalent GPU computing units, forming a resource replenishment amount after meeting the requirement; if the candidate data flow topology reduces task branches, a resource release amount is formed based on the current capacity of the cancelled task node. This generates resource adjustment requirements and records the corresponding task node to be adjusted, the direction of resource change, and the required capacity adjustment scale.

[0061] When determining the target computing power nodes, the system further filters available computing power nodes based on the execution constraints of the task nodes to be adjusted. For batch model computing nodes, the system focuses on verifying whether candidate nodes have the available accelerated computing capabilities, model parameter loading space, and data transmission capabilities to meet model execution requirements. For real-time inference nodes, the system also verifies whether the data transmission latency from the data entry point to the candidate nodes meets the task time limit. In this embodiment, nodes A and B can directly access the current model parameter copy, and their transmission capabilities with the data preprocessing node meet the requirements for parallel data input. Therefore, they are prioritized as target computing power nodes. Although node C has some idle resources, it needs to reload model parameters, and its startup time exceeds the remaining time before the prediction peak is reached. Therefore, it is not the preferred node for this task. The system ultimately determines that node A carries 1.4 equivalent computing units and node B carries 1.1 equivalent computing units. It also supplements the remaining 0.1 equivalent computing units from the resources that the original model computing nodes can continue to utilize, so that the resource requirements corresponding to the new task structure are covered.

[0062] Subsequently, the system allocates workload based on the data size and computational complexity of the four parallel task nodes. For partitions with larger data volumes, a higher workload can be allocated to node A; for data partitions with lower computational complexity, the available computing resources of node B can be shared with other partitions. When a single parallel task node needs to process data across multiple computing nodes, the system establishes sub-workload relationships based on independently divisible data blocks, ensuring that the same data block is bound to a specific computing node during a single execution, avoiding frequent migrations that could lead to loss of intermediate states. The system writes the task node identifier, target computing node identifier, workload, and effective time range into the task workload relationship and associates the updated task workload relationship with the corresponding task node in the candidate data flow topology.

[0063] In resource scaling-down scenarios, if load forecasting indicates that multiple parallel computing branches are consistently under low processing pressure, the candidate data stream topology merges two task nodes. The system first confirms that the corresponding node has no incomplete data, then removes the bearing relationship between the merged task node and the computing power node, marking the released resources as reschedulable so that they can be used for real-time inference or other high-priority computational tasks. After updating the bearing relationship after scaling up or down, the system verifies that each task node has the computing power capacity to meet execution constraints and checks that the data transmission path between upstream and downstream tasks remains reachable from the target computing power node.

[0064] Furthermore, after successful verification, the updated task carrying relationships are written into the candidate data flow topology, forming the target data flow topology for actual computational execution. By converting changes in task structure into explicit changes in computational requirements, and then updating the carrying relationships between task nodes and computing power nodes in conjunction with the status of available resources, data flow topology adjustments and computing power resource scaling can be completed synchronously, reducing resource mismatch caused by insufficient resources after topology expansion and continuous resource occupation after topology contraction.

[0065] In some embodiments, determining the scope of the failure and the execution breakpoint based on task execution progress and data dependencies, maintaining the execution state outside the scope of the failure's impact, adjusting the task carrying relationships and data dependencies within the scope of the failure's impact, and continuing execution from the execution breakpoint includes: Locate the faulty task node based on the abnormal running status, match the task execution progress with the processing status of the corresponding calculated task, and determine the execution breakpoint between completed and incomplete tasks. Track the unfinished tasks and data flow paths associated with the faulty task node along the data dependency relationship, determine the scope of the fault's impact, and freeze the execution status of tasks outside the scope of the fault's impact. Based on the execution constraints of each task node within the scope of the fault and the available resource status of the computing power node, the alternative computing power node is determined, the corresponding task bearing relationship is updated, and the data dependency relationship affected by the fault is rebuilt. Restore the task status and pending data corresponding to the breakpoint, and continue executing the calculation task according to the updated task carrying relationship and data dependency relationship.

[0066] Specifically, this embodiment continues to use the aforementioned edge-cloud collaborative AI intelligent computing cluster as an example. During the execution of the target data flow topology, the system monitors the heartbeat status, task completion progress, data input / output status, and corresponding computing node running status of each task node according to a 1-second sampling period. When a parallel model computing node fails to return an execution response for several consecutive sampling periods, and the corresponding GPU node experiences abnormal load or communication unreachability, the system marks the task node as a faulty task node and reads its associated task batch, data partition, and iteration round.

[0067] For example, a batch iterative calculation task is split into four parallel computing nodes. The third computing node encounters an anomaly while processing data partition C in the 18th iteration. The system confirms from the task execution records that the results of iterations 1 through 17 have been completed and written to the intermediate result storage area. In the 18th iteration, partition C has only completed partial calculations and has not yet generated valid output. Therefore, the execution breakpoint is determined between the completion status of the 17th iteration and the pending status of the 18th iteration. Tasks that have generated complete and valid output are not rolled back; data blocks that have not generated valid output are remarked as pending data.

[0068] Furthermore, after determining the execution breakpoint, the system traces the input source and output destination corresponding to the faulty task node along the data dependencies in the target data flow topology. Taking the aforementioned fault as an example, the input of the third computing node comes from the data preprocessing node, and its output needs to enter the result aggregation node together with the results of the other three parallel computing nodes. Therefore, the system includes the incomplete computation of partition C in round 18, the data flow path between the third computing node and the result aggregation node, and the aggregation state waiting for the partition result in the scope of the fault impact.

[0069] The results of the 18th round of computation already completed by the other three parallel computing nodes remain valid. The real-time inference branch, other batch task branches, and nodes that do not have data dependencies on the current fault are not included in the scope of the fault's impact. The system freezes the status of completed tasks and intermediate results outside the scope of the fault's impact, prohibits the fault recovery process from overwriting the corresponding results, and suspends the final submission of the current batch by the result aggregation node, thereby limiting the recovery process to the actually affected local execution links.

[0070] Subsequently, the system filters alternative computing nodes based on the execution constraints of task nodes within the scope of the fault's impact. For the third computing node, the system reads the model parameter scale, data partition scale, estimated remaining computation, and current iteration stage to determine that the alternative node needs to have the corresponding model runtime space and data access capabilities. The system filters nodes from currently available cloud computing servers that can access the 17th round model state and partition C input data, and compares the available computing power, task queuing status, and data transmission availability of each node. In this embodiment, the backup node D has loaded the same model version, has GPU resources to meet the remaining computational needs, and maintains a usable transmission path with the data preprocessing node and result aggregation node; therefore, node D is determined as the alternative computing node. The system removes the bearing relationship between the faulty task node and the original computing node, updates the subsequent bearing relationship of the third computing node to node D, and records the new bearing start position as the execution breakpoint.

[0071] While updating the task carrying relationships, the system reconstructs the data dependencies affected by the failure. The data transfer connection from the original data preprocessing node to the failed node is switched to the execution instance of the third computing node corresponding to node D. The partition C result generated by node D is reconnected to the original result aggregation node. The system maintains the connection relationships between the other three parallel branches and the result aggregation node unchanged, and uses the task batch identifier, data partition identifier, and iteration round to verify the data correspondence of the new connection to avoid mixing results from different rounds during the recovery process. If there are still unconfirmed data packets at the original failed node, the completed data is filtered out according to the data processing confirmation record, and only the unconfirmed parts are resent to the alternative computing node.

[0072] Furthermore, after completing the local topology restoration, the system loads the model execution state saved at the end of round 17 from the intermediate result storage area, and reads the data to be processed corresponding to partition C in round 18. It then restores the iteration round, data read position, and task progress to node D. Node D continues to complete the calculation of partition C from the execution breakpoint, generates the result, and sends it to the result aggregation node. The result aggregation node matches the restored partition C result with the other three previously frozen and saved partition results according to the same task batch. Once the aggregation conditions are met, it continues to execute the result summary of round 18 and the next iteration, without needing to re-execute rounds 1 to 17 and other completed branches.

[0073] By limiting the scope of the fault's impact based on the task execution progress and data dependencies, and only migrating the carrying relationships and incomplete states of the affected tasks, valid calculation results can be preserved, reducing redundant calculations and data rollbacks during the fault recovery process, thereby improving the continuous execution capability of AI intelligent measurement data streams and the efficiency of local fault recovery.

[0074] In some embodiments, after adjusting the task-bearing relationships and data dependencies within the scope of the fault impact, the method further includes: Based on the adjusted task load relationships and data dependencies, perform a recovery feasibility verification on the scope of the fault impact. When the recovery feasibility verification fails, the adjacent unfinished task nodes are determined along the data dependency relationship based on the unmet execution constraints, and the adjacent unfinished task nodes are included in the scope of the fault impact. The execution breakpoint is redefined based on the expanded scope of the fault impact, and the corresponding task carrying relationships and data dependencies are updated until the recovery conditions are met, at which point execution continues from the redefined execution breakpoint.

[0075] Specifically, this embodiment continues to use the partial fault recovery process of the aforementioned batch iterative calculation task as an example. After the system migrates the fault model calculation node to the alternative computing node and re-establishes the data dependency relationship between the data preprocessing node, the alternative calculation node, and the result aggregation node, it does not immediately restore the data flow, but instead performs a recovery feasibility verification for the current fault impact range.

[0076] The system reads the task execution status of each task node within the fault's impact range, the newly established task carrying relationships, and data dependencies. It checks the recovery computing power, model state availability, input data integrity, and upstream and downstream dependency satisfaction status item by item according to the original task execution constraints. It also simulates the data transmission process of a minimum data batch after the execution breakpoint to determine whether the alternative computing power node can receive the data to be processed, restore the model execution state, and produce effective results that can continue to be used downstream within the allowed time limit.

[0077] Taking the failure of the third parallel model computing node in the 18th iteration as an example, the system initially determined the incomplete computation of partition C in the 18th iteration and the aggregation state awaiting the result as the scope of the failure's impact, and selected a backup node D to carry out the subsequent computation. During the recovery verification, although node D had GPU resources sufficient for the remaining computation, the system found that the computation of partition C in the 18th iteration still relied on a set of temporary normalization parameters generated by the data preprocessing node in this iteration. These parameters were only stored in the volatile cache of the execution link where the original failed node was located and had not yet been written to the shared state storage area. At this time, even if node D could obtain the model state of the 17th iteration and the original input of partition C, it could not recover the computation of the 18th iteration according to the original execution conditions. Therefore, it was determined that the current recovery plan did not meet the task state integrity constraint, and the recovery feasibility verification failed.

[0078] The system traces upstream along data dependencies based on unmet execution constraints to identify adjacent incomplete task nodes that generate temporary normalization parameters. Task execution records confirm that the data preprocessing node has completed data cleaning for partition C in round 18, but its normalization processing has not yet produced a fully recoverable output. Therefore, the incomplete processing portion corresponding to this data preprocessing node is included in the fault's impact range. If the missing state originates downstream—for example, if the result aggregation node has consumed some partition results but has not yet formed a consistent aggregation state—the system extends downstream along data dependencies, including the corresponding incomplete aggregation node within the fault's impact range. For the other three parallel computing nodes that do not have data transfer or state dependencies with the missing state, their completed results remain frozen and are not rolled back as the fault's impact range expands.

[0079] Furthermore, as the scope of the fault expands, the system re-examines the execution progress of the newly added task nodes and redetermines the execution breakpoint based on the most recent task boundary that can simultaneously guarantee the integrity of input data and execution status. In this embodiment, the execution breakpoint is adjusted from the completion of round 17 and the start position of partition C calculation in round 18 to the start position of normalization processing of partition C in round 18, while retaining all results from round 17 and the calculation results of the other three partitions already completed in round 18.

[0080] The system then re-allocates available computing resources based on the expanded computational load, allocating the data preprocessing portion of partition C to edge node E with the corresponding processing environment, and continuing to allocate subsequent model computations to the backup node D. It also establishes data dependencies from the original partition data to node E, from node E's processing results to node D, and from node D's computation results to the result aggregation node. Simultaneously, new task batches, data partitions, and iteration rounds are written into each connection relationship to ensure that the data in the recovery chain belongs to the same execution context.

[0081] Furthermore, after the update is completed, the system performs a recovery feasibility check again. By simulating the complete local link from normalization processing to model calculation and result aggregation in partition C, it checks whether the preprocessing output of node E can meet the input requirements of node D, whether node D can complete the calculation within the remaining time limit, and whether the recovery result can meet the aggregation conditions together with the results of other frozen partitions. If there are still missing states or execution constraints are not met, the adjacent unfinished task nodes are included along the corresponding data dependencies, and the execution breakpoint and bearing relationship are redefined. If all recovery conditions are met, the task suspension state within the scope of the fault is lifted, partition C is processed from the redefined execution breakpoint, and subsequent aggregation and iterative execution are resumed after obtaining valid results.

[0082] By expanding the scope of the fault impact step by step according to the recovery constraints, it is possible to avoid directly triggering the restart of the entire data stream when local recovery is insufficient. This ensures that the recovery scope is only extended to the task nodes required to form complete execution conditions, thereby reducing redundant calculations and recovery resource overhead while ensuring data dependency and execution state consistency.

[0083] In some embodiments, load prediction parameters and data stream orchestration parameters are corrected based on the deviation between the actual operating state and the predicted data stream load state for subsequent data stream orchestration, including: The actual running status is correlated with the corresponding predicted data stream load status according to the task execution relationship, and the load prediction deviation and orchestration execution deviation of each task node are determined. Trace the propagation path of load forecasting deviations and orchestration execution deviations along the data flow topology to determine the corresponding sources and scope of impact of the deviations; Based on the source of the deviation, determine the load forecast parameters or data stream orchestration parameters to be corrected within the scope of influence, and generate candidate parameter versions according to the degree of deviation. The candidate parameter versions are validated by data stream orchestration using historical operating status. Candidate parameter versions that meet the preset stability conditions are used for subsequent data stream load prediction and data stream orchestration.

[0084] Specifically, this embodiment continues to use the aforementioned edge-cloud collaborative AI intelligent computing cluster as an example. After each target data flow topology switch and one evaluation cycle, the system correlates the actual collected data throughput status, task completion status, and resource carrying capacity status with the data flow load status generated before the topology adjustment, according to the task identifier, task node identifier, and execution time window. The evaluation cycle can be set to 120 seconds and aligned with the time granularity consistent with the load prediction. For each task node, the system compares the predicted processing pressure with the actual processing pressure to determine the load prediction deviation; simultaneously, it compares the expected node processing capacity, data backlog changes, and task completion status during topology adjustment with the actual running results to determine the orchestration execution deviation.

[0085] For example, if the system predicts that the model computing node for the batch iterative calculation task requires 400MB of processing power per second during peak periods, and the original computing node is split into 4 parallel task nodes, the actual data input during runtime only reaches 330MB per second, resulting in a negative load prediction bias. If the 4 parallel task nodes have sufficient computing power, but the waiting time of the result aggregation node increases, resulting in an effective throughput of only 300MB per second, then an orchestration execution bias will be further formed.

[0086] The system traces the propagation of deviations along the actual data flow topology in the task execution chain. For load prediction deviations, it first determines whether the deviation has already occurred at the data entry point or the preceding task node. If the predicted input is significantly higher than the actual input, and the deviation appears synchronously at subsequent nodes as the input decreases, the main source of the deviation is located in the load prediction process, without directly adjusting the parallel topology parameters. For orchestration execution deviations, the system traces upstream and downstream from the task node where the actual processing capacity decreases.

[0087] For example, if all four parallel model computing nodes can complete the computation according to the allocated data, but the result aggregation node experiences continuous buffering and waiting due to receiving multiple branch outputs, the system will pinpoint the source of the deviation to the orchestration relationship between the parallel branches and the result aggregation, and determine the scope of influence as the four parallel task nodes, the result aggregation node, and their corresponding connections. If the deviation is only caused by a temporary increase in the load of a certain computing node and does not propagate to other task nodes, the scope of influence will be limited to the corresponding task node and its load relationship to avoid expanding the parameter correction range.

[0088] Furthermore, after identifying the source of the deviation, the system selects parameters directly related to the deviation from the scope of influence for correction. For load prediction deviation, the historical time window length, the prediction weight corresponding to the recent operating status, and the threshold for identifying sudden changes can be adjusted; for orchestration execution deviation, the task splitting granularity, the parallel branch triggering threshold, the node processing capacity estimation parameters, or the topology switching conditions are adjusted according to the actual bottleneck location.

[0089] For example, if the predicted peak value exceeds the actual peak value by more than 15% for three consecutive evaluation periods, the system reduces the sustained impact of recent surges on future load predictions. When the model computation nodes expand to four parallel branches and result aggregation continues to cause backlog, the system reduces the magnitude of a single parallel expansion or increases the constraint weight of the result aggregation node's processing capacity in the candidate topology evaluation. The parameter adjustment amount is determined based on the degree of deviation and the duration of the deviation; the larger the deviation and the longer the duration, the higher the corresponding correction magnitude. However, each correction remains within a preset range to prevent drastic parameter changes caused by a single anomaly. After correction, candidate parameter versions are generated, and the applicable data flow load range, task type, and corresponding deviation source are recorded.

[0090] Furthermore, after the candidate parameter versions are generated, the system selects multiple data windows from historical operational data that include stable loads, centralized task submissions, and load fallback processes for offline orchestration and playback. For the same historical window, load prediction, topology adjustment judgment, and resource matching are performed using both the current parameter version and the candidate parameter versions. The deviation between the predicted load and the historical actual load is compared, and it is also checked whether the generated candidate topology can cover the actual processing requirements and whether unnecessary task splitting and resource expansion are generated.

[0091] For example, if a candidate parameter version reduces the peak prediction bias from 18% to 7%, and can still trigger model computation node expansion in advance in historically high-load samples, while not adding additional parallel branches in low-load samples, then the prediction and orchestration results are considered improved. The system also uses historical windows with different dates and different task combinations to verify the candidate parameter version, avoiding the possibility that the parameters are only applicable to single peak scenarios.

[0092] When the candidate parameter version meets the preset stability conditions in terms of load prediction deviation, task blocking status, and resource adjustment stability in historical playback, the candidate parameter version is registered as the effective version for subsequent data stream load prediction and data stream orchestration; if any key verification status shows significant degradation, the current parameter version continues to be used, and the corresponding deviation sample is retained to participate in the next round of correction.

[0093] By feeding back actual operating results to each task node along with prediction and orchestration results, and locating the source of deviation along the data flow topology, load prediction parameters and data flow orchestration parameters can be updated in a targeted manner according to actual operating changes. This reduces prediction drift and orchestration mismatch in long-term operation, and improves the continuous adaptability and operational stability of elastic data flow orchestration.

[0094] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0095] Figure 2 This is a schematic diagram of the structure of the elastic data stream orchestration system of the adaptive AI intelligent calculation engine provided in the embodiments of this application. Figure 2 As shown, the system includes: The acquisition module 201 is used to acquire the data stream running status, measure task characteristics and computing node resource status, perform time-series association according to task execution relationship, and generate data stream association status. The determination module 202 is used to predict the data flow load status within a time period based on the data flow association status, and match it with the processing capacity of the current data flow topology to determine the topology adjustment requirements; Adjustment module 203 is used to determine the scope of tasks to be adjusted based on topology adjustment requirements and data dependencies of the calculation tasks, perform structural adjustments on task nodes and dependencies within the scope of tasks to be adjusted, and generate candidate data flow topology; The calculation module 204 is used to determine the resource adjustment requirements based on the changes in the calculation requirements of the candidate data flow topology and the available resource status of the computing power nodes, select the target computing power node and update the bearing relationship of the task nodes to obtain the target data flow topology. The execution module 205 is used to execute the measurement task according to the target data flow topology. When an anomaly occurs, it determines the scope of the fault impact and the execution breakpoint based on the task execution progress and data dependency relationship, maintains the execution state outside the scope of the fault impact, adjusts the task carrying relationship and data dependency relationship within the scope of the fault impact, and continues execution from the execution breakpoint. The correction module 206 is used to correct the load prediction parameters and data stream orchestration parameters based on the deviation between the actual operating state and the predicted data stream load state, for subsequent data stream orchestration.

[0096] In some embodiments, Figure 2 The acquisition module 201 acquires the status data corresponding to the data stream processing process, the calculation task execution process, and the computing power node operation process according to a unified sampling period, and performs standardized processing on the status data to form a time-series status sequence. Based on the task nodes and task execution relationships corresponding to the calculation tasks, the time-series status sequence is mapped to a unified time window to establish the association between the data stream status and the corresponding calculation tasks and the computing power nodes. Based on the association, the data processing pressure, computing requirements, and resource availability of each task node are extracted to generate a data stream association status that represents the correspondence between the data stream processing requirements and the computing power carrying status.

[0097] In some embodiments, Figure 2 The determination module 202 extracts changes in data processing demand and computing power capacity within a continuous time window from the data flow association state, generating load time series characteristics; inputs the load time series characteristics into the load prediction model to determine the predicted processing pressure corresponding to each task node within the prediction period, generating the data flow load state; determines the corresponding processing capacity based on the execution relationship of task nodes in the current data flow topology, matches the data flow load state with the processing capacity node by node, and determines the target task nodes with insufficient or redundant processing capacity and the corresponding degree of deviation; based on the dependency position and degree of deviation of the target task nodes in the current data flow topology, generates topology adjustment requirements that characterize the topology adjustment range and direction.

[0098] In some embodiments, Figure 2The adjustment module 203 locates the corresponding target task node according to the topology adjustment requirements, determines the task nodes with execution association with the target task node along the data dependency relationship, and forms the task range to be adjusted; determines the structural adjustment method corresponding to the task range to be adjusted according to the topology adjustment requirements, and adjusts the execution granularity and connection relationship of the task nodes in combination with the separable state and dependency constraints of each task node; keeps the data flow topology outside the task range to be adjusted unchanged, and connects the adjusted task nodes and dependencies to the original data flow topology to generate a candidate data flow topology.

[0099] In some embodiments, Figure 2 The adjustment module 203 performs execution simulation on the candidate data flow topology based on the predicted data flow load status, determines the processing status and data flow status of each task node within the prediction period; matches the processing status with the predicted data flow load status to determine the load carrying capacity of the candidate data flow topology, and when the load carrying capacity meets the preset orchestration conditions, determines the candidate data flow topology as the topology to be switched; keeps the current data flow topology running until the preset switching conditions are met, migrates the task execution status to the topology to be switched and enables the corresponding task nodes and dependencies.

[0100] In some embodiments, Figure 2 The computing module 204 determines the changes in computing requirements corresponding to the task nodes to be adjusted based on the changes in task structure between the candidate data flow topology and the current data flow topology; it matches the changes in computing requirements with the available resource status of the computing power nodes to determine the amount of resource replenishment or release, and generates resource adjustment requirements; based on the resource adjustment requirements and the execution constraints of the task nodes, it determines the target computing power node from the available computing power nodes and determines the corresponding capacity; it updates the capacity relationship between the task nodes to be adjusted and the target computing power node according to the capacity, and associates it with the candidate data flow topology to obtain the target data flow topology.

[0101] In some embodiments, Figure 2 The execution module 205 locates the faulty task node based on the abnormal running state, matches the task execution progress with the processing status of the corresponding calculation task, and determines the execution breakpoint between completed and incomplete tasks; it traces the incomplete tasks and data flow paths associated with the faulty task node along the data dependency relationship, determines the scope of the fault impact, and freezes the execution state of tasks outside the scope of the fault impact; it determines the alternative computing power node based on the execution constraints of each task node within the scope of the fault impact and the available resource status of the computing power node, updates the corresponding task bearing relationship, and reconstructs the data dependency relationship affected by the fault; it restores the task state and pending data corresponding to the execution breakpoint, and continues to execute the calculation task according to the updated task bearing relationship and data dependency relationship.

[0102] In some embodiments, Figure 2The execution module 205 performs a recovery feasibility check on the scope of the fault impact based on the adjusted task carrying relationship and data dependency relationship. When the recovery feasibility check fails, it determines the adjacent unfinished task nodes along the data dependency relationship based on the unmet execution constraints and includes the adjacent unfinished task nodes in the scope of the fault impact. It redetermines the execution breakpoint based on the expanded scope of the fault impact and updates the corresponding task carrying relationship and data dependency relationship until the recovery conditions are met, and then continues execution from the redetermined execution breakpoint.

[0103] In some embodiments, Figure 2 The correction module 206 associates the actual running status with the corresponding predicted data stream load status according to the task execution relationship, determines the load prediction deviation and orchestration execution deviation of each task node; traces the propagation path of the load prediction deviation and orchestration execution deviation along the data stream topology, determines the corresponding deviation source and impact range; determines the load prediction parameter or data stream orchestration parameter to be corrected from the impact range according to the deviation source, and generates candidate parameter versions according to the degree of deviation; uses historical running status to verify the candidate parameter versions for data stream orchestration, and uses the candidate parameter versions that meet the preset stability conditions for subsequent data stream load prediction and data stream orchestration.

[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments.

[0105] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various system embodiments described above.

[0106] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0107] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0108] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0111] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A flexible data stream orchestration method for an adaptive AI intelligent calculation engine, characterized in that, include: Acquire the data stream running status, calculate task characteristics and computing node resource status, perform time-series correlation according to task execution relationships, and generate data stream correlation status; Based on the data flow association status, predict the data flow load status within the time period and match it with the processing capacity of the current data flow topology to determine the topology adjustment requirements; Based on the topology adjustment requirements and the data dependencies of the calculation tasks, the scope of tasks to be adjusted is determined, and the task nodes and dependencies within the scope of tasks to be adjusted are structurally adjusted to generate candidate data flow topologies. Based on the changes in computational requirements of the candidate data flow topology and the available resource status of the computing power nodes, the resource adjustment requirements are determined, the target computing power node is selected, and the bearing relationship of the task nodes is updated to obtain the target data flow topology. The measurement task is executed according to the target data flow topology. When an anomaly occurs, the scope of the fault impact and the execution breakpoint are determined based on the task execution progress and data dependency relationship. The execution state outside the scope of the fault impact is maintained, the task carrying relationship and data dependency relationship within the scope of the fault impact are adjusted, and the execution continues from the execution breakpoint. Based on the deviation between the actual operating status and the predicted data stream load status, the load prediction parameters and data stream orchestration parameters are adjusted for subsequent data stream orchestration.

2. The method according to claim 1, characterized in that, The process of acquiring the data stream's running status, calculating task characteristics, and determining the resource status of computing nodes, and then performing time-series correlation according to task execution relationships to generate a data stream correlation status, includes: The state data corresponding to the data stream processing process, the calculation task execution process, and the computing node operation process are obtained according to a unified sampling period, and the state data is standardized to form a time-series state sequence. Based on the task nodes and task execution relationships corresponding to the measurement tasks, the time sequence state is mapped to a unified time window to establish the association between the data flow state and the corresponding measurement tasks and the computing power nodes. Based on the aforementioned relationships, the data processing pressure, computing requirements, and resource availability of each task node are extracted, and a data flow association state representing the correspondence between data flow processing requirements and computing power carrying capacity is generated.

3. The method according to claim 1, characterized in that, The step of predicting the data flow load status within a time period based on the data flow association status and matching it with the processing capacity of the current data flow topology to determine the topology adjustment requirements includes: Extract the changes in data processing demand and computing power load within a continuous time window from the data stream association state to generate load time-series characteristics; The load time-series characteristics are input into the load prediction model to determine the prediction processing pressure of each task node within the prediction period and generate the data stream load status. Based on the execution relationship of task nodes in the current data flow topology, determine the corresponding processing capacity, match the data flow load status with the processing capacity node by node, and determine the target task nodes with insufficient or redundant processing capacity and the corresponding degree of deviation. Based on the dependency position of the target task node in the current data flow topology and the degree of deviation, a topology adjustment requirement is generated, which characterizes the scope and direction of the topology adjustment.

4. The method according to claim 1, characterized in that, The step of determining the scope of tasks to be adjusted based on the topology adjustment requirements and the data dependencies of the calculation tasks, and structurally adjusting the task nodes and dependencies within the scope of tasks to be adjusted to generate candidate data flow topologies includes: Based on the topology adjustment requirements, locate the corresponding target task node, determine the task nodes with execution association with the target task node along the data dependency relationship, and form the scope of tasks to be adjusted; Based on the topology adjustment requirements, determine the structural adjustment method corresponding to the scope of tasks to be adjusted, and adjust the execution granularity and connection relationship of task nodes in combination with the separable state and dependency constraints of each task node. Keeping the data flow topology outside the scope of the task to be adjusted unchanged, the adjusted task nodes and dependencies are connected to the original data flow topology to generate a candidate data flow topology.

5. The method according to claim 4, characterized in that, After generating the candidate data stream topology, the process also includes: The candidate data flow topology is executed based on the predicted data flow load status to determine the processing status and data flow status of each task node within the prediction period. The processing state is matched with the predicted data stream load state to determine the load carrying capacity of the candidate data stream topology. When the load carrying capacity meets the preset orchestration conditions, the candidate data stream topology is determined as the topology to be switched. The current data flow topology is maintained until the preset switching conditions are met, and the task execution state is migrated to the topology to be switched and the corresponding task nodes and dependencies are enabled.

6. The method according to claim 1, characterized in that, The step of determining resource adjustment needs based on changes in computational requirements and available resource status of computing nodes in the candidate data flow topology, selecting target computing nodes, updating the bearing relationships of task nodes, and obtaining the target data flow topology includes: Based on the changes in task structure between the candidate data flow topology and the current data flow topology, determine the changes in computational requirements corresponding to the task nodes to be adjusted; The changes in computing demand are matched with the available resource status of computing nodes to determine the amount of resources to be added or released, and resource adjustment requirements are generated. Based on the resource adjustment requirements and task node execution constraints, the target computing power node is determined from the available computing power nodes, and the corresponding capacity is determined. Update the carrying relationship between the task node to be adjusted and the target computing power node according to the carrying capacity, and associate it with the candidate data flow topology to obtain the target data flow topology.

7. The method according to claim 1, characterized in that, The steps of determining the scope of the fault impact and the execution breakpoint based on the task execution progress and data dependencies, maintaining the execution state outside the scope of the fault impact, adjusting the task carrying relationships and data dependencies within the scope of the fault impact, and continuing execution from the execution breakpoint include: Locate the faulty task node based on the abnormal operating status, match the task execution progress with the processing status of the corresponding calculated task, and determine the execution breakpoint between completed and incomplete tasks. The system traces the unfinished tasks and data flow paths associated with the faulty task node along the data dependency relationship, determines the scope of the fault's impact, and freezes the execution status of tasks outside the scope of the fault's impact. Based on the execution constraints of each task node within the scope of the fault and the available resource status of the computing power node, a replacement computing power node is determined, the corresponding task bearing relationship is updated, and the data dependency relationship affected by the fault is rebuilt. Restore the task status and pending data corresponding to the breakpoint, and continue executing the calculation task according to the updated task carrying relationship and data dependency relationship.

8. The method according to claim 7, characterized in that, After adjusting the task-bearing relationships and data dependencies within the scope of the fault's impact, the following is also included: Based on the adjusted task carrying relationships and data dependencies, a recovery feasibility verification is performed on the scope of the fault impact. When the recovery feasibility verification fails, adjacent unfinished task nodes are determined along the data dependency relationship based on the unmet execution constraints, and the adjacent unfinished task nodes are included in the scope of the fault impact. The execution breakpoint is redefined based on the expanded scope of the fault impact, and the corresponding task carrying relationships and data dependencies are updated until the recovery conditions are met, at which point execution continues from the redefined execution breakpoint.

9. The method according to claim 1, characterized in that, The step of correcting the load prediction parameters and data stream orchestration parameters based on the deviation between the actual operating state and the predicted data stream load state for subsequent data stream orchestration includes: The actual running status is correlated with the corresponding predicted data stream load status according to the task execution relationship, and the load prediction deviation and orchestration execution deviation of each task node are determined. The propagation paths of the load prediction deviation and orchestration execution deviation are traced along the data flow topology to determine the corresponding sources and scope of influence of the deviations. Based on the source of the deviation, determine the load prediction parameter or data stream orchestration parameter to be corrected from the scope of influence, and generate candidate parameter versions according to the degree of deviation; The candidate parameter versions are verified by data stream orchestration using historical operating status, and the candidate parameter versions that meet the preset stability conditions are used for subsequent data stream load prediction and data stream orchestration.

10. A flexible data stream orchestration system for an adaptive AI intelligent calculation engine, characterized in that, include: The acquisition module is used to acquire the data stream running status, measure task characteristics and computing node resource status, perform time-series correlation according to task execution relationships, and generate data stream correlation status. The determination module is used to predict the data flow load status within a time period based on the data flow association status, and match it with the processing capacity of the current data flow topology to determine the topology adjustment requirements. The adjustment module is used to determine the scope of tasks to be adjusted based on the topology adjustment requirements and the data dependencies of the calculation tasks, and to perform structural adjustments on the task nodes and dependencies within the scope of tasks to be adjusted to generate candidate data flow topologies. The calculation module is used to determine the resource adjustment requirements based on the changes in the calculation requirements of the candidate data flow topology and the available resource status of the computing power nodes, select the target computing power node and update the bearing relationship of the task nodes to obtain the target data flow topology. The execution module is used to execute the measurement task according to the target data flow topology. When an anomaly occurs, it determines the scope of the fault impact and the execution breakpoint based on the task execution progress and data dependency relationship, maintains the execution state outside the scope of the fault impact, adjusts the task carrying relationship and data dependency relationship within the scope of the fault impact, and continues execution from the execution breakpoint. The correction module is used to correct the load prediction parameters and data stream orchestration parameters based on the deviation between the actual operating state and the predicted data stream load state, for subsequent data stream orchestration.