Computing power resource fusion method based on distributed flow pipeline

Abstract

The invention relates to the technical field of distributed computing and computing power scheduling, in particular to a computing power resource fusion method based on a distributed flow pipeline, which comprises the following steps of: disassembling a user computing power request into a flow pipeline unit for packaging computing logic, an input / output interface and a resource demand label; meanwhile, computing power types, real-time load rates, memory occupancy rates, network round-trip delays, geographic positioning and energy consumption data of cloud edge end nodes are collected, a hierarchical topology network framework is constructed based on the collected data, node computing power available values are calculated, an inter-node transmission cost matrix is generated, a fault probability prediction model is constructed, and a fault probability prediction model is constructed. And finally, analyzing a data dependency relationship of the pipeline unit through a four-dimensional joint decision engine, executing dynamic mapping, and preferentially mapping the high-computing-power demand unit to a GPU cluster node, so as to realize non-interruption reconstruction during pipeline topology operation. And the global computing power resource utilization rate, the task operation efficiency and the service stability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing and computing power scheduling, in particular to a computing power resource fusion method based on a distributed streaming pipeline. BACKGROUND

[0002] Distributed computing and computing power scheduling is an important technology. Under the background of accelerating digital transformation and exploding computing power demand, this technology is the key support for improving the utilization rate of global computing power resources and ensuring efficient operation of tasks. It can not only decompose complex computing power requests from users into ordered execution units to adapt to the computing power characteristics of different levels of cloud centers, edge clusters and terminal devices, but also balance computing power utilization efficiency, transmission delay, energy cost and operation stability to meet the needs of real-time data processing, AI model inference and large-scale data computing for diversified tasks. It promotes the upgrade of computing power resources from static allocation to dynamic fusion and is widely applicable to computing power scheduling scenarios in the fields of the Internet, industrial Internet and smart cities. The existing distributed computing power resource scheduling technology faces core problems such as low resource mapping accuracy, weak dynamic adaptation capability and easy interruption of overload reconstruction in actual application. Traditional scheduling technology mainly relies on single-dimensional data to allocate tasks and does not fully integrate multi-dimensional information such as node geographic positioning, real-time load rate, memory occupancy rate, network delay, energy consumption and fault risk, resulting in a disconnection between resource mapping and task demand. Strong real-time tasks are allocated to geographically remote cloud center nodes with high latency, causing task stalls. Computing-intensive tasks are allocated to edge nodes, resulting in low execution efficiency due to insufficient computing power. At the same time, the decision-making process lacks consideration of task data dependency, and tasks with strong data dependency are not deployed to low-latency node domains, exacerbating data transmission loss. Moreover, there is no effective fault probability prediction and energy consumption constraint mechanism. Either the task is interrupted due to the selection of high-fault-risk nodes, or resources are wasted due to the neglect of energy consumption costs. In addition, when the load of the target node exceeds the critical value, traditional technology mainly uses full-task migration, which does not finely disassemble and statelessly migrate task units, easily causing service interruption during migration and affecting user experience. These problems are mutually compounded, resulting in low computing power resource utilization, large task running latency fluctuations and insufficient service stability, making it difficult to meet the needs of efficient, stable and low-consumption computing power scheduling in complex scenarios. To solve this technical problem, we provide a computing power resource fusion method based on a distributed streaming pipeline. SUMMARY

[0003] The purpose of the present application is to provide a computing power resource fusion method based on a distributed streaming pipeline to solve the problems raised in the background technology.

[0004] Due to low resource mapping accuracy, non-fusion of multi-dimensional node data and ignoring task data dependence, the case can improve mapping accuracy and adapt to task demand by disassembling computing power request into streaming units, collecting multi-dimensional node data to construct global resource atlas and matching nodes with four-dimensional decision engine.

[0005] Due to the interruption of overload reconstruction, the traditional full migration causes service interruption, so the case can realize non-interruptible reconstruction and guarantee service stability by disassembling units into sub-units when the load exceeds the limit, migrating to idle nodes and building direct connection channels.

[0006] To achieve the above purpose, one of the purposes of the present application is to provide a computing power resource fusion method based on a distributed streaming pipeline, comprising the following steps: S1, disassemble the user computing power request into sequentially executed units, each unit encapsulating calculation logic, input / output interface and resource demand label to form a streaming pipeline unit, and simultaneously collect the computing power type, real-time load rate, memory occupancy rate, network round-trip delay, node geographic positioning and energy consumption data of the cloud edge node to provide original input for dynamic resource mapping; S2, based on the multi-dimensional data collected in S1, first construct a hierarchical topological network framework according to the node geographic positioning, and fuse the real-time load rate and memory occupancy rate to calculate the available value of each node computing power, combine the network round-trip delay to dynamically generate the transmission cost matrix between nodes, superimpose the energy consumption data and historical fault records to construct a fault probability prediction model, and finally integrate into a global resource atlas with time stamp; S3, based on the global resource atlas, analyze the data dependence relationship of the streaming pipeline unit, and execute dynamic mapping through a four-dimensional joint decision engine, wherein the computing power matching degree dimension calculates the similarity between unit demand and node residual computing power, the path cost dimension predicts the data transmission delay between units according to the transmission cost matrix, the fault tolerance weight dimension selects nodes with fault probability lower than the preset threshold, and the energy consumption constraint dimension excludes energy consumption exceeding the limit area, according to the evaluation result, the high computing power demand unit is preferentially mapped to the GPU cluster node, the unit group with strong data dependence is allocated to the low delay node within the same edge domain, when the target node load reaches the critical value, the pipeline unit on the overloaded node is disassembled into sub-units, migrated to the adjacent idle node and established a direct data channel, realizing non-interruptible reconstruction of pipeline topology at runtime.

[0007] Compared with the prior art, the present application has the following advantages: 1. By disassembling the user computing power request into streaming pipeline units encapsulating calculation logic and resource demand, collecting multi-dimensional data of cloud edge nodes to construct a global resource atlas with time stamp, and matching nodes with a four-dimensional joint decision engine, the technical effect of accurate resource mapping is achieved, the problem of low mapping accuracy of traditional scheduling resources is solved, and the advantages of adapting to different task demands and improving global computing power resource utilization are achieved.

[0008] 2. The technical effect of dynamically adapting to working conditions is achieved by constructing a hierarchical topology network based on node geographical positioning, generating a transmission cost matrix between nodes and a failure probability prediction model, and dynamically adjusting the boundaries of each layer and the link bandwidth, solving the problem of weak dynamic adaptation capability of traditional technology, and having the advantages of balancing transmission delay, energy cost and operation stability.

[0009] 3. The technical effect of overload non-interrupt reconstruction is achieved by disassembling the pipeline unit into stateless subunits when the node load exceeds the limit, migrating to an idle node in the same edge domain and constructing a zero-copy direct connection channel, and scheduling data flow by the routing coordinator, solving the problem of service interruption caused by traditional full migration, and having the advantages of guaranteeing continuous operation of services and improving user experience. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The overall workflow of the present application is shown in the figure. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0012] Please refer to Figure 1 The present embodiment provides a distributed streaming pipeline-based computing resource fusion method, which includes the following steps: S1, the user computing power request is disassembled into sequential execution units, each unit encapsulates the calculation logic, input / output interface and resource demand label to form a streaming pipeline unit, and at the same time, the computing power type, real-time load rate, memory occupancy rate, network round-trip delay, node geographical positioning and energy consumption data of cloud-edge-end nodes are collected in real time to provide original input for dynamic resource mapping; S2, based on the multi-dimensional data collected in S1, first, a hierarchical topology network framework is constructed according to node geographical positioning, and the real-time load rate and memory occupancy rate are combined to calculate the available value of each node computing power, and a node transmission cost matrix is dynamically generated combined with the network round-trip delay, and a failure probability prediction model is constructed by superimposing energy consumption data and historical failure records, and finally integrated into a global resource atlas with time stamp; S3. Based on the global resource graph, the data dependencies of the streaming pipeline units are analyzed, and dynamic mapping is performed through a four-dimensional joint decision engine. Among them, the computing power matching degree dimension calculates the similarity between the unit demand and the remaining computing power of the node; the path cost dimension predicts the data transmission latency between units based on the transmission cost matrix; the fault tolerance weight dimension selects nodes with a failure probability lower than a preset threshold; and the energy consumption constraint dimension excludes energy consumption exceeding the limit. According to the evaluation results, units with high computing power demand are preferentially mapped to GPU cluster nodes, and unit groups with strong data dependencies are assigned to low-latency nodes in the same edge domain. When the load of the target node reaches the critical value, the pipeline units on the overloaded node are decomposed into sub-units, migrated to adjacent idle nodes, and a direct data channel is established to achieve uninterrupted reconfiguration of the pipeline topology during runtime.

[0013] When constructing a hierarchical topology network framework, the system first dynamically divides the physical boundaries into three layers based on the latitude and longitude coordinates of the nodes' geographic locations, with the user's requested location as the center. The terminal device layer covers the nearest distance range, the edge cluster layer covers the mid-distance range, and the cloud center layer covers the long-distance range. Real-time transmission latency is obtained by sending dynamic topology probe packets to nodes between layers. Logical direct connection channels are established between edge nodes with latency below a set threshold and cloud center nodes. Finally, the physical boundary range of each layer is dynamically adjusted according to the node distribution density to achieve elastic scaling of the topology framework.

[0014] When calculating the available computing power of a node, the memory weighting factor calculation rules are first defined for each node based on the three-layer physical boundary. The weighting coefficient is dynamically generated according to the memory occupancy rate. The real-time load rate, the node's peak computing power, and the memory weighting factor are multiplied to obtain the weighted dynamic available computing power value. When a node load rate is detected to rise sharply within a preset time, the virtual computing power buffer pool mechanism is automatically activated to inject a temporary virtual computing power quota into the node, thus avoiding pipeline reorganization triggered by sudden load.

[0015] When generating the inter-node transmission cost matrix, the time series prediction model is first trained by collecting historical round-trip delay data based on the logical direct-connection channel network. The predicted transmission delay between any two nodes in the future time period is output. When the deviation between the measured delay and the predicted value exceeds the tolerance range, redundant bandwidth is automatically added to the corresponding link to compensate for network jitter. Node pairs whose predicted delay exceeds the communication threshold are marked as unreachable. The computational complexity of the decision engine is reduced by using a compressed matrix storage structure.

[0016] When constructing the failure probability prediction model, we first analyze the correlation between energy consumption data and historical failure records, establish a positive correlation model between power consumption fluctuation rate and failure probability, and dynamically adjust the node failure probability threshold according to real-time energy consumption fluctuation data to ensure that the false alarm rate is below an acceptable level. Finally, we align and fuse the available computing power value, transmission cost matrix and failure probability prediction value in three dimensions according to geographical topological coordinates and timestamps to form a global resource heat map with spatiotemporal attributes.

[0017] Further explanation is needed: After completing the multi-dimensional data collection of cloud-edge-device nodes, including cloud servers, edge computing nodes, and terminal device nodes, in step S1, constructing a hierarchical topology network framework is the foundation for the orderly management of computing resources. Only by clearly defining the geographical hierarchy and connection relationships of nodes can the available computing power and transmission costs be accurately calculated subsequently, avoiding the allocation of streaming pipeline units to geographically distant nodes with excessively high latency. The specific implementation method is as follows: When constructing a layered topology network framework, the first step is to dynamically divide the physical boundaries into three layers based on the latitude and longitude coordinates of the nodes' geographic locations, centered on the user's requested location. The latitude and longitude coordinates of the nodes' geographic locations are obtained in real-time by the GPS modules built into each node or by network positioning services. The dynamic division of the three physical boundaries is based on the network environment and node distribution at the user's requested location, setting different distance ranges. The terminal device layer covers the closest distance range, typically within 1 kilometer, mainly including smartphones and smart home device terminal nodes near the user. These nodes have the lowest latency but weakest computing power. The edge cluster layer covers the medium distance range, typically 1-10 kilometers, including edge server clusters deployed in communities and parks, with computing power and latency between those of the terminal and the cloud. The cloud center layer covers a long distance of over 10 kilometers, including cloud servers deployed in city data centers. It boasts the strongest computing power but relatively high latency. During partitioning, it requires real-time calls to map service interfaces to calculate the straight-line distance between each node and the user's requested location, automatically classifying them into the corresponding layer to ensure the layering closely matches the actual geographical distribution. Real-time transmission latency is obtained by sending dynamic topology probe packets to nodes between layers. These dynamic topology probe packets are small data packets containing a unique node identifier and a sending timestamp, avoiding excessive network resource consumption. The system sends these probe packets every 10 seconds from the terminal node corresponding to the user's requested location to nodes in the edge cluster layer and cloud center layer. Upon receiving the packet, the target node immediately returns a response packet containing a receiving timestamp. The system calculates the one-way transmission latency, i.e., the real-time transmission latency, by (receive timestamp - send timestamp) / 2. For example, if the send timestamp is 1620000000000 milliseconds and the receive timestamp is 1620000000010 milliseconds, the real-time transmission latency is 5 milliseconds. Then, logical direct connection channels are established between edge nodes with latency below a set threshold and cloud center nodes. The threshold is dynamically adjusted according to the task type; the threshold is set to 50 milliseconds for tasks with high real-time requirements and 200 milliseconds for non-real-time tasks (data backup). If the real-time transmission latency of an edge node is 30 milliseconds and that of a cloud center node is 180 milliseconds, the system establishes a logical direct connection channel for these nodes using software-defined networking technology. This channel is not... Instead of dedicated physical lines, these are dedicated data transmission links partitioned from the existing network, used solely for data interaction within streaming pipeline units. This avoids conflicts with other network traffic, ensuring transmission stability. Finally, the physical boundary range of each layer is dynamically adjusted based on node distribution density to achieve elastic scaling of the topology framework. Node distribution density refers to the number of nodes per unit area. The system calculates this by statistically analyzing the current number of nodes and coverage area at each layer. If a layer's node distribution is too dense (e.g., 100 nodes within 1-10 kilometers in the edge cluster layer, far exceeding 50 nodes / square kilometer), the physical boundary of that layer is automatically reduced to avoid resource waste due to excessive node concentration. Conversely, if a layer's node distribution is too sparse (e.g., only 2 nodes within 1 kilometer in the terminal device layer), the boundary is expanded.To ensure sufficient nodes at each level to handle tasks, this dynamic adjustment enables the topology framework to scale elastically, adapting to node distribution in different regions. After determining the hierarchical topology network framework, it is necessary to further calculate the available computing power of each node. This value directly determines whether a node can handle streaming pipeline units, avoiding task lag or node overload due to resource misjudgment. The specific implementation method is as follows: When calculating the available computing power of a node, a memory weighting factor calculation rule is first defined for each layer of nodes based on the three-layer physical boundary. The memory weighting factor is a coefficient dynamically adjusted according to the node's memory occupancy rate, used to correct the available computing power value. The rules differ for nodes at different layers due to varying levels of memory resource scarcity. For terminal device layer nodes with 4-8GB of memory, the rule is set to decrease the weighting factor by 0.15 for every 10% increase in memory occupancy. For edge cluster layer nodes with 32-64GB of memory, the rule is to decrease the weighting factor by 0.1 for every 10% increase in memory occupancy. For cloud server nodes with ample memory (128-512GB), the rule is to decrease the weighting factor by 0.05 for every 10% increase in memory occupancy. This tiered rule ensures that the impact of memory on computing power aligns with the hardware characteristics of nodes at each tier. Weighting coefficients are dynamically generated based on memory occupancy. The system collects memory occupancy data for each node in real time and combines it with the corresponding tier's calculation rules to generate weighting coefficients. For example, if a terminal node's occupancy is 30%, the weight is reduced by 0.15 × 3 = 0.45, resulting in a weighting coefficient of 1 - 0.45 = 0.55. If an edge node's occupancy is 50%, the weight is reduced by 0.1 × 5 = 0.5, resulting in a weighting coefficient of 0.5. If a cloud node's occupancy is 40%, the weight is reduced by 0.05 × 4 = 0.2, resulting in a weighting coefficient of 0.8. The weighting coefficient ranges from 0 to 1; a larger value indicates a smaller limitation on computing power from memory. Then, the real-time load rate and node peak values ​​are calculated... Multiplying the load factor by the memory weighting factor yields the weighted dynamic available computing power. Real-time load rate represents the node's current CPU / GPU utilization, while peak computing power is the maximum computing capacity the node's hardware can provide, determined by both factory parameters and hardware testing tools. The calculation logic for the dynamic available computing power is: Real-time load rate × Peak computing power × Memory weighting factor. For example, a cloud node with a real-time load rate of 0.6, a peak computing power of 2000 GFLOPS, and a weighting factor of 0.8 has a dynamic available computing power of 0.6 × 2000 × 0.8 = 960 GFLOPS. This value directly reflects the actual computing power currently available to the node. Finally, when a sharp increase in the node load rate is detected within a preset time, [the system automatically...]. The system automatically activates the virtual computing power buffer pool mechanism, injecting temporary virtual computing power quotas into the node to prevent pipeline reorganization triggered by sudden load increases. The preset time is usually set to 5 minutes, and a sharp increase is defined as the load rate rising from below 50% to above 80% within 5 minutes. The virtual computing power buffer pool mechanism is a virtual computing power resource pool reserved by the system and provided by cloud virtualization technology. When a sharp increase in load is detected, the system automatically injects temporary virtual computing power quotas into the node from the buffer pool. For example, injecting 100 GFLOPS of virtual computing power into the above terminal node increases its dynamic computing power availability value from the original 0.85 × 50 × 0.4 = 17 GFLOPS to (0.85 × 50 + 100) × 0.4 = 48.To achieve 5 GFLOPS, the uninterrupted reconfiguration of the streaming pipeline unit is avoided due to a sudden drop in computing power, ensuring continuous task operation. After clarifying the available computing power of each node, the transmission cost between nodes is the key factor affecting the data interaction efficiency of the streaming pipeline unit. If data between units needs to be transmitted between high-latency nodes, it will cause the task latency to exceed the standard. Therefore, this cost needs to be quantified through a transmission cost matrix. The specific implementation method is as follows: When generating the inter-node transmission cost matrix, the time-series prediction model is first trained based on historical round-trip delay data collected from the logically directly connected channel network. This historical round-trip delay data is extracted from the communication logs of the logically directly connected channels over the past 24 hours, recorded at 1-minute intervals (a total of 1440 data points). Each data point contains the sending node ID, receiving node ID, and round-trip delay. The time-series prediction model uses a Long Short-Term Memory (LSTM) network, trained by the system's algorithm module based on an open-source framework. Inputting the historical round-trip delay sequence, it outputs the predicted transmission delay every 5 minutes within the next hour. This model effectively captures the periodic fluctuations in network latency and outputs the predicted transmission delay between any two nodes within a future time period. After model training, the system outputs the predicted transmission delay for each node pair. The prediction results are initially constructed into a matrix framework in the format of sending node row - receiving node column. For example, the first row and second column of the matrix correspond to the predicted latency from terminal node 1 to edge node 1, and the second row and third column correspond to the predicted latency from edge node 1 to cloud node 1. This ensures that the transmission cost of any two nodes is recorded in a corresponding position. When the deviation between the measured latency and the predicted value exceeds the tolerance range, redundant bandwidth is automatically added to the corresponding link to compensate for network jitter. The tolerance range is set according to the task latency requirements: 5ms for real-time tasks and 10ms for non-real-time tasks. The system collects the measured round-trip latency in real time through a logical direct connection channel and compares it with the predicted value at the same time. For example, if the predicted latency is 20ms and the measured latency is 27ms, the deviation is 7ms. A deviation exceeding 5ms is considered a deviation. When the system determines the acceptable latency range for a given task, it automatically calls the software-defined network's bandwidth adjustment interface to add redundant bandwidth to the link. For example, if the original link bandwidth is 100Mbps, adding 20Mbps of redundant bandwidth brings the total bandwidth to 120Mbps. This additional bandwidth compensates for network jitter, bringing the measured latency back to a reasonable range. Node pairs whose predicted latency exceeds the communication threshold are then marked as unreachable. The communication threshold is the maximum acceptable transmission latency between nodes, determined by the task type: 100ms for real-time tasks and 300ms for non-real-time tasks. If a node pair's predicted latency is 350ms, exceeding the 300ms threshold for non-real-time tasks, then that node pair is marked as unreachable in the transmission cost matrix. For nodes that are unreachable, the subsequent four-dimensional joint decision engine will automatically skip these unreachable node pairs when allocating streaming pipeline units to avoid task failure due to excessive latency. Finally, the computational complexity of the decision engine is reduced by compressing the matrix storage structure. Since most node pairs are unreachable or infrequently interacting, traditional full matrix storage would waste a lot of memory. Therefore, the system adopts a sparse matrix storage structure, storing only node pairs marked as reachable and their predicted latency. Unreachable node pairs do not occupy storage space. At the same time, the latency data of reachable node pairs is integerized to further compress the data volume. This allows the decision engine to avoid traversing invalid data when calling the matrix, reducing the computational complexity from O(n²) to O(k), where k is the number of reachable node pairs, thus improving decision efficiency.After obtaining the available computing power and transmission cost matrix, node failure risk also needs to be considered. Assigning streaming pipeline units to nodes with high failure probabilities can easily lead to task interruptions. Therefore, it is necessary to quantify the risk using a failure probability prediction model and integrate multi-dimensional data to form a global resource map, providing a complete reference for subsequent dynamic mapping. The specific implementation method is as follows: When constructing the fault probability prediction model, the correlation between energy consumption data and historical fault records is first analyzed to establish a positive correlation model between power consumption fluctuation rate and fault occurrence probability. Energy consumption data consists of the power consumption value collected every 10 seconds from each node in real time. Historical fault records are the node fault data stored in the system over the past year, including the fault occurrence time and energy consumption fluctuations in the hour preceding the fault. During the analysis, the power consumption fluctuation rate, i.e., the maximum change in power consumption per unit time, is calculated using statistical tools. It is found that when the power consumption fluctuation rate exceeds 20%, the probability of a node failing within one hour is 40% higher than when the fluctuation rate is below 5%. Based on this pattern, a positive correlation model is established: the higher the power consumption fluctuation rate, the higher the fault occurrence probability. The fault probability value (0-1) output by the model increases linearly with the fluctuation rate. Furthermore, the node fault probability threshold is dynamically adjusted based on real-time energy consumption fluctuation data to ensure a low false alarm rate. Below the acceptable level, the failure probability threshold is the critical value for determining whether a node is high-risk (initially set to 0.3). The system calculates the power consumption fluctuation rate of each node in real time and dynamically adjusts the threshold based on the false alarm rate of the entire system. If the false alarm rate exceeds 1% (acceptable level) in a certain period, the threshold is increased from 0.3 to 0.35 to reduce false alarms. If the false alarm rate is below 0.5%, it is decreased to 0.25 to avoid missing high-risk nodes. Through this dynamic adjustment, it is ensured that the fault warning is accurate without affecting the task undertaking of normal nodes. Finally, the available computing power value, transmission cost matrix, and fault probability prediction value are fused together in three dimensions according to geographic topological coordinates and timestamps to form a global resource heat map with spatiotemporal attributes. The three-dimensional alignment and fusion associates the data of the three dimensions according to geographic topological coordinates (latitude and longitude) - timestamp (accurate to the second): First, the available computing power and predicted failure probability of each node are added with its geographical coordinates and collection timestamp. Then, the node pair latency data in the transmission cost matrix are associated with the coordinates and predicted timestamps of the sending and receiving nodes. Finally, these data are overlaid on a geographic map using a visualization tool to form a global resource heat map. The available computing power is represented by the color intensity, the transmission latency by the line thickness, and the failure probability by the node flashing frequency. It also supports backtracking to historical status by timestamp, allowing operation and maintenance personnel and decision engines to intuitively grasp the resource distribution and risk situation of the entire system, providing a global perspective for the dynamic mapping of streaming pipeline units.

[0018] When the four-dimensional joint decision engine performs dynamic mapping, a hierarchical decision tree architecture is adopted. The specific steps are as follows: Nodes in high-energy-consuming areas are excluded based on energy consumption constraint rules. Nodes with low failure probability are then selected based on fault tolerance weight. The weight ratio of each dimension is dynamically adjusted according to the task type. The weight of the path cost dimension is increased for real-time tasks, and the weight of the computing power matching degree is increased for computationally intensive tasks. Finally, before outputting the mapping scheme, the latency reduction effect after topology reconstruction is verified by virtual pipeline reorganization simulation. If the expected result is not achieved, the decision-making process is retried.

[0019] Further explanation is needed: after obtaining the global resource heatmap with spatiotemporal attributes, the four-dimensional joint decision engine can perform dynamic mapping of streaming pipeline units based on this map through a hierarchical decision tree architecture. This architecture can progressively select nodes and optimize schemes according to the priority of energy saving, followed by reliability, and then adapting to task requirements, ensuring that the final mapping result takes into account energy consumption costs, operational stability, and task performance requirements. The specific implementation method is as follows: When the four-dimensional joint decision engine performs dynamic mapping, a hierarchical decision tree architecture is adopted. The specific steps are as follows: First, high-energy-consuming nodes are excluded based on energy consumption constraint rules. These rules are screening criteria developed by combining the system's long-term energy-saving goals with regional electricity costs. The operations and maintenance team determines these rules based on node energy consumption data and electricity bills from the past six months. Specifically, this involves two judgment logics: First, a single-node energy consumption cap is calculated. The average real-time energy consumption of all nodes is calculated, and 1.4 times this average is set as the upper limit for a single node. Nodes whose real-time energy consumption exceeds this cap are directly marked as high-energy-consuming nodes and excluded. Second, a regional energy consumption threshold is applied. Nodes are divided into several regions based on geographical topology coordinates. If the proportion of high-energy-consuming nodes in a certain region exceeds 50%, or the total energy consumption of that region accounts for more than 25% of the system's total energy consumption, it is determined to be a high-energy-consuming region, and batch processing is performed. Excluding all nodes within the region, even if the energy consumption of some nodes does not exceed the single node limit, the core logic of this approach is to first control energy consumption costs at the global level, avoid allocating flow pipeline units to nodes or regions with excessively high energy consumption, and reduce the potential for failures caused by long-term full-load operation of high-energy-consuming equipment. This lays the foundation for energy conservation in the subsequent screening process. Then, nodes with low failure probability are screened based on the fault tolerance weight dimension. The fault tolerance weight dimension is to convert the failure probability value output by the failure probability prediction model into a quantifiable weight dimension. The specific conversion formula is fault tolerance weight = 1 - failure probability value. During the screening, the engine first extracts the fault tolerance weight of the remaining nodes after energy consumption screening from the global resource heat map, and then sets a fault tolerance weight threshold, setting the fault tolerance weight below 0.All nodes with a fault tolerance score of 7 were removed, leaving only high-reliability nodes. This step further ensures operational stability while maintaining energy efficiency. If streaming pipeline units are assigned to low-fault-tolerant nodes, subsequent node failures will cause task interruptions, increasing the additional cost of pipeline refactoring. Therefore, high-fault-tolerant nodes must be prioritized. Then, the weight ratio of each dimension is dynamically adjusted according to the task type. For real-time tasks, the weight of the path cost dimension is increased; for computationally intensive tasks, the weight of the computing power matching degree is increased. Here, each dimension refers to the four core evaluation dimensions of the four-dimensional joint decision engine: computing power matching degree (measuring the fit between the resource requirements of the streaming pipeline unit and the remaining computing power of the node), path cost, and inter-node data transmission latency assessment based on the transmission cost matrix. Fault tolerance weights have been selected and are used for final scoring. Energy consumption constraints have also been selected and are used for secondary verification. The initial weighting for each dimension is 25%, but it needs to be dynamically adjusted according to the task type to adapt to core requirements: For real-time tasks, the core requirement is low latency, so the weight of the path cost dimension is increased to 40%, while the weights of computing power matching and fault tolerance are each reduced to 20%, and energy consumption constraints are maintained at 10%, ensuring that the engine prioritizes nodes with low transmission latency. For computationally intensive tasks, the core requirement is sufficient computing power, so the weight of the computing power matching dimension is increased to 40%, while the weights of path cost and fault tolerance are each reduced to 20%, and energy consumption constraints are maintained at 10%, ensuring that the engine prioritizes nodes with sufficient computing power and suitable architecture. After adjustment, the engine will sort the nodes according to their scores in the four dimensions from high to low, initially determining the mapping node list for the streaming pipeline unit and forming a basic mapping scheme. Finally, before outputting the mapping scheme, the latency reduction effect after topology reconstruction is verified through virtual pipeline reconfiguration simulation. If the expected results are not achieved, the decision-making process is retried. The virtual pipeline reconfiguration simulation is a verification step that replicates the real-world operating scenario in the system's built-in simulation environment. This simulation environment imports real-time data on node computing power, transmission latency, and failure probability from the global resource heatmap, and loads the computational logic and data dependencies of the streaming pipeline unit to simulate the pipeline topology operation state after the unit is mapped to the target node. During verification, the engine first sets the basic mapping scheme... In this simulation, all streaming pipeline units are deployed to their corresponding target nodes according to data dependencies. The computation process of each unit and the data transmission process between nodes are then simulated. The total latency after topology reconstruction, the sum of the computation latency of each unit and the transmission latency between nodes, are calculated and compared with a preset latency target. If the total simulation latency is lower than the target, the simulation is considered successful, and the final mapping scheme is output. If the total latency exceeds the target, the simulation is considered unsuccessful, and the engine automatically backtracks to the step of adjusting the weight ratio of each dimension, or returns to the step of filtering nodes with low failure probability to re-filter nodes with even lower latency. This iteration is repeated until the simulation verification is successful, ensuring that the final output mapping scheme meets the performance requirements of the task and avoiding latency exceeding the target during actual runtime.

[0020] When jointly optimizing computing power matching degree and path cost, a hardware instruction set compatibility factor is first introduced into the computing power similarity calculation. Nodes with the same computing architecture are given additional affinity scores. The path cost is classified according to the data transmission volume between units. For large data transmission links, the path cost score is required to reach the minimum guarantee line. When the predicted latency exceeds the threshold, the software-defined network capability is automatically invoked to apply for an end-to-end dedicated channel to cover the marked unreachable node pairs.

[0021] When implementing fault tolerance weights and energy consumption constraints, the fault probability threshold is first dynamically adjusted according to the system's operating period. During the maintenance window, the fault tolerance standard is appropriately relaxed. Areas that are repeatedly excluded due to excessive energy consumption are marked as mapping forbidden zones. Subsequent decisions automatically skip nodes in these areas. A green channel mechanism is activated for nodes using renewable energy, giving them extra tolerance in the energy consumption constraint calculation and increasing their fault tolerance weight score, so that green nodes in the heat map can obtain priority mapping qualifications.

[0022] When performing fine-grained mapping of execution units, the admission conditions for GPU cluster nodes are first set. That is, the hardware affinity optimization process is triggered only when the unit's floating-point computation, memory requirements, and computing architecture requirements all meet the standards. For unit groups with strong data dependencies, they are forcibly assigned to nodes in the same domain that meet the requirements of low latency and high bandwidth based on the edge domain partitioning results. The edge domain range is dynamically adjusted according to the measured latency and bandwidth between nodes to ensure that the communication performance within the domain meets the requirements of data-intensive tasks.

[0023] Further explanation is needed: after the four-dimensional joint decision engine completes the basic dimension screening and weight adjustment, in order to further improve the accuracy of the streaming pipeline unit mapping and the data transmission efficiency, it is necessary to jointly optimize the computing power matching degree and path cost. This is because evaluating the two alone can easily lead to problems such as computing power matching but excessively high transmission latency or low latency but insufficient computing power. It is necessary to balance the relationship between the two through collaborative optimization to ensure that the unit can both compute efficiently and transmit data quickly. The specific implementation method is as follows: When jointly optimizing computing power matching and path cost, a hardware instruction set compatibility factor is first introduced into the computing power similarity calculation. This factor measures the degree of compatibility between the hardware instruction set required by the streaming pipeline unit and the actual instruction set of the node. Different hardware architectures correspond to different instruction sets. If the instruction set required by the unit is incompatible with the node's instruction set, software simulation adaptation is required, which consumes an additional 20%–30% of computing power. Therefore, this factor is used to prioritize compatible nodes. In practice, the system first extracts the hardware architecture information of each node from the global resource heatmap, and then reads the resource requirement tags of the streaming pipeline unit. Hardware architecture requirements dictate that nodes employing the same computing architecture receive additional affinity points. The base affinity score is 100 points. If a node's architecture is completely identical to the unit's requirements, 10 affinity points are added to the base score. If the architectures are compatible but not completely identical, only 5 points are added. If the architectures are incompatible, no points are added and 15 points are deducted. This differentiated scoring ensures that computing power similarity calculations reflect the actual impact of hardware adaptation, reducing computing power loss during subsequent operation. Subsequently, path cost sensitivity is tiered based on the data transmission volume between units. The data transmission volume refers to the amount of data that needs to be exchanged between streaming pipeline units, determined by the unit... The input / output interface parameters are determined, and the sensitivity is graded into three levels based on data volume: small (≤100MB, low sensitivity to path cost, some latency is allowed), medium (100MB-1GB, medium sensitivity, latency needs to be controlled), and large (>1GB, high sensitivity, excessive latency will cause task lag). The core logic of the grading is that the larger the data volume, the more significant the impact of transmission latency on the overall efficiency of the task. Path cost requirements need to be set specifically. For large data volume transmission links, the path cost score must reach a minimum guarantee line. The path cost score is a quantitative score converted from the predicted latency in the transmission cost matrix, with a minimum guarantee line. The threshold is set according to sensitivity levels. The minimum guarantee threshold for large data transmission links is set to 80 points. If the path cost score of a large data transmission link is only 70 points (corresponding to a latency of 45ms), the link is determined to be unacceptable and node pairs need to be re-selected to ensure that large data transmission can be completed quickly and avoid becoming a bottleneck for task efficiency. Finally, when the predicted latency exceeds the threshold, the software-defined networking (SDN) capability is automatically invoked to request an end-to-end dedicated channel to cover the marked unreachable node pairs. The predicted latency threshold is set according to the data volume: 30ms for large data links, 100ms for medium data links, and 300ms for small data links.If the predicted latency of a node pair exceeds the corresponding threshold and it was previously marked as unreachable, the system will automatically send a dedicated channel request to the SDN controller. The request includes the sending node ID, receiving node ID, and required bandwidth. After receiving the request, the SDN controller will allocate an independent transmission channel for the node pair in the existing network, isolate other network traffic, and simultaneously schedule network devices along the path to optimize the forwarding path, reducing the predicted latency to within the threshold. This transforms the previously unreachable node pair into a reachable one, ensuring that the data transmission requirements between streaming pipeline units are met and preventing the mapping scheme from failing due to link unreachability. After completing the joint optimization of computing power and path, in order to further ensure the system's operational stability and long-term energy-saving goals, it is necessary to implement targeted and refined management of fault tolerance weights and energy consumption constraints. The system load and regional energy consumption characteristics vary at different times, so the strategy needs to be dynamically adjusted to avoid resource waste or failure risks caused by a one-size-fits-all approach. The specific implementation method is as follows: When implementing fault tolerance weights and energy consumption constraints, the failure probability threshold is first dynamically adjusted based on the system runtime. The system runtime is divided into three categories according to load and maintenance requirements: peak business hours (e.g., 9:00-21:00, high system load, requiring strict stability assurance), off-peak business hours (e.g., 21:00-24:00, medium load), and maintenance windows (e.g., 0:00-6:00, some nodes may undergo maintenance, low load). The failure probability threshold is the critical value for determining whether a node is high-risk (initially set at 0.3, corresponding to a 30% failure probability). The adjustment logic is as follows: During peak hours, the threshold is maintained at 0.3, allowing only low-failure-risk nodes to participate in mapping. During off-peak hours, it is lowered to 0.25 to expand the node selection range. The tolerance standard is appropriately relaxed during maintenance windows, such as raising the threshold to 0.5. At this point, some nodes may temporarily increase their failure probability due to maintenance (e.g., from 0.2 to 0.4). Relaxing the threshold allows them to still participate in mapping, preventing task backlog due to insufficient available nodes during maintenance windows. Furthermore, because the load is low during maintenance periods, even if a small number of nodes fail, tasks can be quickly migrated without excessive concern about stability risks. Regions repeatedly excluded due to excessive energy consumption are marked as mapping exclusion zones. Subsequent decisions automatically skip nodes in these zones. "Repeatedly" is defined as three or more consecutive exclusions. The system records the number of times each region is excluded due to energy consumption constraints. If a region is excluded in three consecutive decisions because the average energy consumption of nodes within the region exceeds the upper limit, it indicates a potential bottleneck in the region's power infrastructure or generally high energy consumption of node hardware. Continuing to include it in the screening process would waste decision-making time. Therefore, the system marks this region as a mapping exclusion zone and marks it with a red block in the global resource heatmap. The subsequent four-dimensional joint decision engine performs the mapping. When this happens, all nodes in the region will be automatically filtered, eliminating the need for energy consumption detection and calculation, thus improving decision-making efficiency. Simultaneously, tasks are guided to areas with better energy efficiency, balancing the global energy consumption distribution. Finally, a green channel mechanism is activated for nodes using renewable energy, granting them extra tolerance in energy consumption constraint calculations and increasing their fault tolerance weight score. This allows green nodes in the heatmap to receive priority mapping. Nodes using renewable energy refer to those powered by clean energy sources such as solar, wind, and hydropower. The system identifies these nodes by reading their metadata information. The extra tolerance is specifically reflected in: The energy consumption limit for this type of node is relaxed from 1.4 times the global average to 1.6 times, allowing it to operate at higher energy consumption. Since renewable energy is more environmentally friendly and aligns with long-term energy conservation goals, strict energy consumption restrictions are unnecessary. The fault tolerance weight score is increased by adding 5 points to the existing fault tolerance weight, enhancing its competitiveness in the selection process. The system will also mark this type of node as green in the global resource heatmap. In subsequent decision-making, the engine will prioritize selecting matching nodes from the green nodes, even if their computing power matching degree or path cost score is slightly lower than non-green nodes. This drives the system towards low energy consumption and environmentally friendly operation. After completing fault tolerance and energy consumption control, refined mapping needs to be performed for high computing power demand units and strong data dependency unit groups. These units have extremely high requirements for hardware adaptability and low communication latency; ordinary mapping methods can easily lead to performance bottlenecks. Special strategies are needed to ensure operational efficiency. The specific implementation method is as follows: When performing fine-grained mapping of execution units, the first step is to set the admission criteria for GPU cluster nodes. This means that the hardware affinity optimization process is triggered only when the unit's floating-point computation load, memory requirements, and computing architecture requirements all simultaneously meet the criteria. GPU cluster nodes are cloud center or edge cluster nodes deployed with high-performance GPUs, primarily handling computationally intensive tasks. The three admission criteria must be set in conjunction with the GPU hardware capabilities: first, floating-point computation load, requiring a single-run computation requirement exceeding 100 GFLOPS, corresponding to 10% of the GPU's single-precision floating-point computation capability, to avoid GPU resource idleness; second, memory requirements, requiring the unit to require more than 8GB of memory during runtime; and third, the computing architecture requirements. The system requires that each unit support the corresponding GPU computing architecture. Hardware affinity optimization is only triggered when all three requirements of the streaming pipeline unit are met. The system will prioritize allocating the unit to nodes in the GPU cluster with high memory idle rates and low GPU load rates, while simultaneously closing unnecessary background processes on those nodes to release GPU resources. This ensures the unit can efficiently utilize GPU computing power and avoids performance degradation caused by resource contention. For unit groups with strong data dependencies, they will be forcibly assigned to nodes within the same domain that meet low latency and high bandwidth requirements, based on the edge domain partitioning results. Strong data dependency unit groups refer to a group of units with frequent data interactions and high latency sensitivity. The criteria for determining this are: More than 80% of the output data of Unit A needs to be used as input for Unit B, and Unit B needs to start calculation within 100ms after receiving the data. The edge domain partitioning result refers to dividing the nodes of the edge cluster layer into multiple independent edge domains according to the standard of measured latency between nodes ≤50ms and bandwidth ≥1Gbps. During forced allocation, the system will first identify all units in the strong data dependency unit group, and then query the target edge domain of each unit. If the units in the unit group belong to different edge domains, they will be forcibly adjusted to the same edge domain. The allocated nodes must meet the requirements of low latency and high bandwidth to ensure that the data transmission within the unit group can be completed quickly and avoid the increase in latency caused by cross-domain transmission. Finally, the edge domain range is dynamically adjusted based on the measured latency and bandwidth between nodes to ensure that the communication performance within the domain meets the requirements of data-intensive tasks. Specifically, the data-intensive task requirements are a measured latency between nodes within the domain ≤ 50ms and a bandwidth ≥ 1Gbps. The system collects measured latency and bandwidth data between all nodes within the edge domain every 5 minutes. The edge domain range will be adjusted if the following situations occur: First, if the measured latency of some nodes within the domain exceeds 50ms, the node will be removed from the edge domain to avoid lowering overall communication performance; second, if the number of available nodes within the domain is too small (e.g., too few...). If the number of nodes exceeds 5, resulting in insufficient nodes to be allocated to the strong data-dependent unit group, the edge domain will be expanded to include surrounding nodes that meet the requirements of latency ≤55ms and bandwidth ≥1Gbps. If the bandwidth of some nodes in the domain drops below 1Gbps, they will be replaced with nodes with higher bandwidth first. If replacement is not possible, the domain range will be reduced to retain only nodes with bandwidth ≥1Gbps. Through this dynamic adjustment, the communication performance in the edge domain is ensured to always meet the low latency and high bandwidth requirements of data-intensive tasks, thus providing a guarantee for the stable operation of the strong data-dependent unit group.

[0024] During the dismantling and migration of pipeline units, when the target node load continuously exceeds the critical value and there are low-load nodes in the neighborhood, the original unit is first dismantled into stateless computing sub-units and a lightweight routing coordinator. The computing sub-units are then migrated to idle nodes in the designated edge domain. Zero-copy direct connection channels between the migrated nodes are built using the user-space data plane development kit. Finally, the routing coordinator takes over the data flow scheduling between sub-units. When a sub-unit fails, it automatically switches to a backup node and rebuilds the direct connection channel, realizing the dynamic update of the transmission cost matrix.

[0025] It needs further explanation that, even after completing the fine-grained mapping of the flow pipeline units, and despite sufficient prior planning, a sudden increase in tasks may still cause a surge in the load on the target node. If not handled promptly, this could lead to task stalls or even interruptions. Therefore, a pipeline unit disassembly and migration mechanism is needed to achieve load balancing without interrupting task execution, thus ensuring the overall stability of the system. The specific implementation method is as follows: During the dismantling and migration of pipeline units, the triggering conditions must first be clarified. Specifically, this occurs when the target node's load consistently exceeds a critical value and a low-load node exists in the neighborhood. The target node refers to the node currently running the streaming pipeline unit. The critical value, defined based on the node's hardware type, is defined as the load rate exceeding the critical value in three consecutive data collections. The neighborhood is a dual geographical and logical range defined by the previously described hierarchical topology network framework. Geographically, it's within 10 kilometers of the target node; logically, it's within the same edge region. Low-load nodes refer to nodes in the neighborhood with a load rate below 40% and available computing power. A node that meets the requirements of a subunit will only trigger the subsequent dismantling and migration process if it simultaneously meets two conditions: overload and the presence of low-load nodes in its neighborhood. This avoids meaningless migration operations. Once the triggering conditions are met, the original unit is first dismantled into stateless computing subunits and a lightweight routing coordinator. Stateless computing subunits refer to submodules that do not rely on local storage data and only obtain data through input interfaces to execute computational logic. The original streaming pipeline unit typically contains continuous computational steps. During dismantling, it is split according to computational steps. Each subunit encapsulates independent computational logic and does not retain any local cached data. If intermediate calculation results are transmitted via subsequent direct connection channels and not stored locally on the node, this stateless design ensures that sub-units can be migrated at any time without worrying about data loss. The lightweight routing coordinator is a small functional module responsible for scheduling data flows between sub-units. Its core functions include recording the migration location of sub-units, forwarding the input and output data of sub-units, and monitoring the operating status of sub-units. During disassembly, the routing coordinator is first started on the original target node, and then the coordinator splits the original unit into 2 to 4 stateless computing sub-units according to the calculation logic of the original unit. The number of splits depends on the resource requirements of the sub-units and the number of low-load nodes. The resource matching process ensures that the computing power requirement of each sub-unit does not exceed the available computing power of the target migration node. The computing sub-unit is then migrated to an idle node within the designated edge domain. This edge domain is the one defined during the fine-grained mapping process described earlier. Selecting nodes within the same edge domain avoids increased transmission latency caused by cross-domain migration. Before migration, the lightweight routing coordinator sends resource probe requests to low-load nodes within the same edge domain to obtain real-time available computing power, memory usage, and network bandwidth data. It then filters out idle nodes that perfectly match the resource requirements of the computing sub-unit and initiates the migration process. The computational logic of the computational subunit is sent to the target idle node via a compressed transmission protocol. The target node loads the computational logic and initializes its runtime environment. The routing coordinator sends a data transmission command to the original node's computational subunit, transmitting the subunit's current input data to the target node's subunit. Upon receiving the data, the target node's subunit starts computation, while the original node's subunit pauses operation. Once the target node's subunit outputs its first computational result, the migration is confirmed as successful, and the original node's subunit completely stops, avoiding data conflicts caused by dual-node operation. After migration, a zero-copy direct connection channel between the migrated nodes is built using a user-space data plane development kit. This kit bypasses the operating system kernel and directly manipulates the hardware network card. Traditional kernel-space data transmission requires forwarding through the kernel buffer, resulting in approximately 20% latency loss. The user-space kit can directly establish a data transmission channel between user space and the network card, significantly reducing latency. A zero-copy direct connection channel means data is directly transferred from one node's memory to another node's memory without intermediate buffer copying. The specific construction process is as follows: Initialize the user-space data plane development kit on the migration node (the idle node where the sub-unit is located) and the original associated node (other sub-unit nodes that need to receive the output data of this sub-unit), configure the network card to user-space mode, establish a memory mapping relationship between the two nodes through the kit, bind the output memory area of ​​node A to the input memory area of ​​node B, and test the channel transmission rate and latency (ensure bandwidth ≥ 1Gbps and latency ≤ 10ms). If the standard is met, mark the channel as available; otherwise, reconfigure the network card parameters until the standard is met. Through this channel, data transmission between sub-units can avoid kernel intervention, ensuring that the transmission efficiency meets the task requirements. After the channel is built, the routing coordinator finally takes over the data flow scheduling between sub-units. The scheduling process must ensure the order and integrity of data transmission. First, the routing coordinator will establish a sub-unit-channel mapping table, recording the input data source channel and output data target channel of each sub-unit. The input of the model inference sub-unit comes from channel 1 of the data preprocessing sub-unit, and the output points to channel 2 of the result output sub-unit. Second, when When a sub-unit completes its computation and outputs data, it sends a data ready notification to the routing coordinator. The coordinator, based on the mapping table, sends a data transmission command to the corresponding target channel, triggering data transmission on the zero-copy direct connection channel. If data congestion occurs, the coordinator activates a temporary caching mechanism, waiting for the target sub-unit to become idle before triggering transmission again. Simultaneously, the coordinator monitors the data transmission status in real time. If data loss is detected, it instructs the source sub-unit to retransmit, ensuring data integrity. To further guarantee reliability, when a sub-unit fails, it automatically switches to a backup node and rebuilds the direct connection channel. The backup node is a low-load node reserved synchronously within the same edge domain during sub-unit migration, with the same resource configuration as the primary migration node. The backup node pre-loads the sub-unit's computation logic and runtime environment but does not start computation, remaining in standby mode. Fault detection is achieved by the routing coordinator sending heartbeat packets to the sub-unit every 2 seconds. If no heartbeat response is received from the sub-unit for three consecutive times, a fault is identified. If the node where the sub-unit resides suddenly loses power, the coordinator immediately executes the switchover process. A start command is sent to the backup node. The backup node loads the latest input data of the subunit, unprocessed data cached by the coordinator, and starts computation. Using the user-space data plane development kit, the zero-copy direct connection channel between the backup node and other subunits is rebuilt, the subunit-channel mapping table is updated, the channel of the faulty subunit is replaced with the new channel of the backup node, and data transmission to the faulty node is stopped. The switching process takes no more than 500ms, far below the task interruption tolerance threshold (1 second), ensuring continuous task operation. Finally, the transmission cost matrix is ​​dynamically updated. The predicted transmission delay between nodes recorded in the transmission cost matrix will change due to subunit migration and channel reconstruction. Therefore, after the direct connection channel is rebuilt, the system will automatically collect the measured transmission delay of the new channel and update the position of the corresponding node pair in the transmission cost matrix with the new delay data. At the same time, if the original node pair was previously marked as reachable, it will be updated to unreachable. The updated transmission cost matrix will be synchronized in real time to the global resource heat map and the four-dimensional joint decision engine to ensure that subsequent new streaming pipeline unit mappings can be based on the latest transmission cost data, avoiding the use of outdated delay information that would lead to an unreasonable mapping scheme.

[0026] This invention decomposes user computing power requests into streaming pipeline units that encapsulate computing logic, input / output interfaces, and resource requirement tags. Simultaneously, it collects computing power type, real-time load rate, memory usage, network round-trip latency, geolocation, and energy consumption data from cloud, edge, and endpoint nodes. Based on the collected data, a hierarchical topology network framework is constructed. The available computing power value of nodes is calculated, a transmission cost matrix between nodes is generated, and a fault probability prediction model is built. This is integrated to form a timestamped global resource map. Finally, a four-dimensional joint decision engine parses the data dependencies of the pipeline units and performs dynamic mapping, prioritizing the mapping of high-computing-power-demand units to GPU cluster nodes. This achieves uninterrupted reconfiguration of the pipeline topology during runtime, improving overall computing power resource utilization, task execution efficiency, and service stability.

[0027] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for integrating computing resources based on distributed streaming pipelines, characterized in that: Includes the following steps: S1. Decompose user computing power requests into sequential execution units. Each unit encapsulates computing logic, input / output interfaces, and resource requirement tags to form a streaming pipeline unit. At the same time, it collects computing power type, real-time load rate, memory usage rate, network round-trip latency, node geolocation, and energy consumption data of cloud-edge-device nodes in real time to provide raw input for dynamic resource mapping. S2. Based on the multi-dimensional data collected by S1, a hierarchical topology network framework is first constructed according to the geographic location of the nodes. The available computing power of each node is calculated by integrating real-time load rate and memory usage rate. The transmission cost matrix between nodes is dynamically generated by combining network round-trip latency. Energy consumption data and historical fault records are superimposed to construct a fault probability prediction model. Finally, it is integrated into a global resource map with timestamps. S3. Based on the global resource graph, the data dependencies of the streaming pipeline units are analyzed, and dynamic mapping is performed through a four-dimensional joint decision engine. Among them, the computing power matching degree dimension calculates the similarity between the unit demand and the remaining computing power of the node; the path cost dimension predicts the data transmission latency between units based on the transmission cost matrix; the fault tolerance weight dimension selects nodes with a failure probability lower than a preset threshold; and the energy consumption constraint dimension excludes energy consumption exceeding the limit. According to the evaluation results, units with high computing power demand are preferentially mapped to GPU cluster nodes, and unit groups with strong data dependencies are assigned to low-latency nodes in the same edge domain. When the load of the target node reaches the critical value, the pipeline units on the overloaded node are decomposed into sub-units, migrated to adjacent idle nodes, and a direct data channel is established to achieve uninterrupted reconfiguration of the pipeline topology during runtime.

2. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When constructing a hierarchical topology network framework, the system first dynamically divides the physical boundaries into three layers based on the latitude and longitude coordinates of the nodes' geographic locations, with the user's requested location as the center. The terminal device layer covers the nearest distance range, the edge cluster layer covers the mid-distance range, and the cloud center layer covers the long-distance range. Real-time transmission latency is obtained by sending dynamic topology probe packets to nodes between layers. Logical direct connection channels are established between edge nodes with latency below a set threshold and cloud center nodes. Finally, the physical boundary range of each layer is dynamically adjusted according to the node distribution density to achieve elastic scaling of the topology framework.

3. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When calculating the available computing power of a node, the memory weighting factor calculation rules are first defined for each node based on the three-layer physical boundary. The weighting coefficient is dynamically generated according to the memory occupancy rate. The real-time load rate, the node's peak computing power, and the memory weighting factor are multiplied to obtain the weighted dynamic available computing power value. When a node load rate is detected to rise sharply within a preset time, the virtual computing power buffer pool mechanism is automatically activated to inject a temporary virtual computing power quota into the node, thus avoiding pipeline reorganization triggered by sudden load.

4. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When generating the inter-node transmission cost matrix, the time series prediction model is first trained by collecting historical round-trip delay data based on the logical direct-connection channel network. The predicted transmission delay between any two nodes in the future time period is output. When the deviation between the measured delay and the predicted value exceeds the tolerance range, redundant bandwidth is automatically added to the corresponding link to compensate for network jitter. Node pairs whose predicted delay exceeds the communication threshold are marked as unreachable. The computational complexity of the decision engine is reduced by using a compressed matrix storage structure.

5. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When constructing the failure probability prediction model, we first analyze the correlation between energy consumption data and historical failure records, establish a positive correlation model between power consumption fluctuation rate and failure probability, and dynamically adjust the node failure probability threshold according to real-time energy consumption fluctuation data to ensure that the false alarm rate is below an acceptable level. Finally, we align and fuse the available computing power value, transmission cost matrix and failure probability prediction value in three dimensions according to geographical topological coordinates and timestamps to form a global resource heat map with spatiotemporal attributes.

6. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When the four-dimensional joint decision engine performs dynamic mapping, a hierarchical decision tree architecture is adopted. The specific steps are as follows: Nodes in high-energy-consuming areas are excluded based on energy consumption constraint rules. Nodes with low failure probability are then selected based on fault tolerance weight. The weight ratio of each dimension is dynamically adjusted according to the task type. The weight of the path cost dimension is increased for real-time tasks, and the weight of the computing power matching degree is increased for computationally intensive tasks. Finally, before outputting the mapping scheme, the latency reduction effect after topology reconstruction is verified by virtual pipeline reorganization simulation. If the expected result is not achieved, the decision-making process is retried.

7. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When jointly optimizing computing power matching degree and path cost, a hardware instruction set compatibility factor is first introduced into the computing power similarity calculation. Nodes with the same computing architecture are given additional affinity scores. The path cost is classified according to the data transmission volume between units. For large data transmission links, the path cost score is required to reach the minimum guarantee line. When the predicted latency exceeds the threshold, the software-defined network capability is automatically invoked to apply for an end-to-end dedicated channel to cover the marked unreachable node pairs.

8. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When implementing fault tolerance weights and energy consumption constraints, the fault probability threshold is first dynamically adjusted according to the system's operating period. During the maintenance window, the fault tolerance standard is appropriately relaxed. Areas that are repeatedly excluded due to excessive energy consumption are marked as mapping forbidden zones. Subsequent decisions automatically skip nodes in these areas. A green channel mechanism is activated for nodes using renewable energy, giving them extra tolerance in the energy consumption constraint calculation and increasing their fault tolerance weight score, so that green nodes in the heat map can obtain priority mapping qualifications.

9. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: When performing fine-grained mapping of execution units, the admission conditions for GPU cluster nodes are first set. That is, the hardware affinity optimization process is triggered only when the unit's floating-point computation, memory requirements, and computing architecture requirements all meet the standards. For unit groups with strong data dependencies, they are forcibly assigned to nodes in the same domain that meet the requirements of low latency and high bandwidth based on the edge domain partitioning results. The edge domain range is dynamically adjusted according to the measured latency and bandwidth between nodes to ensure that the communication performance within the domain meets the requirements of data-intensive tasks.

10. The computing resource fusion method based on distributed streaming pipelines according to claim 1, characterized in that: During the dismantling and migration of pipeline units, when the target node load continuously exceeds the critical value and there are low-load nodes in the neighborhood, the original unit is first dismantled into stateless computing sub-units and a lightweight routing coordinator. The computing sub-units are then migrated to idle nodes in the designated edge domain. Zero-copy direct connection channels between the migrated nodes are built using the user-space data plane development kit. Finally, the routing coordinator takes over the data flow scheduling between sub-units. When a sub-unit fails, it automatically switches to a backup node and rebuilds the direct connection channel, realizing the dynamic update of the transmission cost matrix.

Images