A method and system for isolating and scheduling computing power resources for multiple AI tasks in parallel

Abstract

The disclosure provides a kind of computing power resource isolation and scheduling method and system of multiple AI task concurrency, belong to computer resource scheduling technical field.The method is by monitoring the memory occupancy of computing node, if it is more than fuse threshold value, then trigger state snapshot collection, capture process stack, network connection and cache queue information generation state data set.Initial transmission path is generated from source node to standby node, detects path intersection point and calculates transmission collision probability, if it is more than collision threshold value, then adjust generation transmission path.According to the bandwidth of each path segment and the volume of data slice, calculate the expected transmission time, if it is less than time threshold value, then start data migration.In the process of migration, real-time monitoring link state, if congestion occurs, then filter standby node and dynamically allocate data flow direction to generate target path, continue to complete migration and release source node resource after confirming completion.The disclosure effectively reduces the path conflict rate and data migration delay, improves resource utilization efficiency and system stability.

Description

Technical Field

[0001] This disclosure relates to the field of computer resource scheduling and big data processing technology, specifically to a method and system for isolating and scheduling computing resources for concurrent multi-AI tasks. Background Technology

[0002] In the field of computing resource isolation and scheduling for concurrent multi-AI tasks, ensuring the stable operation of clusters under extremely high load conditions is of decisive importance. Current conventional resource scheduling schemes mostly rely on task queuing or simple proximity allocation mechanisms. This approach ignores the instantaneous impact of sudden massive data surges on the cluster network's physical topology. Consequently, when faced with large-scale task anomalies, the data transfer process itself can easily become a heavy burden that overwhelms the entire computing network. Especially in big data processing scenarios, multiple AI tasks often need to process massive datasets simultaneously (such as petabyte-scale image, video, or text data), which makes the scheduling and management of cluster resources even more challenging.

[0003] In real-world concurrent computing environments, when multiple high-energy-consuming tasks within a cluster simultaneously approach memory limits, the system triggers a batch circuit breaker mechanism to protect the hardware. This batch circuit breaker instantly generates an extremely large amount of task context state data. Once this massive amount of data floods into the cluster network for transfer, it instantly overwhelms the limited communication channels. Due to the varying physical distances and remaining bandwidth of different backup servers, the massive data streams are highly susceptible to path overlap and collisions during transmission, leading to severe network congestion or even transmission paralysis.

[0004] For example, in a large computing center, hundreds of complex image processing tasks simultaneously exhaust memory and are forcibly suspended. The system instantly packages hundreds of huge status files. When these files are transferred to other idle servers, if they are all crammed onto a few backbone network lines, it's like all the cars in the morning rush hour rushing into the same intersection. Not only will no one be able to pass smoothly, but it will also cause the entire computing center's network to be completely locked down.

[0005] Therefore, in such extreme concurrent circuit breaker scenarios, overcoming the differences in physical distance and bandwidth limitations of each backup node, and avoiding cross-conflicts of massive migration data streams on network paths, thereby ensuring the smooth transfer of tasks, has become a key issue in realizing the isolation and scheduling of concurrent computing resources for multiple artificial intelligence tasks.

[0006] The information disclosed in the background section is only for enhancing the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of this, this disclosure provides a method and system for isolating and scheduling computing resources for multiple concurrent AI tasks, which can effectively reduce path conflict rate and data migration latency in a distributed environment, and improve resource utilization efficiency and system stability when multiple tasks are concurrent.

[0008] In a first aspect, embodiments of this application provide a method for isolating and scheduling computing resources for concurrent multi-AI tasks, the method comprising: Monitor the memory usage of computing nodes in a distributed system; If the memory usage exceeds the preset circuit breaker threshold, a state snapshot acquisition mechanism is triggered to obtain the runtime state data of the computing node in order to generate a state dataset. Based on the state dataset, determine the data volume characteristics of the data to be transferred, and determine the network topology characteristics of the backup nodes in the distributed system; Based on the data volume characteristics and the network topology characteristics, an initial transmission path is generated from the source node to each backup node; Detect whether there is a path intersection in the initial transmission path, where the path intersection is the same relay node passed by at least two paths within the same time window; If the path intersection point exists, calculate the transmission collision probability of the path intersection point; If the transmission collision probability exceeds a preset collision threshold, the initial transmission path is adjusted to generate a first transmission path. The estimated transmission time is calculated based on the bandwidth limit of each path segment in the first transmission path and the data fragment size of the data to be transferred. If the estimated transmission time is less than a preset time threshold, then the first transmission path is determined to be the second transmission path, and data migration is initiated according to the second transmission path. During the data migration process, network status parameters of each link are monitored in real time; If network congestion is determined to occur in the link based on the network status parameters, then a backup node that meets the conditions is selected from the network topology features, and the data flow is reallocated according to the dynamic load balancing weight to generate the target transmission path. Continue the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

[0009] Secondly, embodiments of this application provide a computing resource isolation and scheduling system for concurrent multi-AI tasks, the system comprising: The monitoring module is used to monitor the memory usage of computing nodes in a distributed system. The snapshot acquisition module is used to trigger the state snapshot acquisition mechanism if the memory usage rate exceeds the preset circuit breaker threshold, and to obtain the runtime state data of the computing node to generate a state dataset. The feature extraction module is used to determine the data volume characteristics of the data to be transferred and the network topology characteristics of the backup nodes in the distributed system based on the state dataset. The path generation module is used to generate an initial transmission path from the source node to each backup node based on the data volume characteristics and the network topology characteristics. The collision detection module is used to detect whether there is a path intersection in the initial transmission path. The path intersection is the same relay node passed by at least two paths within the same time window. The probability calculation module is used to calculate the transmission collision probability of the path intersection if the path intersection exists. The path adjustment module is used to adjust the initial transmission path to generate a first transmission path if the transmission collision probability exceeds a preset collision threshold. The time estimation module is used to calculate the estimated transmission time based on the bandwidth limit of each path segment in the first transmission path and the data fragment size of the data to be transferred. The migration initiation module is used to determine the first transmission path as the second transmission path and initiate data migration according to the second transmission path if the estimated transmission time is less than a preset time threshold. The status monitoring module is used to monitor the network status parameters of each link in real time during the data migration process; The congestion handling module is used to select backup nodes from the network topology features and reallocate data flow according to dynamic load balancing weights if network congestion is determined to occur in the link based on the network status parameters, so as to generate a target transmission path. The release module is used to continue the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

[0010] This application provides a method and system for isolating and scheduling computing resources for concurrent AI tasks. By monitoring memory usage in real time and triggering a state snapshot mechanism when the circuit breaker threshold is exceeded, state data such as process stacks, network connections, and cache queue depths during node operation are collected, enabling accurate capture of complete context information at the moment of task interruption. By parsing the storage capacity usage ratio and data fragment volume distribution from the state data, and combining the physical hop count, link bandwidth, and transmission latency between backup nodes to construct network topology features, a graph convolutional layer is used to generate node embedding vectors and output the initial transmission path, achieving comprehensive perception of the cluster network physical topology and intelligent path planning. For concurrent traffic contention and link reuse conflicts at path intersections, the transmission collision probability is calculated and the relay node sequence is dynamically replaced, effectively avoiding cross-collisions of massive migration data streams on network paths. During data migration, link throughput attenuation and queue depth changes are monitored in real time. When a sudden change in packet loss rate or a continuous increase in queue depth is detected, congestion determination is immediately triggered, and data flow is redistributed according to dynamic load balancing weights, achieving rapid response and adaptive adjustment to sudden network congestion. This disclosure effectively reduces path conflict rate and data migration latency in distributed environments, and improves resource utilization efficiency and system stability during multi-task concurrency. This disclosure is applicable to the concurrent scheduling of large-scale AI tasks in big data processing scenarios. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall process of a method for isolating and scheduling computing resources for concurrent multi-AI tasks provided in an exemplary embodiment of this application.

[0012] Figure 2 This is a flowchart illustrating the state dataset generation step in a multi-AI task concurrent computing resource isolation and scheduling method provided in another exemplary embodiment of this application.

[0013] Figure 3 This is a flowchart illustrating the network topology feature determination step in a multi-AI task concurrent computing resource isolation and scheduling method provided in another exemplary embodiment of this application.

[0014] Figure 4 This is a flowchart illustrating the initial transmission path generation step in the multi-AI task concurrent computing resource isolation and scheduling method provided in another exemplary embodiment of this application.

[0015] Figure 5 This is a flowchart illustrating the transmission collision probability calculation step in the multi-AI task concurrent computing resource isolation and scheduling method provided in another exemplary embodiment of this application.

[0016] Figure 6This is a flowchart illustrating the path adjustment step in the multi-AI task concurrent computing resource isolation and scheduling method provided in another exemplary embodiment of this application.

[0017] Figure 7 This is a flowchart illustrating the network congestion determination and handling process in the computing resource isolation and scheduling method for concurrent multi-AI tasks provided in another exemplary embodiment of this application.

[0018] Figure 8 This is a diagram of the module composition of a computing resource isolation and scheduling system for concurrent multi-AI tasks provided in another exemplary embodiment of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0020] This disclosure provides a method for isolating and scheduling computing resources for concurrent multi-AI tasks, such as... Figure 1 The method shown is for isolating and scheduling computing resources for concurrent multi-AI tasks. This method may include the following steps: Step S110: Monitor the memory usage of computing nodes in the distributed system.

[0021] Specifically, in a distributed computing cluster, the monitoring agent collects the memory usage of each computing node at a preset period and uses a sliding window algorithm to calculate a weighted average memory usage rate, thereby smoothing out errors caused by instantaneous fluctuations.

[0022] Step S120: If the memory occupancy rate exceeds a preset circuit breaker threshold, a state snapshot acquisition mechanism is triggered to obtain the runtime state data of the computing nodes and generate a state dataset. The circuit breaker threshold is determined based on the upper limit of the hardware physical memory of each computing node in the distributed system and the reserved overhead of the operating system. Specifically, by collecting the total physical memory capacity of each computing node and deducting the fixed reserved memory occupied by the operating system kernel and critical system services (e.g., 5% to 10% of the total memory), the remaining available memory range of 85% to 90% is set as the circuit breaker threshold range. The basis for setting this threshold range is that when the memory occupancy rate exceeds this range, the remaining available memory of the system is insufficient to support the memory allocation request of new tasks, and the page swapping frequency will increase sharply, leading to a significant decrease in task execution efficiency. At this time, triggering the circuit breaker mechanism can terminate the task in time and save the state context before the system performance drops precipitously. In actual deployment, operation and maintenance personnel can fine-tune the threshold range according to the computational intensity characteristics of different business scenarios. For example, the lower limit of the range is 85% for computationally intensive tasks, and the upper limit of the range is 90% for memory access intensive tasks.

[0023] In one embodiment of this disclosure, such as Figure 2 As shown, step S120 may include the following sub-steps: Step S121: Capture process stack information, network connection status, and cache queue depth from the runtime environment of the computing node.

[0024] Step S122: Perform anomaly detection on the captured data, remove abnormal data, and obtain a standard snapshot set.

[0025] Optionally, the Isolation Forest algorithm can be used to perform anomaly detection on the initial snapshot set and remove abnormal snapshot data.

[0026] Step S123: Generate the status dataset based on the standard snapshot set, combined with node identifiers and resource usage details.

[0027] Optionally, principal component analysis can be used to reduce the dimensionality of the multidimensional state dataset, extract principal component feature vectors, obtain a dimensionality-reduced dataset, construct a resource allocation matrix based on the dimensionality-reduced dataset, and if the rank of the resource allocation matrix is ​​less than a preset full rank threshold, a resource reallocation mechanism is triggered to determine the target allocation scheme.

[0028] Specifically, when the memory usage of a node exceeds the preset circuit breaker threshold (e.g., 85%) multiple times (e.g., three times consecutively), the monitoring agent immediately sends a high-priority forced termination command to the task scheduling engine. This instantly cuts off the execution context of the current data processing task by sending a termination signal to the target process. At the moment the task is interrupted, the system kernel probe mechanism is simultaneously activated, delving into the underlying operating system within a very short time window (e.g., 200 milliseconds) to capture the function call stack information of the current process (up to 50 layers). Simultaneously, it scans the system network status table to count the number of Transmission Control Protocol connections in specific states (e.g., waiting to close), and reads the depth of the backlogged message queue in the memory database (e.g., exceeding 10,000 messages). These heterogeneous data extracted from the underlying layer are then fed into the data aggregation engine. The engine uses a hash mapping algorithm to bind the unique physical identifier assigned to the node, the trigger timestamp, and the captured stack, network, and queue resource usage details to key-value pairs, transforming the originally scattered information into structured state records (such as a single record of about 500 kilobytes). Finally, tens of thousands of such records are concurrently written into continuous data blocks in the distributed columnar storage cluster.

[0029] The above method comprehensively captures the complete context information at the moment of task interruption by capturing multi-dimensional runtime data such as process stack information, network connection status, and cache queue depth at the kernel level, avoiding the problem of missing state information caused by relying on a single indicator. Simultaneously, anomaly detection and removal of abnormal snapshot data using the Isolation Forest algorithm, combined with dimensionality reduction using Principal Component Analysis, effectively eliminates redundant information and noisy data, improving the quality of the state dataset.

[0030] Step S130: Based on the state dataset, determine the data volume characteristics of the data to be transferred, and determine the network topology characteristics of the backup nodes in the distributed system.

[0031] In one embodiment of this disclosure, determining the data volume characteristics of the data to be transferred may include: parsing the storage capacity usage ratio and data fragment volume distribution of each computing node from the state dataset; and calculating the number and size of the data blocks to be transferred based on the storage capacity usage ratio and the data fragment volume distribution to obtain the data volume characteristics.

[0032] In one embodiment of this disclosure, such as Figure 3 As shown, determining the network topology characteristics of the standby nodes in the distributed system may include the following sub-steps: Step S131: Use the random forest algorithm to process the storage capacity utilization rate and the data fragmentation amount to obtain the processed storage capacity utilization rate and the processed data fragmentation amount.

[0033] Step S132: Based on the processed storage capacity utilization rate and the processed data fragmentation amount, count the number of data blocks to be transferred and the size of each block to obtain the data volume value.

[0034] Step S133: If the data volume value is greater than the preset volume threshold, then extract the sub-data blocks and their sizes from the data block to obtain the sub-data volume value.

[0035] Step S134: Based on the sub-data volume value, detect the physical hop count, link bandwidth limit, and historical transmission delay records between backup nodes to determine the remaining storage capacity of the backup nodes.

[0036] Step S135: Use a clustering algorithm to cluster the physical hop count, the upper limit of the link bandwidth, and the historical transmission delay records to obtain the clustered physical hop count, the clustered bandwidth value, and the clustered delay value.

[0037] Step S136: Determine the cross-domain transmission cost based on the clustered physical hop count, the clustered bandwidth value, the clustered latency value, and the remaining storage capacity.

[0038] Step S137: If the cross-domain transmission cost is less than a preset cost threshold, then the network topology characteristics are obtained based on the cross-domain transmission cost and the remaining storage capacity.

[0039] Specifically, the random forest algorithm in step S131 processes the storage capacity utilization and data fragmentation as follows: First, construct the training dataset. Extract N samples from the cluster's historical operation logs. Each sample contains storage capacity utilization, total number of data shards, data shard volume distribution features (including mean, variance, maximum shard volume, minimum shard volume), and corresponding data transfer priority labels, forming an N×M dimensional feature matrix, where M is the number of feature dimensions.

[0040] Secondly, the Bootstrap sampling method is used to randomly sample K subsets of the training dataset with replacement, where K is the number of decision trees. For each node of each decision tree, m candidate features are randomly selected from all M features. The optimal splitting feature and splitting threshold are chosen from these m candidate features. The Gini coefficient is used as the splitting quality evaluation index; the smaller the Gini coefficient, the higher the purity of the samples after splitting. K decision trees are recursively grown in this way. Splitting stops when each tree reaches its maximum depth or the number of samples in a node is less than a preset minimum number of samples, ultimately forming a random forest model.

[0041] Then, the current storage capacity utilization and data fragmentation amount are input into the trained random forest model. Each decision tree independently outputs a prediction result (i.e., data transfer priority category or data volume level) according to its node splitting rules. The prediction results of K trees are integrated by majority voting. The frequency of each prediction category in all decision trees is counted. The category with the highest frequency is taken as the final output result, i.e., the processed storage capacity utilization and the processed data fragmentation amount.

[0042] By using the random forest algorithm described above, the ensemble learning mechanism of multiple decision trees can effectively suppress the risk of overfitting of a single decision tree. At the same time, the random feature selection mechanism enhances the model's adaptability to high-dimensional feature spaces, thereby accurately identifying the core features of the data to be transferred and avoiding misjudgments or omissions caused by a single indicator.

[0043] Specifically, the system uses a distributed monitoring component to collect real-time cluster logs, extracting the total capacity and occupied capacity of the source node. It then uses division to determine the storage capacity usage ratio and employs a hash mapping algorithm to scan the underlying files, obtaining the data fragment volume distribution (e.g., primarily exhibiting a normal distribution and concentrated in the 64-128 byte range). If a node triggers a capacity warning threshold, the system automatically traverses its directory tree structure, counts the number of cold data blocks that meet the migration rules, and determines the size of each block. By multiplying the number of blocks by their size, the system accurately calculates the total data volume characteristics of the data to be transferred.

[0044] To ensure a smooth migration of this data, the routing detection module sends probe messages (such as Internet Control Message Protocol (ICMP) messages to candidate backup nodes. It uses a time-to-live (TTL) field decrement mechanism to measure the physical hop count between the source and target nodes. Simultaneously, it injects test traffic packets into the link to determine the current link bandwidth limit and extracts historical communication logs (e.g., the past 30 days) from the time-series database to calculate the average historical transmission latency. The system then reads the remaining storage capacity of the target node, normalizes the hop count, bandwidth, and latency data, and substitutes them into a weighted distance formula (e.g., the weighted Euclidean distance formula) to calculate the cross-domain transmission cost function. The capacity weight and network cost weight can be set according to actual needs (e.g., capacity weight set to 40%, network cost weight set to 60%). Finally, a multidimensional tensor matrix containing all the above parameters is generated as the network topology feature.

[0045] In the above method, the random forest algorithm is used to process storage capacity utilization and data fragmentation, which can accurately identify the core characteristics of the data to be transferred and avoid misjudgment or omission due to a single threshold judgment. By detecting the physical hop count, link bandwidth limit, and historical transmission latency records between backup nodes, and using a clustering algorithm for classification, the network performance differences of backup nodes can be comprehensively quantified. On this basis, the cross-domain transmission cost function is calculated using a weighted distance formula, which unifies the measurement of the remaining storage capacity of the nodes and the network transmission cost, so that the generated network topology features can truly reflect the physical topology structure and link quality differences of the cluster.

[0046] Step S140: Generate initial transmission paths from the source node to each backup node based on the data volume characteristics and the network topology characteristics.

[0047] In one embodiment of this disclosure, such as Figure 4 As shown, step S140 may include the following sub-steps: Step S141: Establish an adjacency matrix between nodes based on the data fragmentation amount and the cross-domain transmission cost.

[0048] Step S142: Extract the link segments and bandwidth values ​​between each node. If the bandwidth value is less than a preset bandwidth threshold, mark the bandwidth sharing conflict status of the corresponding link in the adjacency matrix to obtain an adjacency matrix with conflict markings.

[0049] Step S143: Generate a topology graph based on the adjacency matrix with conflict annotations, and extract features from the topology graph using a graph convolutional network to obtain the node embedding vector of each node.

[0050] Step S144: Calculate the vector similarity between the source node and each backup node based on the node embedding vector of each node, and generate candidate paths based on the vector similarity.

[0051] Step S145: If the number of nodes in the candidate path is greater than a preset node threshold, then remove the excess nodes to update the candidate path.

[0052] Step S146: Extract the connected path sequence from the updated candidate path, and obtain the initial transmission path based on the connected path sequence.

[0053] Specifically, the system first obtains the total amount of data fragments and divides them equally into multiple data fragments to determine the data fragment volume distribution. Simultaneously, it calculates network topology features based on the cross-domain transmission cost function, adding the link delay to the inverse of the available bandwidth with preset weights to obtain the link cost. Then, the system constructs an adjacency matrix containing multiple data center nodes, filling the matrix elements with the aforementioned cost. When it detects that a link simultaneously carries multiple fragment transmissions, causing the total demand to exceed the physical limit, it marks the bandwidth sharing conflict detection result as a conflict state on the corresponding edge of the adjacency matrix. Next, the system inputs the adjacency matrix with conflict annotations into a graph convolutional network model. The first layer aggregates neighbor node information, mapping the initial features to a first dimension (e.g., 64-dimensional). The second layer further extracts node embedding vectors from the topology and outputs them as a dense vector with a second dimension (e.g., 128-dimensional). Finally, the system uses the similarity between node embedding vectors (e.g., cosine similarity) as a heuristic guiding factor to drive a search algorithm (e.g., depth-first search) to find paths from the source node to the backup node, traversing and recording connected nodes sequentially, ultimately generating a candidate path sequence as the initial transmission path.

[0054] In the aforementioned method, by integrating the data fragmentation size and cross-domain transmission cost into the adjacency matrix construction, the matrix elements can simultaneously reflect the dual constraints of storage load and network transmission cost. By marking bandwidth sharing conflict states in the adjacency matrix, potential link contention risks can be identified in advance, avoiding the neglect of bandwidth contention issues caused by multi-task concurrency during the path planning stage. Simultaneously, by extracting node embedding vectors through graph convolutional networks, the multidimensional relationships between nodes (physical hop count, bandwidth, latency, storage capacity, etc.) can be encoded into dense vectors, enabling subsequent path search to be performed in a high-dimensional semantic space, significantly improving the accuracy and intelligence of path planning.

[0055] Step S150: Detect whether there is a path intersection in the initial transmission path. The path intersection is the same relay node that at least two paths pass through in the same time window.

[0056] Step S160: If the path intersection point exists, calculate the transmission collision probability of the path intersection point.

[0057] In one embodiment of this disclosure, such as Figure 5 As shown, step S160 may include the following sub-steps: Step S161: Extract the time window of each path passing through the intersection of the paths. If the time windows of at least two paths overlap and converge at the same relay node, then obtain the concurrent traffic value converged at the relay node.

[0058] Step S162: Based on the concurrent traffic value, extract the concurrent traffic contention intensity and transmission delay fluctuation range of the path intersection.

[0059] Step S163: Determine the buffer overflow risk coefficient of the relay node based on the historical cache records of the relay node.

[0060] Step S164: Calculate the product of the concurrent traffic contention intensity and the buffer overflow risk coefficient as the transmission collision probability.

[0061] Specifically, the system uses a search algorithm (such as depth-first search) to traverse the initial transmission path set from the source node to the destination node, extracting the routing topology sequence of multiple path segments. Subsequently, the system collects traffic data for each path segment in real time and calculates the link reuse rate statistics, setting a single path occupancy threshold. When the link reuse rate statistics of a relay node exceed this threshold, the system automatically identifies and marks it as a path intersection. Next, the system uses a timestamp comparison algorithm to perform concurrent detection on this intersection, setting a time window. If at least two paths simultaneously request the relay node's forwarding resources within the time window, the system extracts the concurrent traffic contention intensity of the intersection through the traffic monitoring module (e.g., calculated using a queuing theory model). Simultaneously, it performs variance analysis on historical latency logs to extract the transmission latency fluctuation range. The system further calculates the buffer overflow risk coefficient based on the upper limit of the latency fluctuation range and the relay node's current remaining memory capacity, using a smoothing algorithm (such as exponential smoothing). Finally, the concurrent traffic contention intensity is multiplied by the buffer overflow risk coefficient to calculate the transmission collision probability of the path intersection.

[0062] In the above method, a time window comparison algorithm accurately identifies path intersections, avoiding misjudgments caused by relying solely on topology. By extracting the intensity of concurrent traffic contention and the range of transmission latency fluctuations, and combining historical cache records to calculate the buffer overflow risk coefficient, the congestion risk at path intersections can be quantitatively assessed. Multiplying the intensity of concurrent traffic contention by the buffer overflow risk coefficient yields the transmission collision probability, comprehensively reflecting the matching degree between traffic load intensity and node processing capacity, thus avoiding path optimization deviations caused by subjective judgment.

[0063] Step S170: If the transmission collision probability exceeds a preset collision threshold, the initial transmission path is adjusted to generate a first transmission path. The collision threshold is determined based on historical congestion event statistics of the cluster network and the buffer capacity of the path intersection point. Specifically, by analyzing the network transmission logs of the cluster over a specific period (e.g., 30 days), historical records of buffer overflows leading to packet loss and retransmission at each relay node under different concurrent traffic intensities are statistically analyzed. All transmission collision probability values ​​calculated before historical overflow events are arranged in ascending order, for example, the value at the 75th percentile is used as the baseline threshold. Simultaneously, combining the physical memory buffer size of the relay node and the average size of data fragments, the time window for normal buffer occupancy of a single path and the overflow threshold during multi-path concurrency are calculated. These two factors are then combined to determine the collision threshold, for example, 0.6. This threshold indicates that when the transmission collision probability exceeds 60%, the probability of buffer overflow at the path intersection leading to data retransmission will significantly increase. Triggering path adjustment at this point can proactively avoid risks before congestion occurs.

[0064] In one embodiment of this disclosure, such as Figure 6 As shown, step S170 may include the following sub-steps: Step S171: Traverse all nodes in the initial transmission path, calculate the overlap between adjacent nodes, and obtain the overlap sequence.

[0065] Step S172: Based on the overlap sequence, select the set of consecutive nodes with the highest overlap, and mark the path segments corresponding to the set of consecutive nodes as objects to be adjusted.

[0066] Step S173: Construct the reachability matrix of neighboring nodes based on the network topology features, and obtain the node communication delay matrix within the reachability matrix.

[0067] Step S174: Based on the node communication delay matrix, select the node with the smallest delay in the reachability matrix as the replacement node.

[0068] Step S175: Replace the relay node sequence in the object to be adjusted with the replacement node to generate the first transmission path.

[0069] Specifically, the system calculates the probability of transmission collision for the current service using a probability assessment model. It then reads a pre-established collision threshold from the threshold register and uses a numerical comparison algorithm to determine if the current probability exceeds the threshold, triggering a route optimization process. During this process, the system traverses the initial transmission path node sequence, using a centrality algorithm (such as betweenness centrality) to calculate the concurrent traffic reuse rate between adjacent nodes. It identifies a path segment with the highest overlap metric, classifying it as a network congestion bottleneck and marking it as an object to be adjusted. To find the optimal detour path, the system extracts the reachability matrix of neighboring nodes from the network topology feature database and uses the shortest path first algorithm to traverse and calculate the link delay weights within the matrix. It finds that the end-to-end delay via a certain backup node is significantly lower than other candidate nodes, thus locking it as the replacement node with the lowest delay. The system automatically issues flow table modification commands, removing the original relay nodes from the objects to be adjusted, inserting the selected replacement node into the corresponding position, reconstructing a new node sequence, and writing this updated sequence into the global routing table to generate the first transmission path for actual data forwarding.

[0070] In the above method, by calculating the overlap between adjacent nodes and selecting the set of consecutive nodes with the highest overlap, network congestion bottlenecks are accurately located, avoiding resource waste caused by global path reconstruction. By constructing a reachability matrix of neighboring nodes and selecting the alternative node with the lowest latency, the detour cost is minimized while ensuring path connectivity. This allows the adjusted path to both avoid congested nodes and maintain low transmission latency, achieving the best balance between path optimization effect and resource overhead.

[0071] Step S180: Calculate the estimated transmission time based on the upper limit of the bandwidth of each path segment in the first transmission path and the data fragment volume of the data to be transferred.

[0072] In one embodiment of this disclosure, step S180 may include: extracting the upper limit of bandwidth and the data fragment volume of each path segment in the first transmission path; calculating the quotient of the data fragment volume of each path segment divided by the corresponding upper limit of bandwidth to obtain the single-segment transmission time of each path segment; and summing the single-segment transmission time of each path segment to obtain the estimated transmission time.

[0073] Specifically, the system scans the currently selected first transmission path using a network topology awareness module, identifying that the path consists of multiple path segments. It then extracts the bandwidth limit and corresponding data fragment size for each path segment. Next, the system calls a time estimation algorithm to perform division operations, calculating the transmission time of each segment. Finally, an accumulator adds the transmission times of each segment to obtain the total estimated transmission time. The system then reads a preset time threshold from the database and uses a numerical comparator to determine whether the estimated transmission time is less than the threshold condition.

[0074] In the above method, the estimated transmission time is obtained by calculating the quotient of the data fragment volume of each path segment and the bandwidth limit, and then summing them up. This quantifies the path topology and bandwidth resource status into a unified time indicator, making the evaluation of path quality comparable. By comparing the estimated transmission time with a preset time threshold, a clear and quantifiable standard is provided for path confirmation, avoiding decision-making biases caused by subjective judgment.

[0075] Step S190: If the estimated transmission time is less than a preset time threshold, the first transmission path is determined as the second transmission path, and data migration is initiated according to the second transmission path. The time threshold is determined jointly based on the maximum allowable interruption duration of the data migration task and the business continuity requirements. Specifically, in a multi-AI task concurrent scenario, if the migration of state data for a single task exceeds this time threshold, the source node's memory resources will be occupied for a long time and cannot be released, thus affecting the normal scheduling and execution of subsequent queued tasks. The time threshold is determined based on: using the average data transmission bandwidth within the rack where the source node is located as a benchmark, combined with the expected maximum value of the total volume of data to be migrated, the theoretical shortest migration time is calculated and multiplied by an engineering safety factor to obtain the recommended range of values ​​for the time threshold. In actual deployment, the system administrator can dynamically adjust the threshold within the range of values ​​according to the actual bandwidth of the cluster network and task priorities. Higher priority tasks use a smaller threshold to ensure rapid resource release, while lower priority tasks can use a larger threshold to tolerate a longer migration window.

[0076] Specifically, when the estimated transmission time is less than a preset time threshold, the system triggers a configuration persistence instruction to write the routing table of the current first transmission path into memory for retention. Based on the data fragment size, it sends memory allocation requests to relay nodes in the path and records the pre-allocated buffer capacity to be locked by each relay node in the status table. After capacity locking is completed, the system updates the status identifier of the path to the second transmission path and registers it with the global routing state machine. The main control program then calls the underlying transmission interface and issues a data migration instruction containing the target node identifier and routing label to the source node according to the set fragment size to initiate the physical migration process of the underlying data stream.

[0077] In the above method, by immediately fixing the routing configuration and pre-allocating relay node buffer capacity after path confirmation, path failure or insufficient caching caused by resource contention after data migration starts is avoided. By updating and registering the path status identifier to the global routing state machine, traceable management and rapid switching of path status are achieved, reserving a flexible operation interface for possible subsequent dynamic adjustments.

[0078] Step S200: During the data migration process, monitor the network status parameters of each link in real time.

[0079] Step S210: If network congestion is determined to occur in the link based on the network status parameters, then a backup node that meets the conditions is selected from the network topology features, and the data flow is redistributed according to the dynamic load balancing weight to generate the target transmission path.

[0080] In one embodiment of this disclosure, such as Figure 7 As shown, determining that a link is experiencing network congestion may include the following sub-steps: Step S211: Real-time sampling of the instantaneous throughput decay rate and queue depth accumulation rate of each link.

[0081] Step S212: If the packet loss rate of the link is detected to exceed the preset mutation threshold, or the cumulative rate of the queue depth shows an upward trend for multiple consecutive sampling periods, then it is determined that the link is experiencing network congestion.

[0082] Step S213: Select standby nodes from the network topology features whose link quality assessment scores are higher than the current congested node and whose remaining storage capacity meets the data fragment volume requirements.

[0083] Step S214: Redistribute data flow according to dynamic load balancing weights to generate target transmission paths.

[0084] Specifically, the system segments the data file into data fragments according to the second transmission path and sends them sequentially to relay nodes along the route. Simultaneously, the system's built-in probe program calculates the instantaneous throughput decay rate of each link using a smoothing algorithm (such as an exponentially weighted moving average algorithm) at preset intervals, and reads data from the cached interfaces of each node to calculate the queue depth accumulation rate. At a certain relay node, the probe program, by comparing the sent and acknowledged sequence numbers, discovers that the packet loss rate suddenly jumps above a preset mutation threshold. At the same time, the queue depth accumulation rate of this node shows an upward trend over several consecutive sampling periods. Based on these indicators, the system uses an inference algorithm (such as a fuzzy logic inference algorithm) to determine that the link has experienced severe network congestion.

[0085] Upon triggering the congestion assessment command, the system immediately invokes the network-wide topology state matrix to extract candidate nodes within a one-hop radius of the currently congested node. A multi-dimensional attribute decision algorithm is used to calculate the link quality assessment score for each candidate node, selecting a backup node with a higher assessment score than the current congested node and sufficient remaining storage capacity to meet data fragment caching requirements. The system then adds this backup node to the routing table and assigns a weight coefficient to it using a dynamic load balancing algorithm based on the inverse of bandwidth and latency. Subsequent data fragments originally destined for the congested node are redirected to this backup node according to this weight ratio, thereby generating a target transmission path that bypasses the congested area.

[0086] In the above method, by sampling the instantaneous throughput decay rate and queue depth accumulation rate in real time, performance degradation signals can be captured in the early stages of network congestion, avoiding the need to trigger countermeasures only after congestion worsens. By using sudden changes in packet loss rate and continuous increases in queue depth as joint judgment conditions, frequent path switching caused by misjudgment of a single indicator is effectively reduced. By selecting backup nodes with link quality assessment scores higher than the current congested node and sufficient storage capacity from network topology characteristics, and reallocating data flow according to dynamic load balancing weights, the processing capacity of backup nodes is guaranteed, and traffic balancing among multiple paths is achieved, minimizing the impact of congestion on the overall progress of data migration.

[0087] Step S220: Continue to complete the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

[0088] Specifically, the system updates the routing table based on the target transmission path, updating the next-hop node number in the routing table to the corresponding backup node, and continues to send the remaining data fragments to the target node via the transmission control protocol. During data push, the source node's built-in timer periodically sends status query requests to the backup node to obtain the acknowledgment signal returned by the backup node and the integrity verification result of the local storage index generated based on a filter (such as a Bloom filter). Simultaneously, it analyzes whether the arrival delay of the acknowledgment signal exceeds a threshold to determine the current link congestion situation and dynamically adjusts the sending window. When the source node finds that all status data fragments have generated corresponding index key-value pairs in the backup node's distributed hash table, and the checksum of the backup node's data calculated using a hash algorithm (such as Secure Hash Algorithm 256) is completely consistent with the checksum initially generated by the source node, it determines that data consistency and integrity have met the standards. The system then writes a transfer completion status flag to the global state registry and calls the underlying memory reclamation function to forcibly clear the cache space allocated in the source node for this transmission, remounting the released physical memory pages to the operating system's available memory pool for other processes to schedule.

[0089] In the above method, by periodically querying the data reception confirmation signal of the backup node and the integrity verification result of the local storage index, data loss or corruption during transmission can be detected in a timely manner, and data consistency and integrity can be ensured through verification and comparison. By releasing the source node's memory resources only after confirming that all data shards have been migrated, the risk of data loss caused by premature release before migration is complete is avoided, ensuring the atomicity and reliability of task state transitions. The released memory resources are remounted to the available memory pool for scheduling by other processes, realizing rapid recovery and efficient reuse of computing resources.

[0090] In summary, the computing resource isolation and scheduling method and system for concurrent multi-AI tasks provided in this disclosure can accurately capture the complete context information at the moment of task interruption by collecting massive amounts of state data such as process stacks, network connections, and cache queue depths during node runtime through real-time monitoring of memory occupancy and triggering a state snapshot mechanism when the circuit breaker threshold is exceeded. By parsing the storage capacity usage ratio and data fragment volume distribution from the state data, and combining the physical hop count, link bandwidth, and transmission latency between backup nodes to construct network topology features, a graph convolutional layer is used to generate node embedding vectors and output the initial transmission path, achieving comprehensive perception of the cluster network physical topology and intelligent path planning. For concurrent traffic competition and link reuse conflicts at path intersections, the transmission collision probability is calculated and the relay node sequence is dynamically replaced, effectively avoiding cross-collisions of massive migration data streams on network paths. During data migration, the link throughput decay and queue depth changes are monitored in real time. When a sudden change in packet loss rate or a continuous increase in queue depth is detected, congestion determination is immediately triggered, and the data flow is redistributed according to dynamic load balancing weights, achieving rapid response and adaptive adjustment to sudden network congestion. This disclosure effectively reduces path conflict rate and data migration latency in distributed environments, and improves resource utilization efficiency and system stability during multi-task concurrency. This disclosure is applicable to the concurrent scheduling of large-scale AI tasks in big data processing scenarios.

[0091] This disclosure also provides a computing resource isolation and scheduling system for concurrent multi-AI tasks, such as... Figure 8 As shown, the system includes a monitoring module 101, a snapshot acquisition module 102, a feature extraction module 103, a path generation module 104, a collision detection module 105, a probability calculation module 106, a path adjustment module 107, a time estimation module 108, a migration initiation module 109, a status monitoring module 1010, a congestion handling module 1011, and a completion release module 1012. Among them: Monitoring module 101 is used to monitor the memory usage of computing nodes in a distributed system; The snapshot acquisition module 102 is used to trigger the state snapshot acquisition mechanism to obtain the runtime state data of the computing node and generate a state dataset if the memory occupancy rate exceeds a preset circuit breaker threshold. The feature extraction module 103 is used to determine the data volume characteristics of the data to be transferred and the network topology characteristics of the backup nodes in the distributed system based on the state dataset. The path generation module 104 is used to generate an initial transmission path from the source node to each backup node based on the data volume characteristics and the network topology characteristics. Collision detection module 105 is used to detect whether there is a path intersection in the initial transmission path, wherein the path intersection is the same relay node passed by at least two paths within the same time window; The probability calculation module 106 is used to calculate the transmission collision probability of the path intersection if the path intersection exists. The path adjustment module 107 is used to adjust the initial transmission path to generate a first transmission path if the transmission collision probability exceeds a preset collision threshold. The time estimation module 108 is used to calculate the estimated transmission time based on the bandwidth limit of each path segment in the first transmission path and the data fragment volume of the data to be transferred. The migration initiation module 109 is used to determine the first transmission path as the second transmission path and initiate data migration according to the second transmission path if the estimated transmission time is less than a preset time threshold. The status monitoring module 1010 is used to monitor the network status parameters of each link in real time during the data migration process; The congestion handling module 1011 is used to select backup nodes from the network topology features and reallocate data flow according to dynamic load balancing weights if network congestion is determined to occur in the link based on the network status parameters, so as to generate a target transmission path. The release module 1012 is used to continue the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

[0092] It should be noted that the embodiments of the computing resource isolation and scheduling system for concurrent multi-AI tasks provided in this application can be used to execute the processing flow of the embodiments of the computing resource isolation and scheduling method for concurrent multi-AI tasks in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the above method embodiments.

[0093] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A method for isolating and scheduling computing resources for concurrent multi-AI tasks, characterized in that, include: Monitor the memory usage of computing nodes in a distributed system; If the memory usage exceeds the preset circuit breaker threshold, a state snapshot acquisition mechanism is triggered to obtain the runtime state data of the computing node in order to generate a state dataset. Based on the state dataset, determine the data volume characteristics of the data to be transferred, and determine the network topology characteristics of the backup nodes in the distributed system; Based on the data volume characteristics and the network topology characteristics, an initial transmission path is generated from the source node to each backup node; Detect whether there is a path intersection in the initial transmission path, where the path intersection is the same relay node passed by at least two paths within the same time window; If the path intersection point exists, calculate the transmission collision probability of the path intersection point; If the transmission collision probability exceeds a preset collision threshold, the initial transmission path is adjusted to generate a first transmission path. The estimated transmission time is calculated based on the bandwidth limit of each path segment in the first transmission path and the data fragment size of the data to be transferred. If the estimated transmission time is less than a preset time threshold, then the first transmission path is determined to be the second transmission path, and data migration is initiated according to the second transmission path. During the data migration process, network status parameters of each link are monitored in real time; If network congestion is determined to occur in the link based on the network status parameters, then a backup node that meets the conditions is selected from the network topology features, and the data flow is reallocated according to the dynamic load balancing weight to generate the target transmission path. Continue the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

2. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The step of acquiring the runtime state data of the computing node to generate a state dataset includes: Retrieve process stack information, network connection status, and cache queue depth from the runtime environment of the computing node; Anomaly detection is performed on the captured data, and abnormal data is removed to obtain a standard snapshot set; The status dataset is generated based on the standard snapshot set, combined with node identifiers and resource usage details.

3. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The determination of the data volume characteristics of the data to be transferred includes: The storage capacity usage ratio and data fragment volume distribution of each computing node are analyzed from the state dataset. Based on the storage capacity usage ratio and the data fragment volume distribution, the number of data blocks to be transferred and the size of each block are counted to obtain the data volume characteristics.

4. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, Determining the network topology characteristics of the backup nodes in the distributed system includes: The storage capacity utilization rate and the data fragmentation amount are processed using the random forest algorithm to obtain the processed storage capacity utilization rate and the processed data fragmentation amount. Based on the processed storage capacity utilization rate and the processed data fragmentation amount, the number of data blocks to be transferred and the size of a single block are counted to obtain the data volume value. If the data volume value is greater than the preset volume threshold, then sub-data blocks and their sizes are extracted from the data block to obtain the sub-data volume value; Based on the sub-data volume value, the physical hop count between backup nodes, the link bandwidth limit, and historical transmission delay records are detected to determine the remaining storage capacity of the backup nodes. A clustering algorithm is used to cluster the physical hop count, the link bandwidth limit, and the historical transmission delay records to obtain the clustered physical hop count, the clustered bandwidth value, and the clustered delay value. The cost of cross-domain transmission is determined based on the physical hop count after clustering, the bandwidth value after clustering, the latency value after clustering, and the remaining storage capacity. If the cross-domain transmission cost is less than a preset cost threshold, the network topology characteristics are obtained based on the cross-domain transmission cost and the remaining storage capacity.

5. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 4, characterized in that, The generation of initial transmission paths from the source node to each backup node includes: Based on the data fragmentation amount and the cross-domain transmission cost, establish an adjacency matrix between nodes; Extract the link segments and bandwidth values ​​between each node. If the bandwidth value is less than a preset bandwidth threshold, mark the bandwidth sharing conflict status of the corresponding link in the adjacency matrix to obtain an adjacency matrix with conflict markings. A topology graph is generated based on the adjacency matrix with conflict annotations, and features are extracted from the topology graph using a graph convolutional network to obtain the node embedding vector of each node. Based on the node embedding vectors of each node, calculate the vector similarity between the source node and each backup node, and generate candidate paths based on the vector similarity. If the number of nodes in the candidate path is greater than a preset node threshold, then the excess nodes are removed to update the candidate path. Extract the connected path sequence from the updated candidate path, and obtain the initial transmission path based on the connected path sequence.

6. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The calculation of the transmission collision probability at the path intersection includes: Extract the time window of each path that passes through the intersection of the paths. If the time windows of at least two paths overlap and converge at the same relay node, then obtain the concurrent traffic value that converges at the relay node. Based on the concurrent traffic value, extract the concurrent traffic contention intensity and transmission delay fluctuation range of the path intersection point; The buffer overflow risk coefficient of the relay node is determined based on the historical cache records of the relay node; The product of the concurrent traffic contention intensity and the buffer overflow risk coefficient is calculated as the transmission collision probability.

7. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The step of adjusting the initial transmission path to generate a first transmission path includes: Traverse all nodes in the initial transmission path, calculate the overlap between adjacent nodes, and obtain the overlap sequence. Based on the overlap sequence, the set of consecutive nodes with the highest overlap is selected, and the path segments corresponding to the set of consecutive nodes are marked as objects to be adjusted. Based on the network topology features, a reachability matrix of neighboring nodes is constructed, and a node communication delay matrix within the reachability matrix is ​​obtained; Based on the node communication delay matrix, the node with the smallest delay in the reachability matrix is ​​selected as the replacement node; The relay node sequence in the object to be adjusted is replaced with the replacement node to generate the first transmission path.

8. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The step of calculating the estimated transmission time based on the bandwidth limit of each path segment in the first transmission path and the data fragment size of the data to be transferred includes: Extract the upper limit of bandwidth and data fragment size of each path segment in the first transmission path; Calculate the quotient of the data fragment volume of each path segment divided by the corresponding bandwidth limit to obtain the transmission time of a single segment of each path segment; The estimated transmission time is obtained by summing the transmission times of each path segment.

9. The method for isolating and scheduling computing resources for concurrent multi-AI tasks according to claim 1, characterized in that, The step of determining network congestion on a link based on the network status parameters includes: Real-time sampling of the instantaneous throughput decay rate and queue depth accumulation rate of each link; If the packet loss rate of a link exceeds a preset mutation threshold, or if the cumulative rate of the queue depth shows an upward trend for several consecutive sampling periods, then the link is determined to be congested.

10. A system for implementing the computing resource isolation and scheduling method for concurrent multi-AI tasks as described in any one of claims 1-9, characterized in that, include: The monitoring module is used to monitor the memory usage of computing nodes in a distributed system. The snapshot acquisition module is used to trigger the state snapshot acquisition mechanism if the memory usage rate exceeds the preset circuit breaker threshold, and to obtain the runtime state data of the computing node to generate a state dataset. The feature extraction module is used to determine the data volume characteristics of the data to be transferred and the network topology characteristics of the backup nodes in the distributed system based on the state dataset. The path generation module is used to generate an initial transmission path from the source node to each backup node based on the data volume characteristics and the network topology characteristics. The collision detection module is used to detect whether there is a path intersection in the initial transmission path. The path intersection is the same relay node passed by at least two paths within the same time window. The probability calculation module is used to calculate the transmission collision probability of the path intersection if the path intersection exists. The path adjustment module is used to adjust the initial transmission path to generate a first transmission path if the transmission collision probability exceeds a preset collision threshold. The time estimation module is used to calculate the estimated transmission time based on the bandwidth limit of each path segment in the first transmission path and the data fragment size of the data to be transferred. The migration initiation module is used to determine the first transmission path as the second transmission path and initiate data migration according to the second transmission path if the estimated transmission time is less than a preset time threshold. The status monitoring module is used to monitor the network status parameters of each link in real time during the data migration process; The congestion handling module is used to select backup nodes from the network topology features and reallocate data flow according to dynamic load balancing weights if network congestion is determined to occur in the link based on the network status parameters, so as to generate a target transmission path. The release module is used to continue the data migration according to the target transmission path, and release the memory resources occupied by the source node after confirming that the data migration is completed.

Images