Heterogeneous supercomputing parallel task allocation method and system based on stream job scheduling

Abstract

The application provides a heterogeneous supercomputer parallel task allocation method and system based on flow job scheduling, and relates to the technical field of heterogeneous supercomputer system parallel task scheduling. The method comprises the following steps: obtaining a parallel task flow to be scheduled, constructing a task dependency directed graph and collecting estimated resource consumption metadata of each task; using a rehearsal strategy to generate a plurality of feasible allocation schemes and storing them in a candidate scheme pool; simultaneously issuing each scheme to a plurality of independent computing nodes for parallel execution, monitoring the execution progress of each group and activating the scheme that first reaches the preset progress threshold as the main scheme; after the main scheme is activated, the computing resource allocation ratio of the main job channel task and the flanking accumulation and lag pool task is dynamically adjusted according to the connectivity index. The application can effectively shorten the overall completion time of the parallel task flow, improve the resource utilization rate of the heterogeneous computing nodes, and realize adaptive optimization of the scheduling strategy.

Description

Technical Field

[0001] This invention relates to the field of parallel task scheduling technology for heterogeneous supercomputing systems, and more specifically, to a method and system for allocating parallel tasks in heterogeneous supercomputing systems based on streaming job scheduling. Background Technology

[0002] As the scale of problems in fields such as scientific computing and engineering simulation continues to expand, heterogeneous supercomputing systems have become the mainstream computing platform supporting large-scale parallel computing tasks. In these systems, different types of computing nodes exhibit significant differences in computational power and memory access characteristics, while parallel task flows often have complex data dependencies. Existing task scheduling methods mostly employ static resource allocation strategies, determining the resource allocation scheme upon task submission and maintaining it until task completion. This makes it difficult to dynamically adjust based on actual resource consumption and node load changes during task execution. Furthermore, existing methods lack effective feedback correction mechanisms when faced with discrepancies between estimated and actual resource consumption, making it difficult for scheduling strategies to continuously adapt to the actual operating environment. These factors collectively constrain the resource utilization efficiency and job completion performance of heterogeneous supercomputing systems. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for heterogeneous supercomputing parallel task allocation based on streaming job scheduling, the method comprising: Obtain the parallel task flow to be scheduled, parse the data dependency relationship between each task in the parallel task flow, construct the task dependency directed graph, and collect the estimated resource consumption metadata of each task. Based on the task dependency directed graph and the estimated resource consumption metadata, N feasible allocation schemes are generated using a pre-simulation strategy, where N≥2. Each feasible allocation scheme includes the mapping relationship between each task and heterogeneous computing nodes, as well as the resource allocation strategy of each computing node. All feasible allocation schemes are stored in the alternative scheme pool. The N feasible allocation schemes in the alternative scheme pool are simultaneously distributed to N independent computing nodes for parallel execution. The execution progress of each computing node is monitored, and the scheme that first reaches the preset progress threshold on the critical path is activated as the main scheme, while the remaining schemes are terminated or downgraded to flanking buffer tasks. After the main scheme is activated, each group of computing nodes dynamically adjusts the computing resource allocation ratio between the main operation channel tasks and the side storage tank tasks according to the current connectivity index. The connectivity index is dynamically adjusted based on the real-time load of the computing nodes, the backlog of main channel tasks, and the water level of the side storage tanks. During task execution, the actual resource consumption data of each task is collected in real time. The actual resource consumption data is compared with the estimated resource consumption metadata for each task to generate an individual deviation signature for each task on its assigned computing node type. The individual deviation signature is injected back into the flank stagnation tank level regulator of the corresponding computing node to update the stagnation ratio control parameters of the computing node; when the actual execution delay of the task propagates to the subsequent task along the edge of the task-dependent directed graph, the deviation propagation coefficient is calculated and the weight parameters of the corresponding path in the pre-play strategy are corrected. Periodically perform cluster analysis on all accumulated individual deviation signatures, identify deviation patterns and mark unreliable prediction combinations, and remove the computing node types corresponding to the unreliable prediction combinations from the candidate resource pool of subsequent pre-playing strategies. The pre-simulation strategy is updated with the corrected path weight parameters and the eliminated candidate resource pool for scheduling decisions in the next round of parallel task flow.

[0004] In addition, the present invention also provides a heterogeneous supercomputing parallel task allocation system based on streaming job scheduling, which is used to store and execute the above-mentioned methods.

[0005] Based on the above, the heterogeneous supercomputing parallel task allocation method based on streaming job scheduling provided in this application generates multiple feasible allocation schemes with resource allocation conservatism gradients through a pre-simulation strategy and executes them in parallel. This transforms the selection process from static prediction to dynamic selection based on actual execution progress, effectively overcoming the problem of performance degradation caused by prediction deviation when a single static allocation scheme faces uncertainty in resource demand.

[0006] Secondly, the individual deviation signature back-injection mechanism enables each computing node to adjust the lag ratio control parameters in real time based on the deviation between the actual resource consumption and the estimated value of the task, thus realizing adaptive optimization of the node-level resource allocation strategy. Meanwhile, periodic cluster analysis identifies unreliable prediction combinations from a global perspective, corrects the estimated data and candidate resource pool in the pre-simulation strategy, and makes the scheduling strategy continuously approach the optimal adaptation state of the actual computing environment during continuous operation, significantly improving the prediction accuracy of the pre-simulation strategy and the reliability of scheduling decisions.

[0007] Furthermore, the connectivity index-driven dynamic resource allocation mechanism enables each computing node to autonomously adjust the ratio of computing resources between the main operation channel and the flank tasks based on changes in main channel task backlog, flank reservoir level, and real-time load. This ensures the efficiency of critical path task execution while maximizing the use of idle computing resources. The early warning path replacement mechanism identifies paths with delay risks in advance through real-time calculation of the deviation propagation coefficient and selects complementary segments from the flank reservoir for replacement, effectively curbing the spread of execution delays of individual tasks along task dependencies to the entire task flow.

[0008] The complete closed-loop feedback system constructed in this application enables the scheduling strategy of heterogeneous supercomputing systems to have the ability to self-sensing, self-adjusting and self-optimizing. In scientific computing scenarios such as nuclear reactor design calculations, which have large-scale parallel computing requirements and complex dependencies between tasks, it can effectively shorten the overall completion time of parallel task flows, improve the resource utilization of heterogeneous computing nodes, and reduce the risk of scheduling mismatch caused by prediction deviations. Attached Figure Description

[0009] Fig. 1 This is a flowchart illustrating the method provided by the present invention; Fig. 2 This is a schematic diagram comparing the activation time of the present invention with that of the prior art. Fig. 3 This is a schematic diagram illustrating the variation of the resource consumption prediction deviation rate between the present invention and existing technologies with the number of scheduling rounds. Detailed Implementation

[0010] like Figs. 1-3 As shown in the figure, this technical solution provides a method and system for parallel task allocation in heterogeneous supercomputing based on streaming job scheduling. The technical details are explained in detail below with reference to a specific application scenario.

[0011] This embodiment uses neutron transport calculation under multiple schemes in a nuclear reactor core as an application scenario. During the nuclear reactor design process, designers need to perform neutron transport equation solving calculations for different fuel arrangement schemes, control rod insertion schemes, and combustible poison configuration schemes to obtain key parameters such as the effective multiplication factor, power distribution, and burnup characteristics under each scheme. Each calculation scheme includes multiple computational tasks with data dependencies. For example, the front-end mesh generation task depends on the output of the geometric modeling task, the neutron transport solving task depends on the output of the mesh generation task, the burnup calculation task depends on the output of the neutron transport solving task, and the subsequent power distribution post-processing task depends on the outputs of both burnup calculation and neutron transport. These computational tasks need to be executed on a heterogeneous supercomputing system, which includes various types of computing resources such as CPU computing nodes, GPU computing nodes, and domestically produced accelerator card computing nodes. Different types of computing nodes have significant differences in floating-point operation capabilities, memory bandwidth, and memory access latency.

[0012] Step 1: Obtain the parallel task flow and construct the task dependency directed graph.

[0013] When designers submit a batch of core neutron transport calculation schemes to the client, the central scheduler receives these parallel task streams to be scheduled. Each calculation scheme corresponds to an independent parallel task stream, and each task stream contains multiple computational tasks and the data dependencies between these tasks. The scheduler first resolves the dependencies between tasks in each task stream—for example, whether a task depends on the output file of another task as input—and constructs a task dependency directed graph for each task stream based on these dependencies. Each node in this directed graph represents a computational task, and each directed edge represents a data dependency, with the edge pointing from the dependent task to the dependent task. Simultaneously, the scheduler also collects estimated resource consumption metadata for each task, including estimated CPU computation, GPU computation, memory usage, video memory usage, storage read / write operations, and estimated execution time. This estimated data can be obtained based on historical statistics or empirical formulas; for example, for the neutron transport solution task, its estimated computation can be estimated based on the grid size, energy group number, and scattering order.

[0014] Step 2: Generate feasible allocation schemes and store them in the alternative scheme pool. The scheduler generates multiple feasible allocation schemes based on the task dependency directed graph and estimated resource consumption metadata, using a pre-analysis strategy. The specific execution method of the pre-analysis strategy is as follows.

[0015] The scheduler extracts all directed paths from the start node to the end node from the task dependency directed graph. For each directed path, the estimated resource consumption metadata of each task on that path is accumulated according to a preset weight coefficient to obtain the path weight. The path weight reflects the total consumption of that path in a certain resource dimension. A path with a larger weight means that more total resources are required to complete all tasks on that path, and it is more likely to become a bottleneck restricting the completion time of the entire task flow. After sorting all directed paths in descending order of path weight, the directed path with the largest path weight is selected and marked as the critical path, and each task on that critical path is marked as a critical task.

[0016] Then, for each critical task on the critical path, heterogeneous computing nodes that meet its estimated resource consumption metadata are assigned as candidate nodes. For example, for computationally intensive neutron transport solving tasks, GPU nodes or domestically produced accelerator card nodes are preferred; for memory-intensive mesh generation tasks, large-memory CPU nodes are preferred. This generates a sequence of candidate nodes for the critical path. Using this candidate node sequence as a constraint skeleton, non-critical tasks are allocated according to their dependencies and estimated resource consumption metadata—that is, under the premise of satisfying dependency constraints, appropriate types of computing nodes are allocated to each non-critical task according to its resource requirements, ultimately generating a complete task-to-node mapping scheme. This mapping scheme includes information on which specific computing node each task is assigned to, as well as the allocation strategy for various resources on each computing node, such as the number of CPU cores, the number of GPUs, and memory quotas.

[0017] To generate multiple feasible allocation schemes with different resource allocation strategies, the scheduler makes differentiated adjustments during critical path marking and filling allocation operations. Specifically, a lag ratio control parameter is set for each critical task on the critical path. This parameter represents the upper limit of the ratio of the computational workload of flanking lag pool tasks to the computational workload of main channel tasks on the computing node where the critical task is located during its execution. In the first generation of feasible allocation schemes, the lag ratio control parameter for each critical task is set to 0.2, allowing the computing node where each critical task is located to reserve resources for critical path tasks while allowing some tasks in the flanking lag pool to occupy no more than 20% of the node's computing resources. In the second generation of feasible allocation schemes, the lag ratio control parameter is set to 0.4, allowing flanking lag pool tasks to occupy a larger proportion of computing resources, thus forming a scheme with lower resource allocation conservatism. Subsequently, with each generation operation, the lag ratio control parameter is increased in increments of 0.2 until N feasible allocation schemes with different lag ratio control parameters are generated. In this embodiment, N is set to 5. The five feasible allocation schemes generated are sorted in ascending order of the lag ratio control parameter and stored in the alternative scheme pool. The scheme with the smallest lag ratio control parameter has the highest resource allocation conservatism, that is, it is most inclined to ensure the resource supply for the critical path task; the scheme with the largest lag ratio control parameter has the lowest resource allocation conservatism, that is, it is most inclined to allow flanking tasks to occupy computing resources.

[0018] Step 3: Execute each scheme in parallel and activate the main scheme. The scheduler simultaneously distributes five feasible allocation schemes from the alternative scheme pool to five independent computing nodes for parallel execution. Resources are isolated between the independent computing nodes in each group, and computing nodes within the same group share the group's resource quota. For example, the first group of computing nodes executes the most conservative scheme with a latency ratio control parameter of 0.2, the second group executes the scheme with a latency ratio control parameter of 0.4, and so on. Each group of computing nodes synchronously starts task execution according to the task-to-node mapping relationship and resource allocation strategy in its respective allocation scheme.

[0019] During task execution, each independent computing node in a group maintains a local progress counter, which records the number of critical tasks on the critical path that the group has completed. Each computing node reports its local progress counter value to the central scheduler at fixed time intervals, such as every 10 seconds. After receiving the reported count values ​​from each group, the central scheduler compares each group's count value with the total number of critical tasks on the critical path in the corresponding scheme for that group, calculating the completion percentage. When the completion percentage of any group first reaches a preset progress threshold (set to 60% in this embodiment), the central scheduler marks the group's currently executing feasible allocation scheme as the winning scheme and activates it as the primary scheme. Subsequently, the central scheduler sends termination or degradation instructions to the remaining groups. The termination instruction requires the remaining groups to immediately stop executing their current tasks and release all occupied computing resources; the degradation instruction requires the remaining groups to convert their currently executing tasks to flanking slack pool tasks, continuing to execute these tasks using idle computing resources, but no longer participating in resource contention for primary channel tasks. The specific groups to which termination instructions and degrade instructions are sent can be dynamically determined based on the current actual execution progress and resource usage of each group—groups that are close to the winning solution and have completed a significant amount of work will receive degrade instructions first, while groups that are significantly behind will receive termination instructions.

[0020] Step 4: Dynamically adjust the resource allocation ratio between the main channel and the flank retention ponds. After the main scheme is activated, each group of computing nodes dynamically adjusts the computing resource allocation ratio between the main job channel tasks and the flanking stagnation pool tasks according to the current connectivity index. Each computing node independently maintains its local flanking stagnation pool level, which is equal to the ratio of the estimated total computational load of all flanking stagnation pool tasks currently queued for execution in that computing node's local flanking stagnation pool to the preset level capacity. The preset level capacity is equal to the maximum computational load that the computing node's total computing power can handle per unit time. For example, for a CPU node, if it can perform 10 billion floating-point operations per second, then the preset level capacity is the computational load corresponding to 10 billion floating-point operations per second.

[0021] Each computing node collects its local real-time load rate and main channel task backlog at a fixed monitoring period, such as every 30 seconds. The real-time load rate is equal to the ratio of the actual computational load performed by the computing node per unit time to the preset water level capacity. The main channel task backlog is equal to the estimated total computational load occupied by the main channel tasks currently assigned to the computing node but not yet completed, including the remaining computational load of tasks currently being executed and the estimated computational load of tasks not yet started.

[0022] Each computing node takes the flank reservoir water level, real-time load rate, and main channel task backlog as input parameters and inputs them into a pre-set connectivity index calculation function. The calculation logic of this function is as follows: First, the main channel task backlog is normalized by dividing the current main channel task backlog by the preset water level capacity to obtain a normalized value between 0 and 1. Then, this normalized value is divided by the weighted sum of the flank reservoir water level and the real-time load rate. The weighting coefficients can be preset according to the actual system operation; for example, the weight of the flank reservoir water level is set to 0.6, and the weight of the real-time load rate is set to 0.4. The value output by this calculation function is the connectivity index value at the current moment.

[0023] Each computing node compares its connectivity index value with preset high and low connectivity thresholds. The high connectivity threshold can be set to 0.8, and the low connectivity threshold can be set to 0.3. When the connectivity index value is higher than 0.8, the computing node is determined to be in a main channel priority state, and the computing resource allocation ratio for the next monitoring cycle is set to 7:3, that is, 70% of the computing resources are allocated to the main operation channel tasks, and 30% to the flanking stagnation pool tasks. When the connectivity index value is lower than 0.3, the computing node is determined to be in a flanking priority state, and the computing resource allocation ratio for the next monitoring cycle is set to 3:7, that is, 30% of the computing resources are allocated to the main operation channel tasks, and 70% to the flanking stagnation pool tasks. When the connectivity index value is between 0.3 and 0.8, the computing node is determined to be in a balanced state, and the current computing resource allocation ratio remains unchanged. After each computing node completes the ratio adjustment for the current monitoring cycle, it waits for the next monitoring cycle to arrive, re-collects input parameters, and repeats the above comparison and ratio setting operations.

[0024] Step 5: Generate individual deviation signatures During task execution, each computing node collects real-time data on the actual resource consumption of each task, including actual CPU computation, actual GPU computation, actual memory usage, and actual execution time. After each task completes, the actual resource consumption data is compared item by item with the estimated resource consumption metadata, and the deviation value for each item is calculated. The deviation value is calculated by subtracting the estimated value from the actual value and then dividing by the estimated value, resulting in a percentage-based relative deviation. The task type, the assigned computing node type, and the calculated deviation value are combined to form the task's individual deviation signature. For example, if a neutron transport solving task is assigned to a GPU node and its actual execution time is 15% longer than the estimated execution time, the generated individual deviation signature will include three elements: task type is neutron transport solving, node type is GPU, and the deviation value is positive 15%.

[0025] Step Six: Re-inject deviation signature and correct pre-simulation strategy parameters The generated individual deviation signatures are injected back into the flanking stagnation tank level regulator of the corresponding computing node to update the stagnation ratio control parameters of that computing node. Each computing node's flanking stagnation tank level regulator has a built-in stagnation ratio control parameter register that stores the current stagnation ratio control parameter value of that computing node. When any task on that computing node completes execution and generates a corresponding individual deviation signature, the deviation value in the signature is extracted. This deviation value is compared with a preset deviation dead zone threshold, which is set to 5% in this embodiment. When the absolute value of the deviation value is less than or equal to 5%, the deviation between the estimated and actual values ​​is considered to be within an acceptable range, and the current stagnation ratio control parameter value of that computing node remains unchanged. When the deviation value is positive and the absolute value is greater than 5%, it indicates that the computing node's current resource consumption estimate for this type of task is insufficient, and the stagnation ratio control parameter value is adjusted in an increasing direction with an adjustment step size of 0.05, allowing flanking stagnation tank tasks to occupy a larger proportion of computing resources. When the deviation is negative and its absolute value is greater than 5%, it indicates that the computing node is currently overestimating the resource consumption of this type of task. The lag ratio control parameter is adjusted in a decreasing direction with an adjustment step size of 0.03, thus limiting the proportion of computing resources occupied by flanking lag pool tasks. The adjustment step size in the increasing direction is larger than the adjustment step size in the decreasing direction. This makes the response sensitivity of the lag ratio control parameter higher when resource consumption is underestimated than when it is overestimated. This is done to prioritize resource supply for critical path tasks—when underestimating, the allowed resource proportion for flanking tasks is rapidly increased to alleviate pressure on the main channel; when overestimating, resources are recovered more slowly to avoid over-conservatism. The adjusted lag ratio control parameter value is written to a register and used to control the injection rate of flanking lag pool tasks when the computing node subsequently executes main channel tasks.

[0026] Meanwhile, when the actual execution delay of a critical task propagates along the edges of the task-dependent directed graph to subsequent tasks, the deviation propagation coefficient is calculated and the weight parameters of the corresponding path in the rehearsal strategy are adjusted. Specifically, when the actual end time of any critical task is later than its estimated end time, the actual execution delay of the task is calculated, which is the difference between the actual end time and the estimated end time. In the task-dependent directed graph, starting with the critical task, all its direct and indirect successor tasks are traversed along all outgoing edges. For each traversed successor task, its weighted deviation propagation coefficient is calculated. This coefficient is equal to the actual execution delay multiplied by the topological level weight of the successor task on the critical path. The topological level weight is negatively correlated with the number of edges between the successor task and the starting node. For example, the weight of a successor task with a distance of 1 hop is 1, the weight of a successor task with a distance of 2 hops is 0.5, the weight of a successor task with a distance of 3 hops is 0.33, and so on. The weighted bias propagation coefficient is superimposed on the path weight parameter corresponding to the subsequent task in the pre-simulation strategy, thereby increasing the path weight of the path where the subsequent task is located, and thus increasing the probability that the path will be marked as the critical path in the next round of pre-simulation.

[0027] Step 7: Cluster analysis and removal of unreliable prediction combinations The scheduler periodically performs cluster analysis on all accumulated individual deviation signatures. Specifically, it extracts the task type feature, the node type feature of the assigned computing node, and the deviation value from each individual deviation signature, combining these three elements to form a three-dimensional deviation feature vector. Using a 24-hour cycle, the three-dimensional deviation feature vectors corresponding to all accumulated individual deviation signatures within the current cycle are input into a clustering algorithm for unsupervised clustering. In this embodiment, K-means clustering is used, with a preset number of clusters of 5 to 10, outputting multiple deviation clusters. The cluster center deviation value and intra-cluster sample density of each deviation cluster are calculated. When the intra-cluster sample density of a deviation cluster is higher than a preset density threshold and the absolute value of the cluster center deviation value is greater than a preset deviation tolerance threshold (e.g., the density threshold is set to at least 50 samples per cluster, and the deviation tolerance threshold is set to 20%), this deviation cluster is marked as a high-deviation cluster.

[0028] The combination of task type features and node type features common to all samples in the high-bias cluster is extracted and used as an unreliable prediction combination. For example, if all samples in a cluster are fuel consumption calculation tasks executed on domestic accelerator card nodes and the deviation value is greater than 20%, then "fuel consumption calculation task + domestic accelerator card node" is marked as an unreliable prediction combination. This unreliable prediction combination is removed from the candidate resource pool of subsequent pre-simulation strategies, meaning that subsequent pre-simulation strategies will no longer consider domestic accelerator card nodes as candidate nodes for fuel consumption calculation tasks. Simultaneously, a correction operation is triggered on the estimated resource consumption metadata of the corresponding fuel consumption calculation task in the pre-simulation strategy, replacing the original estimated resource consumption metadata with the average value of the actual resource consumption data of each sample in the high-bias cluster.

[0029] Step 8: Update the rehearsal strategy The pre-simulation strategy is updated using the revised path weight parameters and the candidate resource pool after removing unreliable predictions, and used for scheduling decisions in the next round of parallel task flows. When receiving new parallel task flows in the future, the updated pre-simulation strategy will use the revised path weights to identify critical paths and the updated candidate resource pool to allocate appropriate computing nodes to each task, thereby gradually improving the adaptability of the pre-simulation strategy to the actual computing environment.

[0030] In alternative implementations, the time period for the clustering analysis described above can be adjusted based on the actual task arrival frequency. For scenarios with high task arrival frequency, the period can be shortened to 12 hours or even 6 hours to more quickly capture changes in deviation patterns; for scenarios with low task arrival frequency, the period can be extended to 48 hours or 72 hours to ensure sufficient sample size is accumulated for clustering analysis within each period. Parameters such as the deviation dead zone threshold, deviation tolerance threshold, and density threshold can also be adjusted accordingly based on the accuracy requirements and resource availability of the specific application scenario.

[0031] Based on the above implementation scheme for neutron transport calculation scenario with multiple schemes in nuclear reactor core, the following further explains the generation strategy of feasible allocation schemes, the differentiated adjustment method, and the parallel execution of multiple schemes and the creation of the main scheme activation mechanism.

[0032] Regarding the specific generation method of feasible allocation schemes in the rehearsal strategy, the central scheduler has a pre-built path enumeration and weight calculation module. After constructing the task dependency directed graph and collecting the estimated resource consumption metadata of each task, this module starts from the starting node of the directed graph, traverses along the directed edges to the ending node, and extracts all directed paths. For each extracted directed path, the estimated resource consumption metadata of all tasks on the path is accumulated according to a unified dimension, and the accumulated result is the path weight of the path. Taking the neutron transport calculation scenario as an example, if a path successively includes a grid generation task, a neutron transport solution task, and a burnup calculation task, with estimated computational loads of 80 units, 500 units, and 200 units respectively, then the path weight of this path is 780. After completing the weight calculation of all paths, the scheduler sorts all directed paths according to their path weights from largest to smallest, and marks the directed path with the largest path weight in the sorting result as the critical path. Each task on this critical path is marked as a critical task.

[0033] Once the critical path is determined, the scheduler allocates suitable heterogeneous computing nodes to each critical task on the critical path. The allocation is based on the matching degree between the estimated resource consumption metadata of each critical task and the resource capabilities of each computing node. For example, for neutron transport solving tasks with high computational density and suitable for parallel acceleration, computing nodes equipped with multiple graphics processing units or domestically produced accelerator cards are preferred; for mesh generation tasks with frequent memory access but relatively simple computational logic, computing nodes with large memory capacity are preferred. After selecting computing nodes for each critical task on the critical path according to the above allocation principles, the sequence of these nodes constitutes the candidate node sequence for the critical path. The scheduler uses this candidate node sequence as the constraint skeleton of the overall mapping scheme to perform fill-in allocation for non-critical tasks outside the critical path.

[0034] The specific method of filling allocation is as follows: determine the execution order of non-critical tasks based on their dependencies in the task dependency directed graph. For each non-critical task to be allocated, select a node that can meet its estimated resource consumption requirements from the node where its direct predecessor task is located or from available nodes of the same type, until all non-critical tasks are allocated to specific computing nodes. At this point, a complete task-to-node mapping scheme is generated.

[0035] To obtain multiple optimal mapping schemes, the scheduler repeatedly performs the critical path marking and padding allocation operations described above. Each complete round of operations generates a feasible allocation scheme, each scheme containing the mapping relationship between all tasks and compute nodes, as well as the resource allocation strategy on each compute node. These schemes are sequentially stored in the candidate scheme pool.

[0036] Regarding the specific implementation of the gradient difference in resource allocation conservatism among feasible allocation schemes, the scheduler makes differentiated adjustments to the node selection in the candidate node sequence of the critical path during the repeated generation of schemes. The core mechanism of the differentiated adjustment lies in setting a lag ratio control parameter for each critical task on the critical path. This parameter, as an upper limit of the ratio, is used to control the proportional relationship between the computational workload of the flanking lag pool tasks and the computational workload of the main channel tasks on the computing node where the critical task is located during execution. When generating the first feasible allocation scheme, the scheduler uniformly sets the lag ratio control parameter of each critical task to a first preset value, which is greater than zero and less than 1; in this embodiment, it is set to 0.2. This means that in this scheme, while ensuring that the critical path tasks obtain no less than 80% of the computing resources, the computing nodes where each critical task is located allow tasks in the flanking lag pool to use no more than 20% of the node's computing resources, reflecting a high degree of resource allocation conservatism.

[0037] When generating the second feasible allocation scheme, the scheduler uniformly sets the lag ratio control parameter for each critical task to a second preset value, which is greater than the first preset value; in this embodiment, it is set to 0.4. At this time, the computing nodes where each critical task resides allow flanking lag pool tasks to occupy no more than 40% of the computing resources, while the proportion of computing resources available for the main channel tasks decreases to no less than 60%. This scheme is less conservative in resource allocation than the first scheme. Subsequently, each time the critical path marking and filling allocation operation is performed, the scheduler increments the lag ratio control parameter by a preset step size; in this embodiment, the step size is set to 0.2.

[0038] If the preset total number of feasible allocation schemes N is 5, then the lag ratio control parameters corresponding to the 5 schemes generated sequentially are 0.2, 0.4, 0.6, 0.8, and 1.0, respectively. When the parameter value is 1.0, it means that the proportion of computing resources occupied by the flanking lag pool tasks can reach the same as that of the main channel tasks. After generating all N schemes, the scheduler sorts these schemes according to their corresponding lag ratio control parameter values ​​from smallest to largest. The scheme with the smallest lag ratio control parameter value has the highest resource allocation conservatism, and the scheme with the largest value has the lowest resource allocation conservatism. The sorted schemes are stored in the candidate scheme pool in sequence. There is a correlation between the initial value of the lag ratio control parameter, the step size, and the total number of schemes N: the value of N determines the number of schemes; the larger N is, the wider the coverage of candidate schemes, but the more computing node groups are occupied during parallel execution; the size of the step size determines the density of the conservatism interval between adjacent schemes; the smaller the step size, the more subtle the difference in conservatism between schemes, and the larger the step size, the more significant the difference between schemes. These three parameters can be set comprehensively based on the total number of available computing nodes and the desired scheduling granularity.

[0039] Regarding the specific mechanism for simultaneously issuing and selecting the best of multiple alternative solutions from the pool, after the pool is constructed, the central scheduler issues the N feasible allocation solutions to N independent computing nodes. Resources are isolated between the independent computing nodes in each group; that is, computing resources occupied by nodes in one group will not be preempted or shared by other groups, and computing nodes within the same group share the group's resource quota. Upon receiving the allocation solution, each computing node in each group synchronously starts task execution according to the task-to-node mapping relationship and resource allocation strategy defined in the solution. Each group progresses independently on its own critical path, without affecting others.

[0040] Each group of independent computing nodes maintains a local progress counter during execution. This counter counts the number of critical tasks completed on the critical path corresponding to the group's solution. Each computing node reports its local progress counter value to the central scheduler at fixed time intervals, such as every 5 seconds. The central scheduler receives the reported data from all N groups of nodes in real time. For each group, it compares the reported count value with the total number of critical tasks on the critical path in that group's solution to calculate the completion percentage. This percentage reflects the execution progress of that group's solution on its own critical path. When the completion percentage of any group first reaches a preset progress threshold (set to 60% in this embodiment), the central scheduler immediately marks the group's currently executing feasible allocation scheme as the winning scheme and activates it as the primary scheme for formal execution.

[0041] After the main program is activated, the central scheduler immediately sends termination or degradation instructions to the remaining independent computing nodes. The termination instruction instructs the corresponding computing node group to immediately stop all currently executing tasks and release all computing resources occupied by the group, including CPU cores, graphics processing unit memory, memory space, and storage resources. The degradation instruction, however, does not release resources or stop tasks upon receiving it. Instead, it marks all currently executing tasks and those yet to be executed as flanking lag tasks. These downgraded tasks continue to utilize the idle computing resources of their respective computing nodes but no longer participate in resource contention for the main job channel. That is, the main job channel tasks have absolute priority in resource allocation, and flanking lag tasks can only use the remaining computing resources not occupied by the main job channel tasks.

[0042] In practice, the central scheduler determines which instruction to send based on the current progress of each group: for groups whose progress is significantly behind the winning solution and the gap exceeds a preset threshold, a termination instruction is sent to quickly reclaim resources; for groups whose progress is close to the winning solution and the gap is within the preset threshold, a degradation instruction is sent to preserve their completed work for possible future use. After the termination or degradation operation is completed, the main solution exclusively uses all activated computing resources to continue execution until the entire parallel task flow is completed.

[0043] Based on the above implementation scheme for neutron transport calculation in a multi-scheme nuclear reactor core, the following further explains the cluster analysis of individual deviation signatures, the calculation and application of deviation propagation coefficients, and the creation of a path replacement mechanism during the execution of the main scheme.

[0044] Regarding the periodic clustering analysis of individual bias signatures and the identification of unreliable predictive combinations, the central scheduler is equipped with a bias signature accumulation and storage module and a clustering analysis engine. This storage module collects individual bias signatures from each computing node according to a fixed time window. Each signature contains three elements: task type characteristics, the node type characteristics of the assigned computing node, and the bias value. Taking a nuclear reactor computing scenario as an example, a complete individual bias signature can be represented as a neutron transport solution task, a graphics processing unit node, with a bias value of +18%, or a burnup calculation task, a domestic accelerator card node, with a bias value of -12%.

[0045] The clustering analysis engine performs analysis operations at preset time intervals; in this embodiment, the interval is set to 24 hours. At the beginning of each period, the engine reads all accumulated individual deviation signatures from the cumulative storage module and maps the task type feature, node type feature, and deviation value in each signature into a feature vector in three-dimensional space. After completing the transformation of all vectors, the engine uses the K-means clustering algorithm to perform unsupervised clustering on these three-dimensional feature vectors, with the number of clusters preset to 8. After the algorithm iterates and converges, it outputs multiple deviation clusters, each containing several deviation signatures with similar feature combinations.

[0046] After clustering is completed, the clustering analysis engine calculates two statistics for each biased cluster: cluster centroid deviation and in-cluster sample density. Cluster centroid deviation is the arithmetic mean of the deviation values ​​of all samples within the cluster; in-cluster sample density is the number of individual biased signatures contained in the cluster. When the in-cluster sample density of a biased cluster is higher than a preset density threshold, and the absolute value of the cluster centroid deviation is greater than a preset bias tolerance threshold, the engine marks the cluster as a high-bias cluster. In this embodiment, the density threshold is set to 50 samples, and the bias tolerance threshold is set to 20%. For example, if a cluster contains 120 individual biased signatures, and the deviation values ​​of these signatures are all between +25% and +35%, then the cluster simultaneously meets the two conditions of a density higher than 50 and an absolute value of centroid deviation greater than 20%, and is marked as a high-bias cluster.

[0047] The engine then extracts a combination of task type features and node type features common to all samples in the high-bias cluster and marks this combination as an unreliable prediction combination. For example, if all 120 samples in the high-bias cluster correspond to a scenario where fuel consumption calculation tasks are executed on domestic accelerator card nodes, then the combination of fuel consumption calculation tasks and domestic accelerator card nodes is marked as an unreliable prediction combination.

[0048] After marking, the scheduler removes the unreliable predicted combination from the candidate resource pool of subsequent pre-planning strategies. This means that domestically produced accelerator card nodes will no longer be considered as candidate allocation targets for fuel consumption calculation tasks when generating feasible allocation schemes later. Simultaneously, the scheduler triggers a correction operation on the estimated resource consumption metadata for the corresponding fuel consumption calculation task in the pre-planning strategy. The correction method involves calculating the average of the actual resource consumption data of all samples in the high-bias cluster and replacing the original estimated resource consumption metadata with this average. After the above removal and correction operations are completed, the updated candidate resource pool and estimated data will take effect in the next round of scheduling decisions.

[0049] Regarding the mechanism for calculating the deviation propagation coefficient and correcting path weights when the actual execution delay propagates along the task-dependent directed graph, this mechanism is executed by the deviation propagation tracing module deployed in the central scheduler. During task execution, the deviation propagation tracing module is triggered when the actual end time of any critical task on the critical path is later than its estimated end time. This module first calculates the actual execution delay of the critical task by subtracting the estimated end time from the actual end time; the difference is the delay in seconds.

[0050] After calculating the latency, the tracking module traverses all direct and indirect successor tasks in the task dependency directed graph, starting with the critical task as the initial node and following all outgoing edges. The traversal can use either breadth-first or depth-first graph traversal. For each traversed successor task, the tracking module calculates a weighted deviation propagation coefficient. This coefficient is calculated by multiplying the actual execution latency by the topological level weight of the successor task. The topological level weight is negatively correlated with the number of directed edges between the successor task and the initial node.

[0051] The specific scheme adopted in this embodiment is as follows: the successor task that is 1 edge away from the starting node has a topological layer weight of 1; the successor task that is 2 edges away has a weight of 0.5; the successor task that is 3 edges away has a weight of 0.33; the successor task that is 4 edges away has a weight of 0.25, and so on. The weighted deviation propagation coefficient of each successor task is superimposed on the path weight parameter corresponding to that successor task in the pre-simulation strategy, thereby increasing the path weight of the path containing that successor task. After the path weight is increased, when the path is re-evaluated in the next pre-simulation stage, the probability of it being marked as a critical path will increase accordingly, thus enabling the scheduler to allocate more resources to the tasks on that path in subsequent scheduling.

[0052] When the actual execution delay propagates along the task dependency directed graph to the terminal node on the critical path (i.e., when all subsequent tasks traversed from the initial critical task include the terminal node of the critical path), the tracing module records the total cumulative deviation propagation at that terminal node. This total is the weighted sum of the delays from all intermediate nodes during the propagation from the initial critical task to the terminal node. The tracing module further calculates the deviation sensitivity index of the critical path, calculated by dividing the total cumulative deviation propagation by the original path weight of the critical path. This index reflects the sensitivity of the critical path to the execution delay of a single task; a larger index value indicates that the delay of any task on the path will cause a more significant overall delay amplification effect.

[0053] During the execution of the main scheme, whenever the weighted deviation propagation coefficient propagates along the task dependency directed graph to any subsequent task, the deviation propagation tracking module synchronously triggers the real-time calculation of the deviation sensitivity index of the path where that subsequent task is located. The deviation sensitivity index value calculated in real time is sent to the local early warning monitoring module of each computing node in the form of data packets through the broadcast channel of the central scheduler. After receiving the index, the local early warning monitoring module stores it in its local cache for use in determining the current early warning path during the execution of the main scheme. The local early warning monitoring module of each computing node continuously monitors the real-time calculated value of the deviation sensitivity index it receives. When the real-time calculated value of the deviation sensitivity index of any path exceeds the preset sensitivity threshold for the first time, the module marks the path as the current early warning path and generates an early warning record for the path, which is stored in the early warning record cache of each computing node. In this embodiment, the sensitivity threshold is set to 1.5, that is, an early warning is triggered when the total accumulated deviation propagation reaches 1.5 times the original path weight.

[0054] The early warning record cache stores the time of each early warning and the identification information of the warned path. When the same critical path is marked as the current early warning path a certain number of times within a preset time window reaches a preset frequency threshold, the path is marked as a historically high-sensitivity path. In this embodiment, the time window is set to 72 hours, and the frequency threshold is set to 3 times. Once a path is marked as a historically high-sensitivity path, the marking information is sent back to the central scheduler. In subsequent rehearsal strategies, the scheduler assigns each critical task on the historically high-sensitivity path an initial value of the lag ratio control parameter that is higher than that of other critical tasks, so that these critical tasks can obtain more resource guarantees during execution. The increment value of the lag ratio control parameter is dynamically determined based on the difference between the deviation sensitivity index of the path and the preset sensitivity threshold. The larger the difference, the larger the increment, but the maximum increment does not exceed 0.3.

[0055] Regarding the task segment replacement operation triggered by early warning during the execution of the main scheme, this operation is completed collaboratively by the local early warning monitoring module on each computing node and the central scheduler. The local early warning monitoring module continuously monitors the deviation sensitivity index after receiving the real-time broadcast. When the real-time calculated value of the deviation sensitivity index of any path first exceeds the preset sensitivity threshold and is marked as the current early warning path, the preparation process for the replacement operation is immediately initiated.

[0056] The first step in the replacement operation is progress monitoring. The local early warning monitoring module monitors in real time the proportion of completed critical tasks on the current early warning path to the total number of critical tasks on that path. When this proportion reaches a first monitoring threshold (set to 50% in this embodiment), indicating that at least half of the critical tasks on the path have been completed, the module begins to calculate the first deviation rate of the completed critical tasks. The first deviation rate is calculated by comparing the actual cumulative execution time of the completed critical tasks with the estimated cumulative execution time of these tasks, subtracting the estimated cumulative execution time from the actual cumulative execution time, and then dividing by the estimated cumulative execution time to obtain the deviation rate in percentage form.

[0057] When the first deviation rate exceeds a preset deviation rate threshold (set to 30% in this embodiment), the local early warning monitoring module sends a replacement request to the central scheduler. Upon receiving the replacement request, the central scheduler searches the flanking stagnation pools of each computing node for available unfinished task segments. These unfinished task segments originate from previously terminated or downgraded feasible allocation schemes. Each segment carries a resource allocation conservatism identifier for its original scheme, recorded as a numerical value of the stagnation ratio control parameter. The search criteria are that the segment has not yet been completed and its contained tasks correspond to or are substitutable with subsequent critical tasks not yet executed on the current early warning path in terms of task type.

[0058] If an incomplete task segment that meets the criteria is found, the central scheduler selects a segment with a higher resource allocation conservatism than the main scheme as a candidate replacement segment. Here, higher resource allocation conservatism means that the original feasible allocation scheme to which the segment belongs has a larger lag ratio control parameter value than the main scheme; that is, this scheme allows tasks in the flanking lag pool to occupy a larger proportion of computing resources during execution. After selection, the central scheduler performs a replacement operation: replacing the subsequent critical task that has not yet been executed on the current warning path with the candidate replacement segment, allocating the computing resources originally reserved for the replaced critical task to the candidate replacement segment, and changing the status flag of the candidate replacement segment in the flanking lag pool from pending or queued to active. After the replacement is completed, the candidate replacement segment becomes part of the main scheme and continues to execute.

[0059] After the replacement operation is completed, the local early warning monitoring module records the execution time of the replacement operation and the change in deviation rate before and after the replacement. This change represents the improvement in deviation rate. The execution time and the improvement in deviation rate are sent back as feedback parameters to the deviation sensitivity index calculation module of the central scheduler. This module uses the feedback parameters to correct the determination frequency threshold of historically high-sensitivity paths and the sensitivity threshold of the current early warning path. The correction method is as follows: when a path achieves a significant improvement in deviation rate through replacement, the sensitivity threshold for triggering early warnings for that path is appropriately lowered, making it easier to trigger early warnings and obtain replacement opportunities in subsequent executions; at the same time, the frequency threshold required for the path to be marked as a historically high-sensitivity path is appropriately increased to avoid the path being marked as a historically high-sensitivity path too early. In this embodiment, the correction range for the sensitivity threshold is 1.2 to 1.8, and the correction range for the frequency threshold is 2 to 5 times.

[0060] When the current warning path improves its deviation rate to the preset improvement target through this replacement operation (in this embodiment, the improvement target is set to a decrease in the absolute value of the deviation rate to below 15%), the local warning monitoring module removes the marker from the warning record cache. This cleared marker is no longer counted in the preset frequency threshold statistics within the preset time window; that is, if the path triggers a warning again later, the trigger count restarts from the cleared state. Through this mechanism, paths that frequently trigger warnings but successfully improve their deviation rate through replacement operations will not be prematurely marked as historically high-sensitivity paths due to accumulated warning counts, thus avoiding unnecessary over-protection of resources.

[0061] Based on the above implementation scheme for neutron transport calculation in multiple schemes of nuclear reactor core, the following section further explains the specific mechanism for each calculation node to dynamically adjust the resource allocation ratio of the main operation channel task and the flank stagnation pool task according to the connectivity index, as well as the specific operation of updating the stagnation ratio control parameters after the individual deviation signature is reinjected into the flank stagnation pool water level regulator.

[0062] Regarding the mechanism for independently adjusting resource allocation ratios on each compute node, after the main scheme is activated, a local resource scheduling agent process runs on each compute node. This agent process independently maintains the local flanking stagnation pool level. The flanking stagnation pool is implemented as a task waiting queue located in the compute node's local memory, containing all computational tasks marked as flanking stagnation pool tasks but not yet completed. The level is calculated by summing the estimated computational load of all currently queued flanking stagnation pool tasks and dividing the sum by the preset level capacity. The preset level capacity is defined as the maximum computational load that the compute node's total computing power can handle per unit time. The specific value of the preset level capacity varies for different types of compute nodes. For example, for a GPU node configured with dual graphics processing units, the preset level capacity, measured in floating-point operations per second, can be set to 80% of the node's theoretical peak performance of its dual graphics processing units, reserving some resources for system overhead and data transmission. The preset capacity of a CPU node is obtained by multiplying the theoretical floating-point operation capability of all its CPU cores by a utilization coefficient.

[0063] The local resource scheduling agent process performs resource ratio adjustment operations at a fixed monitoring cycle, which is set to 30 seconds in this embodiment. At the beginning of each cycle, the agent process collects three local parameters. The first parameter is the real-time load rate, which is equal to the ratio of the actual amount of computation executed by the computing node per unit time to the preset water level capacity. The actual amount of computation executed is obtained by reading the performance counters of each computing unit on the node, such as the number of instructions executed by the CPU node or the kernel execution time of the GPU node. The second parameter is the backlog of main channel tasks, which is equal to the estimated total amount of computation occupied by all main job channel tasks currently assigned to the computing node but not yet completed. The tasks not yet completed here include two types of tasks: the remaining estimated amount of computation for tasks that have started execution but have not yet ended, and the total estimated amount of computation for tasks that have been assigned to the node but have not yet started execution. The third parameter is the water level value of the aforementioned flank storage tank.

[0064] After collecting the three parameters, the agent process inputs them into a locally pre-stored connectivity index calculation function. The specific calculation logic of this function is as follows: First, the backlog of main channel tasks is divided by the preset water level capacity to obtain a normalized backlog ratio, which is between 0 and 1. Then, the water level values ​​of the flank storage tanks and the real-time load rate are weighted and summed. In this embodiment, the weight of the flank storage tank water level value is 0.6, and the weight of the real-time load rate is 0.4. Finally, the normalized backlog ratio is divided by the above weighted sum, and the quotient is the connectivity index value at the current moment. The physical meaning of this index value is: the higher the normalized backlog ratio, the more severe the backlog of main channel tasks, requiring priority to ensure resources for main channel tasks; the larger the weighted sum, the higher the backlog of flank tasks and the current load, and the more limited the additional resources that main channel tasks can obtain. A higher connectivity index value indicates that the current main channel task backlog of the computing node is relatively serious, and more resources need to be allocated to the main operation channel tasks; a lower index value indicates that the main channel task backlog is relatively light, and more resources can be allocated to the side storage pool tasks.

[0065] After obtaining the connectivity index value, the agent process compares this value with preset high and low connectivity thresholds. In this embodiment, the high connectivity threshold is set to 0.75, and the low connectivity threshold is set to 0.25. When the connectivity index value is higher than 0.75, the agent process determines that the computing node is currently in a main channel priority state and sets the computing resource allocation ratio for the next monitoring cycle to a first ratio value. This first ratio value specifically means that in the next 30-second cycle, the computing node will allocate approximately 75% of its computing resources to the main operation channel tasks and approximately 25% to the flanking stagnation pool tasks. When the connectivity index value is lower than 0.25, the agent process determines that the computing node is currently in a flanking priority state and sets the computing resource allocation ratio for the next monitoring cycle to a second ratio value, i.e., approximately 30% of the computing resources are allocated to the main operation channel tasks and approximately 70% to the flanking stagnation pool tasks. When the connectivity index value is between 0.25 and 0.75, the agent process determines that the computing node is currently in a balanced state, maintaining the current computing resource allocation ratio unchanged without any adjustments.

[0066] After completing the above judgment and ratio setting, the agent process writes the set ratio value into the configuration register of the local resource allocator. The resource allocator then executes task scheduling according to this ratio during subsequent monitoring cycles. The agent process then waits for the next monitoring cycle to arrive, at which time it re-collects the three input parameters and repeats the above comparison and ratio setting operations. There is an interrelationship between the specific values ​​of the high connectivity threshold, low connectivity threshold, and the first and second ratio values: the high connectivity threshold and low connectivity threshold together determine the sensitivity of resource allocation ratio switching. The wider the interval between the two thresholds, the larger the range in which the resource allocation ratio remains stable, the slower the system responds to fluctuations in the connectivity index, but the higher the scheduling stability; the narrower the interval between the two thresholds, the more sensitive the system responds to fluctuations, but may trigger frequent ratio switching. The difference between the first and second ratio values ​​determines the degree of difference in resource allocation between the main channel priority state and the flanking priority state. The larger the difference, the more obvious the resource tilt; the smaller the difference, the more similar the allocation in the two states. The selection of the above parameters can be adjusted according to the relative importance of the main channel tasks and flanking tasks in the specific application scenario.

[0067] Regarding the specific operation of injecting individual deviation signatures back into the flank reflux tank level regulator and updating the reflux ratio control parameters, each computing node's flank reflux tank level regulator internally has a reflux ratio control parameter register. This register stores the current reflux ratio control parameter value of the computing node in floating-point form. This parameter value determines the maximum injection rate of flank reflux tank tasks during the execution of the main operation channel task on that computing node, i.e., the maximum number of tasks or computational loads allowed to be extracted from the flank reflux tank and started per unit time.

[0068] When any task on the computing node is completed, the individual deviation signature corresponding to that task is sent to the local flanking reservoir level regulator. The level regulator extracts the deviation value from this signature. The deviation value here is defined as the percentage obtained by subtracting the estimated resource consumption from the actual resource consumption and dividing by the estimated resource consumption. The level regulator compares this deviation value with a preset deviation dead zone threshold. The purpose of setting the deviation dead zone threshold is to avoid frequent fluctuations in the reservoir ratio control parameter due to small deviations. In this embodiment, the deviation dead zone threshold is set to 5%. When the absolute value of the deviation value is less than or equal to 5%, the level regulator determines that the current deviation between the estimated and actual values ​​is within an acceptable range and maintains the current reservoir ratio control parameter value of the computing node unchanged.

[0069] When the deviation is positive and its absolute value is greater than 5%, it indicates that the current resource consumption estimate for this type of task by the computing node is insufficient, and the actual resource consumption of the task exceeds the estimate. At this time, the water level regulator adjusts the storage ratio control parameter value of the computing node in an increasing direction with a first adjustment step size. After the storage ratio control parameter value is increased, the upper limit of the proportion of computing resources that the flank storage pool tasks can occupy is increased, which is equivalent to the main channel tasks being able to release more resources to the flank tasks in subsequent executions, thereby alleviating the resource shortage in the main channel caused by insufficient estimation. In this embodiment, the first adjustment step size is set to 0.05.

[0070] When the deviation is negative and its absolute value is greater than 5%, it indicates that the computing node has overestimated the resource consumption for this type of task, and the actual resource consumption of the task is lower than the estimate. At this time, the water level regulator adjusts the storage ratio control parameter of the computing node to decrease it using the second adjustment step size. After the storage ratio control parameter value decreases, the upper limit of the proportion of computing resources that the flank storage pool task can occupy is reduced, which is equivalent to reserving more redundant resources for the main channel task in subsequent execution, avoiding resource idleness due to overestimation. In this embodiment, the second adjustment step size is 0.03.

[0071] The design where the first adjustment step size is larger than the second adjustment step size results in a higher response sensitivity of the sluggishness ratio control parameter when resource consumption is underestimated compared to when resource consumption is overestimated. This asymmetric design is based on the consideration that in heterogeneous supercomputing environments, underestimation of resource consumption can lead to execution delays in main channel tasks due to resource shortages. This delay propagates along the task dependency directed graph and may cause the entire task flow to exceed its completion time, with negative impacts typically more severe than resource idleness. Therefore, a faster adjustment speed is adopted when underestimation occurs to quickly increase the resource availability of flanking tasks, while a slower recovery speed is adopted when overestimation occurs to avoid frequent adjustments due to excessive conservatism.

[0072] After the water level regulator completes the above adjustments, it writes the adjusted lag ratio control parameter value into the lag ratio control parameter register, replacing the original old value. The value in this register is read by the local resource scheduling agent process when executing main job channel tasks, and is used to control the injection rate of tasks in the flanking lag pool. Specifically, at the beginning of each resource allocation cycle, the agent process reads the lag ratio control parameter value in the register, multiplies the estimated total computational load of tasks waiting to be executed in the current flanking lag pool by this parameter value, and obtains the maximum computational load allowed to be scheduled from the flanking lag pool within that cycle. Then, it executes the flanking task scheduling operation with this value as the upper limit. Through the above injection and update mechanism, the lag ratio control parameters of each computing node are continuously adaptively adjusted according to the actual deviation during task execution, causing the resource allocation strategy of each computing node to gradually converge towards the resource consumption characteristics of the task type it actually undertakes.

[0073] Based on the above implementation scheme for neutron transport calculation scenario in the multi-scheme nuclear reactor core, the following further explains the independence of the current early warning path marker and the historical high-sensitivity path marker, the retrieval and screening mechanism of candidate replacement segments in the flank stagnation pool, and the pre-charging operation driven by the change of connectivity index.

[0074] Regarding the independence of current warning path marking and historical high-sensitivity path marking in terms of time scale and judgment criteria, these two markings serve different scheduling decision purposes, and their triggering conditions and application scopes are clearly distinguished. The marking of the current warning path is determined solely based on the deviation sensitivity index calculated in real-time within the currently executing parallel task flow, without relying on any historical execution data. Specifically, when the real-time calculated deviation sensitivity index of a path exceeds a preset sensitivity threshold for the first time during the execution of the current task flow, that path is marked as the current warning path. This marking is entirely based on the real-time execution status of the current task flow and is used to trigger real-time replacement decisions during the execution of the main plan, i.e., determining whether it is necessary to select alternative task segments from the flank buffer to replace subsequent critical tasks that have not yet been executed on the current warning path. The validity period of this marking is limited to the execution period of the current task flow; once the current task flow is completed or the path's deviation rate is improved through replacement operations, the marking becomes invalid or is cleared.

[0075] The marking of historically high-sensitivity paths is based on whether the number of times the same critical path is marked as a current warning path within a preset time window reaches a preset frequency threshold. This marking requires accumulating historical data across multiple rounds of task flow execution, with a time scale much longer than that of current warning path marking. In specific implementation, the central scheduler maintains a global path warning count table, which records the identifier of each path that has been marked as a current warning path and its cumulative marking count within a preset time window. When the cumulative marking count of a path reaches the preset frequency threshold, the path is marked as a historically high-sensitivity path. This marking is used for the allocation of the initial value of the lag ratio control parameter in the pre-simulation strategy of the next round of scheduling tasks. That is, when generating feasible allocation schemes in the future, the critical tasks on historically high-sensitivity paths are assigned an initial value of the lag ratio control parameter higher than that of ordinary critical tasks, so that they can obtain more abundant resource guarantees during execution. These two marking criteria are independent of each other. The triggering of a current warning path does not require the path to have a historically high-sensitivity mark, and the marking of a historically high-sensitivity path does not require the path to be in a warning state at the current moment. When a path has both types of tags, the two tags take effect independently, serving real-time replacement decisions and subsequent pre-simulation strategy adjustments, respectively.

[0076] Once a path is marked as a current warning path, if a replacement operation is triggered during the current execution process and the deviation rate is improved to the preset improvement target through this replacement operation, the mark is removed from the warning record cache. Furthermore, the mark removed is no longer counted in the mark count statistics within the preset time window. This means that although the path previously triggered a warning, because the replacement operation successfully improved the execution deviation, this warning is not used as a basis for determining whether the path is a historically high-sensitivity path. This design avoids paths being prematurely marked as historically high-sensitivity paths due to the accumulation of single, remediable deviations, thereby reducing unnecessary excessive resource protection in subsequent task flow scheduling.

[0077] Regarding the retrieval and filtering mechanism for incomplete task segments in the flanking stagnation pool, before the central scheduler searches the flanking stagnation pool for any incomplete task segments from terminated or downgraded schemes, each computing node maintains a survival score for each incomplete task segment in the flanking stagnation pool. This score is a comprehensive evaluation index, consisting of a weighted sum of three components: the normalized value of resource allocation conservatism, the normalized value of deviation direction complementarity, and the normalized value of local cache hit rate, with weighting coefficients of 0.4, 0.3, and 0.3, respectively.

[0078] The normalized value of resource allocation conservatism is calculated as follows: take the lag ratio control parameter value of the original feasible allocation scheme to which the incomplete task segment belongs, divide it by the maximum lag ratio control parameter value of all schemes in the alternative scheme pool, and the normalized result is between 0 and 1. The larger the value, the lower the conservatism of the original scheme to which the segment belongs, that is, the higher the proportion of resources allowed for flanking tasks when it is executed.

[0079] The calculation method for the deviation direction complementarity is as follows: First, determine the deviation direction of the executed portion of the main scheme on the current warning path, which is the positive or negative sign of the difference between the actual resource consumption and the estimated resource consumption in the executed critical task. A positive value indicates insufficient resource consumption estimation, and a negative value indicates excessive resource consumption estimation. Then, determine the deviation direction of the executed portion on the same or similar path as the current warning path in the original feasible allocation scheme to which the incomplete task segment belongs. The deviation direction complementarity is equal to the degree of opposition between the two deviation directions. That is, if the main scheme shows an overall positive deviation of insufficient estimation on the current warning path, while the original scheme to which the incomplete task segment belongs shows an overall negative deviation of excessive estimation on the corresponding path, the complementarity is high, which is taken as 1 in this embodiment; if the two deviation directions are the same, the complementarity is 0; if they are partially complementary, the intermediate value between 0 and 1 is taken based on the actual comparison of the deviation values. The design logic of this component is that selecting a segment with a deviation direction opposite to or different from the main scheme to replace the subsequent tasks on the current warning path may offset the accumulated deviation effect generated by the main scheme, thereby improving the overall execution efficiency.

[0080] The local cache hit rate is calculated as follows: The amount of data already residing in the local cache of the current computing node out of all the input data required for the incomplete task segment is counted. This residing data amount is divided by the total amount of input data required for the segment; the resulting ratio is the local cache hit rate. A higher ratio indicates lower data transfer overhead and higher execution efficiency when activating the segment for execution.

[0081] When multiple unfinished task segments in the flank buffer pool have a resource allocation conservatism higher than the main scheme, the central scheduler calculates the survival score of each candidate segment and selects the segment with the highest survival score as the final candidate replacement segment. After the candidate replacement segment is activated, executed, and completes the replacement, its survival score is fed back as a feedback parameter to the central scheduler's pre-simulation feasible allocation scheme generation module. In subsequent pre-simulations, when generating new feasible allocation schemes, this module advances the initial ranking of the original scheme with a high survival score in the candidate scheme pool. Specifically, when a task segment generated by an original scheme exhibits a high survival score during the replacement operation, the buffer ratio control parameter combination of this scheme type is prioritized in subsequent scheduling rounds. That is, when generating N schemes, this combination is placed at the front of the ranking queue, giving it a higher probability of being selected in the candidate scheme pool.

[0082] Regarding the pre-charge mechanism driven by connectivity index changes, after each computing node collects local real-time load rate and main channel task backlog at a fixed monitoring period, the node maintains a connectivity index change rate register locally. This register stores the first-order difference value of the connectivity index between the previous monitoring period and the current monitoring period, i.e., the connectivity index value of the current period minus the connectivity index value of the previous period. The sign of this difference value reflects the direction of change of the connectivity index, and its absolute value reflects the rate of change.

[0083] When the first difference of the connectivity index is negative and its absolute value exceeds a preset rate of decline threshold, it indicates that the connectivity index is rapidly decreasing, meaning the ratio of the main channel task backlog to the flanking task level and real-time load is rapidly decreasing. This trend suggests that within a future time window, the main channel task backlog will continue to decrease below the low level, at which point the computing node will experience a shortage of main channel tasks. At this point, the local resource scheduling agent process triggers a task pre-charging operation from the flanking to the main channel. The specific execution method of this pre-charging operation is as follows: the agent process selects the unfinished task segment with the smallest estimated computational load from the computing node's flanking slack pool, pre-loads it into the computing node's local execution cache, and allocates a reserved computing resource share to it. This reserved computing resource share is equal to a preset percentage of the computing node's current idle computing resources; in this embodiment, this percentage is set to 60%. After loading, the task segment is in a pending execution state. Once the main channel task backlog decreases below the low level, the pre-charged task can be immediately put into execution, avoiding the time overhead of scheduling tasks from the flanking slack pool.

[0084] When the first difference of the connectivity index is positive and its absolute value exceeds the preset rate of increase threshold, it indicates that the connectivity index is rising rapidly, meaning the ratio of the main channel task backlog to the flanking task level and real-time load is increasing rapidly. This trend suggests that within a future time window, the main channel task backlog will continue to rise above the high level, at which point the compute node will face a shortage of main channel resources. At this point, the local resource scheduling agent process triggers a task pre-charge operation from the main channel to the flanking direction. The specific execution method of this pre-charge operation is as follows: the agent process selects the task with the shallowest dependency depth from the main job channel tasks currently allocated to the compute node but not yet started. The shallowest dependency depth means that the task has the smallest number of edges from the termination node in the task dependency directed graph, that is, the task has the highest deferredness, and delaying its execution has the least impact on the overall task flow completion time. After selecting the task, it is transferred from the main job channel task queue to the flanking stagnation pool of the compute node, and the current connectivity index value of the compute node is actively reduced. The connectivity index value is reduced by multiplying the current calculated value by a decay coefficient less than 1; in this embodiment, the decay coefficient is set to 0.85. The decayed connectivity index value causes subsequent resource allocation decisions to favor a flanking priority, thereby allowing the computing node to release computing resources in advance and reserve processing capacity for the upcoming peak of main channel tasks.

[0085] After triggering a precharge operation in any direction, the compute node reports the execution record of this precharge operation to the central scheduler. The execution record includes the precharge direction, i.e., from the flank to the main channel or from the main channel to the flank, the task identifier of the precharged task, and the connectivity index change rate value at the time of precharge triggering. After collecting the precharge execution records from each compute node, the central scheduler adjusts the connectivity index change rate threshold of each compute node according to the frequency and distribution of the records. The specific adjustment logic is as follows: when the precharge direction prediction accuracy in the precharge operation record reported by a compute node is higher than the preset accuracy threshold, it indicates that the current change rate threshold setting is reasonable and remains unchanged; when the prediction accuracy is lower than the preset accuracy threshold, the absolute value of the change rate threshold is appropriately increased to reduce the trigger sensitivity of the precharge operation and reduce the number of false precharges. In this embodiment, the accuracy threshold is 70%, and the adjustment step of the change rate threshold is 10% of its current value. Through the above dynamic adjustment mechanism, the precharge trigger threshold of each compute node gradually adapts to its actual load fluctuation characteristics, thereby improving the accuracy and effectiveness of the precharge operation.

[0086] This embodiment also provides a heterogeneous supercomputing parallel task allocation system based on streaming job scheduling. This system includes a central scheduler and multiple groups of parallel computing nodes. The central scheduler is deployed on a dedicated server, which is equipped with at least one high-performance multi-core CPU and machine-readable storage media for storing executable instructions and runtime data. The central scheduler connects to each group of computing nodes via a high-speed internal network and is responsible for control plane operations such as receiving and parsing parallel task streams, constructing task dependency directed graphs, generating feasible allocation schemes, managing the alternative scheme pool, activating winning schemes, and updating pre-launch strategies. Each group of computing nodes contains multiple heterogeneous computing nodes, which can be CPU-type computing nodes, graphics processing unit-type computing nodes, or domestically produced accelerator card-type computing nodes. Computing nodes within the same group share the group's resource quota, and computing resources are isolated between groups. Each computing node executes specific computing tasks under the instructions of the central scheduler. Each computing node runs a local resource scheduling agent process and a water level regulator, which are responsible for performing local task scheduling, resource allocation ratio adjustment, flanking reservoir management, connectivity index calculation, and deviation signature reinjection and other data surface operations.

[0087] The central scheduler communicates asynchronously with each computing node via message queues. The types of instructions issued by the central scheduler to the computing nodes include scheme issuance instructions, termination instructions, degradation instructions, pre-charge instructions, and replacement instructions. The data types reported by each computing node to the central scheduler include progress counts, individual deviation signatures, pre-charge execution records, and replacement feedback parameters. The central scheduler internally deploys the following functional modules: a path enumeration and weight calculation module, responsible for extracting all directed paths from the task dependency directed graph and calculating path weights; a critical path marking module, responsible for marking critical paths and critical tasks based on path weights; a scheme generation module, responsible for performing fill-in allocation operations and generating feasible allocation schemes with different hysteresis ratio control parameters; a winning scheme determination module, responsible for receiving progress reports from each group and determining the winning scheme; a deviation propagation tracking module, responsible for calculating the deviation propagation coefficient and deviation sensitivity index; a clustering analysis engine, responsible for clustering individual deviation signatures and identifying unreliable prediction combinations; and a global early warning counter maintenance module, responsible for recording the number of early warnings for each path within the current time window and marking historically highly sensitive paths. The above modules are stored in software form on the machine-readable storage medium of the central scheduler and are loaded and executed by the central processing unit.

[0088] The local resource scheduling agent process deployed on each compute node includes the following sub-modules: a connectivity index calculation sub-module, responsible for collecting local load parameters and calculating the connectivity index according to the monitoring cycle; a resource allocation ratio adjustment sub-module, responsible for setting the resource allocation ratio for the next cycle based on the comparison result of the connectivity index value and the threshold; a flanking stagnation pool management sub-module, responsible for maintaining the local flanking stagnation pool task queue, calculating the water level value, and maintaining the survival score of each incomplete task segment; a pre-charge operation execution sub-module, responsible for monitoring the connectivity index change rate and triggering the pre-charge operation in the corresponding direction; and a connectivity index change rate register, used to store the first-order difference value of the connectivity index between the previous monitoring cycle and the current monitoring cycle. The flanking stagnation pool water level regulator on each compute node has a built-in stagnation ratio control parameter register, which stores the current stagnation ratio control parameter value of the node. The water level regulator receives the individual deviation signature from the task completion callback function and updates the parameter value in the register according to the deviation value.

[0089] When the system executes a complete parallel task allocation process, the interaction between data flow and control flow is as follows: After designers submit multiple core neutron transport calculation schemes on the client, the central scheduler parses the parallel task flow corresponding to each scheme and constructs a task dependency directed graph. The constructed directed graph and its estimated resource consumption metadata are input into the path enumeration and weight calculation module. After this module outputs all directed paths and their weights, the critical path marking module marks the critical path and critical tasks. The scheme generation module generates N feasible allocation schemes based on the critical path information and the set values ​​of the hysteresis ratio control parameters and stores them in the candidate scheme pool. The central scheduler distributes the N schemes from the candidate scheme pool to N independent computing nodes, and each group of computing nodes starts execution simultaneously.

[0090] During execution, each computing node continuously reports its local progress count to the central scheduler. Upon receiving the reported data, the winning scheme determination module compares the progress of each group in real time. Once a group's progress reaches a preset threshold, the module sends a main scheme activation notification to that group and simultaneously sends termination or downgrade instructions to the remaining groups. After the main scheme is activated, the local resource scheduling agent process of each computing node independently adjusts the resource allocation ratio according to its respective connectivity index. After the tasks on each computing node are completed, the individual deviation signature is sent back to the local flanking reservoir level regulator to update the reservoir ratio control parameters, and also reported to the central scheduler's clustering analysis engine for periodic clustering analysis. When the deviation propagation tracking module detects a delay in a critical task, it calculates the deviation propagation coefficient and deviation sensitivity index, broadcasting the warning information to the local warning monitoring module of each computing node. The local warning monitoring module triggers a replacement decision based on the warning information, selecting candidate replacement segments from the flanking reservoir and performing the replacement operation. The feedback parameters of the replacement operation are sent back to the central scheduler's scheme generation module to adjust the ranking of each scheme in subsequent rehearsals. The connectivity index change rate register of each computing node triggers a pre-charge operation determination after each monitoring cycle update. The execution record of the pre-charge operation is reported to the central scheduler to correct the change rate threshold of each node.

[0091] The central scheduler and each computing node communicate via a high-speed internal network. The server hosting the central scheduler is equipped with dual redundant power supplies and disk arrays to ensure high availability of the scheduling service. Each computing node is equipped with local solid-state storage disks to cache input data and intermediate results required for task execution. Input data for task segments in the flanking reflux pool is cached in the local storage of a computing node after it is first loaded onto that node. When the segment is activated for execution, there is no need to read the data from global storage again, thereby reducing data transfer overhead. The machine-readable storage medium of the central scheduler stores the executable instructions of the above modules. When these instructions are executed by the central processing unit, the system executes the heterogeneous supercomputing parallel task allocation method based on streaming job scheduling described in the preceding embodiments.

[0092] In alternative implementations, the central scheduler's functionality can be distributed across multiple management nodes. One master management node coordinates scheduling decisions, while the remaining nodes act as hot standby nodes, taking over scheduling services in case of host failure. The local resource scheduling agent processes on each compute node can also be configured differently based on the specific hardware configuration of the compute node. For example, GPU-intensive nodes can be configured with larger flanking buffer capacities to fully utilize their parallel computing capabilities, while CPU-intensive nodes can be configured with smaller capacities to reduce context switching overhead.

[0093] It should be further clarified that: In this application, the main job channel task refers to the computational task that is being executed or awaiting execution by each computing node after the main scheme is activated, and that belongs to the critical path of the directed graph dependent on the main scheme task. Corresponding to the main job channel task, the flanking stagnation pool task refers to the incomplete task fragments in the terminated or downgraded alternative schemes, as well as tasks transferred from the main channel to the flanking channel during the execution of the main scheme due to pre-charging operations. The flanking buffer task and the flanking stagnation pool task refer to the same concept in this application, namely, tasks that have been marked as not participating in the main channel resource competition and can be delayed. The alternative scheme pool is a logical data structure maintained by the central scheduler. Its physical implementation is an ordered list in the central scheduler's memory. Each entry in the list stores a complete feasible allocation scheme, including the scheme number, the mapping table from each task to the computing node, the resource allocation strategy of each computing node, and the corresponding stagnation ratio control parameter value. The lifecycle of the alternative scheme pool begins when the scheme is generated and stored in the pool, and ends when the winning scheme is selected and the remaining schemes are terminated or downgraded. At the start of the next round of scheduling, the original pool content is cleared and refilled.

[0094] Regarding the source and storage mechanism of unfinished task fragments in terminated scenarios, when a group of computing nodes receives a termination instruction, the group does not immediately discard all execution context. Before the termination instruction is issued, the central scheduler sends a context save request to the group. The group's nodes package the intermediate computation results of the partially executed but incomplete tasks, the input data loaded into memory, and the remaining work description of the tasks into an unfinished task fragment, write it to the shared storage volume of the physical node group to which the group belongs, and register it in the global index table of the central scheduler's flank storage pool. After completing the context save, the group releases all computing resources. The time required for the above context save operation is usually between tens and hundreds of milliseconds, far less than the time required to re-execute the task, so it will not have a significant impact on the overall resource reclamation efficiency. Groups that receive a degradation instruction do not perform a context save operation, but instead directly convert the currently executing task queue as a flank storage pool task and continue to execute it in place using idle resources.

[0095] Regarding the progress comparison benchmark between the various schemes, in step three, the progress count values ​​reported by the independent computing nodes of each group represent the number of completed critical tasks on the critical path of that group's own scheme. Different schemes may have different critical paths due to varying lag ratio control parameters, leading to differences in node selection and thus the critical paths themselves. However, the critical paths of each scheme originate from the same original task dependency directed graph, with the same start and end nodes. Therefore, although the critical paths of each scheme may differ slightly in intermediate nodes due to resource allocation differences, their path lengths are similar. Progress comparison between groups is based on the proportion calculated independently within each scheme. The winning scheme is determined by a group reaching the 60% threshold first. This condition reflects the group's execution efficiency under its own allocation strategy, rather than competition between different schemes in absolute time. When a scheme completes 60% of its critical path tasks first, it indicates that the resource allocation strategy of that scheme is most suitable in the current actual execution environment, making it reasonable to activate it as the primary scheme.

[0096] Regarding the relationship between the lag ratio control parameter and the resource allocation conservatism, the typical range of the lag ratio control parameter in the various embodiments of this application is a real number between 0 and 1. The closer the parameter value is to 0, the more the scheme tends to concentrate computing resources on critical path tasks during execution, and the lower the proportion of resources allowed to be occupied by flanking lag pool tasks, i.e., the higher the resource allocation conservatism. The closer the parameter value is to 1, the more the scheme tends to allow flanking tasks and main channel tasks to share computing resources during execution, and the lower the resource allocation conservatism. The first preset value is taken as 0.2 in the embodiments of this application, the second preset value is taken as 0.4, and the preset step size is taken as 0.2. The above values ​​can be adjusted according to the computing intensity ratio of the main task and flanking tasks in the actual system. For example, when flanking tasks are generally lightweight post-processing tasks, the first preset value can be increased to 0.3 to make fuller use of idle resources; when flanking tasks are generally heavyweight computing tasks, the first preset value can be decreased to 0.1 to more strictly protect the main channel resources. The typical range for the total number of schemes N is 3 to 8. A larger value is chosen when computing node resources are abundant to increase the coverage of candidate schemes, and a smaller value is chosen when computing node resources are scarce to reduce the resource overhead caused by parallel execution. When N is large and the total number of available computing nodes in the system is insufficient to support the parallel execution of N groups of independent nodes, the scheduler dynamically reduces the number of schemes in the candidate scheme pool after the scheme is generated based on the current resource availability. Specifically, the reduction method is to delete some schemes from both the high conservatism end and the low conservatism end, and prioritize retaining schemes whose hysteresis ratio control parameters are distributed in the middle range.

[0097] Regarding the weighting coefficients and normalization method in the connectivity index calculation function, the normalization benchmark value of the main channel task backlog is the preset water level capacity of the computing node. The normalization method is to divide the current backlog by the preset water level capacity to obtain a dimensionless normalized backlog ratio. In the connectivity index calculation formula, the sum of the weight of the flanking reservoir water level value and the weight of the real-time load rate is 1, which are taken as 0.6 and 0.4 respectively in this embodiment. These two weight coefficients can be configured differently according to the hardware characteristics of the computing node. For example, for nodes with more processor cores, the weight of the flanking reservoir water level value can be increased to make fuller use of multi-core parallel capabilities; for nodes with relatively tight computing resources, the weight of the real-time load rate can be increased to avoid overload. In this embodiment, the high connectivity threshold is taken as 0.75, the low connectivity threshold is taken as 0.25, and the interval width between the two thresholds is 0.5. This interval width determines the stable range in which the resource allocation ratio remains unchanged. The larger the width, the more stable the system is but the slower the response to load changes; the smaller the width, the faster the system response but may cause frequent switching of the allocation ratio. In this embodiment, the first ratio is 75% for the main channel and 25% for the flanks. The second ratio is 30% for the main channel and 70% for the flanks. The difference between the two is 45 percentage points. This difference determines the degree of difference in resource allocation between the main channel priority state and the flank priority state. The larger the difference, the more obvious the resource tilt. The smaller the difference, the more similar the two states become.

[0098] Regarding the threshold parameters in cluster analysis and their relationship with the deviation dead zone threshold in real-time backinjection, the preset density threshold in this embodiment is set to 50 samples, the preset deviation tolerance threshold is set to 20%, and the preset time period is set to 24 hours. The deviation dead zone threshold is used for real-time backinjection determination in step six and is set to 5%. The deviation tolerance threshold is used for cluster analysis determination in step seven and is set to 20%. The difference in their values ​​is due to the different decision scales they serve: the deviation dead zone threshold is used for real-time fine-tuning of a single task; setting a smaller value can promptly capture subtle deviations and perform gradual corrections; the deviation tolerance threshold is used for determining the statistical trend of batch samples; setting a larger value can filter out random fluctuations and only identify statistically significant systematic deviation patterns. There is no subordinate or linkage relationship between the two; they are set independently.

[0099] Regarding the statistical scope of the flank storage tank water level, the calculation only includes the estimated computational load of tasks currently queued and not yet started in the flank storage tank. Flank tasks that have started execution but have not yet ended are not included in the water level calculation. This design is based on the consideration that tasks already in execution are consuming computational resources, and their remaining computational load cannot be changed by adjusting the injection rate. Only queued tasks that have not yet started are constrained by the storage ratio control parameter. The water level is updated by increasing it each time a new task fragment enters the flank storage tank and decreasing it each time a task is removed from the flank storage tank and begins execution.

[0100] Regarding the values ​​of the first monitoring threshold, preset improvement target, preset sensitivity threshold, and preset frequency threshold, the first monitoring threshold is set to 50% in the replacement operation triggering condition. This means that the evaluation of whether replacement is necessary only begins when more than half of the critical tasks on the current warning path have been completed. Premature replacement may lead to incorrect decisions due to insufficient available information, while premature replacement may result in the loss of replacement value due to insufficient remaining tasks. The preset improvement target is set to a deviation rate absolute value decreasing to below 15%. This value is higher than the deviation dead zone threshold of 5% but lower than the deviation rate threshold for triggering replacement of 30%, ensuring the feasibility of the replacement operation target. The preset sensitivity threshold is set to 1.5, and the preset frequency threshold is set to 3 times within a 72-hour time window. These values ​​can be adjusted according to the actual distribution of historical sensitive paths. In this embodiment, the first adjustment step size is set to 0.05, and the second adjustment step size is set to 0.03, with a difference of 0.02. This difference ensures that the response sensitivity when the estimate is insufficient is higher than the response sensitivity when the estimate is excessive. The larger the difference, the more significant the asymmetry, but an excessively large difference may cause oscillations in the hysteresis ratio control parameter during adjustment.

[0101] Regarding the timing of the pre-execution strategy update, the revised path weight parameters and the candidate resource pool after removing unreliable prediction combinations will not be switched until the currently executing parallel task flow completes all tasks. The current task flow continues to execute according to the pre-execution strategy parameters when the main scheme was activated until completion. The updated pre-execution strategy will take effect when the next round of parallel task flow arrives at the central scheduler. This design is based on the consideration that the currently executing task flow has already completed scheme generation and resource allocation according to the original pre-execution strategy. Switching strategies midway would lead to a loss of comparability between schemes and may introduce unpredictable resource contention.

[0102] Regarding the high connectivity threshold and low connectivity threshold, the initial connectivity index threshold for each computing node is uniformly issued by the central scheduler. However, during operation, each computing node can request adjustments from the central scheduler based on its local pre-charge operation execution records. After collecting pre-charge execution records from all computing nodes, the central scheduler uniformly issues the revised threshold to all computing nodes every 24 hours for global coordination, ensuring a consistent basis for threshold adjustments among nodes. In this embodiment, the initial values ​​of the high connectivity threshold and low connectivity threshold are set by the system administrator based on the typical task load characteristics of the supercomputing platform. In the absence of prior knowledge, the default values ​​of 0.75 and 0.25 given in this implementation method are used.

[0103] Regarding the resource ownership of downgraded nodes, the computing resources of a node group that has been downgraded to continue executing tasks in the flanking buffer pool still physically belong to the group itself. However, control over resource usage shifts to the overall scheduling framework of the main scheme. Specifically, the local resource scheduling agent process of this node group still adjusts the resource allocation ratio locally, but its adjustment instructions come from the global policy parameters broadcast by the central scheduler after the main scheme is activated, rather than continuing to execute the independent scheduling policy in the original scheme. The main scheme can call on the idle computing resources of the downgraded node group to execute flanking tasks, but when the main channel task resources are scarce, the downgraded node group must still prioritize the execution of main channel tasks locally.

[0104] Regarding data consistency between the real-time back-injection and periodic clustering analysis mechanisms for individual bias signatures, both mechanisms use the same individual bias signature data source, which is stored as a database table in the persistent storage of the central scheduler. The real-time back-injection mechanism executes synchronously with each signature written to the database via triggers, marking the back-injected status on the signature record upon completion. The periodic clustering analysis mechanism reads all signature records within the current period upon periodic triggering, regardless of whether they have been back-injected. Both mechanisms operate on rows of records in the same data table, ensuring concurrency safety through database row-level locking, thus eliminating read-write inconsistencies caused by data races.

[0105] Regarding the timeliness of current early warning path markings and historical high-sensitivity path markings, the effective scope of a current early warning path marking is a single replacement decision process, that is, from the time a path is marked as a current early warning path until the deviation rate improvement reaches the preset improvement target through a replacement operation or the task flow to which the path belongs is completed. The effective scope of a historical high-sensitivity path marking is from the time the marking is generated until the next global update of the pre-simulation strategy. Each time the pre-simulation strategy is updated, the central scheduler re-evaluates whether all historical high-sensitivity path markings need to be continued or revoked based on the early warning records within the most recent preset time window. After a path is marked as a historical high-sensitivity path in a certain round, the initial value increment of the lag ratio control parameter obtained by that path in subsequent scheduling rounds is re-determined according to the latest marking status at each pre-simulation strategy update, and does not accumulate indefinitely.

[0106] Regarding the physical meaning of deviation direction complementarity, a positive deviation direction indicates that the actual resource consumption of the task on that path consistently exceeds the estimated value, meaning the solution tends to underestimate the task's resource requirements; a negative deviation direction indicates that the actual resource consumption consistently falls below the estimated value, meaning the solution tends to overestimate the task's resource requirements. When a path exhibiting a positive deviation in the main solution is replaced by a segment exhibiting a negative deviation in the original solution, the deviation directions of the two solutions are opposite, and the overall deviation may be neutralized after combined execution. Deviation direction complementarity is an indicator that measures this neutralization potential; the higher the complementarity, the greater the possibility of offsetting the cumulative effect of deviation after replacement. When both deviation directions of the two solutions are positive, the complementarity is 0. In this case, replacement does not help improve the deviation rate and is therefore not the first choice for candidate replacement segments.

[0107] Regarding the impact of the replacement operation on the stability of the main solution, the replacement operation occurs between a subsequent critical task that has not yet been executed on the current warning path of the main solution and a candidate replacement segment. The task to be replaced has been confirmed to have a high risk of delay, and continued execution may cause the completion time of the entire task flow to exceed the deadline. The candidate replacement segment comes from the original solution with a higher resource allocation conservatism than the main solution, meaning that it allows flanking tasks to occupy a larger proportion of computing resources in the original solution. This means that the segment has stronger resource elasticity, and using this segment to execute when there is resource shortage on the current warning path helps to alleviate the risk of delay. After the replacement operation is completed, the remaining execution path of the main solution changes, but the status of the main solution remains unchanged. The tasks that were replaced in the flanking stagnation pool are converted into flanking stagnation pool tasks and continue to be executed using idle resources. There will be no new resource contention due to the replacement operation.

[0108] The above implementation method uses a multi-scheme neutron transport calculation scenario in a nuclear reactor core as an example to describe in detail the heterogeneous supercomputing parallel task allocation method based on streaming job scheduling proposed in this application. This method generates multiple feasible allocation schemes with resource allocation conservatism gradients through a pre-draft strategy and executes them in parallel. It combines a flanking stagnation pool mechanism to achieve dynamic optimization activation of the main scheme, utilizes real-time back-injection of individual deviation signatures and periodic cluster analysis to achieve adaptive correction of the pre-draft strategy, and supplements it with a connectivity index-driven resource ratio adjustment and early warning path replacement mechanism to construct a complete closed-loop scheduling system.

[0109] It should be noted that the parameter values ​​given in the above embodiments, including the initial value, step size, threshold settings, monitoring period, and weighting coefficients of the hysteresis ratio control parameters, are exemplary configurations provided to facilitate understanding and implementation by those skilled in the art. In actual deployment, they can be appropriately adjusted according to the hardware characteristics, typical task load characteristics, and performance optimization goals of the specific supercomputing platform. The scope of protection covered by this application is not limited to the specific application scenarios and parameter configurations described above. Any technical solutions obtained by making equivalent transformations or reasonable modifications to the above embodiments without departing from the core technical concept of this application fall within the scope of protection claimed in this application. After reading the above embodiments, those skilled in the art can apply the scheduling methods and system architecture described therein to other scientific and technological computing fields with similar heterogeneous computing needs and parallel task dependence characteristics without creative effort.

Claims

1. A method for parallel task allocation in heterogeneous supercomputing based on streaming job scheduling, characterized in that, The method includes: Obtain the parallel task flow to be scheduled, parse the data dependency relationship between each task in the parallel task flow, construct the task dependency directed graph, and collect the estimated resource consumption metadata of each task. Based on the task dependency directed graph and the estimated resource consumption metadata, N feasible allocation schemes are generated using a pre-simulation strategy, where N≥2. Each feasible allocation scheme includes the mapping relationship between each task and heterogeneous computing nodes, as well as the resource allocation strategy of each computing node. All feasible allocation schemes are stored in the alternative scheme pool. The N feasible allocation schemes in the alternative scheme pool are simultaneously distributed to N independent computing nodes for parallel execution. The execution progress of each computing node is monitored, and the scheme that first reaches the preset progress threshold on the critical path is activated as the main scheme, while the remaining schemes are terminated or downgraded to flanking buffer tasks. After the main scheme is activated, each group of computing nodes dynamically adjusts the computing resource allocation ratio between the main operation channel tasks and the side storage tank tasks according to the current connectivity index. The connectivity index is dynamically adjusted based on the real-time load of the computing nodes, the backlog of main channel tasks, and the water level of the side storage tanks. During task execution, the actual resource consumption data of each task is collected in real time. The actual resource consumption data is compared with the estimated resource consumption metadata for each task to generate an individual deviation signature for each task on its assigned computing node type. The individual deviation signatures are injected back into the flanking reservoir level regulator of the corresponding computing node to update the reservoir ratio control parameters. When the actual execution delay of the task propagates to the subsequent task along the edge of the task-dependent directed graph, the deviation propagation coefficient is calculated and the weight parameters of the corresponding path in the pre-simulation strategy are corrected. All accumulated individual deviation signatures are periodically clustered to mark unreliable prediction combinations and remove their corresponding computing node types from the candidate resource pool. The pre-simulation strategy is updated with the corrected path weight parameters and the removed candidate resource pool.

2. The method according to claim 1, characterized in that, The process of generating N feasible allocation schemes using a pre-simulation strategy includes: Extract all directed paths from the start node to the end node from the task dependency directed graph, calculate the path weight of each directed path, select the directed path with the largest path weight as the critical path, and mark each task on the critical path as a critical task. Assign heterogeneous computing nodes that meet the estimated resource consumption metadata to each critical task on the critical path as candidate nodes, and generate a candidate node sequence for the critical path. Using the candidate node sequence as a constraint skeleton, non-critical tasks are allocated according to their dependencies and estimated resource consumption metadata to generate a complete task-node mapping scheme, which serves as the feasible allocation scheme. Repeat the critical path marking and the filling allocation operation, and make differential adjustments to the node selection in the candidate node sequence each time, so that the feasible allocation schemes generated in each generation form a gradient difference in the conservatism of resource allocation, until N feasible allocation schemes are generated.

3. The method according to claim 2, characterized in that, The differential adjustment of node selection in the candidate node sequence includes: Set a lag ratio control parameter for each critical task on the critical path. The lag ratio control parameter represents the upper limit of the ratio of the computation amount of the flank lag pool task to the computation amount of the main channel task on the computing node where the critical task is executed. When generating feasible allocation schemes multiple times, the stagnation ratio control parameters of each key task are set to the first preset value and the second preset value, which are increased sequentially, and then increased in increments according to the preset step size until N feasible allocation schemes with different stagnation ratio control parameters are generated. The N feasible allocation schemes are sorted in ascending order according to the values ​​of their respective storage ratio control parameters. The scheme with the smallest storage ratio control parameter value has the highest resource allocation conservatism, and the scheme with the largest value has the lowest resource allocation conservatism.

4. The method according to claim 1, characterized in that, The step of simultaneously distributing N feasible allocation schemes from the candidate scheme pool to N independent computing nodes for parallel execution, monitoring the execution progress of each computing node group, and activating the scheme that first reaches a preset progress threshold on the critical path as the primary scheme includes: Each feasible allocation scheme in the pool of alternative schemes is assigned a set of independent computing nodes, and the resources of each set of independent computing nodes are isolated from each other. Each group of independent computing nodes is instructed to start task execution synchronously according to the corresponding feasible allocation scheme, and each group of independent computing nodes maintains a local progress counter that records the number of critical tasks that have been completed. Each computing node reports the count value of its local progress counter to the central scheduler, which then compares each set of count values ​​with the total number of critical tasks on the critical path corresponding to that set. When the ratio of the count value of any group of independent computing nodes to the total number of critical tasks reaches the preset progress threshold for the first time, the central scheduler marks the feasible allocation scheme currently being executed by that group as the winning scheme, activates it as the main scheme, and sends a termination instruction or a downgrade instruction to the other groups. The termination instruction is used to instruct the immediate cessation of task execution and release of computing resources. The degradation instruction is used to instruct the currently executing task to be converted and marked as a flanking stagnation pool task, which continues to be executed using idle computing resources without participating in the main channel task competition.

5. The method according to claim 1, characterized in that, The periodic clustering analysis of all accumulated individual bias signatures to label unreliable predictive combinations includes: Extract the task type features, node type features, and deviation values ​​from each individual deviation signature, and combine them to form a three-dimensional deviation feature vector; Using a preset time period as the interval, the accumulated three-dimensional deviation feature vectors within the current period are input into the clustering algorithm for unsupervised clustering, and multiple deviation clusters are output. Calculate the cluster center deviation value and intra-cluster sample density for each biased cluster. When the intra-cluster sample density of any biased cluster is higher than a preset density threshold and the absolute value of the cluster center deviation value is greater than a preset deviation tolerance threshold, the biased cluster is marked as a high-bias cluster. Extract the combination of task type features and node type features common to all samples in the high-bias cluster as the unreliable prediction combination; The unreliable prediction combination is removed from the candidate resource pool of the subsequent pre-simulation strategy, and the estimated resource consumption metadata corresponding to the task type characteristics in the pre-simulation strategy is corrected.

6. The method according to claim 1, characterized in that, The step of calculating the deviation propagation coefficient and correcting the weight parameters of the corresponding path in the pre-rehearsal strategy when the actual execution delay of the task propagates along the edges of the task-dependent directed graph to subsequent tasks includes: When the actual end time of any critical task is later than its estimated end time, calculate the actual execution delay of that critical task. Starting with the critical task as the starting node, all its direct and indirect successor tasks are traversed. For each traversed successor task, a weighted deviation propagation coefficient is calculated. The weighted deviation propagation coefficient is equal to the actual execution delay multiplied by the topological hierarchy weight of the successor task on the critical path. The weighted bias propagation coefficient is superimposed onto the path weight parameter corresponding to the subsequent task in the pre-simulation strategy; When the actual execution delay propagates to the terminal node on the critical path, the total cumulative deviation propagation is recorded, and the ratio of the total cumulative deviation propagation to the original path weight of the critical path is determined as the deviation sensitivity index of the critical path. During the execution of the main scheme, the deviation sensitivity index calculated in real time is broadcast to the local early warning monitoring module of each computing node; When the real-time calculated value of the deviation sensitivity index exceeds the preset sensitivity threshold for the first time, the critical path is marked as the current warning path; When the number of times the same critical path is marked as the current warning path within a preset time window reaches a preset frequency threshold, the critical path is marked as a historical high-sensitivity path, and in subsequent rehearsal strategies, its critical tasks are assigned an initial value of the hysteresis ratio control parameter that is higher than that of other critical tasks.

7. The method according to claim 6, characterized in that, When any path is marked as the current warning path, the following replacement operation is also triggered: When the proportion of the number of completed key tasks on the current warning path reaches the first monitoring threshold, the first deviation rate between the actual cumulative execution time and the estimated cumulative execution time of the completed key tasks is calculated. When the first deviation rate exceeds the preset deviation rate threshold, retrieve whether there are unfinished task segments from the flanking stagnation pool of each current computing node from the feasible allocation scheme that has been terminated or downgraded. If such a fragment exists, an unfinished task fragment with a higher resource allocation conservatism than the main scheme is selected as a candidate replacement fragment, and the candidate replacement fragment is used to replace the subsequent critical tasks that have not yet been executed on the current warning path. Record the execution time and deviation rate improvement of this replacement operation, and use them as feedback parameters to inject back into the deviation sensitivity index calculation module in the pre-simulation strategy; When the deviation rate of the current early warning path improves to the preset improvement target through this replacement operation, the marker is cleared from the early warning record cache, and the marker is not included in the statistics of the preset frequency threshold within the preset time window.

8. The method according to claim 1, characterized in that, The computing nodes in each group dynamically adjust the computing resource allocation ratio between the main operation channel tasks and the flanking stagnation pool tasks according to the current connectivity index, including: Each computing node independently maintains its local flank storage tank water level value, which is equal to the ratio of the estimated total computational load of all flank storage tank tasks currently queued for execution in the local flank storage tank of that computing node to the preset water level capacity. Each computing node collects the real-time load rate and main channel task backlog of its local computing node, inputs the water level of the flank storage tank, the real-time load rate and the main channel task backlog into a local connectivity index calculation function, and outputs the connectivity index value at the current moment. Each computing node compares the connectivity index value with preset high connectivity thresholds and low connectivity thresholds. When the connectivity index value is higher than the high connectivity threshold, it is determined to be in a main channel priority state, and the computing resource allocation ratio for the next monitoring cycle is set to a first ratio value, wherein the computing resource allocation ratio for the main operation channel tasks is higher than that for the flanking stagnation pool tasks. When the connectivity index value is lower than the low connectivity threshold, it is determined to be in a flanking priority state, and the computing resource allocation ratio for the next monitoring cycle is set to a second ratio value, wherein the computing resource allocation ratio for the flanking stagnation pool tasks is higher than that for the main operation channel tasks. When the connectivity index value is between the high connectivity threshold and the low connectivity threshold, it is determined to be in a balanced state, and the current computing resource allocation ratio remains unchanged.

9. The method according to claim 1, characterized in that, The step of reinjecting the individual deviation signature into the flanking reservoir water level regulator of the corresponding computing node to update the reservoir ratio control parameters includes: Each computing node's flanking refrigeration tank level regulator has a built-in refrigeration ratio control parameter register that stores the current refrigeration ratio control parameter value of the computing node. The refrigeration ratio control parameter value is used to control the maximum injection rate of the flanking refrigeration tank task during the execution of the main operation channel task by the computing node. When any task on the computing node is completed and a corresponding individual deviation signature is generated, the deviation value in the individual deviation signature is extracted. The deviation value is equal to the actual resource consumption of the task minus the estimated resource consumption. The deviation value is compared with a preset deviation dead zone threshold. When the absolute value of the deviation value is less than or equal to the deviation dead zone threshold, the current hysteresis ratio control parameter value is maintained unchanged. When the deviation value is positive and its absolute value is greater than the deviation dead zone threshold, the hysteresis ratio control parameter value is adjusted to increase by a first adjustment step size; when the deviation value is negative and its absolute value is greater than the deviation dead zone threshold, the hysteresis ratio control parameter value is adjusted to decrease by a second adjustment step size. The adjusted hysteresis ratio control parameter value is written into the hysteresis ratio control parameter register, replacing the original hysteresis ratio control parameter value.

10. A heterogeneous supercomputing parallel task allocation system based on streaming job scheduling, characterized in that, include: One or more processors; A machine-readable storage medium, connected to the processor, is used to store executable instructions of the processor; when the executable instructions are executed by the one or more processors, the system performs the heterogeneous supercomputing parallel task allocation method based on streaming job scheduling as described in any one of claims 1 to 9.

Images