Large model reasoning scheduling method based on off-network computing power server

Abstract

The invention relates to the technical field of server scheduling, and discloses a large model reasoning scheduling method based on an off-network computing power server, which comprises the following steps of: collecting hardware information of the off-network computing power server, and constructing a server resource vector; decomposing a large model reasoning task into a plurality of sub-tasks, and constructing task modeling; according to the hardware information of the off-network server and the task dependency relationship, performing task division by using an optimization method; performing dynamic task scheduling on the sub-tasks based on the current load state of the server; according to the real-time calculation load of the server, dynamically migrating part of tasks to other servers to ensure the load balance among the servers; and executing the distributed task, monitoring the running state of the server and the task completion condition, and dynamically adjusting a task distribution strategy according to feedback data. Through modeling task requirements and server resources, in combination with real-time monitoring and optimal scheduling, efficient calculation and resource utilization maximization of large model reasoning in an off-network environment are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of server scheduling, and specifically to a large model reasoning scheduling method based on off-grid computing servers. Background Art

[0002] With the rapid development of artificial intelligence technology, the widespread application of large models (such as Transformer, GPT, etc.) in natural language processing, computer vision and other fields has led to an exponential growth in their demand for reasoning computing. The reasoning process of large models usually involves a large number of complex computing operations, especially matrix operations, attention mechanism calculations, etc. These tasks not only require high-performance computing resources, but also place higher requirements on task allocation and scheduling strategies. In the traditional centralized computing environment, such large-scale reasoning tasks are usually completed through a stable network connection and a cloud computing cluster. However, as more task scenarios extend to off-grid edge devices and distributed servers, the management of this computing demand has become more complicated.

[0003] Generally speaking, under good network conditions, task scheduling can rely on centralized control and global information support, and scheduling strategies can be adjusted dynamically in real time to achieve multi-node collaborative computing. However, in an off-grid computing server environment, it is difficult to achieve efficient collaboration between computing servers due to the lack of a stable network connection. In addition, server hardware resources in an off-grid environment are usually heterogeneous (such as different CPU, GPU performance, and memory capacity), and task allocation and scheduling require more sophisticated optimization strategies. Traditional scheduling methods often do not accurately model resource requirements, making it difficult to meet the diverse computing needs in an off-grid environment, resulting in reduced computing efficiency.

[0004] Existing scheduling technologies generally rely on a stable network environment to collect server resource information and task progress status in real time in off-grid or weak network environments. When the network is unstable or missing, it is difficult for the scheduling system to obtain the task execution status in a timely manner, resulting in delayed task allocation and unreasonable scheduling. In addition, due to the lack of accurate resource demand modeling and dynamic feedback optimization, existing technologies usually adopt static strategies for task allocation, which fail to fully consider the real-time changes in server load during task execution. The problems of idle resources and single-point overload are difficult to avoid, resulting in low overall resource utilization.

[0005] In addition, in multi-server collaborative computing, the data communication cost between tasks is ignored or not fully optimized, which further increases the overall delay of computing. Especially in large model reasoning scenarios with complex task dependencies, data transmission delays may even become the main bottleneck of the system. It can be seen that the existing technology lacks a set of efficient scheduling mechanisms that can achieve dynamic resource allocation, task optimization and load balancing in the off-grid computing server environment, and it is difficult to adapt to complex computing needs and resource distribution characteristics.

[0006] Therefore, the present invention proposes a large model reasoning scheduling method based on an off-grid computing server to solve the deficiencies of the prior art. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a large model reasoning scheduling method based on off-grid computing power servers, which solves the problems of unreasonable resource allocation of large model reasoning tasks, low dynamic scheduling efficiency and load imbalance caused by the lack of a stable network in the off-grid computing power server environment.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: a large model reasoning scheduling method based on an off-grid computing server comprises the following steps: S1. Collect the hardware information of the off-grid computing server and build a server resource vector; S2, decompose the large model reasoning task into multiple subtasks and build task modeling; S3, dividing tasks using optimization methods based on the hardware information and task dependencies of the off-network server; S4. Dynamically schedule subtasks based on the current load status of the server; S5. Dynamically migrate some tasks to other servers based on the real-time computing load of the server to ensure load balance between servers; S6. Execute the assigned tasks, monitor the server's operating status and task completion status, and dynamically adjust the task allocation strategy based on feedback data.

[0009] Preferably, the task modeling includes: A task-resource matrix is ​​constructed, where the elements of the matrix represent the computational cost required for each task on each server, and the computational cost is calculated based on the CPU requirement, GPU requirement, and memory requirement of the task and the available resources of the server.

[0010] Preferably, the task division includes: The task-resource matrix is ​​decomposed into low-rank by using matrix decomposition method, and the task is divided into multiple sub-task sets; Each subtask set corresponds to a server, and the computing requirements of the subtask set are matched with the available resources of the server.

[0011] Preferably, the matrix decomposition method includes using a tensor decomposition method to optimize the decomposition of the task-resource matrix, and the goal of task division is to minimize the migration cost of tasks between servers and the imbalance of computing load.

[0012] Preferably, the dynamic task scheduling is implemented based on a game theory optimization model, and the game theory optimization model includes the following contents: Model each server as a player in the game; Each player’s utility function consists of two parts: the computational cost of the subtask and the ratio of the server’s current load to its total resource capacity; Assign tasks according to the maximization result of the utility function and design load balancing adjustments.

[0013] Preferably, the calculation of the utility function includes: The computational cost of the task on the server; The ratio of the current server load to the total server computing power; The weight factor of task scheduling is used to balance the computing cost and load balancing.

[0014] Preferably, the load balancing adjustment is implemented through a dynamic load balancing model, which determines the dynamic migration strategy of the task according to the current load of the server and the task migration cost.

[0015] Preferably, the execution of the assigned tasks includes: Real-time monitoring of server operating status, including CPU, GPU and memory utilization; Adjust the task allocation strategy according to the server's operating status, and prioritize allocating new tasks to servers with lower loads; Reallocate unfinished tasks to ensure the system's fault tolerance.

[0016] Preferably, the load balancing model uses a differential dynamic system to model the dynamic changes of server loads. The changes in server loads consist of three parts: load attenuation, spontaneous balancing, and task migration. The ultimate goal is to make the loads of all servers approach the global average load.

[0017] The present invention also provides a scheduling system for a large model reasoning scheduling method, comprising the following modules: Task initialization module, used to collect server hardware information and demand information of large model tasks; Task division module, used to optimize the task division and generate a set of subtasks; Dynamic scheduling module, used to monitor the server load status in real time and dynamically allocate tasks; Load balancing module, used to dynamically migrate tasks according to the load balancing model; The execution and feedback module is used to execute tasks and monitor the completion of tasks in real time, and provide feedback data to optimize scheduling strategies.

[0018] The present invention provides a large model reasoning scheduling method based on an off-grid computing server. It has the following beneficial effects: 1. This invention achieves efficient matching of tasks and resources by accurately modeling server resource status and task requirements, and combining tensor decomposition and game theory to optimize dynamic scheduling strategies. Through dynamic load migration and task scheduling optimization, the performance degradation of a single server due to resource bottlenecks is effectively avoided, and the overall efficiency of large model reasoning and the utilization of computing resources are significantly improved.

[0019] 2. Aiming at the problem that off-grid computing servers cannot rely on centralized network scheduling, the present invention builds a distributed scheduling mechanism independent of the network environment. Through localized task decomposition, division and dynamic adjustment strategies, the system can run efficiently in the absence of network or instability, and has strong environmental adaptability.

[0020] 3. The present invention dynamically adjusts the task distribution of the server by real-time monitoring of the server load status, combining the differential dynamics load balancing model and task migration strategy, effectively avoiding the problems of single-point overload and idle resources, while improving the stability of task execution and the overall robustness of the system.

[0021] 4. The present invention adapts to various types of large-model reasoning tasks through task priority scheduling, real-time feedback optimization and dynamic concurrency adjustment mechanism. Whether it is a task with high real-time requirements or a large-scale reasoning task that requires high computing power support, the present invention can flexibly adjust the scheduling strategy to ensure the rapid completion of tasks and efficient use of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of the steps of the large model reasoning scheduling method based on the off-grid computing server of the present invention; Figure 2 This is a module framework diagram of the large model reasoning and scheduling system based on off-grid computing power servers of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Please refer to the attached Figure 1 The embodiment of the present invention provides a large model reasoning scheduling method based on an off-grid computing server, comprising the following steps: S1. Collect the hardware information of the off-grid computing server and build a server resource vector; In an off-grid computing server environment, in order to achieve efficient scheduling of large model inference tasks, the system first needs to understand the resource status of the computing server. This step is the basis for subsequent task decomposition, division, and dynamic scheduling. By comprehensively collecting and modeling the server's hardware information, it can provide the necessary data support for resource matching and task allocation. In general, there is no direct network connection between off-grid servers, so the collection of hardware information and the initialization of resource status need to be completed locally. To ensure the accuracy of scheduling, the information collected in this step includes not only static hardware parameters, but also needs to be combined with real-time resource status to provide dynamic support for subsequent steps.

[0025] In this embodiment, the hardware information of the off-grid computing server mainly includes the CPU computing power, GPU computing power and memory capacity of each server. This information is collected through hardware monitoring tools, and the specific implementation methods include calling system-level API interfaces, parsing server hardware configuration files, or directly running test scripts. For example, the CPU computing power can be obtained by reading the number of logical cores and main frequency information of the system; the GPU computing power can be estimated based on the floating-point computing capability of the GPU (such as FLOPS); the memory capacity is confirmed by reading the physical memory size of the server.

[0026] Specifically, the hardware resources of each server can be modeled as a resource vector:

[0027] in, is the total CPU power of the server, is the total GPU computing power of the server, in floating point operations. The total memory capacity of the server.

[0028] In one possible implementation, a global resource matrix can be constructed by collecting hardware information from multiple servers in the cluster.

[0029]

[0030] Each row of the matrix contains the resource information of a server. Servers As an option, in order to better adapt to the real-time requirements of task allocation, the collected hardware information can also be dynamically updated in combination with the current resource usage status of the server. Generally, the real-time resource usage information of the server includes the current CPU utilization, GPU utilization, and memory usage, which are respectively , , In this way, the available resource vector of the server can be further defined as:

[0031] in, is the CPU computing power currently available on the server, is the GPU computing power currently available on the server, The current available memory capacity of the server In a possible implementation, the real-time load data of the server can be obtained through a hardware monitoring tool. Such real-time data can be used for subsequent dynamic task scheduling and load balancing operations.

[0032] In addition, in order to improve the accuracy of information collection, in this embodiment, the collection tool can sample the resources multiple times and calculate the average value. For example, the GPU computing power can be estimated by monitoring the average value of 10 samples of the floating-point operation peak value:

[0033] in, For the GPU floating point computing performance at multiple sampling times. Through multiple sampling, the impact of instantaneous fluctuations can be effectively filtered out to obtain more stable resource information.

[0034] In some embodiments, the resource information of the server can also be normalized in combination with the task requirement characteristics. Specifically, the hardware resource vector of the server can be normalized to a range of 0 to 1 to better adapt to the task allocation algorithm. For example, the normalized resource vector can be defined as:

[0035] in, 、 、 They are the maximum resource values ​​of CPU, GPU and memory in the cluster respectively. is the normalized resource vector.

[0036] In another implementation, in order to support multiple types of tasks, the server resource information can be further expanded. For example, for tasks that require storing large amounts of data, disk / Performance parameters This extension can provide more dimensional data support for the task allocation algorithm, making the allocation strategy more flexible.

[0037] Through the above methods, the collection and modeling process of server hardware information is completed. Subsequent steps will use this resource information to reasonably decompose and allocate large model reasoning tasks, thereby achieving optimal resource utilization and efficient task scheduling.

[0038] Step S2: Decompose the large model reasoning task into multiple subtasks and construct task modeling; The main task of this step is to decompose the complex large model reasoning process into multiple subtasks and build a computing demand model for each subtask. Through the decomposition and modeling of subtasks, the resource distribution characteristics of the off-grid computing server can be better matched, thereby providing data support for subsequent task division, dynamic scheduling and other links. Since the hardware information of the off-grid computing server has been collected in step S1, this step needs to be refined and modeled according to the resource status of the server and the logical characteristics of the task itself.

[0039] In general, large model reasoning tasks can be decomposed into multiple computing stages, which may involve modules such as feedforward network calculations, attention mechanisms, and nonlinear activation function processing. The computing requirements of each module have different characteristics, and there are dependencies between some computing modules. For example, attention calculations need to be performed after the feedforward network calculations are completed. In order to clearly describe these logical relationships, the present invention comprehensively represents the execution characteristics of each subtask and its demand for resources by constructing a task dependency graph and a task requirement vector.

[0040] In some embodiments, this step also needs to consider the dynamics of the tasks during the large model reasoning process. Since the length and shape of the model input and the intermediate results generated during the reasoning process may affect the computing requirements, the task modeling needs to have a certain degree of scalability and flexibility to ensure that the decomposed subtasks can adapt to different computing nodes.

[0041] The large model reasoning task is first decomposed into a series of interrelated subtasks. Each subtask is modeled by a task demand vector, which represents the demand for computing resources during the calculation process of the subtask. The specific definition is as follows:

[0042] in, is the resource requirement vector of the subtask, The CPU resource requirement of the task, usually expressed in the form of the number of cores. The demand for GPU resources for the task, The memory resource requirement of the task.

[0043] Specifically, the acquisition of the computational demand vector can be determined in the following way: According to the structural characteristics of the large model, analyze the computational complexity of each task module. For example, the feedforward network is usually a matrix multiplication operation, and its computational requirements are directly related to the input dimension and the size of the weight matrix; Combined with the hardware architecture characteristics of the off-grid computing server, the requirements of each subtask for CPU, GPU and memory resources are further refined. For example, convolution operations may mainly rely on the GPU, while activation function processing relies more on the CPU; In some embodiments, the actual resource consumption of different task modules on the server can be measured by offline testing to obtain a more accurate task demand vector.

[0044] In order to represent the dependency relationship between subtasks, the present invention further constructs a task dependency graph (DAG graph). The DAG graph consists of nodes and directed edges, where: Each node represents a subtask; each directed edge represents the execution dependency between subtasks. The output of the task is , then add a slave node to the DAG graph. Point to Node edge.

[0045] As an option, the generation of the DAG graph can refer to the network structure of the large model. For example, in the reasoning process of the Transformer model, the attention mechanism needs to rely on the output of the input embedding layer, so the subtask nodes of the attention layer should rely on the embedding layer nodes. In this case, the structure of the DAG graph can not only reflect the order between tasks, but also provide an important basis for the subsequent task division.

[0046] In a possible implementation, the task decomposition process can also be combined with the technical features of dynamic task modeling. For input sequences with variable lengths (such as natural language processing tasks), the computational requirements of subtasks may be adjusted as the input length changes. To adapt to this feature, the present invention proposes a task decomposition method based on dynamic modeling: First, according to the length of the input sequence , parameterize the resource requirements of each subtask. For example, for matrix multiplication operations, its GPU requirements can be expressed as:

[0047] in , The parameters related to the task module are determined through experiments; then, the subtask division strategy is adjusted according to the dynamic requirements of the task. For example, for long sequence inputs, the matrix multiplication task can be further decomposed into multiple subtasks to reduce the resource usage of a single task.

[0048] In some embodiments, the task modeling process of the present invention also includes an adaptability analysis of task requirements. Specifically, the subtask requirement vector is matched with the server resource vector obtained in step S1 to determine the execution environment suitable for each subtask. The calculation formula for the matching analysis is as follows:

[0049] in, For server tasks The suitability score of , , are the CPU, GPU and memory resources of the server respectively. , , Respectively for tasks Requirements for CPU, GPU, and memory.

[0050] By calculation , which can provide a reference for subsequent task division, so that each task can be assigned to the most suitable server for execution.

[0051] In addition, the modeling of communication requirements must also be considered during task modeling. Communication requirements are usually expressed as data dependencies between tasks, defined as:

[0052] in For the task Output to the task Modeling of communication requirements for data volumes can help determine the task division scheme to minimize data transmission across servers. Through the above task decomposition and modeling process, the present invention lays a solid foundation for subsequent task division and scheduling. The computational requirement vector of the subtask and the task dependency graph can not only accurately describe the resource requirements of the task, but also reflect the logical relationship between tasks, thereby providing support for the dynamic allocation of tasks. The above content can be further expanded in combination with specific implementation scenarios to adapt to different types of large model reasoning tasks.

[0053] S3, dividing tasks using optimization methods based on the hardware information and task dependencies of the off-network server; In the technical solution of the present invention, the main task of step S3 is to further divide the task into multiple subtask sets based on the server hardware information collected in step S1 and the task model constructed in step S2. Each subtask set consists of related subtasks and is assigned to a specific server for execution. The goal of task division is to achieve the best match between computing resources and task requirements, while minimizing the communication cost across servers, thereby providing a basis for subsequent dynamic scheduling.

[0054] In general, task division needs to consider the computing requirements of the task, the available computing power of the server, and the dependencies between tasks. For example, for GPU-intensive tasks, they need to be assigned to servers with higher GPU computing power first; for tasks with a large amount of data dependencies, they should be assigned to the same server as much as possible to reduce the communication overhead of data transmission. As an option, this step optimizes the calculation by constructing a task-resource matrix and implements task division using methods such as tensor decomposition. Specifically, the task division in this step must meet the following conditions: The resource requirements of each subtask set cannot exceed the available computing power of the target server; Minimize the amount of data transferred across servers; Follow the dependencies of tasks.

[0055] In order to achieve the best match between tasks and server resources, we first build a task-resource matrix , the matrix is ​​defined as follows:

[0056] in, For the task The computational cost on the server, , , Respectively for tasks CPU requirements, GPU requirements, and memory requirements , , They are the server's CPU computing power, GPU computing power, and memory capacity.

[0057] In some embodiments, the task-resource matrix can also be expanded into a dynamic matrix to add the currently available resource information of the server.

[0058] In order to optimize the task division results, the task-resource matrix The optimization can be performed by tensor decomposition. Specifically, the task-resource matrix is ​​decomposed into two low-rank matrices:

[0059] in, is the task feature matrix, describing the task in The performance of potential calculation factors, is the server feature matrix, describing the server The ability of potential computing factors, is the rank of matrix decomposition, satisfying .

[0060] Through matrix decomposition, we can extract the potential matching relationship between tasks and servers, and thus divide the tasks into several subtask sets. The specific division method is:

[0061] in For the set of tasks assigned to the server, the goal of task division is to minimize all The sum of .

[0062] In order to further reduce the communication overhead of task partitioning, this embodiment also considers the impact of task dependencies on the partitioning results. The dependencies between tasks can be expressed through the communication demand matrix Represents that the matrix element Defined as:

[0063] in For the task Output to the task The amount of data.

[0064] When dividing tasks, tasks with close dependencies are assigned to the same server first. The communication optimization goal can be expressed as follows:

[0065] in, is the communication weight coefficient, which is used to balance the computational cost and the communication cost. They are respectively the task sets assigned to the servers.

[0066] In order to improve the flexibility of task division, the present invention can adopt a multi-stage division strategy. First, tasks are divided into coarse-grained task groups, and the tasks in each task group have a strong dependency relationship and are suitable as the same subtask set. Then, according to the resource requirements of each task group, it is further refined to ensure that each subtask set can be efficiently executed on the target server.

[0067] For example, in the Transformer structure of a large model, coarse-grained division can be performed according to the hierarchy of the model, and the feedforward network and attention mechanism of each layer can be regarded as a task group. Subsequently, based on the GPU computing power of the server, the task group is further divided into smaller computing units to be assigned to different servers for execution. The result of task division will directly affect the subsequent scheduling efficiency. Therefore, this embodiment also combines task adaptability analysis to ensure that the divided subtask set can meet the resource conditions of the target server. The scoring function of adaptability analysis is as follows:

[0068] in, For server tasks The suitability score of , , are the available resources of the server, , , are the resource requirements of the tasks respectively.

[0069] By calculating the adaptability score, the task division results can be further optimized to ensure maximum resource utilization. Through the above task division method, the present invention realizes the comprehensive optimization of computing cost, communication cost and task dependency, and provides a high-quality initial task distribution scheme for subsequent dynamic scheduling and load balancing. The result of task division not only improves the utilization efficiency of resources, but also reduces the data transmission overhead across servers. The above method can be flexibly adjusted according to the specific server configuration and task requirements.

[0070] S4. Dynamically schedule subtasks based on the current load status of the server; In the technical solution of the present invention, step S4 is a key link for dynamically optimizing the initial task division result generated by step S3. Since the hardware resource status of the off-grid computing power server may fluctuate dynamically over time, for example, the load of some servers increases or the resources are insufficient during the execution process, the initial task division scheme may no longer apply. The purpose of this step is to monitor the load status of each server in real time, and dynamically schedule subtasks according to the monitoring data to ensure the rationality of task allocation, thereby further improving the efficiency of task execution. In general, dynamic task scheduling needs to comprehensively consider the current load status of the server, the computing requirements of the task, and the priority of the task. On this basis, the present invention realizes the dynamic scheduling of tasks by constructing a task scheduling model and adopting a game theory optimization strategy. In this design, the server is modeled as a rational player in the game, and the goal of each player is to maximize its own utility function and achieve balanced optimization of task allocation on a global scale.

[0071] In some embodiments, dynamic scheduling also needs to combine the execution priority of tasks and allocate resources to high-priority tasks. In addition, the scheduling system needs to update the server status in real time to cope with sudden resource changes.

[0072] Dynamic task scheduling uses the server load status as the core scheduling basis. Specifically, the real-time load status of the server Defined as:

[0073] in, is the load status of the server. The total resource demand of the currently running task is the total demand of the running task on the CPU, GPU, and memory. The total server resource is the sum of the server's CPU, GPU, and memory.

[0074] Based on the load status of the server, the present invention designs a dynamic scheduling mechanism, the scheduling goal of which is to minimize the load imbalance of all servers while meeting the resource requirements of the task.

[0075] Specifically, In this embodiment, a game theory optimization model for task scheduling is constructed. The main contents of the game model include: Player Definition: Each server is modeled as a player in the game.

[0076] Strategy space: Each player’s strategy is the set of tasks assigned to him / her For each task , players can choose whether to accept the task.

[0077] Utility function: The utility function of each server is defined as:

[0078] in, Indicates the task The computation cost on the server is defined by the task-resource matrix in step S3. Indicates the server load status. Indicates the total computing capacity of the server. is a weight factor used to balance the task computation cost and the optimization goal of load balancing.

[0079] By maximizing the utility function, each server will give priority to tasks that suit its own resource conditions when allocating tasks, while avoiding excessive load on itself.

[0080] In one possible implementation, dynamic scheduling of tasks can be achieved through the following process: First, get the real-time load status of the server ; Then, based on the resource requirements of the task and the available resources of the server Calculating the suitability score , the formula is as follows:

[0081] in, For the task Suitability score on the server, , , They are respectively the CPU, GPU and memory resources currently available on the server. , , Respectively for tasks CPU, GPU, and memory requirements.

[0082] Based on the scoring results, tasks with higher adaptability are preferentially assigned to suitable servers.

[0083] When the load on some servers reaches a bottleneck (for example > 0.8), in this embodiment, the task migration mechanism will be triggered to migrate some tasks on the high-load server to the low-load server. The decision basis for task migration is the migration cost Compatibility score with the target server The migration cost is defined as follows:

[0084] in, For the task The amount of data transfer, , , For task resource requirements, , , The total server resources.

[0085] In some embodiments, dynamic scheduling can also be combined with the execution priority of tasks. Tasks with high priority will be assigned to servers with lower loads. For example, for tasks with high real-time requirements, their priority Can be set to a higher value. The dynamic scheduling model combined with priority can be expressed as:

[0086] By introducing task priorities, the dynamic scheduling system can flexibly adjust the allocation strategy according to the importance of the task to ensure that critical tasks are completed first.

[0087] Through the above dynamic task scheduling mechanism, the present invention effectively solves the problem of task allocation of off-grid computing servers in load fluctuation environments. This method can not only dynamically adapt to resource changes, but also maximize the resource utilization of the server, while ensuring the real-time execution of tasks and the balance of the overall system.

[0088] S5. Dynamically migrate some tasks to other servers based on the real-time computing load of the server to ensure load balance between servers; In the present invention, the main function of step S5 is to achieve dynamic load balancing between off-grid computing servers. Since the dynamic scheduling mechanism of step S4 may not be able to completely avoid load imbalance, especially during task execution, some servers are close to resource bottlenecks due to task accumulation, while other servers may have more idle resources. Therefore, this step monitors the load status of each server in real time, calculates the migration strategy based on the load dynamic model, and migrates some tasks on high-load servers to low-load servers to ensure that the computing resources of the entire cluster are fully utilized.

[0089] In general, load migration needs to comprehensively consider the following factors: the real-time load status of the server, the communication cost of task migration, and the resource adaptability of the target server. In some embodiments, it is also necessary to combine the dependencies of the tasks to ensure that the migration tasks do not affect the logical order of the overall calculation. In addition, in order to avoid the adverse effects of frequent migration operations on performance, this step sets a certain trigger threshold for the migration conditions. For example, the migration is triggered only when the load status of the high-load server exceeds the preset value.

[0090] In this embodiment, the core of load migration is to monitor the load status of the server in real time and calculate the specific solution of task migration based on the load dynamic model. Defined as:

[0091] In one possible implementation, when the load status of a server Exceeding the set threshold (For example, 0.8), the system will start the load migration mechanism to migrate some tasks on the high-load server to the low-load server.

[0092] Specifically, this embodiment implements task migration through the following three steps: task selection, target server selection, and migration cost calculation.

[0093] From the high-load server task queue, select tasks with high computing requirements for migration, giving priority to the following types of tasks: Long-running tasks; Tasks that require high GPU resources; A task that has weak dependencies on other tasks.

[0094] Specific selection tasks Priority Calculated by the following formula:

[0095] in, , , is the weight coefficient, which is set according to the specific application scenario. , , Respectively for tasks GPU, CPU and memory requirements, For the task The amount of communication required with other tasks.

[0096] Prioritize Migration Higher-level tasks can reduce interference with other tasks.

[0097] Select the target server that is most suitable for the migration task from the low-load servers. The target server is selected based on the suitability score. , and its calculation formula is:

[0098] in, Indicates the server's task The suitability score of , , Respectively represent the currently available CPU, GPU, and memory resources of the server.

[0099] Prioritize servers with higher adaptability scores as target servers.

[0100] The ultimate migration goal is to minimize while maximizing .

[0101] In a possible implementation, the load balancing model can further describe the dynamic changes of the server load through differential dynamics equations. The load change of the server can be expressed as:

[0102] in, is the natural attenuation coefficient of the server load after the task is completed, For Server To the server The speed factor of the migration task, The load increase caused by the newly added tasks.

[0103] Through this equation, the load change trend of the server can be simulated and predicted, providing a basis for decision-making on task migration.

[0104] In some embodiments, task migration also needs to be combined with the priority and execution stage of the task. For tasks with higher priority, it is not recommended to migrate during execution unless the load status of the server where it is located is seriously exceeded (for example, >0.95). In addition, tasks are more suitable for migration in the early stages of execution (such as when they are just assigned), while migration should be avoided as much as possible when the execution is close to completion. The timing of migration can be quantified by the following formula:

[0105] in, is the priority of task migration, is the current execution time of the task, The mid-point in the execution of the task.

[0106] Through this formula, the migration priority of the task can be dynamically adjusted, thereby reducing the impact of migration on the task execution time.

[0107] As an option, this embodiment also supports batch task migration. When the server load exceeds the standard, multiple tasks can be migrated at the same time to quickly relieve the load pressure. The selection basis of batch tasks is the total migration cost. The minimization formula is as follows:

[0108] in, is the set of tasks that need to be migrated from the server. is the migration cost of a single task.

[0109] Through the above method, the load between servers can be quickly balanced to ensure efficient execution of tasks.

[0110] Through the above-mentioned task migration mechanism, the present invention realizes the dynamic adjustment of off-grid computing servers under high load conditions, and further optimizes the global utilization efficiency of computing resources. The task migration process comprehensively considers the communication cost, target adaptability and execution priority, thereby significantly improving the flexibility and robustness of the system while ensuring the quality of task completion. The mechanism can be further expanded and optimized according to different task scenarios to adapt to different resource distribution and computing requirements.

[0111] S6. Execute the assigned tasks, monitor the server's operating status and task completion status, and dynamically adjust the task allocation strategy based on feedback data.

[0112] In the technical solution of the present invention, step S6 is the closed-loop part of the entire scheduling process. Based on the resource information collection, task modeling, task division, dynamic scheduling and load balancing implemented in steps S1 to S5, the assigned tasks need to be executed on the off-grid computing server. At the same time, the task execution process is not static, and the server operation status may be dynamically affected by external factors or task progress. Therefore, real-time monitoring of the status of task execution and dynamic adjustment of the scheduling strategy based on the monitoring results are key links to improve system efficiency and robustness.

[0113] Generally speaking, the monitoring of the task execution process includes two core aspects: one is the running status of the server, which is mainly reflected in the utilization of hardware resources such as CPU, GPU and memory; the other is the completion of the task, such as the execution progress of the current task and the queue of unfinished tasks. As an option, feedback data can be collected in real time through the resource monitoring module and provide feedback to the allocation strategy of the scheduling system.

[0114] Specifically, this step implements a closed-loop optimization method with dynamic adjustment through monitoring and feedback mechanism to cope with abnormal situations in task execution and uneven resource distribution.

[0115] In this embodiment, the execution of tasks on off-grid computing servers is determined by the allocation scheme generated by the task scheduling module. Each server executes the assigned subtasks according to the scheduling results, and reports its operating status and task progress to the monitoring module in real time during the execution process.

[0116] In order to fully understand the task execution process, a monitoring parameter set is designed in this step to reflect the server's operating status and task completion status in real time. Monitoring parameters include but are not limited to: : The CPU utilization of the server, expressed as the proportion of CPU resources occupied by the currently running tasks; : The GPU utilization of the server, expressed as the proportion of GPU computing power occupied by the current running task; : The memory utilization of the server, expressed as the proportion of memory resources occupied by the currently running tasks; :Task The execution progress on the server is expressed as the ratio of the completed computational amount of the task to the total computational amount.

[0117] In some embodiments, the monitoring module can obtain the above parameters through built-in hardware sensors and system APIs. For example, the utilization of the GPU Can be collected in real time through runtime tools (such as NVIDIA-smi command), memory utilization It can be obtained through the memory management interface of the operating system.

[0118] In this step, the monitoring data will be transmitted to the feedback module of the scheduling system in real time. The feedback module analyzes the running status and task completion of the server and dynamically adjusts the scheduling strategy based on the analysis results.

[0119] The adjustment strategies of the feedback module include but are not limited to the following situations: For the assigned tasks, when their execution progress Falling behind expectations (e.g. ), the feedback module will increase the priority of the task to ensure it is completed as soon as possible.

[0120] When the server hardware utilization approaches the bottleneck (e.g. or ), the feedback module triggers the task migration mechanism and reallocates some tasks to servers with low load.

[0121] For tasks that are still in the queue, the feedback module will optimize the resource distribution of the server based on the monitoring data to shorten the waiting time of unfinished tasks.

[0122] In one possible implementation, the scheduling system can achieve balanced utilization of server resources by dynamically adjusting the task allocation ratio. , each task The resource requirement is To optimize the distribution ratio, the system calculates the load weight of each server. , the formula is as follows:

[0123] in , , are the current hardware utilization of the server respectively; , , They are the total resources of the server respectively.

[0124] The system prioritizes assigning new tasks to load weights Minimum number of servers to achieve load balancing.

[0125] As an option, the monitoring module also supports fault detection and fault tolerance mechanisms. For example, when a server stops running due to hardware failure or abnormal status, the monitoring module will trigger an alarm and migrate all unfinished tasks to other servers. Fault detection criteria may include: the resource utilization of the server is continuously zero (such as and ); The execution progress of the task remains unchanged for a long time (such as remain constant over a period of time).

[0126] In this case, the feedback module will prioritize allocating the affected tasks to the currently optimal server and recalculate the global scheduling plan.

[0127] In this case, the feedback module will prioritize allocating the affected tasks to the currently optimal server and recalculate the global scheduling plan.

[0128] Through the above-mentioned task execution and feedback mechanism, the present invention establishes a closed-loop optimization system. The system monitors the server status and task progress in real time during the task execution process, and ensures the efficient operation of the system by dynamically adjusting the allocation strategy. This design not only improves the flexibility and robustness of task execution, but also can adapt to different computing needs in an environment where resources change dynamically. The above content can be further expanded and optimized in combination with specific application scenarios to cope with different types of tasks and system architectures.

[0129] In summary, the present invention provides a large-model reasoning scheduling method based on off-grid computing power servers, and constructs an intelligent scheduling mechanism suitable for off-grid environments through systematic resource information collection, task decomposition modeling, task division, dynamic scheduling, load balancing, and feedback adjustment. This method achieves efficient matching of tasks and resources by accurately modeling server resource status and task requirements, and optimizes the task execution process through dynamic load migration and real-time feedback, thereby improving reasoning speed, resource utilization, and system stability. The present invention effectively solves the problems of resource imbalance and task scheduling complexity in view of the computing power distribution characteristics of off-grid environments, and has broad application prospects and technical feasibility.

[0130] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A large model reasoning scheduling method based on an off-grid computing server, characterized in that: The following steps are involved: Collect the hardware information of off-grid computing servers and build server resource vectors; Decompose the large model reasoning task into multiple subtasks and build task modeling; According to the hardware information and task dependencies of the off-grid server, the task is divided using the optimization method; Dynamically schedule subtasks based on the current load status of the server; Dynamically migrate some tasks to other servers based on the real-time computing load of the server to ensure load balance between servers; Execute assigned tasks, monitor the server's operating status and task completion status, and dynamically adjust task allocation strategies based on feedback data.

2. The large model reasoning scheduling method based on off-grid computing server according to claim 1 is characterized in that: The task modeling includes: Construct a task-resource matrix, where the elements of the matrix represent the computational cost required for each task on each server, and the computational cost is calculated based on the CPU requirement, GPU requirement, and memory requirement of the task and the available resources of the server; Use directed acyclic graphs to describe the dependencies between tasks.

3. The large model reasoning scheduling method based on off-grid computing server according to claim 1 is characterized in that: The task division includes: The task-resource matrix is ​​decomposed into low-rank by using matrix decomposition method, and the task is divided into multiple sub-task sets; Each subtask set corresponds to a server, and the computing requirements of the subtask set are matched with the available resources of the server.

4. The large model reasoning scheduling method based on off-grid computing server according to claim 3 is characterized in that: The matrix decomposition method includes using a tensor decomposition method to optimize the decomposition of the task-resource matrix, and the goal of task division is to minimize the migration cost of tasks between servers and the imbalance of computing load.

5. The large model reasoning scheduling method based on off-grid computing server according to claim 1 is characterized in that: The dynamic task scheduling is implemented based on a game theory optimization model, which includes the following contents: Model each server as a player in the game; Each player’s utility function consists of two parts: the computational cost of the subtask and the ratio of the server’s current load to its total resource capacity; Assign tasks according to the maximization result of the utility function and design load balancing adjustments.

6. The large model reasoning scheduling method based on off-grid computing server according to claim 5 is characterized in that: The calculation of the utility function includes: The computational cost of the task on the server; The ratio of the current server load to the total server computing power; The weight factor of task scheduling is used to balance the computing cost and load balancing.

7. The large model reasoning scheduling method based on off-grid computing server according to claim 5 is characterized in that: The load balancing adjustment is implemented through a dynamic load balancing model, which determines the dynamic migration strategy of the task according to the current load of the server and the task migration cost.

8. The large model reasoning scheduling method based on off-grid computing server according to claim 1 is characterized in that: The tasks of executing the assignment include: Real-time monitoring of server operating status, including CPU, GPU and memory utilization; Adjust the task allocation strategy according to the server's operating status, and prioritize allocating new tasks to servers with lower loads; Reallocate unfinished tasks to ensure the system's fault tolerance.

9. The large model reasoning scheduling method based on off-grid computing server according to claim 7 is characterized in that: The load balancing model uses a differential dynamic system to model the dynamic changes of server loads. The changes in server loads consist of three parts: load attenuation, spontaneous balancing, and task migration. The ultimate goal is to make the loads of all servers approach the global average load.

10. A scheduling system for a large model reasoning scheduling method, according to any one of claims 1-9, wherein the large model reasoning scheduling method based on an off-grid computing server is characterized in that: Includes the following modules: Task initialization module, used to collect server hardware information and demand information of large model tasks; Task division module, used to optimize the task division and generate a set of subtasks; Dynamic scheduling module, used to monitor the server load status in real time and dynamically allocate tasks; Load balancing module, used to dynamically migrate tasks according to the load balancing model; The execution and feedback module is used to execute tasks and monitor the completion of tasks in real time, and provide feedback data to optimize scheduling strategies.

Images