Large model computing power dynamic allocation method and device, equipment, storage medium and product

Abstract

The application discloses a large model computing power dynamic allocation method and device, equipment, storage medium and product, relates to the computer technical field, and the method comprises receiving a large model task request; task characteristics are extracted according to the large model task request, and the task characteristics are converted into task characteristic vectors;The resource use data of each computing node in the cluster is collected and quantitatively processed to obtain a cluster resource state matrix;Based on the task characteristic vector and the resource state matrix, the optimal computing power allocation scheme is calculated by a multi-objective optimization algorithm;Based on the optimal computing power allocation scheme, a single large model task is allocated to multiple computing nodes and executed in parallel;In the execution process, power self-adaptive adjustment and load balancing processing are carried out.The application solves the problem that the existing static resource allocation mechanism cannot dynamically adjust resource occupation according to task real-time load, resulting in low computing power resource utilization, coexistence of resource idling and contention, improves resource utilization and task execution efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium and product for dynamic allocation of computing power in large models. Background Technology

[0002] Currently, large-scale model computing power allocation systems generally adopt static resource quotas and fixed scheduling strategies. At the hardware level, a fixed number of GPU cards and corresponding CPU and memory resources are usually pre-allocated to each model instance. Each computing node runs the allocated tasks independently, and the resource quota remains unchanged throughout the instance's lifecycle. At the software level, a first-come, first-served scheduling method is adopted, and a uniform resource allocation rule is applied to all types of tasks. Once a task is allocated, the resource usage is not adjusted during execution.

[0003] The aforementioned static allocation and fixed scheduling mechanisms have many drawbacks: the pre-allocation method cannot dynamically adjust resources according to real-time load, resulting in a large amount of idle computing power during off-peak periods and severe resource contention and queuing during peak periods, leading to low overall resource utilization; the use of indiscriminate scheduling strategies for different types of tasks cannot guarantee the quality of service (QoS) and response timeliness of core services, and critical tasks may be blocked by non-critical tasks; for large models with hundreds of billions of parameters, it is difficult to cope with sudden traffic surges, lacks foresight and stability, and the reliability of the system is difficult to guarantee. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, equipment, storage medium and product for dynamic allocation of computing power in large models, which aims to achieve dynamic adaptation of computing power resources to real-time load, realize efficient utilization of computing power, prioritize core tasks, and improve system stability and reliability.

[0005] To achieve the above objectives, this application proposes a method for dynamic allocation of computing power for large-scale models, the method comprising: Receive large model task requests; Extract task features based on the large model task request, and convert the task features into task feature vectors; Real-time collection and quantification of resource usage data from each computing node in the cluster yields a cluster resource status matrix. Based on the task feature vector and resource status matrix, the optimal computing power allocation scheme is calculated using a multi-objective optimization algorithm; Based on the optimal computing power allocation scheme, a single large model task is allocated to multiple computing nodes for parallel execution; During the execution of large model tasks, adaptive adjustment of computing power and load balancing are performed.

[0006] In one embodiment, the step of extracting task features based on the large model task request and converting the task features into a task feature vector includes: Extract basic attribute information of the task based on the large model task request; Predict the computational load of a task using a prediction model trained on historical data; Identify dependencies between tasks to construct a task dependency graph; The basic attribute information, task computation amount, and dependency information are converted into a uniform-dimensional numerical task feature vector and added to the task queue by one-hot encoding and min-max normalization.

[0007] In one embodiment, the real-time acquisition and quantification of resource usage data from each computing node in the cluster to obtain a cluster resource status matrix includes: Real-time collection of resource usage data from each computing node in the cluster; The collected resource usage data is preprocessed for anomaly removal and smoothing. The available computing power index for each computing node is calculated based on the preprocessed data. The resource usage data and available computing power index are quantified to obtain a cluster resource status matrix, which reflects the resource usage and available computing power of each computing node in the cluster in real time.

[0008] In one embodiment, the step of calculating the optimal computing power allocation scheme based on the task feature vector and resource state matrix using a multi-objective optimization algorithm includes: Calculate the overall priority of each task in the task queue based on the task feature vector; Using the cluster resource state matrix as constraints, a multi-objective optimization model is constructed with the goal of maximizing system throughput and minimizing average task response time. The multi-objective optimization model is solved by an optimization algorithm to generate the mapping relationship between each task and computing node and the resource quota scheme. The execution order and resource quota of the tasks are determined according to the comprehensive priority.

[0009] In one embodiment, the step of allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme includes: Based on the generated mapping relationship between each task and computing node and the resource quota scheme, the large model computing graph structure is loaded and parsed. Select the appropriate sharding strategy based on the task type to split the task into multiple sharded tasks with fine granularity; The sharded tasks are mapped to corresponding computing nodes, and data synchronization between the sharded tasks is achieved through a distributed communication framework, enabling the sharded tasks to be executed in parallel.

[0010] In one embodiment, during the execution of a large model task, adaptive adjustment of computing power and load balancing are performed, including: During the execution of the sharding task, the load rate of each computing node is monitored in real time and the system load balance is calculated. When the system load balancing exceeds a preset threshold, a load balancing mechanism is triggered to migrate the incremental sharded tasks on overloaded nodes to lightly loaded nodes, and / or adjust the sharding position of model parameters online to achieve dynamic rebalancing of computing resources.

[0011] In one embodiment, the method further includes: The status of computing nodes and the execution status of sharded tasks are monitored in real time through heartbeat detection and task progress reporting mechanisms. Abnormal events are detected by combining rule detection and machine learning models, and the severity level of the abnormality is determined. Based on the severity level of the abnormality, graded fault recovery measures are implemented.

[0012] In one embodiment, the method further includes: Once all sharded tasks have been completed, the execution results are aggregated and all computing resources used by the tasks are reclaimed. The cluster resource status matrix is ​​then updated, and the task execution data is stored in the historical task database.

[0013] In one embodiment, before receiving the large model task request, the method further includes: Through system initialization and resource registration operations, the hardware configuration information and available resources reported by each computing node are obtained and added to the cluster resource topology map after verification.

[0014] In one embodiment, predicting the computational load of the task using a prediction model trained on historical data includes: By using a gradient boosting tree model trained on historical data, the execution time and resource consumption of a task can be predicted based on the task type, input data size, and model version number.

[0015] In one embodiment, the preprocessing of anomaly removal and smoothing of the collected resource usage data includes: The collected resource usage data is processed by using the 3σ criterion to remove outliers and by using the sliding window averaging method to smooth out transient fluctuations. Time series analysis was performed on the preprocessed data to calculate the moving average, variance, and trend rate of change of resource usage.

[0016] In one embodiment, calculating the available computing power index of each computing node based on the preprocessed data includes: Based on the preprocessed data, a weighted summation method is used to quantitatively evaluate the overall resource status of the computing nodes and calculate the available computing power index of each computing node. The overall resource status includes: GPU memory utilization, GPU computing utilization, CPU utilization, and network bandwidth utilization.

[0017] In one embodiment, calculating the overall priority of each task in the task queue based on the task feature vector includes: Based on the timeliness requirements, business importance, and computational workload of the tasks, the overall priority of each task in the task queue to be processed is calculated.

[0018] In one embodiment, the step of selecting a corresponding sharding strategy based on the task type to finely split the task into multiple sharded tasks includes: For inference tasks, a partitioning strategy combining pipelined parallelism and tensor parallelism is adopted. Pipeline parallelism distributes different layers of the model to different computing nodes, while tensor parallelism partitions a single tensor along different dimensions to different computing nodes. For the training task, a sharding strategy combining data parallelism and model parallelism is adopted. Data parallelism distributes training data to different nodes in batches, while model parallelism distributes model parameters to different nodes.

[0019] In one embodiment, the method further includes at least one of the following: The load balancing mechanism is as follows: when the load balancing degree exceeds a preset threshold, the minimum load priority strategy is adopted to migrate the sharded tasks with long execution time and large amount of computation on the node with the highest load rate to the node with the lowest load rate. The incremental migration includes migrating the execution status and necessary data of the fragment tasks; The segmentation position for adjusting model parameters online is to gradually migrate model parameters without interrupting task execution.

[0020] In one embodiment, the step of performing graded fault recovery measures according to the severity level of the anomaly includes: For minor anomalies, automatic retry processing is performed; For general exceptions, the failed sharded tasks will be reassigned to other normal nodes for execution; For severe anomalies, the faulty node is isolated, and all tasks on the faulty node are migrated to other available nodes, while an alarm message is issued.

[0021] Furthermore, to achieve the above objectives, this application also proposes a dynamic allocation device for large-scale model computing power, the device comprising: The receiving module is used to receive requests for large model tasks; The feature extraction and quantization module is used to extract task features according to the large model task request and convert the task features into task feature vectors. The data acquisition and quantization module is used to collect and quantify the resource usage data of each computing node in the cluster in real time to obtain the cluster resource status matrix. The scheme calculation module is used to calculate the optimal computing power allocation scheme based on the task feature vector and resource status matrix using a multi-objective optimization algorithm; The computing power allocation module is used to allocate a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme. The load balancing module is used to adaptively adjust computing power and balance load during the execution of large model tasks.

[0022] In addition, to achieve the above objectives, this application also proposes a large model computing power dynamic allocation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model computing power dynamic allocation method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the large model computing power dynamic allocation method described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the large model computing power dynamic allocation method described above.

[0025] The technical solutions proposed in this application have at least the following technical effects: receiving large model task requests; extracting task features based on the large model task requests and converting the task features into task feature vectors; collecting and quantifying resource usage data of each computing node in the cluster in real time to obtain a cluster resource state matrix; calculating the optimal computing power allocation scheme based on the task feature vectors and the resource state matrix using a multi-objective optimization algorithm; allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme; and performing adaptive adjustment of computing power and load balancing during the execution of large model tasks. The technical solutions of this application realize a complete computing power scheduling process from task reception, feature extraction, resource monitoring, scheduling decision-making, parallel execution to dynamic adjustment, solving the problem that existing static resource allocation mechanisms cannot dynamically adjust resource usage according to real-time task load, resulting in low computing power resource utilization and the coexistence of resource idleness and contention, significantly improving the overall utilization rate of cluster resources and task execution efficiency. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the large-scale model computing power dynamic allocation system module involved in the embodiments of this application; Figure 2 This is a schematic diagram of the large model load perception and feature extraction module involved in the embodiments of this application; Figure 3 This is a schematic diagram of the computing resource status monitoring and quantification module involved in the embodiments of this application; Figure 4 This is a schematic diagram of the dynamic allocation decision and priority scheduling module involved in the embodiments of this application; Figure 5 This is a flowchart illustrating the first embodiment of the dynamic allocation method for large-scale computing power in this application. Figure 6 This is a schematic diagram of the overall process of an embodiment of the dynamic allocation method for large-scale computing power in this application; Figure 7 This is a schematic diagram of the module structure of the large-scale computing power dynamic allocation device in an embodiment of this application; Figure 8This is a schematic diagram of the hardware operating environment involved in the dynamic allocation method of large model computing power in the embodiments of this application.

[0029] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0032] The main solution of this application is as follows: receiving large model task requests; extracting task features based on the large model task requests and converting the task features into task feature vectors; collecting and quantifying resource usage data of each computing node in the cluster in real time to obtain a cluster resource status matrix; calculating the optimal computing power allocation scheme based on the task feature vectors and the resource status matrix using a multi-objective optimization algorithm; allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme; and performing adaptive adjustment of computing power and load balancing during the execution of the large model task. This application's technical solution realizes a complete computing power scheduling process from task reception, feature extraction, resource monitoring, scheduling decision-making, parallel execution to dynamic adjustment, solving the problem that existing static resource allocation mechanisms cannot dynamically adjust resource usage according to real-time task load, resulting in low computing power resource utilization and the coexistence of resource idleness and contention, significantly improving the overall utilization of cluster resources and task execution efficiency.

[0033] This application's embodiments take into account that, due to existing technologies, current mainstream large-scale model computing power allocation systems generally adopt static resource quotas and fixed scheduling strategies. At the hardware level, a fixed number of GPU cards and corresponding CPU and memory resources are typically pre-allocated to each large-scale model instance, with each computing node independently running its allocated tasks. At the software level, a first-come, first-served scheduling method is used, applying uniform resource allocation rules to all types of tasks; once a task is allocated, resource usage is not adjusted during execution. Some systems support simple manual resource expansion, but cannot automatically adjust resource allocation ratios based on real-time load. This presents the following drawbacks: The pre-allocation mechanism allocates fixed GPU memory and computing resources to each model instance, which cannot dynamically adjust resource usage according to real-time task load. This results in a large amount of computing power being idle during off-peak hours, while resource contention and task queuing occur during peak hours.

[0034] Using the same scheduling strategy for different types of tasks, such as inference requests, fine-tuning tasks, and batch predictions, cannot guarantee the service quality and response time of core businesses.

[0035] For large models with hundreds of billions of parameters, the lack of a dynamic computing power redistribution mechanism when deploying multiple GPUs in a sharded manner can easily lead to a significant increase in the inference latency of the entire model instance when the load of a certain GPU card fluctuates.

[0036] It cannot automatically adjust the number of model instances based on the task queue length and resource usage, making it difficult to cope with sudden traffic surges.

[0037] Most scheduling is based on the current instantaneous state, without considering the periodic changes and trend predictions of task load, resulting in insufficient foresight and stability of scheduling decisions.

[0038] Therefore, this application provides a method for dynamic allocation of computing power for large models, which can solve technical problems such as how to dynamically adjust computing power allocation according to the real-time load of tasks, improve resource utilization and avoid contention, how to ensure the service quality of core businesses through differentiated priority scheduling, how to achieve dynamic computing power redistribution in the multi-card deployment scenario of large models to avoid single nodes dragging down overall performance, how to automatically detect load imbalance and implement computing power rebalancing, how to build a full life cycle monitoring and hierarchical anomaly recovery mechanism to improve system reliability, and how to make forward-looking and stable scheduling decisions by combining historical data and multi-objective optimization algorithms.

[0039] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, large-model computing power dynamic allocation device, or system capable of realizing the above functions. The following description uses a large-model computing power dynamic allocation system as an example to illustrate the various embodiments.

[0040] Reference Figure 1 As shown, the large-model computing power dynamic allocation system involved in this embodiment includes a large-model load perception and feature extraction module, a computing power resource status monitoring and quantification module, a dynamic allocation decision and priority scheduling module, a task sharding and parallel execution management module, a computing power adaptive adjustment and load balancing module, and an execution status monitoring and anomaly handling module.

[0041] The large model load awareness and feature extraction module is used to capture various large model tasks entering the system in real time, extract task type features, computational load features, timeliness features, and dependency features, and construct a unified task feature vector representation. This module is the entry point of the entire system, responsible for connecting with external systems and user task requests, and providing basic data support for subsequent scheduling decisions.

[0042] like Figure 2 As shown, the large model load perception and feature extraction module specifically includes a task capture unit, a feature extraction unit, and a feature encoding unit.

[0043] The task capture unit utilizes Java NIO technology to handle high-concurrency task requests, capable of processing a large number of concurrent requests simultaneously without blocking. Through integration with mainstream message queue middleware such as RabbitMQ and Kafka, it achieves asynchronous processing and decoupling of task requests, improving system reliability and scalability. The task capture unit also incorporates request rate limiting and circuit breaking mechanisms, automatically limiting the incoming request rate when the system load is too high to prevent system overload.

[0044] The feature extraction unit performs deep analysis of task requests, extracting not only basic task attribute information but also using machine learning models to more accurately predict the computational load of tasks. A gradient boosting tree model trained on historical task execution data can predict task execution time and resource consumption based on features such as task type, input data size, and model version number, achieving higher prediction accuracy compared to traditional statistical analysis methods. The feature extraction unit can also identify complex dependencies between tasks, including sequential, parallel, and conditional dependencies, constructing a complete task dependency graph to ensure tasks are executed in the correct order.

[0045] The feature encoding unit is used to convert non-numerical task features into numerical feature vectors that can be processed by the algorithm. For categorical features such as task type and model version number, one-hot encoding is used to avoid the numerical relationship between different categories from misleading the algorithm. For numerical features such as input data size and expected completion time, min-max normalization is used to map them to the [0,1] interval to eliminate differences between different units. The final generated task feature vector has a fixed dimension and can be directly input into the subsequent scheduling decision algorithm.

[0046] The computing resource status monitoring and quantification module is used to collect and quantify the hardware resource usage of all computing nodes in the cluster in real time, generating a resource status matrix that includes GPU memory utilization, GPU computing utilization, CPU utilization, memory utilization, and network bandwidth utilization. This module is the "eyes" of the system, responsible for sensing the cluster's resource status in real time and providing accurate basis for resource allocation decisions.

[0047] like Figure 3 As shown, the computing resource status monitoring and quantification module includes a resource acquisition unit, a data preprocessing unit, and a resource quantification unit.

[0048] The resource acquisition unit adopts a distributed acquisition architecture, deploying a lightweight acquisition agent on each computing node to collect resource usage data for that node. The acquisition agent is developed in Java, featuring cross-platform compatibility and low resource consumption, and will not significantly impact the performance of large-scale model tasks running on the computing nodes. The collected raw data is transmitted over the network to the central node for centralized processing. Compression algorithms are used during transmission to reduce data transmission volume and improve efficiency.

[0049] The data preprocessing unit is used to perform a series of processing operations on the collected raw data in order to improve the quality and usability of the data.

[0050] The resource quantification unit uses a weighted summation method to quantitatively evaluate the comprehensive resource status of computing nodes and calculates the available computing power index for each node. Through quantification, the complex multidimensional resource status is transformed into a single numerical indicator, facilitating subsequent resource allocation decisions.

[0051] The dynamic allocation decision and priority scheduling module is used to calculate the optimal computing power allocation scheme based on task feature vectors and resource state matrices using a multi-objective optimization algorithm. It also determines the execution order and resource quotas of tasks according to task priorities. This module is the core of the entire system, responsible for making resource allocation and task scheduling decisions, directly impacting the overall system performance and service quality.

[0052] like Figure 4 As shown, the dynamic allocation decision and priority scheduling module includes a priority calculation unit, a resource allocation optimization unit, and a task scheduling unit.

[0053] The priority calculation unit comprehensively considers the timeliness requirements, business importance, and computational workload of the task to calculate the overall priority of each task.

[0054] The resource allocation optimization unit constructs a multi-objective optimization model aimed at maximizing the overall system throughput and minimizing the average task response time. This model considers resource constraints of computing nodes, resource requirements of tasks, and dependencies between tasks, making it a complex combinatorial optimization problem. This invention employs an improved genetic algorithm to solve this model. Based on the traditional genetic algorithm, it introduces an adaptive crossover and mutation probability mechanism, which automatically adjusts the crossover and mutation probabilities according to the evolutionary state of the population, improving the algorithm's convergence speed and solution accuracy. Simultaneously, an elite retention strategy is adopted, directly preserving the individuals with the highest fitness in each generation to the next generation, ensuring that the algorithm does not lose the optimal solution.

[0055] The task scheduling unit sorts the task queue from highest to lowest based on the overall priority of the tasks, ensuring that high-priority tasks are processed first. The task scheduling unit also supports a task preemption mechanism; when a higher-priority task arrives, the currently executing lower-priority task can be paused, resources allocated to the higher-priority task, and execution of the lower-priority task resumed after the higher-priority task completes.

[0056] The task sharding and parallel execution management module is used to perform fine-grained sharding of large model inference and training tasks according to the computation graph structure and resource allocation scheme, map the sharded tasks to corresponding computing nodes for execution, and manage communication and data synchronization between sharded tasks. This module is key to realizing distributed computing of large models, enabling the allocation of a single large model task to multiple computing nodes for parallel execution, significantly improving task processing speed.

[0057] The task sharding and parallel execution management module includes a computation graph parsing unit, a task sharding unit, and a parallel execution unit. The computation graph parsing unit loads the computation graph structure of the large model. It can identify the node types in the computation graph, including convolutional layers, fully connected layers, attention layers, etc., and calculate the computational cost and memory usage of each node. Simultaneously, it parses the edges in the computation graph to determine the input-output dependencies between nodes, providing a basis for subsequent task sharding.

[0058] The task sharding unit is used to perform fine-grained sharding of tasks based on resource allocation schemes and the parallelism characteristics of the computation graph, employing a combination of sharding strategies. For large model inference tasks, a sharding strategy combining pipelined parallelism and tensor parallelism is used. For large model training tasks, a sharding strategy combining data parallelism and model parallelism is used.

[0059] The parallel execution unit, based on the Java concurrency programming framework and distributed computing framework, manages the parallel execution of sharded tasks. It achieves high-speed communication and data synchronization between different computing nodes through the gRPC remote procedure call mechanism. gRPC, based on the HTTP / 2 protocol, features low latency and high throughput, making it ideal for communication between distributed systems. The parallel execution unit maintains a global task status table, recording the execution status, progress, and node of each sharded task, enabling real-time monitoring of the execution status of all sharded tasks. When a sharded task completes, it automatically triggers the execution of its dependent sharded tasks, ensuring the entire task executes correctly according to the dependencies in the computation graph.

[0060] The adaptive computing power adjustment and load balancing module monitors the load changes of each computing node in real time during task execution. When load imbalance or resource bottlenecks occur, it dynamically adjusts the allocation ratio of sharded tasks and the sharding position of model parameters to achieve dynamic rebalancing of computing resources. This module can effectively solve the load imbalance problem that occurs during task execution, and improve the system's resource utilization and task execution efficiency.

[0061] The adaptive computing power adjustment and load balancing module includes a load monitoring unit, a load balancing decision unit, and a dynamic adjustment unit. During task execution, the load monitoring unit collects resource usage data from each computing node and the execution progress data of the sharded tasks at fixed time intervals. The load monitoring unit sends the collected data to the load balancing decision unit in real time for analysis. The load balancing decision unit calculates the system's load balance. Based on the load balancing decision results, the dynamic adjustment unit dynamically adjusts the allocation ratio of sharded tasks and the sharding positions of the model parameters.

[0062] The execution status monitoring and exception handling module is used to monitor the execution status of all tasks throughout their entire lifecycle. When a task execution failure, timeout, or computing node failure is detected, the module automatically triggers task rescheduling and fault recovery mechanisms to ensure the stable operation of the system. This module is responsible for ensuring the reliability and availability of the system, ensuring that tasks can be completed successfully.

[0063] The execution status monitoring and anomaly handling module includes a status monitoring unit, an anomaly detection unit, and a fault recovery unit. The status monitoring unit employs a combination of heartbeat detection and task progress reporting mechanisms to monitor the execution status of computing nodes and tasks in real time. Simultaneously, the status monitoring unit provides a visual monitoring interface that displays the overall system operation, cluster resource usage, task execution status, and anomaly alarm information, facilitating management and maintenance by system administrators.

[0064] The anomaly detection unit combines rule-based anomaly detection and machine learning anomaly detection methods to perform real-time analysis of monitoring data, detect anomalies such as computing node failures, task execution timeouts, task execution failures, and resource exhaustion, and calculate the severity of the anomalies.

[0065] The fault recovery unit takes appropriate fault recovery measures based on the severity and type of the anomaly. For minor anomalies...

[0066] This application, through the collaborative work of the above modules, realizes a complete dynamic computing power allocation system for large models. It can dynamically adjust the computing power allocation according to the real-time load of tasks, ensure the service quality of core businesses through differentiated priority scheduling, realize dynamic computing power redistribution in the multi-card deployment scenario of large models, automatically detect load imbalance and implement computing power rebalancing, build a full life cycle monitoring and hierarchical anomaly recovery mechanism, and make forward-looking and stable scheduling decisions by combining historical data and multi-objective optimization algorithms.

[0067] Based on the above system architecture, this application provides a method for dynamic allocation of computing power for large models, referring to... Figure 5 , Figure 5 This is a flowchart illustrating the first embodiment of the dynamic allocation method for large-scale computing power in this application.

[0068] In this embodiment, the dynamic allocation method for large model computing power includes steps S10 to S60. The following provides a detailed explanation of each step.

[0069] like Figure 5 As shown, the first embodiment of this application proposes a method for dynamic allocation of computing power for large models, the method comprising: Step S10: Receive a large model task request; This embodiment provides a method for dynamic allocation of computing power for large models. The method first performs system initialization and resource registration operations. System initialization refers to the sequential activation of various functional modules when starting the entire computing power scheduling system, including a large model load perception and feature extraction module, a computing power resource status monitoring and quantification module, a dynamic allocation decision and priority scheduling module, a task sharding and parallel execution management module, a computing power adaptive adjustment and load balancing module, and an execution status monitoring and exception handling module. The specific system architecture can be found in [reference needed]. Figure 1 As shown.

[0070] Resource registration refers to each compute node proactively sending a registration request to the resource management center after startup, reporting its hardware configuration information and current available resources. Hardware configuration information includes, but is not limited to, CPU model and number of cores, memory size, GPU model and number, and network bandwidth. Upon receiving the registration request, the resource management center verifies the compute node. If verification is successful, the node is added to the cluster resource topology graph and assigned a unique node identifier. The cluster resource topology graph is a data structure that records all available compute nodes and their configuration information, used for subsequent resource scheduling decisions. Through these initialization and registration operations, the system establishes a comprehensive understanding of the overall cluster resources, laying the foundation for subsequent task scheduling.

[0071] After system initialization is complete, the system enters the task receiving phase. The system receives large model task requests, which refer to tasks submitted by users or upper-layer applications that require large models to perform calculations. These include, but are not limited to, large model inference requests (such as users sending questions to chatbots), large model fine-tuning tasks (such as adjusting the parameters of a pre-trained model using a specific dataset), and batch prediction tasks (such as performing sentiment analysis on a large amount of text data).

[0072] The system listens in real-time to the task request port through a task capture unit, receiving large-scale task requests from the front-end interface, the scheduled task maker, and third-party systems. It performs format and validity checks on the received task requests, verifying the completeness of request parameters, their conformity to the prescribed format, and the appropriate permissions of the requesting user. Invalid or malicious requests are rejected outright with corresponding error messages. For valid task requests, a unique task identifier is generated, and the task request is stored in the task database.

[0073] Step S20: Extract task features according to the large model task request, and convert the task features into task feature vectors; Next, the system extracts task features based on the large model task request and converts these features into task feature vectors. These task features include basic task attribute information (such as task type, model version number, input data size, etc.), computational cost information (such as expected GPU computation and memory usage), and dependency information (such as whether the task depends on the output of other tasks). After feature extraction, non-numerical task features are converted into numerical feature vectors that can be processed by the algorithm through feature encoding. For categorical features such as task type and model version number, one-hot encoding is used to avoid misleading the algorithm due to the numerical relationship between different categories. For numerical features such as input data size and expected completion time, min-max normalization is used to map them to the [0,1] interval to eliminate differences between different units. The final generated task feature vector has a fixed dimension and can be directly input into the subsequent scheduling decision algorithm.

[0074] In one implementation method, the final generated task feature vector is added to the task queue to await processing.

[0075] Step S30: Collect and quantify the resource usage data of each computing node in the cluster in real time to obtain the cluster resource status matrix. Then, the system collects and quantifies the resource usage data of each computing node in the cluster in real time to obtain a cluster resource status matrix. The resource usage data includes GPU memory utilization, GPU computing utilization, CPU utilization, memory utilization, and network bandwidth utilization. The collection method involves deploying a lightweight collection agent on each computing node, responsible for collecting the resource usage data of that node. The collected data is transmitted over the network to the central node for centralized processing. Quantification processing involves converting the collected multi-dimensional raw resource data into quantitative indicators that reflect the overall resource status of the nodes through a series of algorithms, ultimately forming the cluster resource status matrix. This matrix reflects the resource usage and available computing power of each computing node in the cluster in real time.

[0076] Step S40: Based on the task feature vector and resource state matrix, calculate the optimal computing power allocation scheme using a multi-objective optimization algorithm; Subsequently, based on the task feature vector and resource state matrix, the system calculates the optimal computing power allocation scheme using a multi-objective optimization algorithm. Specifically, the system constructs a multi-objective optimization model with the optimization objectives of maximizing the overall system throughput and minimizing the average task response time. This model considers the resource constraints of computing nodes (such as the upper limit of GPU memory and the number of available CPU cores), the resource requirements of tasks (such as the amount of GPU memory and computing power required by the task), and the dependencies between tasks. An improved genetic algorithm is used to solve this model to obtain the optimal computing power allocation scheme and determine the computing nodes and resource quotas allocated to each task.

[0077] Step S50: Based on the optimal computing power allocation scheme, the single large model task is allocated to multiple computing nodes for parallel execution; Next, based on the optimal computing power allocation scheme, the system distributes a single large model task across multiple computing nodes for parallel execution. Due to the enormous computational load, a single large model task (such as a Transformer model inference task with hundreds of billions of parameters) cannot be completed within an acceptable timeframe by a single computing node; therefore, it needs to be split across multiple computing nodes for parallel execution. The system analyzes the computation graph structure of the large model, identifies the parallelism characteristics within the computation graph, performs fine-grained task partitioning according to the computation graph structure and resource allocation scheme, maps the partitioned tasks to corresponding computing nodes for execution, and manages communication and data synchronization between the partitioned tasks.

[0078] Step S60: During the execution of the large model task, adaptive adjustment of computing power and load balancing are performed.

[0079] During the execution of large model tasks, the system performs adaptive adjustment of computing power and load balancing. Specifically, during task execution, the system monitors the load changes of each computing node in real time. When load imbalances or resource bottlenecks occur, it dynamically adjusts the allocation ratio of task shards and the sharding position of model parameters to achieve dynamic rebalancing of computing resources. For example, if the load on a GPU card suddenly increases, causing computation to slow down, the system migrates some of the task shards from that card to a less loaded GPU card to eliminate the performance bottleneck.

[0080] This embodiment, through the above technical solution, specifically involves receiving large model task requests; extracting task features based on the large model task requests and converting the task features into task feature vectors; collecting and quantifying resource usage data from each computing node in the cluster in real time to obtain a cluster resource status matrix; calculating the optimal computing power allocation scheme using a multi-objective optimization algorithm based on the task feature vectors and the resource status matrix; allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme; and performing adaptive adjustment of computing power and load balancing during the execution of the large model task. This technical solution implements a complete computing power scheduling process from system initialization, task reception, feature extraction, resource monitoring, scheduling decision-making, parallel execution to dynamic adjustment, solving the problem of low computing power resource utilization caused by the inability of existing static resource allocation mechanisms to dynamically adjust resource usage according to real-time task load, and significantly improving the overall utilization rate of cluster resources and task execution efficiency.

[0081] Based on the first embodiment described above, a second embodiment of this application is proposed. In this embodiment, the steps of extracting task features according to the large model task request and converting the task features into task feature vectors are refined.

[0082] As one implementation method, the step of extracting task features based on the large model task request and converting the task features into a task feature vector includes: Extract basic attribute information of the task based on the large model task request; Predict the computational load of a task using a prediction model trained on historical data; Identify dependencies between tasks to construct a task dependency graph; The basic attribute information, task computation amount, and dependency information are converted into a uniform-dimensional numerical task feature vector and added to the task queue by one-hot encoding and min-max normalization.

[0083] Specifically, firstly, the basic attribute information of the task is extracted based on the large model task request. The basic attribute information refers to the inherent attributes of the task itself, including but not limited to the task type (such as inference task, fine-tuning task, batch prediction task, etc.), model version number, input data scale (such as the length of input text, image resolution, etc.), expected completion time (the user's expected task completion time limit), etc.

[0084] Secondly, the computational load of the task is predicted using a prediction model trained on historical data. Specifically, a gradient boosting tree model trained on historical data is used to predict the execution time and resource consumption of the task based on the task type, input data size, and model version number.

[0085] Gradient Boosting Decision Tree (GBDT) is an ensemble learning algorithm that iteratively adds decision trees to fit the residuals of the previous round, offering advantages such as high prediction accuracy and the ability to handle nonlinear relationships. In this embodiment, a large amount of historical task execution data was pre-collected, including the task type, input data size, model version number, and label information such as actual execution time, actual GPU memory usage, and actual GPU computational cost for each task. This data was used to train the gradient boosting tree model. After training, when a new task arrives, the task type, input data size, and model version number are input into the trained gradient boosting tree model, which then outputs the predicted execution time and resource consumption. Compared to traditional statistical analysis methods (such as estimations based on means or percentiles), the gradient boosting tree model can capture complex interactions between different features, resulting in higher prediction accuracy.

[0086] Next, the dependencies between tasks are identified to construct a task dependency graph. A task dependency refers to the fact that the execution of one task depends on the output of another task as input. Common dependencies include sequential dependencies (task B can only begin after task A is completed), parallel dependencies (tasks C and D can be executed simultaneously without interdependence), and conditional dependencies (the execution of task E depends on whether a certain condition is met, and this condition is determined by the execution result of task F). The feature extraction unit identifies these dependencies by parsing the dependency description information in the task request or by analyzing the logical relationship between the task input and output data, and constructs a complete task dependency graph. The task dependency graph is a directed acyclic graph (DAG), where nodes represent tasks and directed edges represent dependencies between tasks. The purpose of constructing the task dependency graph is to ensure that tasks are executed in the correct order, avoiding execution errors caused by unmet dependencies. For example, in a large model fine-tuning task, the data preprocessing task must be executed first, followed by the model training task, and finally the model evaluation task. These three tasks constitute a sequential dependency, and the task dependency graph clearly expresses this execution order.

[0087] Finally, the basic attribute information, task computational complexity, and dependency information are converted into a uniform-dimensional numerical task feature vector and added to the task queue using one-hot encoding and min-max normalization. One-hot encoding is a method of converting categorical variables into numerical variables. For each categorical feature, a new binary feature column is created, with each value corresponding to a column. A value of 1 indicates that the sample belongs to that category, and a value of 0 indicates that it does not. Min-max normalization is a method of scaling numerical features to a specified interval, mapping all numerical features to the [0,1] interval to eliminate differences between different units of measurement. Through the above encoding and normalization processes, all task features are converted into numerical feature vectors with fixed dimensions and consistent numerical ranges, which can be directly input into subsequent scheduling decision algorithms for computation.

[0088] This embodiment achieves accurate extraction and standardized encoding of task features through the above technical solution, enabling the scheduling decision algorithm to make optimal scheduling decisions based on high-quality task feature data. It solves the problem of low scheduling decision accuracy caused by insufficient task feature extraction and non-standard feature representation in the prior art, and improves the accuracy and scientific nature of scheduling decisions.

[0089] Based on any of the above embodiments, a third embodiment of this application is proposed. In this embodiment, the step of collecting and quantifying the resource usage data of each computing node in the real-time cluster to obtain the cluster resource status matrix is ​​refined.

[0090] As one implementation method, the real-time acquisition and quantification of resource usage data from each computing node in the cluster to obtain a cluster resource status matrix includes: Real-time collection of resource usage data from each computing node in the cluster; The collected resource usage data is preprocessed for anomaly removal and smoothing. The available computing power index for each computing node is calculated based on the preprocessed data. The resource usage data and available computing power index are quantified to obtain a cluster resource status matrix, which reflects the resource usage and available computing power of each computing node in the cluster in real time.

[0091] Specifically, firstly, resource usage data from each computing node in the cluster is collected in real time. The resource collection unit adopts a distributed collection architecture, deploying a lightweight collection agent on each computing node to collect its local resource usage data. The collection agent is developed in Java, featuring cross-platform compatibility and low resource consumption, and will not significantly impact the performance of large model tasks running on the computing node. The collected resource usage data includes GPU memory utilization, GPU computational utilization, CPU utilization, memory utilization, and network bandwidth utilization. The collection agent collects this data at a fixed collection period (e.g., once every 1 second). The raw data is transmitted over the network to the central node for centralized processing. Compression algorithms are used during transmission to reduce data transmission volume and improve efficiency.

[0092] Secondly, the collected resource usage data undergoes anomaly removal and smoothing preprocessing. The raw data collected may contain outliers (such as data points far exceeding the normal range due to system fluctuations or collection errors) and noise (such as brief instantaneous fluctuations). These outliers and noise can affect the accuracy of subsequent resource status assessments. Therefore, preprocessing is necessary.

[0093] Specifically, the 3σ criterion can be used to remove outliers. The 3σ criterion (also known as the Raida criterion) is an outlier detection method based on a normal distribution. Assuming the data follows a normal distribution, the probability of data falling outside the mean ± 3 standard deviations is only 0.27%, and can be considered an outlier. For each resource indicator, its mean and standard deviation are calculated within a certain time window. Data points exceeding the mean ± 3 standard deviations are marked as outliers and removed.

[0094] Then, a sliding window averaging method is used for smoothing to eliminate instantaneous fluctuations. The sliding window averaging method involves taking the average of a certain number of data points before and after each data point (e.g., 5 before and 5 after) as the smoothed value for that point. The window size can be adjusted according to actual needs. By using sliding window averaging, the impact of instantaneous fluctuations on resource status judgment can be effectively eliminated, making the trend of resource status changes clearer.

[0095] In addition, time series analysis is performed on the preprocessed data to calculate the moving average of resource usage (reflecting the average level of resources), variance (reflecting the degree of resource fluctuation), and trend change rate (reflecting the direction and speed of resource changes), providing support for subsequent resource forecasting and scheduling decisions.

[0096] Then, the available computing power index for each computing node is calculated based on the preprocessed data. Specifically, a weighted summation method is used to quantitatively evaluate the overall resource status of the computing nodes and calculate the available computing power index for each node.

[0097] The overall resource status includes GPU memory utilization, GPU computing utilization, CPU utilization, and network bandwidth utilization.

[0098] The formula for calculating the available computing power index is as follows:

[0099] in, This represents the available computing power index of the i-th computing node. This represents the GPU computing utilization of the i-th computing node. This represents the GPU memory utilization rate of the i-th computing node. This represents the CPU utilization of the i-th computing node. This represents the network bandwidth utilization of the i-th computing node. , , , These are the weighting coefficients for GPU computation, GPU memory, CPU, and network bandwidth, respectively, and they satisfy... .

[0100] The weighting coefficients can be adjusted according to different application scenarios. For large model tasks that primarily rely on GPU computing, the weighting coefficients for GPU computing and GPU memory can be appropriately increased; for tasks requiring significant CPU computing and network transmission, the weighting coefficients for CPU and network bandwidth can be appropriately increased. Through quantization, the complex multidimensional resource status is transformed into a single numerical indicator, facilitating subsequent resource allocation decisions.

[0101] Finally, the resource usage data and available computing power index are quantified to obtain the cluster resource status matrix. The cluster resource status matrix is ​​an N×M matrix, where N is the number of computing nodes in the cluster, and M represents the number of resource indicators for each node (including GPU memory utilization, GPU computing utilization, CPU utilization, memory utilization, network bandwidth utilization, available computing power index, etc.). The cluster resource status matrix reflects the resource usage and available computing power of each computing node in the cluster in real time.

[0102] Through the above steps, this embodiment achieves comprehensive, real-time, and accurate monitoring and quantification of cluster resource status, transforming complex multidimensional resource status into structured matrix data, providing an accurate data foundation for subsequent resource allocation decisions, and solving the problems of incomplete resource status monitoring and low data quality in existing technologies, which lead to insufficient basis for scheduling decisions.

[0103] Based on any of the above embodiments, a fourth embodiment of this application is proposed. In this embodiment, the step of calculating the optimal computing power allocation scheme based on the task feature vector and the resource state matrix using a multi-objective optimization algorithm is refined.

[0104] As one implementation method, the step of calculating the optimal computing power allocation scheme based on the task feature vector and resource state matrix using a multi-objective optimization algorithm includes: Calculate the overall priority of each task in the task queue based on the task feature vector; Using the cluster resource state matrix as constraints, a multi-objective optimization model is constructed with the goal of maximizing system throughput and minimizing average task response time. The multi-objective optimization model is solved by an optimization algorithm to generate the mapping relationship between each task and computing node and the resource quota scheme. The execution order and resource quota of the tasks are determined according to the comprehensive priority.

[0105] First, the overall priority of each task in the task queue is calculated based on the task feature vector. Specifically, the overall priority of each task in the task queue is calculated based on the task's timeliness requirements, business importance, and computational complexity. The formula for calculating the overall priority is:

[0106] in, This represents the overall priority of the j-th task. This represents the timeliness coefficient of the task. This represents the business importance coefficient of the task. This indicates the estimated computational cost of the task. This represents the maximum estimated computational load for all pending tasks in the system. , , The weighting coefficients are timeliness, business importance, and computational load, respectively, and satisfy the following conditions: .

[0107] The timeliness coefficient is calculated based on the difference between the expected completion time of the task and the current time. The smaller the difference, the higher the timeliness requirement of the task, and the larger the timeliness coefficient. The business importance coefficient is pre-configured by the system administrator according to business needs, with core business tasks having a higher business importance coefficient. The computational workload coefficient is inversely proportional to the estimated computational workload of the task; tasks with less computational workload have higher priority, which can improve the overall throughput of the system.

[0108] Then, using the cluster resource state matrix as constraints, a multi-objective optimization model is constructed with the goal of maximizing system throughput and minimizing average task response time. The multi-objective optimization model is solved by an optimization algorithm to generate the mapping relationship between each task and computing node and the resource quota scheme. The execution order and resource quota of the tasks are determined according to the comprehensive priority.

[0109] Multi-objective optimization refers to an optimization problem that seeks the optimal trade-off solution among multiple conflicting objectives. In this embodiment, system throughput refers to the number of tasks the system completes per unit time; maximizing system throughput means processing as many tasks as possible. Average task response time refers to the average time elapsed from task submission to completion; minimizing average task response time means completing tasks as quickly as possible. There is a certain conflict between these two objectives: to increase throughput, the system may tend to execute smaller tasks first, but this could lead to excessively long response times for larger tasks; to reduce average response time, the system may tend to execute time-sensitive tasks first, but this could reduce overall throughput. Therefore, a multi-objective optimization model is needed to balance these two objectives.

[0110] In this embodiment, the multi-objective optimization model considers the following constraints: resource constraints of computing nodes, resource requirements of tasks, and dependencies between tasks (dependent tasks can only begin execution after the dependent task has been completed). This is a complex combinatorial optimization problem, and it is impossible to find an exact optimal solution in polynomial time. Therefore, this embodiment uses an improved genetic algorithm to solve the model. A genetic algorithm is a global optimization algorithm that simulates natural selection and genetic mechanisms, searching for the optimal solution in the solution space through operations such as selection, crossover, and mutation. This embodiment improves upon the traditional genetic algorithm by introducing an adaptive crossover and mutation probability mechanism, which automatically adjusts the crossover and mutation probabilities according to the evolutionary state of the population. Higher crossover and mutation probabilities are used in the early stages of evolution to enhance global search capabilities, while lower crossover and mutation probabilities are used in the later stages to enhance local search capabilities, thereby improving the algorithm's convergence speed and solution accuracy. Simultaneously, an elite retention strategy is adopted, directly retaining the individual with the highest fitness in each generation to the next generation, ensuring that the algorithm does not lose the optimal solution.

[0111] After solving the multi-objective optimization model, the optimal computing power allocation scheme is obtained. This scheme includes the computing node assigned to each task (i.e., on which node the task will execute) and the resource quota (i.e., how much GPU memory, CPU cores, and RAM are allocated to the task). Simultaneously, the task scheduling unit sorts the task queue from highest to lowest based on the overall priority of the tasks, ensuring that high-priority tasks are processed first. When the available resources of a computing node meet the resource requirements of a task, the task is immediately assigned to that node for execution; when available resources are insufficient, low-priority tasks are placed in a waiting queue and scheduled again after resources are released. The task scheduling unit also supports a task preemption mechanism. When a higher-priority task arrives, the currently executing low-priority task can be paused, resources allocated to the high-priority task, and execution of the low-priority task resumed after the high-priority task completes.

[0112] Through the above steps, this embodiment realizes intelligent resource allocation decision-making based on multi-objective optimization, which can find the optimal balance between throughput and response time in large-scale task scheduling scenarios. At the same time, it ensures the priority execution of core business and time-sensitive tasks through a comprehensive priority mechanism, solving the problem that the scheduling strategy in the prior art is too simple and cannot guarantee the service quality of core business.

[0113] Based on any of the above embodiments, a fifth embodiment of this application is proposed. In this embodiment, the step of allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme is refined.

[0114] As one implementation method, the step of allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme includes: Based on the generated mapping relationship between each task and computing node and the resource quota scheme, the large model computing graph structure is loaded and parsed. Select the appropriate sharding strategy based on the task type to split the task into multiple sharded tasks with fine granularity; The sharded tasks are mapped to corresponding computing nodes, and data synchronization between the sharded tasks is achieved through a distributed communication framework, enabling the sharded tasks to be executed in parallel.

[0115] First, based on the generated mapping relationships between tasks and computation nodes and the resource allocation scheme, the large model computation graph structure is loaded and parsed. A computation graph is a data flow graph representation method used in deep learning frameworks to describe the computation process of a model. A computation graph consists of nodes and edges; nodes represent computational operations (such as convolution, fully connected layers, attention calculations, etc.), and edges represent the flow of tensor data (such as feature maps, weight matrices, etc.). The computation graph parsing unit loads the computation graph structure corresponding to the large model task, identifies the node types in the computation graph (including convolutional layers, fully connected layers, attention layers, etc.), and calculates the computational cost and memory usage of each node. Simultaneously, the edges in the computation graph are parsed to determine the input-output dependencies between nodes, providing a basis for subsequent task partitioning.

[0116] Secondly, the task sharding unit performs fine-grained sharding of tasks using a combination of sharding strategies based on the resource allocation scheme and the parallelism characteristics of the computation graph. Specifically, a sharding strategy is selected according to the task type to finely divide the task into multiple sharded tasks. Task sharding refers to the process of splitting a large model task into multiple subtasks that can be executed in parallel according to a certain strategy. In this embodiment, different sharding strategies are used depending on the task type.

[0117] Specifically, for inference tasks, a partitioning strategy combining pipelined parallelism and tensor parallelism is adopted. Pipeline parallelism involves assigning different layers of the model to different computation nodes. After the previous node completes its computation, the result is passed to the next node for further computation, forming a pipeline. Tensor parallelism involves assigning a single tensor along different dimensions to different computation nodes. Each node is responsible for computing only a portion of the tensor, and the results are finally aggregated. For example, for multiple attention heads in a multi-head attention mechanism, different heads can be assigned to different computation nodes for parallel computation. Pipeline parallelism is suitable for handling models with deep layers, while tensor parallelism is suitable for handling models with large widths. Combining the two can fully utilize the computational parallelism of the model.

[0118] For large model training tasks, a sharding strategy combining data parallelism and model parallelism is adopted. Data parallelism involves dividing the training data into multiple batches and distributing them to different computing nodes. Each node uses the same model parameters for training, and then the gradients are aggregated to update the model parameters. Model parallelism involves dividing the model parameters into multiple parts and distributing them to different computing nodes. Each node is only responsible for updating the parameters of its own part.

[0119] Then, the sharded tasks are mapped to corresponding computing nodes, and data synchronization between the sharded tasks is achieved through a distributed communication framework, enabling the sharded tasks to be executed in parallel. The mapping operation refers to assigning each sharded task to the corresponding computing node determined in the above steps.

[0120] As one implementation method, the parallel execution unit manages the parallel execution of sharded tasks based on Java concurrency frameworks (such as ExecutorService and Fork / Join frameworks) and distributed computing frameworks (such as Apache Spark and Hadoop MapReduce). High-speed communication and data synchronization between different computing nodes are achieved through the gRPC remote procedure call mechanism. gRPC, based on the HTTP / 2 protocol, features low latency and high throughput, making it very suitable for communication between distributed systems. The parallel execution unit maintains a global task status table, recording the execution status, progress, and node of each sharded task, enabling real-time monitoring of the execution status of all sharded tasks. When a sharded task completes its execution, it automatically triggers the execution of its dependent sharded tasks, ensuring that the entire task executes correctly according to the dependencies in the computation graph.

[0121] Through the above steps, this embodiment achieves fine-grained sharding and efficient parallel execution of large model tasks. It can split a single large model task into multiple subtasks and execute them simultaneously on multiple computing nodes, significantly shortening the execution time of large model tasks and solving the problem that a single node cannot accommodate or the execution time is too long for a large model with hundreds of billions of parameters.

[0122] Based on any of the above embodiments, a sixth embodiment of this application is proposed. In this embodiment, the operation of adaptive adjustment of computing power and load balancing is further defined in the process of executing large model tasks.

[0123] As one implementation method, during the execution of large model tasks, adaptive adjustment of computing power and load balancing are performed, including: During the execution of the sharding task, the load rate of each computing node is monitored in real time and the system load balance is calculated. When the system load balancing exceeds a preset threshold, a load balancing mechanism is triggered to migrate the incremental sharded tasks on overloaded nodes to lightly loaded nodes, and / or adjust the sharding position of model parameters online to achieve dynamic rebalancing of computing resources.

[0124] First, during the execution of the sharding task, the load rate of each computing node is monitored in real time, and the system load balancing is calculated. During task execution, the load monitoring unit collects resource usage data and sharding task execution progress data from each computing node at fixed time intervals. Resource usage data includes GPU computing utilization, GPU memory usage, CPU usage, memory usage, and network bandwidth usage; execution progress data includes the completed computation, remaining computation, and estimated completion time of the sharding task. The formula for calculating the system load balancing is:

[0125] in, Indicates the load balancing degree of the system. This represents the total number of nodes participating in the calculation. This represents the load rate of the i-th computing node. This represents the average load rate across all computing nodes. Load balance is an important indicator of how evenly the load is distributed in a system. When the system's load balance exceeds a preset threshold, it indicates a significant load imbalance, requiring the triggering of a load balancing mechanism.

[0126] Secondly, when the system load balancing exceeds a preset threshold, a load balancing mechanism is triggered to migrate incremental sharded tasks from overloaded nodes to lightly loaded nodes, and / or adjust the sharding positions of model parameters online to achieve dynamic rebalancing of computing resources. The preset threshold is a pre-defined upper limit for load balancing.

[0127] The load balancing mechanism is as follows: when the load balancing degree exceeds a preset threshold, the minimum load priority strategy is adopted to migrate the sharded tasks with long execution time and large amount of computation on the node with the highest load rate to the node with the lowest load rate.

[0128] Specifically, first, identify the computing node with the highest load rate and select the sharded tasks with longer execution times and higher computational loads on that node as candidate migration tasks; then, identify the computing node with the lowest load rate as the target node; finally, comprehensively consider the cost and benefits of task migration to determine the optimal migration scheme.

[0129] The dynamic adjustment unit dynamically adjusts the allocation ratio of sharded tasks and the sharding position of model parameters based on the load balancing decision results. For sharded task migration, an incremental migration method is used. Incremental migration includes migrating the execution state and necessary data of the sharded task. Incremental migration means migrating only the execution state and necessary data of the task, rather than migrating all the data of the entire task, thereby reducing the amount of data transferred and downtime during the migration process.

[0130] Online adjustment of model parameter shard locations involves gradually migrating model parameters without interrupting task execution. Online adjustment means making adjustments while the task is running continuously, without stopping the task. Specifically, for adjusting the shard locations of model parameters, a gradual migration approach is used. First, a small portion of the model parameters is migrated from overloaded nodes to less loaded nodes, and the system's performance is observed. Then, more parameters are gradually migrated until the load is balanced. This method avoids system jitter caused by a large-scale migration at once. Simultaneously, data communication paths are optimized, allocating sharded tasks with data dependencies to the same or adjacent compute nodes as much as possible, reducing cross-node data transfer and improving overall system performance.

[0131] Through the above steps, this embodiment achieves dynamic load balancing during task execution, which can promptly detect and resolve load imbalance issues, avoid the phenomenon of some nodes being overloaded while others are idle, improve the system's resource utilization and task execution efficiency, and solve the problems in the prior art of lacking a dynamic computing power redistribution mechanism and the tendency of single-node load fluctuations to lead to increased overall task execution delays.

[0132] Based on any of the above embodiments, a seventh embodiment of this application is proposed. In this embodiment, the method further includes anomaly monitoring and fault recovery processing, specifically including the following operations.

[0133] First, by combining heartbeat detection with task progress reporting mechanisms, the status of computing nodes and the execution status of sharded tasks are monitored in real time.

[0134] Heartbeat detection is a mechanism used to detect whether a node is alive. Specifically, the status monitoring unit sends a heartbeat packet to all computing nodes at regular intervals (e.g., every 3 seconds). The computing nodes respond immediately upon receiving the heartbeat packet. If no response is received from a computing node within a specified time, the node is considered to have failed (e.g., node crash, network interruption, etc.).

[0135] The task progress reporting mechanism refers to each shard task periodically reporting its execution progress and status information to the status monitoring unit during execution. The status monitoring unit then writes this information into a time-series database (such as InfluxDB or Prometheus) for storage and analysis. Time-series databases are specifically designed for storing and processing time-series data, enabling efficient storage and retrieval of timestamped data. Simultaneously, the status monitoring unit provides a visual monitoring interface that displays the overall system operation, cluster resource usage, task execution status, and anomaly alerts, facilitating management and maintenance by system administrators.

[0136] Secondly, rule-based detection and machine learning models are combined to detect abnormal events and determine their severity. Rule-based detection uses pre-defined rules, such as task execution time exceeding a preset threshold or compute node heartbeat timeout, to quickly detect known anomalies. Machine learning-based anomaly detection uses an Isolation Forest algorithm trained on historical monitoring data. Isolation Forest is a tree-based anomaly detection algorithm that constructs multiple isolation trees by randomly selecting features and split points. Abnormal data points, due to their different characteristics, are more easily isolated (i.e., shorter paths) and thus identified as anomalies. The Isolation Forest algorithm can detect unknown anomalies, improving anomaly detection coverage. The two methods are used together to perform real-time analysis of monitoring data, detecting anomalies such as compute node failures, task timeouts, task failures, and resource exhaustion, and calculating the severity of the anomalies using the following formula:

[0137] in, This indicates the severity of the k-th anomaly. The coefficient represents the scope of influence of the anomaly. A coefficient representing the duration of the anomaly. , These are the weighting coefficients for the scope and duration of influence, respectively, and they satisfy... Rule-based anomaly detection uses preset anomaly rules, such as task execution time exceeding a threshold, GPU memory usage exceeding a threshold, and compute node heartbeat timeout, to quickly detect known anomalies. Machine learning-based anomaly detection uses an isolated forest algorithm trained on historical monitoring data to detect unknown anomalies, improving anomaly detection coverage. The anomaly detection unit can also calculate the severity of anomalies, classifying them into three levels: minor, moderate, and severe, providing a basis for subsequent fault recovery.

[0138] Finally, graded fault recovery measures are implemented based on the severity level of the anomaly. Specifically, for minor anomalies, automatic retry processing is performed. Retry means re-executing the failed operation. For example, if a data transmission failure is caused by a brief network fluctuation during task execution, the system automatically retransmits the data, and after 1-2 retries, the network returns to normal and the data transmission succeeds. For general anomalies, failed sharded tasks are reassigned to other normal nodes for execution. For example, when a sharded task fails due to insufficient node resources, the system reschedules the sharded task to another node in the cluster with sufficient resources for re-execution. For severe anomalies, the faulty node is isolated, and all tasks on the faulty node are migrated to other available nodes, while an alarm message is issued. Node isolation means marking the node as unavailable and no longer assigning new tasks to it; task migration means migrating all tasks currently executing on the faulty node to other normal nodes for continued execution; the alarm message refers to a notification message (such as email, SMS, system pop-up, etc.) sent to the system administrator, informing the administrator that a severe anomaly has occurred and its specific details. In addition, this embodiment also supports the breakpoint resume function. When an abnormal interruption occurs during task execution, it can continue execution from the last saved checkpoint, avoiding starting from the beginning and saving a lot of time and resources.

[0139] Through the above steps, this embodiment realizes full lifecycle task monitoring and hierarchical anomaly recovery, which can detect and handle various abnormal situations in a timely manner, automatically trigger task rescheduling and fault recovery, ensure the continuous and stable operation of the system, and solve the problems of lack of a complete task monitoring and anomaly handling mechanism, inability to automatically recover when task execution fails or node failures, and insufficient system reliability in the prior art.

[0140] Based on any of the above embodiments, an eighth embodiment of this application is proposed. In this embodiment, the method further includes task completion and resource release processing, specifically including the following operations.

[0141] First, after all task shards have completed execution, the execution results are aggregated and all computing resources used by the tasks are reclaimed. "All task shards completed" means that the parallel execution unit detects that all task shards in the global task status table are in the "execution completed" state. At this point, the parallel execution unit aggregates the execution results of each task shard to generate the final task execution result. The aggregation method depends on the task type: for inference tasks, it may be necessary to concatenate the outputs from each shard in sequence; for training tasks, it may be necessary to average the gradients calculated by each node before updating the model parameters. Resource reclamation refers to the system releasing all computing resources allocated to the task back to the cluster resource pool for use by other tasks. Reclaimed resources include GPU memory, CPU resources, memory resources, and network bandwidth resources. After resource reclamation, the system updates the cluster resource status matrix, reflecting the released resources in the matrix so that subsequent tasks can be aware of the availability of these resources.

[0142] Then, the task execution result is returned to the task requester, and the execution data of this task, including task execution time, resource consumption, and sharding, is recorded in the historical task database for subsequent algorithm optimization and system improvement.

[0143] Through the above steps, this embodiment realizes the return of results, release of resources and archiving of data after the task is completed, forming a complete closed loop from task access to completion and release, providing data support for the continuous optimization of the system.

[0144] The complete process of this embodiment can be referred to Figure 6 As shown.

[0145] Compared with the prior art, the technical solution of this embodiment has the following advantages: This application employs a dynamic resource allocation mechanism by real-time sensing of cluster resource status and task load characteristics. This mechanism accurately allocates computing resources based on actual task requirements, avoiding resource idleness and waste caused by static resource allocation. A multi-objective optimization algorithm is used for resource allocation decisions, combined with a task priority scheduling mechanism, ensuring the priority execution of core business tasks. Furthermore, fine-grained task sharding and parallel execution techniques significantly shorten the execution time of large model tasks. Real-time monitoring of system load changes during task execution dynamically adjusts the allocation ratio of sharded tasks and the sharding position of model parameters, effectively solving the problem of load imbalance between computing nodes and preventing some nodes from being overloaded while others are idle. Full lifecycle task execution status monitoring and a robust anomaly handling mechanism promptly detect and handle various anomalies, automatically triggering task rescheduling and fault recovery, ensuring the continuous and stable operation of the system.

[0146] Furthermore, to achieve the above objectives, this application also proposes a dynamic allocation device for large-scale model computing power, such as... Figure 7 As shown, the device includes: Receiver module 10 is used to receive large model task requests; Feature extraction and quantization module 20 is used to extract task features according to the large model task request and convert the task features into task feature vectors; The data acquisition and quantization module 30 is used to collect and quantify the resource usage data of each computing node in the cluster in real time to obtain the cluster resource status matrix. The scheme calculation module 40 is used to calculate the optimal computing power allocation scheme based on the task feature vector and resource status matrix through a multi-objective optimization algorithm; The computing power allocation module 50 is used to allocate a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme. The load balancing module 60 is used to perform adaptive adjustment of computing power and load balancing during the execution of large model tasks.

[0147] The large-model computing power dynamic allocation device provided in this application adopts the large-model computing power dynamic allocation method in the above embodiments, which solves the problem that the existing static resource allocation mechanism cannot dynamically adjust resource occupation according to the real-time task load, resulting in low computing power resource utilization and the coexistence of resource idleness and contention. It significantly improves the overall utilization rate of cluster resources and task execution efficiency. Compared with the prior art, the beneficial effects of the large-model computing power dynamic allocation device provided in this application are the same as those of the large-model computing power dynamic allocation method provided in the above embodiments, and other technical features in the large-model computing power dynamic allocation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0148] In addition, to achieve the above objectives, this application also proposes a large model computing power dynamic allocation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model computing power dynamic allocation method as described above.

[0149] The following is for reference. Figure 8This document illustrates a structural schematic diagram of a large-model computing power dynamic allocation device suitable for implementing embodiments of this application. The large-model computing power dynamic allocation device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The large-scale computing power dynamic allocation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 8 As shown, the large-scale computing power dynamic allocation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 or the program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the large-scale computing power dynamic allocation device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the large-scale computing power dynamic allocation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a large-scale computing power dynamic allocation device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0151] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0152] The large-model computing power dynamic allocation device provided in this application adopts the large-model computing power dynamic allocation method in the above embodiments, which solves the problem that the existing static resource allocation mechanism cannot dynamically adjust resource occupation according to the real-time task load, resulting in low computing power resource utilization and the coexistence of resource idleness and contention. It significantly improves the overall utilization rate of cluster resources and task execution efficiency. Compared with the prior art, the beneficial effects of the large-model computing power dynamic allocation device provided in this application are the same as those of the large-model computing power dynamic allocation method provided in the above embodiments, and other technical features in the large-model computing power dynamic allocation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0154] In addition, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the large model computing power dynamic allocation method described above.

[0155] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0156] The aforementioned computer-readable storage medium may be included in the large-scale computing power dynamic allocation device; or it may exist independently and not be assembled into the large-scale computing power dynamic allocation device.

[0157] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the large-model computing power dynamic allocation device, the large-model computing power dynamic allocation device: receives large-model task requests; extracts task features based on the large-model task requests and converts the task features into task feature vectors; collects and quantifies resource usage data of each computing node in the cluster in real time to obtain a cluster resource status matrix; calculates the optimal computing power allocation scheme based on the task feature vectors and the resource status matrix using a multi-objective optimization algorithm; allocates a single large-model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme; and performs adaptive computing power adjustment and load balancing during the execution of the large-model task.

[0158] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0161] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described large-scale computing power dynamic allocation method. This solves the problem that existing static resource allocation mechanisms cannot dynamically adjust resource usage according to real-time task load, leading to low computing power resource utilization and the coexistence of resource idleness and contention. It significantly improves the overall utilization of cluster resources and task execution efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large-scale computing power dynamic allocation method provided in the above embodiments, and will not be elaborated upon here.

[0162] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the large model computing power dynamic allocation method described above.

[0163] The computer program product provided in this application solves the problems of low computing power utilization and the coexistence of resource idleness and contention caused by the inability of existing static resource allocation mechanisms to dynamically adjust resource occupation according to real-time task load. It significantly improves the overall utilization of cluster resources and task execution efficiency. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the large-model computing power dynamic allocation method provided in the above embodiments, and will not be repeated here.

[0164] The technical solutions proposed in this application have at least the following technical effects: receiving large model task requests; extracting task features based on the large model task requests and converting the task features into task feature vectors; collecting and quantifying resource usage data of each computing node in the cluster in real time to obtain a cluster resource state matrix; calculating the optimal computing power allocation scheme based on the task feature vectors and the resource state matrix using a multi-objective optimization algorithm; allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme; and performing adaptive adjustment of computing power and load balancing during the execution of large model tasks. The technical solutions of this application realize a complete computing power scheduling process from task reception, feature extraction, resource monitoring, scheduling decision-making, parallel execution to dynamic adjustment, solving the problem that existing static resource allocation mechanisms cannot dynamically adjust resource usage according to real-time task load, resulting in low computing power resource utilization and the coexistence of resource idleness and contention, significantly improving the overall utilization rate of cluster resources and task execution efficiency.

[0165] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for dynamic allocation of computing power for large-scale models, characterized in that, The method includes: Receive large model task requests; Extract task features based on the large model task request, and convert the task features into task feature vectors; Real-time collection and quantification of resource usage data from each computing node in the cluster yields a cluster resource status matrix. Based on the task feature vector and resource status matrix, the optimal computing power allocation scheme is calculated using a multi-objective optimization algorithm; Based on the optimal computing power allocation scheme, a single large model task is allocated to multiple computing nodes for parallel execution; During the execution of large model tasks, adaptive adjustment of computing power and load balancing are performed.

2. The method as described in claim 1, characterized in that, The step of extracting task features based on the large model task request and converting the task features into a task feature vector includes: Extract basic attribute information of the task based on the large model task request; Predict the computational load of a task using a prediction model trained on historical data; Identify dependencies between tasks to construct a task dependency graph; The basic attribute information, task computation amount, and dependency information are converted into a uniform-dimensional numerical task feature vector and added to the task queue by one-hot encoding and min-max normalization.

3. The method as described in claim 2, characterized in that, The real-time acquisition and quantification of resource usage data from each computing node in the cluster yields a cluster resource status matrix, including: Real-time collection of resource usage data from each computing node in the cluster; The collected resource usage data is preprocessed for anomaly removal and smoothing. The available computing power index for each computing node is calculated based on the preprocessed data. The resource usage data and available computing power index are quantified to obtain a cluster resource status matrix, which reflects the resource usage and available computing power of each computing node in the cluster in real time.

4. The method as described in claim 3, characterized in that, The step of calculating the optimal computing power allocation scheme based on the task feature vector and resource state matrix using a multi-objective optimization algorithm includes: Calculate the overall priority of each task in the task queue based on the task feature vector; Using the cluster resource state matrix as constraints, a multi-objective optimization model is constructed with the goal of maximizing system throughput and minimizing average task response time. The multi-objective optimization model is solved by an optimization algorithm to generate the mapping relationship between each task and computing node and the resource quota scheme. The execution order and resource quota of the tasks are determined according to the comprehensive priority.

5. The method as described in claim 4, characterized in that, The method of allocating a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme includes: Based on the generated mapping relationship between each task and computing node and the resource quota scheme, the large model computing graph structure is loaded and parsed. Select the appropriate sharding strategy based on the task type to split the task into multiple sharded tasks with fine granularity; The sharded tasks are mapped to corresponding computing nodes, and data synchronization between the sharded tasks is achieved through a distributed communication framework, enabling the sharded tasks to be executed in parallel.

6. The method as described in claim 5, characterized in that, During the execution of large model tasks, adaptive adjustment of computing power and load balancing are performed, including: During the execution of the sharding task, the load rate of each computing node is monitored in real time and the system load balance is calculated. When the system load balancing exceeds a preset threshold, a load balancing mechanism is triggered to migrate the incremental sharded tasks on overloaded nodes to lightly loaded nodes, and / or adjust the sharding position of model parameters online to achieve dynamic rebalancing of computing resources.

7. The method as described in claim 6, characterized in that, The method further includes: The status of computing nodes and the execution status of sharded tasks are monitored in real time through heartbeat detection and task progress reporting mechanisms. Abnormal events are detected by combining rule detection and machine learning models, and the severity level of the abnormality is determined. Based on the severity level of the abnormality, graded fault recovery measures are implemented.

8. The method as described in claim 6, characterized in that, The method further includes: Once all sharded tasks have been completed, the execution results are aggregated and all computing resources used by the tasks are reclaimed. The cluster resource status matrix is ​​then updated, and the task execution data is stored in the historical task database.

9. The method as described in claim 1, characterized in that, Before receiving the large model task request, the process also includes: Through system initialization and resource registration operations, the hardware configuration information and available resources reported by each computing node are obtained and added to the cluster resource topology map after verification.

10. The method as described in claim 2, characterized in that, The method of predicting task computation using a prediction model trained on historical data includes: By using a gradient boosting tree model trained on historical data, the execution time and resource consumption of a task can be predicted based on the task type, input data size, and model version number.

11. The method as described in claim 3, characterized in that, The preprocessing of the collected resource usage data for anomaly removal and smoothing includes: The collected resource usage data is processed by using the 3σ criterion to remove outliers and by using the sliding window averaging method to smooth out transient fluctuations. Time series analysis was performed on the preprocessed data to calculate the moving average, variance, and trend rate of change of resource usage.

12. The method as described in claim 11, characterized in that, The calculation of the available computing power index for each computing node based on the preprocessed data includes: Based on the preprocessed data, a weighted summation method is used to quantitatively evaluate the overall resource status of the computing nodes and calculate the available computing power index of each computing node. The overall resource status includes: GPU memory utilization, GPU computing utilization, CPU utilization, and network bandwidth utilization.

13. The method as described in claim 4, characterized in that, The step of calculating the comprehensive priority of each task in the task queue based on the task feature vector includes: Based on the timeliness requirements, business importance, and computational workload of the tasks, the overall priority of each task in the task queue to be processed is calculated.

14. The method as described in claim 5, characterized in that, The step of selecting the appropriate sharding strategy based on the task type to finely split the task into multiple sharded tasks includes: For inference tasks, a partitioning strategy combining pipelined parallelism and tensor parallelism is adopted. Pipeline parallelism distributes different layers of the model to different computing nodes, while tensor parallelism partitions a single tensor along different dimensions to different computing nodes. For the training task, a sharding strategy combining data parallelism and model parallelism is adopted. Data parallelism distributes training data to different nodes in batches, while model parallelism distributes model parameters to different nodes.

15. The method as described in claim 6, characterized in that, It also includes at least one of the following: The load balancing mechanism is as follows: when the load balancing degree exceeds a preset threshold, the minimum load priority strategy is adopted to migrate the sharded tasks with long execution time and large amount of computation on the node with the highest load rate to the node with the lowest load rate. The incremental migration includes migrating the execution status and necessary data of the fragment tasks; The segmentation position for adjusting model parameters online is to gradually migrate model parameters without interrupting task execution.

16. The method as described in claim 7, characterized in that, The step of implementing graded fault recovery measures based on the severity level of the anomaly includes: For minor anomalies, automatic retry processing is performed; For general exceptions, the failed sharded tasks will be reassigned to other normal nodes for execution; For severe anomalies, the faulty node is isolated, and all tasks on the faulty node are migrated to other available nodes, while an alarm message is issued.

17. A dynamic allocation device for computing power of large models, characterized in that, The device includes: The receiving module is used to receive requests for large model tasks; The feature extraction and quantization module is used to extract task features according to the large model task request and convert the task features into task feature vectors. The data acquisition and quantization module is used to collect and quantify the resource usage data of each computing node in the cluster in real time to obtain the cluster resource status matrix. The scheme calculation module is used to calculate the optimal computing power allocation scheme based on the task feature vector and resource status matrix using a multi-objective optimization algorithm; The computing power allocation module is used to allocate a single large model task to multiple computing nodes for parallel execution based on the optimal computing power allocation scheme. The load balancing module is used to adaptively adjust computing power and balance load during the execution of large model tasks.

18. A device for dynamically allocating computing power for large-scale models, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model computing power dynamic allocation method as described in any one of claims 1 to 16.

19. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the large model computing power dynamic allocation method as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the large model computing power dynamic allocation method as described in any one of claims 1 to 16.

Images