Resource scheduling method and device of training platform and electronic equipment

Abstract

Embodiments of the present disclosure provide a resource scheduling method and device of a training platform and an electronic device, and relate to the technical field of cluster scheduling, and are used for allocating computing nodes in a computing cluster. The method comprises: obtaining a first running state of a first computing cluster and a second running state of a second computing cluster; wherein one of the first computing cluster and the second computing cluster is used for running a model inference task, and the other is used for running a model training task; the running state of the computing cluster running the model inference task is a running state after a preset time period, which is estimated based on a current running state and a historical running state; in the case that the first running state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second running state indicates that there is an idle node in the second computing cluster, the idle node is scheduled to the first computing cluster, so that the idle node executes the task of the first computing cluster.

Description

Technical Field

[0001] This disclosure relates to the field of cluster scheduling technology, and in particular to a resource scheduling method, apparatus and electronic equipment for a training and promotion platform. Background Technology

[0002] In recent years, generative artificial intelligence applications have experienced explosive growth, widely applied in various fields such as natural language processing, image generation, and intelligent recommendation, significantly changing people's lives and work. Large-scale models, as the core technological support for generative AI, place extremely high demands on the real-time performance, stability, and cost-effectiveness of services due to their typical online business characteristics. At the same time, large-scale model training, as the foundation for technological iteration, is characterized by computational intensity, huge resource consumption, and long training cycles. Its distributed tasks also constitute another significant load on the continuous and stable occupation of cluster resources.

[0003] However, the underlying computing infrastructure supporting large-scale model training and inference currently faces multiple severe challenges. On the one hand, large-scale model inference services need to cope with the dual pressures of a rapid expansion in model parameter size and an exponential increase in user requests, and the load exhibits highly dynamic fluctuations. Traditional static resource allocation schemes are insufficient to achieve efficient and economical real-time services. On the other hand, the huge demand and long-term occupation of GPU resources and / or other computing resources for large-scale model training tasks can easily lead to resource contention and queuing delays in multi-tenant cluster environments, affecting R&D efficiency. Therefore, it is necessary to explore effective ways to efficiently and fairly schedule and allocate limited computing resources to collaboratively meet the high throughput requirements of training tasks and the low latency requirements of inference services. Summary of the Invention

[0004] The purpose of this disclosure is to provide a resource scheduling method, apparatus, and electronic device for a training and inference platform, which is used to reasonably schedule the computing resources of the training and inference platform by combining the running status of the computing cluster of the running model inference service and the running status of the running model training task.

[0005] To achieve the above objectives, the embodiments of this disclosure provide the following technical solutions: On the one hand, a resource scheduling method for a training and inference platform is provided, including: obtaining a first operating state of a first computing cluster and a second operating state of a second computing cluster; wherein, one of the first computing cluster and the second computing cluster is used to run model inference tasks, and the other is used to run model training tasks; the operating state of the computing cluster running model inference tasks is the operating state after a preset time period estimated based on the current operating state and historical operating states; the operating state of the computing cluster running model training tasks is the current operating state; when the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are idle nodes in the second computing cluster, the idle nodes are scheduled to the first computing cluster, so that the idle nodes execute the tasks of the first computing cluster.

[0006] In the resource scheduling method of the aforementioned training and inference platform, the running status of the first and second computing clusters used for running model inference tasks (state within a future preset time period) and model training tasks (current state) are obtained respectively. When the load of the first computing cluster is greater than or equal to a preset load threshold and there are idle nodes in the second computing cluster, the idle nodes are scheduled to execute tasks on the first computing cluster. This allows for dynamic and reasonable allocation of computing resources based on the actual running status of the computing clusters and future predictions. Therefore, it can improve the overall efficiency of computing task processing and avoid problems such as idle and wasted computing resources or delayed task processing due to insufficient resources caused by static resource allocation or failure to consider future state predictions, which are common in related technologies.

[0007] In one possible implementation, idle nodes are scheduled to the first computing cluster so that they can execute tasks of the first computing cluster. This includes: determining the required number of nodes based on a first operating state; selecting the required number of target idle nodes in the second computing cluster in descending order of idle duration if the number of idle nodes in the second computing cluster is greater than the required number of nodes; wherein the idle duration is used to characterize the duration during which the idle nodes have not executed inference and / or training tasks before the current time point; and scheduling the target idle nodes to the first computing cluster.

[0008] In the resource scheduling method of the aforementioned training and push platform, the required number of nodes is determined based on the actual operating status of the first computing cluster. When there are sufficient idle nodes in the second computing cluster, target idle nodes are selected for scheduling in descending order of idle time. Simultaneously, nodes with shorter idle times remain in the original cluster to execute any subsequent tasks. This allows for dynamic optimization of resource allocation based on the real-time cluster load. It prioritizes the use of long-idle nodes to improve resource utilization while reducing cold-start overhead by minimizing the frequent migration of nodes with short idle times. This ensures task execution efficiency while avoiding resource waste and increased cold-start latency caused by rigid resource scheduling strategies in related technologies.

[0009] In one possible implementation, the running state of the computing cluster running the model inference task is obtained by inputting the current running state of the computing cluster running the model inference task into the running state prediction model. The operational status prediction model was trained in the following manner: The historical running states are divided into multiple sample running states by using a preset sliding window; the size of the preset sliding window is related to the sample running states. Based on multiple sample running states, the initial running state prediction model is trained until the iteration stopping condition is met, thus obtaining the running state prediction model.

[0010] The resource scheduling method of the aforementioned training and push platform employs a sliding window mechanism associated with the current operating state of the samples to train the operating state prediction model, enabling it to adapt to load change characteristics at different time scales. Furthermore, when the model is input with current and historical operating states, it can determine the most predictive historical data segment based on the sliding window, thereby generating future state prediction results that better reflect actual load trends, avoiding the problem of excessive prediction bias caused by static windows in related technologies.

[0011] In one possible implementation, when the first computing cluster is used to run the model inference task, the method further includes: determining whether a target node exists in the second computing cluster when a first running state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold and a second running state indicates that there are no idle nodes in the second computing cluster; wherein the target node is used to characterize the node scheduled from the first computing cluster to the second computing cluster; if the existence of the target node is determined, estimating the estimated impact of returning the target node to the first computing cluster on the model training task; and scheduling the target node, whose estimated impact is less than a preset impact threshold, to the first computing cluster.

[0012] In the resource scheduling method of the above training and inference platform, when the first computing cluster is overloaded and the second computing cluster has no idle nodes, the impact of reclaiming the target node (the computing node borrowed from the computing cluster running the model inference task to the model training task) on the training task can be estimated. This can balance the resource requirements of inference and training tasks while ensuring the stability of the model training task, and avoid the risk of training task interruption or performance degradation caused by forced node reclamation in related technologies.

[0013] In one possible implementation, if there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are nodes in a protected state in the computing cluster running the model inference task, a target running node is selected from the protected nodes in descending order of the number of times the inference task has been executed, and the inference tasks of the unassigned nodes are assigned to the target running node; wherein, the protected state is used to characterize that the node is within a preset time interval after the completion of the inference task; or, if there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are no protected nodes in the computing cluster running the model inference task, a target running node is selected from the idle nodes in ascending order of idle time, and the inference tasks of the unassigned nodes are assigned to the target running node; wherein, the idle time is used to characterize the duration for which the idle node has not executed an inference task and / or a training task before the current time point.

[0014] In one possible implementation, the preset time interval corresponding to the node is positively correlated with the number of times the node continuously transitions from the protected state to the running state; the running state is the state of running inference tasks.

[0015] In the resource scheduling method of the aforementioned training and inference platform, unassigned model inference tasks are prioritized for scheduling to computing nodes that are in a protected state and have been executed a large number of times consecutively. This ensures that the target running node is able to continuously process similar inference tasks in the near future. Simultaneously, its computing environment, model weights, and cached data (such as feature maps and intermediate calculation results) are kept in a highly ready state. This avoids the latency overhead caused by model reloading, memory defragmentation, and cache rebuilding during the cold start of idle nodes, thereby improving task response speed and achieving synergistic optimization of inference service throughput and resource utilization.

[0016] In addition, by prioritizing the allocation of new tasks to highly active nodes, the system ensures that critical nodes remain in a hot state to maintain service quality, while also creating opportunities for inactive nodes to exit the protection state as early as possible and be released for training tasks. This optimizes the overall turnover efficiency and utilization of cluster resources.

[0017] Secondly, in the absence of protected nodes in the computing cluster for model inference tasks, prioritizing nodes with shorter idle times to undertake new inference tasks ensures that nodes that have recently run inference tasks retain some computational context (such as cached data and loaded model weight fragments), reducing cold start overhead and inference latency. Simultaneously, it allows nodes with longer idle times to remain continuously "available," reserving a readily usable resource pool for training tasks. This optimizes the overall utilization efficiency of cluster resources while ensuring low latency for inference services.

[0018] In one possible implementation, the method further includes: splitting the model inference task into at least one computation phase and at least one communication phase; wherein the computation phase is the operation phase performed by the computation node for the model inference task; the communication phase is the phase in which different communication nodes exchange data for the model inference task; and when running the model inference task, the at least one computation phase and the at least one communication phase are executed in parallel in the time dimension.

[0019] In the resource scheduling method of the above training and inference platform, by splitting the model inference task into a computation stage and a communication stage and achieving overlapping of runtime, the overall execution time of the task can be shortened and the communication overhead can be reduced by utilizing the parallel processing mechanism of computation and communication.

[0020] In one possible implementation, the method further includes: obtaining the real-time communication overhead of the model training task and the resource allocation of the computing cluster; adjusting the hyperparameters of the training task based on the communication overhead and resource allocation; wherein the hyperparameters include batch size and / or learning rate.

[0021] In the resource scheduling method of the above training and push platform, by acquiring the communication overhead and computing cluster resource allocation of the model training task in real time, and adjusting the training hyperparameters based on the communication overhead and computing cluster resource allocation, the training process can be self-optimized according to the actual cluster load and communication efficiency. Thus, while ensuring the model convergence quality, the training delay caused by resource competition or communication bottlenecks can be effectively reduced, and the training task can be executed efficiently.

[0022] On the other hand, a resource scheduling device for a training and inference platform is provided. The device includes: an acquisition unit and a scheduling unit; the acquisition unit is used to acquire a first operating state of a first computing cluster and a second operating state of a second computing cluster; wherein, one of the first computing cluster and the second computing cluster is used to run a model inference task, and the other is used to run a model training task; the operating state of the computing cluster running the model inference task is the operating state after a preset time period estimated based on the current operating state and the historical operating state; the operating state of the computing cluster running the model training task is the current operating state; the scheduling unit is used to schedule idle nodes to the first computing cluster when the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are idle nodes in the second computing cluster, so that the idle nodes execute the tasks of the first computing cluster.

[0023] In one possible implementation, the scheduling unit is specifically used to: determine the number of nodes required based on a first operating state; if the number of idle nodes in the second computing cluster is greater than the number of nodes required, select the number of target idle nodes in descending order of idle duration; wherein the idle duration is used to characterize the duration during which the idle node has not performed inference tasks and / or training tasks before the current time point; and schedule the target idle nodes to the first computing cluster.

[0024] In one possible implementation, the running state of the computing cluster for the running model inference task is obtained by inputting the current running state of the computing cluster for the running model inference task into the running state prediction model; wherein, the running state prediction model is trained in the following way: the historical running states are divided into multiple sample running states by using a preset sliding window; the size of the preset sliding window is related to the sample running states; based on the multiple sample running states, the initial running state prediction model is trained until the iteration stopping condition is met, and the running state prediction model is obtained.

[0025] In one possible implementation, when the first computing cluster is used to run model inference tasks, the scheduling unit is further configured to: determine whether a target node exists in the second computing cluster when a first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and a second operating state indicates that there are no idle nodes in the second computing cluster; wherein the target node is used to characterize a node scheduled from the first computing cluster to the second computing cluster; when it is determined that a target node exists, estimate the estimated impact of returning the target node to the first computing cluster on the model training task; and schedule the target node, whose estimated impact is less than a preset impact threshold, to the first computing cluster.

[0026] In one possible implementation, the scheduling unit is further configured to: when there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are nodes in a protected state in the computing cluster running the model inference task, select a target running node from the nodes in the protected state in descending order of the number of times the inference task has been executed, and assign the inference tasks of the unassigned nodes to the target running node; wherein, the protected state is used to characterize that the node is within a preset time interval after the completion of the inference task; or, when there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are no nodes in the protected state in the computing cluster running the model inference task, select a target running node from the idle nodes in ascending order of idle duration, and assign the inference tasks of the unassigned nodes to the target running node; wherein, the idle duration is used to characterize the duration for which the idle node has not executed an inference task and / or a training task before the current time point.

[0027] In one possible implementation, the preset time interval corresponding to the node is positively correlated with the number of times the node continuously transitions from the protected state to the running state; the running state is the state of running inference tasks.

[0028] In one possible implementation, the apparatus further includes: a processing unit; the processing unit is configured to: divide the model inference task into at least one computation stage and at least one communication stage; wherein the computation stage is a computation stage performed by a computation node for the model inference task; the communication stage is a stage for data exchange between different communication nodes for the model inference task; and when running the model inference task, the at least one computation stage and the at least one communication stage are executed in parallel in the time dimension.

[0029] In one possible implementation, the acquisition unit is further configured to acquire the real-time communication overhead of the model training task and the resource allocation of the computing cluster; the processing unit is further configured to adjust the hyperparameters of the training task based on the communication overhead and resource allocation; wherein the hyperparameters include batch size and / or learning rate.

[0030] On the other hand, an electronic device is provided, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the resource scheduling method of the training and push platform described above.

[0031] On the other hand, this application provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in the first aspect and any of its possible implementations.

[0032] On the other hand, this application provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0033] The resource scheduling device, electronic device, computer-readable storage medium, and computing program product of the above-mentioned training and promotion platform have the same beneficial technical effects as the resource scheduling method of the training and promotion platform provided in some of the above embodiments, and will not be described again here. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this disclosure, the accompanying drawings used in some embodiments of this disclosure will be briefly described below. Obviously, the drawings described below are only drawings of some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams and are not intended to limit the actual size of the product, the actual flow of the method, the actual timing of the signals, etc. involved in the embodiments of this disclosure.

[0035] Figure 1 This is a schematic diagram of the hardware structure of a resource scheduling system for a training and promotion platform according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a resource scheduling method for a training and promotion platform according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating resource scheduling according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating yet another type of resource scheduling according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating a node state according to an exemplary embodiment; Figure 6 This is a schematic diagram illustrating yet another type of resource scheduling according to an exemplary embodiment; Figure 7 This is a schematic diagram illustrating yet another node state according to an exemplary embodiment; Figure 8 This is a schematic diagram illustrating yet another type of resource scheduling according to an exemplary embodiment; Figure 9 This is a schematic diagram illustrating an overlapping mechanism according to an exemplary embodiment; Figure 10 This is a schematic diagram illustrating a model lightweighting according to an exemplary embodiment; Figure 11 This is a schematic diagram illustrating hyperparameter tuning according to an exemplary embodiment; Figure 12 This is a schematic diagram illustrating yet another type of hyperparameter tuning according to an exemplary embodiment; Figure 13 This is a schematic diagram illustrating yet another type of hyperparameter tuning according to an exemplary embodiment; Figure 14 This is a schematic diagram illustrating the scaling up and down of a computing node according to an exemplary embodiment; Figure 15 This is a schematic diagram illustrating a resource scheduling method for a training and promotion platform according to an exemplary embodiment; Figure 16 This is a block diagram illustrating a resource scheduling device for a training and promotion platform according to an exemplary embodiment; Figure 17 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0036] The technical solutions in some embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments provided in this disclosure are within the scope of protection of this disclosure.

[0037] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, materials, or characteristics may be included in any suitable manner in any one or more embodiments or examples.

[0038] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0039] In recent years, generative artificial intelligence applications have experienced explosive growth, widely applied in various fields such as natural language processing, image generation, and intelligent recommendation, significantly changing people's lives and work. Large-scale models, as the core technological support for generative AI, place extremely high demands on the real-time performance, stability, and cost-effectiveness of services due to their typical online business characteristics. At the same time, large-scale model training, as the foundation for technological iteration, is characterized by computational intensity, huge resource consumption, and long training cycles. Its distributed tasks also constitute another significant load on the continuous and stable occupation of cluster resources.

[0040] Meanwhile, large model training typically involves computational intensity, high resource consumption, and long training cycles, posing significant challenges to distributed training systems. Especially in multi-tenant environments, where multiple large model training tasks share cluster resources, how to efficiently and fairly allocate limited computing resources has become a key issue in improving training efficiency and reducing operating costs.

[0041] However, the underlying computing infrastructure supporting large-scale model training and inference currently faces multiple severe challenges. On the one hand, large-scale model inference services need to cope with the dual pressures of a rapid expansion in model parameter size and an exponential increase in user requests, and the load exhibits highly dynamic fluctuations. Traditional static resource allocation schemes are insufficient to achieve efficient and economical real-time services. On the other hand, the huge demand and long-term occupation of GPU resources and / or other computing resources for large-scale model training tasks can easily lead to resource contention and queuing delays in multi-tenant cluster environments, affecting R&D efficiency. Therefore, it is necessary to explore effective ways to efficiently and fairly schedule and allocate limited computing resources to collaboratively meet the high throughput requirements of training tasks and the low latency requirements of inference services.

[0042] Based on this, this application provides a resource scheduling method for a training and inference platform. By obtaining the running status of the first and second computing clusters used for running model inference tasks (state within a future preset time period) and model training tasks (current state), respectively, and when the load of the first computing cluster is greater than or equal to a preset load threshold and there are idle nodes in the second computing cluster, the idle nodes are scheduled to the first computing cluster to execute tasks. This method can achieve dynamic and reasonable allocation of computing resources based on the actual running status of the computing clusters and future predictions, thereby improving the overall computing task processing efficiency and avoiding problems such as idle and wasted computing resources or delayed task processing due to insufficient resources caused by static resource allocation or failure to consider future state predictions in related technologies.

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] like Figure 1 As shown, Figure 1 This is a schematic diagram of the hardware structure of a resource scheduling system for a training and promotion platform, according to an exemplary embodiment.

[0045] In one possible implementation, the resource scheduling system of the training and push platform may include a resource scheduling device 101, a first computing cluster 102, and a second computing cluster 103. Optionally, Figure 1 The resource scheduling device 101 of the training and promotion platform can establish communication connections with the first computing cluster 102 and the second computing cluster 103 respectively.

[0046] The first computing cluster 102 and the second computing cluster 103 can each be composed of multiple computing nodes. Each computing node can be a server, a computing host, or a hardware device equipped with a dedicated computing chip (such as a computing device containing a GPU / TPU / NPU). The specific implementation method of the computing nodes is not limited in this application embodiment.

[0047] In this embodiment, one of the first computing cluster 102 and the second computing cluster 103 is used to run the model inference task, and the other is used to run the model training task. Since the model training task mainly utilizes the high-precision floating-point computing power of the GPU for large-scale matrix operations, gradient calculations, and parameter updates, the entire process involves frequent backpropagation and optimizer steps, placing extremely high demands on the GPU's memory capacity, bandwidth, and multi-GPU interconnect performance. The model inference task, on the other hand, focuses on using the trained model for forward computation, typically using lower precision to improve throughput, and relies on the GPU's high memory access bandwidth to quickly load model weights to meet the requirements of low latency and high concurrency real-time response.

[0048] Therefore, as a feasible implementation method, the first computing cluster 102 and the second computing cluster 103 can be GPU computing clusters, that is, each computing node is equipped with a GPU computing chip and is a computing device with GPU computing power processing capability.

[0049] Alternatively, as another feasible implementation, the computing cluster can also be a hybrid cluster including GPUs and / or CPUs and other computing resources to adapt to model inference and training tasks of different scales and requirements. This application does not limit the specific implementation of the first computing cluster 102 and the second computing cluster 103.

[0050] In this embodiment of the application, for ease of explanation, the first computing cluster is used to perform model inference tasks and the second computing cluster is used to perform model training tasks. However, this description is not a limitation. In actual application, the second computing cluster can also be used to perform model inference tasks and the first computing cluster can be used to perform model training tasks.

[0051] Model training refers to the process of iteratively optimizing model parameters based on labeled or unlabeled sample datasets using specific algorithms and computing resources. The core objective is to enable the model to learn the features, patterns, and relationships within the data, thus equipping it with the ability to predict, analyze, or make decisions for specific scenarios. This task typically demands high computing power, requiring multi-node, high-performance computing clusters to handle massive amounts of data and update parameters. Once trained, the model will possess stable task processing capabilities and can be deployed to perform inference operations in real-world business scenarios.

[0052] Model inference refers to the process of deploying a trained and optimized model to a computing environment, receiving input data from real-world business scenarios, and performing calculations, analyses, and outputs results using the model's learned features and parameter systems. It is the core component for the practical application of models. This task prioritizes computational efficiency, response speed, and resource utilization, and can typically be flexibly deployed on general-purpose servers, GPU / FPGA heterogeneous computing clusters, or edge computing nodes according to business needs, adapting to different real-world application scenarios such as real-time requirements and high concurrency.

[0053] For example, in the field of natural language processing, the model training task can train a model similar to Bidirectional Encoder Representations from Transformers (BERT) to understand the deep semantics of text for sentiment analysis or text classification. The model inference task then uses this trained BERT model to judge the sentiment of comments entered by users in real time.

[0054] In the context of large language models, model training typically involves a multi-stage process of building and optimizing the model to learn the fundamental rules of language and world knowledge from massive amounts of internet text data, ultimately forming a large language model with general capabilities. Inference tasks involve deploying the trained large language model online to directly serve users' real-time requests, such as generating coherent answers to user questions, performing code completion, or creating creative text.

[0055] In this embodiment of the application, the resource scheduling device 101 is used to obtain the first operating state of the first computing cluster 102 and the second operating state of the second computing cluster 103.

[0056] The operating status of the first computing cluster 102 (running model inference tasks) is estimated based on the current operating status and historical operating status over a preset time period. The operating status of the second computing cluster 103 (running model training tasks) is the current operating status. The current operating status refers to the real-time resource demand exhibited by the computing cluster within the latest one or several monitoring sampling periods. Historical operating status refers to the operating data records of the computing cluster over a past period (e.g., three months), which may include resource demand and task volume for different time periods (e.g., weekdays, weekends, holidays). In model inference tasks, since usage may vary significantly over time (e.g., weekdays vs. weekends, daytime vs. nighttime), historical operating status may also include the operating status corresponding to specific time points (e.g., 10:00 AM, 5:00 PM, etc.).

[0057] It's important to note that model inference tasks, as online services, are extremely sensitive to response time. User request volume fluctuates significantly over time, especially during peak hours, and exhibits a periodic pattern, with higher request volumes during working hours and lower volumes during rest periods. Therefore, to ensure the accurate execution of model inference tasks, the resource requirements of the primary computing cluster can be estimated in advance to guarantee service quality and avoid performance degradation due to insufficient resources. By predicting the operational status after a preset time period, resource allocation can be dynamically adjusted to effectively address changes in request volume and ensure the stability and efficiency of the inference service. Therefore, for model inference tasks, it is necessary to determine the operational status of the primary computing cluster after a preset time period to proactively address load changes during different usage periods.

[0058] Furthermore, since the resource requirements of model training tasks primarily depend on the current training steps and algorithm execution, their resource requirements in the short term do not exhibit the predictable periodic fluctuations seen in inference tasks. Therefore, the current operating state can accurately reflect its resource requirements over the next period. Consequently, for model training tasks, the current operating state can be directly used as the operating state of the second computing cluster, and resource scheduling and training strategy adjustments can be based on this.

[0059] In this embodiment, the operating status can be obtained through communication with some control nodes in the computing cluster. These control nodes can be servers or nodes specifically responsible for cluster resource management and monitoring. They collect resource usage information of each computing node in the cluster in real time and summarize and analyze it. By establishing communication connections with these control nodes, for example, by using a specific communication protocol and periodically sending query requests, the current operating status data of the computing cluster can be obtained. For historical operating status, it can be queried and extracted from historical data records stored by the control nodes. This historical data is usually stored in a database for subsequent analysis and prediction. In this way, the operating status information of the computing cluster can be obtained in a timely and accurate manner, providing support for resource scheduling of model inference tasks and model training tasks. For example, taking the first and second computing clusters as GPU computing clusters, the current operating status can refer to the real-time resource demand status of the GPU computing cluster in at least one recent monitoring sampling period (e.g., sampling once per minute), reflecting the busy level of the GPU computing cluster at the current moment. Historical operating status refers to the operating data records of the GPU computing cluster in the past three months, including information such as the resource demand and workload of the GPU computing cluster in different time periods. From a time perspective, the runtime data records include runtime data for different types of dates, such as weekdays, weekends, and holidays. For example, on weekdays, the utilization and workload of the GPU computing cluster may be relatively high due to the high demand from users for model inference or training; while on weekends or holidays, the utilization and workload of the GPU computing cluster may be relatively low.

[0060] In this embodiment of the application, the resource scheduling device 101 is further configured to schedule idle nodes to the first computing cluster 102 when the first operating state indicates that the load of the first computing cluster 102 is greater than or equal to a preset load threshold and the second operating state indicates that there are idle nodes in the second computing cluster 103, so that the idle nodes can perform tasks of the first computing cluster 102.

[0061] Understandably, when the load on the first computing cluster 102 is greater than or equal to a preset load threshold, it indicates that the first computing cluster 102 is currently facing significant resource pressure, its resource utilization has approached or reached its limit, task processing speed has decreased, and there may even be task backlog or response timeouts. Therefore, node scheduling is necessary to ensure the efficiency, stability, and timeliness of tasks running on the first computing cluster 102.

[0062] Optionally, Figure 1 The resource scheduling device 101 of the training and promotion platform can be a terminal, a server, or other types of electronic equipment. Figure 1The diagram shown is merely an example of the device configuration of the resource scheduling device 101 of the training and promotion platform and does not constitute a limitation thereof. For example, when the resource scheduling device 101 of the training and promotion platform is a server, the server can be a single server, or it can be a server cluster composed of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This application does not impose any limitations in this regard.

[0063] When the resource scheduling device 101 of the training and push platform is a terminal, the terminal can be a mobile terminal, such as a computer with a mobile terminal, or a portable, pocket-sized, handheld, or computer-built-in mobile device that exchanges voice and / or data with the wireless access network, such as a mobile phone, tablet computer, laptop computer, netbook, or personal digital assistant (PDA). This application does not impose any limitations on this. It should be noted that the structure illustrated in the embodiments of this application does not constitute a limitation on the resource scheduling system of the training and push platform. It may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. For example, the resource scheduling device 101, the first computing cluster 102, and the second computing cluster 103 of the training and push platform can be independent components, or the resource scheduling device 101 of the training and push platform can be a node in the first computing cluster 102 or the second computing cluster 103. Optionally, the function of the resource scheduling device 101 can also be undertaken by any management node in the first computing cluster 102 or the second computing cluster 103, or it can be distributed across multiple computing nodes to collaboratively complete resource scheduling. This application does not impose specific limitations in this regard.

[0064] In some embodiments, Figure 2 This is a flowchart illustrating a resource scheduling method for a training and push platform according to an exemplary embodiment. The execution entity of the resource scheduling method for the training and push platform can be... Figure 1 The resource scheduling device 101 (hereinafter referred to as the resource scheduling device) of the training and promotion platform in the middle, such as Figure 2 As shown, the resource scheduling method of this training and promotion platform includes the following steps: S201-S202.

[0065] S201. Obtain the first running status of the first computing cluster and the second running status of the second computing cluster.

[0066] The operating status of the first computing cluster (taking the model inference task as an example) is the operating status after a preset time period, estimated based on the current operating status and historical operating status. The operating status of the second computing cluster (taking the model training task as an example) is the current operating status.

[0067] In one possible implementation, the resource scheduling device can determine the current operating status of the computing cluster based on the current resource occupancy of the computing cluster and the tasks of the computing nodes to be allocated.

[0068] The resource scheduling device can first collect the real-time resource usage of the computing cluster, including the current usage percentage and remaining availability of hardware resources such as computing power, storage, and bandwidth of each computing node in the cluster, as well as task information to be allocated to each computing node, including task type, task computing power requirement, task execution priority, and the range of nodes to be allocated to the task. Then, it can comprehensively analyze and evaluate the above resource data and task data to accurately determine the current operating status of the computing cluster, providing accurate and reliable decision-making basis for subsequent operations such as dynamic allocation of cluster resources, reasonable task scheduling, and node load balancing.

[0069] In one possible implementation, the resource scheduling device can input the current operating state of the first computing cluster into the operating state prediction model to obtain the estimated operating state of the first computing cluster after a preset time period. The operating state prediction model can be a hybrid architecture model of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network, or it can be a deep neural network model. This application does not impose specific limitations on this.

[0070] The operational status prediction model can be trained based on the historical operational status of the first computing cluster. Because the model's inference tasks exhibit significant time-varying and dynamic periodic characteristics, their resource requirements fluctuate regularly at different time points. This periodic fluctuation pattern forms identifiable characteristic patterns in historical data, such as more inference tasks during weekday morning rush hours and fewer inference tasks late at night on weekends. This reveals an inherent pattern in the historical data regarding resource requirements changing over time. Therefore, the operational status prediction model can be used to predict the operational status of the first computing cluster after a preset time period. The specific training process of the operational status prediction model is detailed in the following embodiment and will not be repeated here.

[0071] As a feasible approach, after obtaining the predicted operating status from the operating status prediction model, the predicted operating status can be post-processed to adjust and refine it by combining business knowledge and expert experience, for example, by adjusting the operating status upwards.

[0072] In one possible implementation, the resource scheduling device can acquire the first operating status and the second operating status in real time or according to a first preset period. For example, the first preset period can be 3 minutes or 5 minutes. This application does not impose specific limitations on this.

[0073] S202. When the first running state indicates that the load of the first computing cluster is greater than or equal to the preset load threshold, and the second running state indicates that there are idle nodes in the second computing cluster, the idle nodes are scheduled to the first computing cluster so that the idle nodes can execute the tasks of the first computing cluster.

[0074] Here, an idle node refers to a computing node in the computing cluster that is not currently running an inference task or a training task. Optionally, the preset load threshold can be set according to actual needs. For example, the preset load threshold can be 100% or 90%. This application does not impose specific restrictions on this.

[0075] When the load of the first computing cluster is greater than or equal to a preset load threshold, it indicates that the computing cluster is operating under high load. Existing computing resources cannot efficiently handle the current and pending computing tasks. To ensure the execution efficiency of various computing tasks and avoid task blocking or delays, it is necessary to consider using idle nodes from other computing clusters to supplement computing power and share the load. Therefore, it is necessary to further determine whether there are idle nodes in the second computing cluster. If there are no idle nodes in the second computing cluster, the idle nodes can be directly scheduled to the first computing cluster, allowing them to execute tasks from the first computing cluster. In one possible implementation, the resource scheduling device can determine the number of nodes required by the first computing cluster based on the first operating state. Then, if the number of idle nodes in the second computing cluster is greater than the number of nodes required, it can select the number of target idle nodes according to their idle duration from highest to lowest, and schedule these target idle nodes to the first computing cluster.

[0076] In another possible implementation, the resource scheduling device can schedule idle nodes to the first computing cluster if the number of idle nodes in the second computing cluster is less than or equal to the number of nodes required.

[0077] For example, such as Figure 3 As shown, Figure 3 A schematic diagram of resource scheduling is shown. Figure 3In this process, the resource scheduling device monitors new tasks (including inference tasks and / or training tasks). If the new task includes an inference task, the operating status of the second computing cluster (running model training tasks) and the first computing cluster (running model inference tasks) are immediately determined through information synchronization. If the new task only includes training tasks, the operating status of the second computing cluster and the first computing cluster are determined through periodic polling of information synchronization. If the new task only includes training tasks, and if the load on the second computing cluster exceeds a preset load threshold (execution of tasks 1, 2, 3, 4, and 5 causes the load to exceed the preset load threshold), and there are idle nodes in the first computing cluster, the idle nodes in the first computing cluster are scheduled to the second computing cluster to execute the model training task (training task 6).

[0078] For example, such as Figure 4 As shown, Figure 4 This diagram illustrates yet another type of resource scheduling. Figure 4 In this system, the resource scheduling device monitors new tasks (including inference tasks and / or training tasks). If the new task includes an inference task, the system immediately determines the operating status of the second computing cluster (running model training tasks) and the first computing cluster (running model inference tasks) through information synchronization. If the new task only includes training tasks, the system determines the operating status of the second computing cluster and the first computing cluster through periodic polling of information synchronization. Furthermore, when the load on the first computing cluster exceeds a preset load threshold and there are idle nodes in the second computing cluster (executing tasks 1 and 2), the idle nodes in the second computing cluster can be scheduled to the first computing cluster to execute model inference tasks.

[0079] It should be noted that the above embodiment uses the example of the first computing cluster (running the model inference task) calling a node of the second computing cluster for illustration. In actual applications, the node running the model training task can also call the computing nodes in the computing cluster running the model inference task. This application does not impose specific limitations in this regard.

[0080] Based on the above technical solution, this application can obtain the running status of the first and second computing clusters used to run model inference tasks (state within a future preset time period) and model training tasks (current state), respectively. When the load of the first computing cluster is greater than or equal to a preset load threshold and there are idle nodes in the second computing cluster, the idle nodes are scheduled to execute tasks in the first computing cluster. This can achieve dynamic and reasonable allocation of computing resources based on the actual running status of the computing clusters and future predictions, thereby improving the overall computing task processing efficiency and avoiding problems such as idle computing resources or task processing delays due to insufficient resources caused by static resource allocation or failure to consider future state predictions in related technologies.

[0081] In some embodiments, the model inference task exhibits certain regularities, and may be accompanied by sudden load spikes during specific periods (such as business peaks), leading to significant differences in the operational status at different times. Therefore, in order to accurately determine the actual load patterns, the operational status prediction model can be trained by dividing the data into samples of different lengths.

[0082] It should be noted that the device used to train the operational status prediction model can be the aforementioned resource scheduling device or other electronic devices, and this application embodiment does not impose any restrictions on this.

[0083] As a feasible implementation method, the operational status prediction model is trained in the following way: S301. The historical operating status is divided into multiple sample operating statuses by using a preset sliding window.

[0084] The size of the preset sliding window is related to the sample's running status.

[0085] It is understandable that a preset sliding window may experience sudden load spikes during specific periods (such as peak business hours), leading to significant differences in operational status at different times. This high degree of temporal imbalance makes it difficult for traditional static resource allocation or fixed-time-window prediction methods to determine the true load patterns. Therefore, this application employs a dynamic sliding window technique to construct the training dataset, where the sliding window size can be adaptively adjusted based on the resource load corresponding to the historical operational status of the samples.

[0086] For example, for an operating state reflecting low current resource load, and the corresponding time interval is in the low period of inference tasks (such as 2:00 AM to 5:00 AM), the number of model inference task requests is low and fluctuates gently during this period. In this case, a larger sliding window (such as a window length of 1 hour and a sliding step of 10 minutes) can be used to fully include the long-term change characteristics of the operating state during this period, helping the model learn the periodic patterns under low load scenarios. On the other hand, for an operating state reflecting high current resource load, and the corresponding time interval is in the peak business period (such as 10:00 AM to 12:00 PM on weekdays), the number of model inference task requests increases explosively during this period, accompanied by frequent fluctuations. In this case, a smaller sliding window (such as a window length of 10 minutes and a sliding step of 1 minute) can be used to capture the short-term abrupt changes in the operating state during this period, enabling the model to respond promptly to sudden load changes.

[0087] Optionally, when constructing the training dataset based on dynamic sliding windows, a dynamic weight adjustment mechanism for the sliding windows can be introduced. Each sliding window is assigned a corresponding weight based on the importance or drastic change in the operational status within the time period it covers. For example, a higher weight can be assigned to a small sliding window during peak business periods, as the short-term abrupt changes it captures are crucial for accurately predicting future operational status; while a lower but reasonable weight can be assigned to a large sliding window during off-peak business periods, considering the relatively stable long-term periodic patterns it learns. This allows for a greater emphasis on the impact of data during critical periods during model training, further enhancing the model's adaptability to different load scenarios.

[0088] S302. Based on multiple sample running states, train the initial running state prediction model until the iteration stopping condition is met to obtain the running state prediction model.

[0089] As a feasible approach, a dynamic loss function can be used in the training process of the operational state prediction model. The model can be trained by combining the mean absolute error (MAE) and the peak penalty term. In other words, MAE is used to handle data noise and the peak penalty term is used to suppress gradient explosion caused by outliers.

[0090] As can be seen, this application constructs a training dataset by adaptively adjusting the window size based on the historical resource load of the samples using dynamic sliding window technology. This captures the temporal characteristics under different load scenarios, enabling the operational status prediction model to perceive the resource load change trends at different times, predict business peaks and troughs in advance, and achieve on-demand dynamic allocation of resources, thereby improving resource utilization efficiency. Simultaneously, the use of a dynamic loss function to train the model effectively handles data noise and suppresses gradient explosion, improving the accuracy of the operational status prediction model.

[0091] In some embodiments, such as Figure 5 As shown, the node states in the first computing cluster (running model inference tasks) include running state, protected state, and idle state. The protected state indicates that a node is within a preset time interval after completing its inference task (running state complete). Nodes in the protected state are reserved for inference tasks and are prohibited from being scheduled to the second computing cluster to execute training tasks. This avoids communication overhead, memory fragmentation, and task execution latency fluctuations caused by cold starts during frequent alternations between training and inference tasks, thus ensuring low-latency stability of the inference service. The idle state indicates that if a computing node remains in the protected state for longer than a preset threshold (i.e., protection period timeout) and no new inference task request is detected, the node's state is automatically switched to idle. Nodes in the idle state are allowed to be scheduled to the second computing cluster. The running state indicates the state of a computing node while executing a model inference task; that is, the state of a node in the idle state or the protected state after receiving an inference task and executing the inference task. In the running state, computing nodes are not allowed to be called by the second computing cluster.

[0092] The preset time interval corresponding to a node is positively correlated with the number of times the node continuously transitions from a protected state to a running state. The running state refers to the state of running inference tasks.

[0093] In one possible implementation, for a computing node in a protected state, the resource scheduling device can record the number of inference tasks n completed during the protected period using a counter (n increments by 1 for each request processed), which is the number of consecutive transitions n from the protected state to the running state, and calculate a preset time interval based on the following formula: Tp = Tpmin + n × Interval.

[0094] Here, Tp can be used to represent a preset time interval. Tpmin can be used to represent the minimum time interval (e.g., Tpmin is 60 seconds for general inference services and 120 seconds for important inference services). n can be used to represent the number of inference tasks completed within the protection period, that is, the number of consecutive inference tasks run. Interval can be used to represent the interval duration (the higher the importance of the inference task, the larger the interval; for example, the interval for general services is 30 seconds, and the interval for important services is 60 seconds).

[0095] In addition, to prevent the protection time from being extended indefinitely due to excessively high request volume in a short period, a preset time interval upper limit Tpmax can be set (its value is also related to the importance of the inference task; for example, Tpmax is 600 seconds for general services and 1200 seconds for important services). When the computing node reaches the preset time interval, its corresponding run count counter n is reset to zero. In one possible implementation, the resource scheduling device can schedule or allocate tasks to the computing nodes in the first computing cluster based on the status of the computing nodes in the first computing cluster.

[0096] As a feasible implementation method, the resource scheduling method of the training and inference platform provided in this application embodiment further includes: when there are inference tasks with unallocated nodes in the first computing cluster and nodes in the first computing cluster in a protected state, selecting target running nodes from the nodes in the protected state in descending order of the number of times the inference tasks are executed, and allocating the inference tasks with unallocated nodes to the target running nodes.

[0097] Alternatively, if there are inference tasks with unassigned nodes in the first computing cluster and there are nodes in the first computing cluster that are under protection, the resource scheduling device can select a target running node from the nodes under protection in descending order according to a preset time interval, and assign the inference tasks with unassigned nodes to the target running node.

[0098] It should be noted that by prioritizing the scheduling of unassigned model inference tasks to computing nodes that are in a protected state and have been executed a large number of times consecutively, or computing nodes with a long preset time interval, it can be ensured that the target running node, due to its continuous processing of similar inference tasks recently, maintains a high state of readiness in its computing environment, model weights, and cached data (such as feature maps and intermediate calculation results). This avoids the latency overhead caused by model reloading, memory defragmentation, and cache reconstruction during the cold start of idle nodes, thereby improving task response speed and achieving synergistic optimization of inference service throughput and resource utilization.

[0099] In addition, by prioritizing the allocation of new tasks to computing nodes that are in a protected state and have been executed a lot of times or computing nodes with a long preset time interval, it not only ensures the continuous hot state of key nodes to maintain service quality, but also creates opportunities for low-activity nodes to exit the protected state as early as possible and be released for training tasks, thereby optimizing the overall turnover efficiency and comprehensive utilization of cluster resources.

[0100] As another feasible implementation, if there are inference tasks on unassigned nodes in the first computing cluster, and the first computing cluster does not have any protected nodes, target running nodes are selected from the idle nodes in ascending order of idle duration, and the inference tasks on the unassigned nodes are assigned to the target running nodes. Here, idle duration represents the length of time an idle node has not executed inference tasks and / or training tasks before the current time point.

[0101] In this embodiment, when there are no protected nodes in the first computing cluster, nodes with shorter idle times are preferentially selected to undertake new inference tasks. This ensures that nodes that have recently run inference tasks retain some computing context (such as cached data, loaded model weight fragments, etc.), reducing cold start overhead during task startup and lowering inference latency. Simultaneously, nodes with longer idle times remain in a "borrowable" state, reserving an immediately available resource pool for training tasks. This optimizes the overall utilization efficiency of cluster resources while ensuring low latency for inference services.

[0102] In some embodiments, combined with Figure 5 ,like Figure 6 As shown, the state of nodes in the first computing cluster also includes a borrowing state. The borrowing state is used to characterize computing nodes in the first computing cluster that are scheduled to the second computing cluster.

[0103] As a feasible implementation method, the resource scheduling method of the training and inference platform provided in this application embodiment further includes: when the first computing cluster is used to run model inference tasks, the resource scheduling device can determine whether there is a target node in the second computing cluster when the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are no idle nodes in the second computing cluster. If the existence of a target node is determined, the estimated impact of returning the target node to the first computing cluster on the model training task is estimated. Target nodes with an estimated impact less than a preset impact threshold are scheduled to the first computing cluster.

[0104] In this context, the target node represents the node scheduled from the first computing cluster (running the model inference task) to the second computing cluster; that is, a computing node in the first computing cluster that is in a borrowed state. Since the model inference task needs to provide real-time services and decision support to users, its timely response is crucial to user experience. Significant delays could lead to serious consequences such as user churn and business losses. The training task, on the other hand, mainly optimizes and improves the model and does not directly participate in real-time business interactions. It can usually be carried out relatively flexibly in the background and has lower requirements for immediacy. Therefore, priority should be given to ensuring the inference task to improve the user experience.

[0105] It should be noted that, in order to minimize the negative impact on the training task and achieve reasonable and efficient allocation of training and promotion platform resources, and to avoid the training task being severely affected by blindly reclaiming the target node, this application may consider the degree of impact on the training task after reclamation of the target node, and schedule the target node in combination with the degree of impact.

[0106] The model inference task can include multiple sub-training tasks. The degree of impact refers to the level of performance impact caused by reclaiming a node on the model training task. For sub-training tasks that use data parallel processing and run on multiple computing nodes, reclaiming one node will usually only linearly reduce the training throughput without causing the task to be interrupted. Therefore, the degree of impact on the model training task is relatively small. On the contrary, if the sub-training task itself only depends on a single computing node to run, reclaiming that node will directly cause the training task to be interrupted, requiring a restart or recovery from a checkpoint. Therefore, the degree of impact is large.

[0107] For example, such as Figure 7 As shown, Figure 7 Training task 1, which employs a data parallel strategy, is currently running on 3 computing nodes. If one of the nodes (the target node) is reclaimed, the sub-training task can be automatically adjusted to continue execution on 2 nodes in parallel. The training throughput is expected to decrease by about 33%, which does not exceed the preset impact threshold (50%), so the target node can be reclaimed.

[0108] Figure 7 Neutron training task 2 occupies only 1 computing node. Reclaiming this node would cause the training to be completely interrupted, and the impact on the training task would be 100%, which exceeds the preset impact threshold (50%). Therefore, it will not be reclaimed.

[0109] Based on the above technical solution, by estimating the impact on the training task after reclaiming the target node (the computing node borrowed from the computing cluster running the model inference task to the model training task) when the first computing cluster is overloaded and the second computing cluster has no idle nodes, it is possible to balance the resource requirements of inference and training tasks while ensuring the stability of the model training task, and avoid the risk of training task interruption or performance degradation caused by forced node reclamation in related technologies.

[0110] As another feasible implementation method, the resource scheduling device allocates resources to computing nodes in the computing cluster running inference tasks in a priority order from high to low: protected state, idle state, and borrowed state.

[0111] For example, such as Figure 8 As shown, Figure 8In the resource scheduling process of (1) and (2), when a new inference task arrives in the first computing cluster, the resource scheduling device can prioritize the computing nodes with more historical processing times under the protection state (in order, protection node A and protection node B) to allocate the inference task.

[0112] In the resource scheduling process of (3) and (4), when there are no protected nodes, when a new inference task arrives in the first computing cluster, the resource scheduling device can give priority to selecting computing nodes with shorter idle time in the idle state (in order of idle node C, idle node D, and idle node E) to allocate inference tasks.

[0113] In the resource scheduling process of (5) and (6), when there are no idle nodes and there are borrowed nodes, the resource scheduling device can initiate a resource reclamation mechanism to select the target with the least impact on the training task after reclamation from the computing nodes occupied by the training task. Specifically, the resource scheduling device can traverse all borrowed computing nodes and evaluate the impact of reclamation on the training task on each node. For example, if task J1 adopts a large-scale data parallel strategy and occupies multiple computing nodes for training, after reclamation of one computing node, it can still switch to a small-scale data parallel mode to continue execution, and the impact is relatively controllable. However, if task J2 only relies on a single computing node, the reclamation operation will cause training interruption, and the impact will be greater. Therefore, the resource scheduling device can prioritize reclamation of the computing nodes occupied by task J1.

[0114] In some embodiments, in order to improve inference efficiency, the process of computing nodes running model inference tasks can also be optimized. The resource scheduling method of the training and inference platform provided in this application also includes: S901-S902.

[0115] S901. The model inference task is divided into at least one computation stage and at least one communication stage.

[0116] The computation phase is the computation stage performed by computing nodes for the model inference task; the communication phase is the stage where different communication nodes exchange data for the model inference task. In other words, the computation phase is a continuous sequence of operations that perform numerical calculations within a single computing node during the inference process, without requiring cross-device data transfer. The communication phase refers to the operations during the inference process that require data exchange between different computing nodes or between a node and memory.

[0117] For example, in a distributed neural network inference scenario, the computation phase may include forward propagation computations completed within a single computing node (such as matrix operations in convolutional layers and fully connected layers), while the communication phase involves the transmission of intermediate results between different nodes (such as cross-node data synchronization after feature map segmentation).

[0118] S902. When running a model inference task, at least one computation phase and at least one communication phase are executed in parallel over time.

[0119] In one possible implementation, when running a model inference task, the resource scheduling device can interleave at least one computation phase with at least one communication phase in parallel runtime to reduce communication overhead.

[0120] For example, such as Figure 9 As shown, Figure 9 A schematic diagram of an overlapping mechanism is shown. For example, for Figure 9 The Transformer architecture model in the code includes two main computational stages: attention layer computation and forward layer computation, as well as two related communication stages: distribution communication and merging communication. The resource scheduler can achieve time overlap by managing the execution flow of two inference batches (Batch 0 and Batch 1) in parallel: when Batch 0 executes its attention layer computation (computation stage), the resource scheduler can synchronously schedule Batch 1 for its distribution communication (communication stage); subsequently, when Batch 1 begins its attention layer computation, Batch 0 enters the distribution communication stage. This allows the communication operation time of one batch to be covered by the computation operation time of the other batch, thus masking the communication latency that would otherwise occur during serial execution, improving the utilization of computational units and the overall inference throughput of the system.

[0121] In one embodiment, the resource scheduling device can also optimize model inference from the perspectives of hybrid expert model parallel load balancing optimization and model lightweighting optimization, so as to improve model inference efficiency.

[0122] In one possible implementation, the resource scheduling device implements differentiated load balancing strategies for different computational stages of the hybrid expert model. During the pre-filling stage of the attention layer, a dynamic token rearrangement mechanism can intelligently group the input sequence, allocating tokens with similar expert activation patterns to the same computational nodes to achieve computational load balancing. Simultaneously, a prefetching strategy based on expert popularity is employed to prioritize loading high-frequency expert parameters into computational nodes, reducing pipeline bubbles.

[0123] During the decoding phase, the resource scheduling device can ensure a balanced cache usage across computing nodes through a hierarchical key-value cache management strategy, implement a priority-aware preemptive scheduling mechanism, and dynamically adjust the distribution of experts based on real-time expert call frequency statistics.

[0124] In the forward computing layer, the resource scheduling device can adaptively select the computing implementation method according to the real-time load intensity. When the number of tokens is small, group matrix multiplication is used, and when the number of tokens is large, dense matrix multiplication is used. At the same time, it combines a hybrid parallel strategy of tensor parallelism, pipeline parallelism and expert parallelism to configure the optimal parallel scheme for different stages.

[0125] In one possible implementation, the resource scheduling device can employ a low-rank adapter fine-tuning quantization method to achieve model lightweighting. For example... Figure 10 As shown, the resource scheduling device can train the low-rank adapter module based on the pre-trained model parameters after freezing 4-bit quantization.

[0126] Specifically, the resource scheduling device can quantize a pre-trained single model from 16 bits to 4 bits of precision; then freeze all quantization parameters unchanged; next, add a trainable low-rank adapter; during fine-tuning, the gradient is backpropagated through the quantized base model, updating only the adapter parameters; thus, at the final inference, an architecture combining a 4-bit base model and an adapter (16 bits) is used, while keeping the optimizer state (32 bits) unchanged, reducing memory usage and computational requirements.

[0127] In another implementation, the resource scheduling device can also optimize model training, including hybrid expert model parallel load balancing optimization and training communication optimization.

[0128] In one possible implementation, the resource scheduling device can dynamically update its deviation parameters based on recent load conditions from experts.

[0129] Specifically, after each training step, the resource scheduling device can count the number of tokens processed by each expert as a load estimate. Based on this estimate, the bias parameters of the experts are adjusted in real time, so that the probability of experts with high loads being selected in routing decisions is reduced accordingly, thereby achieving load balancing. At the same time, for high-load experts, redundant replicas are created and distributed across different graphics processors, so that the original expert and replica experts are reasonably allocated to different computing nodes. During routing decisions, the load is distributed across multiple expert replicas, effectively alleviating the load pressure at a single point.

[0130] In one possible implementation, the resource scheduling device can employ a bidirectional pipeline scheduling strategy to achieve overlapping execution of computation and communication. Addressing the complex communication requirements in hybrid expert model training, it overcomes the limitations of traditional unidirectional pipeline scheduling by simultaneously inputting micro-batches from both ends of the pipeline—one end receiving micro-batches for forward computation, and the other end receiving micro-batches for reverse computation. This maximizes pipeline resource utilization, reduces idle waiting time for computation units, and improves training efficiency.

[0131] Based on the above technical solution, this application can shorten the overall execution time of the task and reduce communication overhead by splitting the model inference task into a computation stage and a communication stage and achieving overlapping of the running time.

[0132] In some embodiments, in order to improve the training efficiency of the model training task, the resource scheduling method of the training and push platform provided in this application embodiment may further include steps 1101-1102.

[0133] S1101. Obtain the real-time communication overhead of the model training task and the resource allocation of the computing cluster.

[0134] In one possible implementation, the resource scheduling device can collect the communication overhead of the model training task and the real-time resource allocation of the computing cluster during the model training process.

[0135] For example, regarding communication overhead, the resource scheduling device can collect data on the latency of gradient synchronization between different computing nodes, the bandwidth utilization of cross-server communication links, and the completion time of a specific set of communication operations to determine the communication overhead. Regarding monitoring resource allocation, the resource scheduling device can obtain the total number of computing nodes allocated to the current model training task, the specific distribution topology of the computing nodes across servers (e.g., whether they are concentrated on the same server or scattered across different servers), and the resource occupancy status of different computing nodes in the computing cluster to determine the resource allocation situation.

[0136] S1102. Adjust the hyperparameters of the training task based on communication overhead and resource allocation.

[0137] The hyperparameters corresponding to the training task may include key parameters that affect training efficiency, such as batch size and / or learning rate.

[0138] In one embodiment, the resource scheduling device can dynamically adjust the resources and hyperparameters of the training task by sensing the dynamic changes in cluster resource status and task load in real time.

[0139] For example, such as Figure 11As shown, when it is detected that there are sufficient computing resources for the running model training task and the number of training tasks to be processed is relatively small, the resource scheduling device can identify the high-priority task (e.g., task 1) in the currently running tasks, expand the capacity of the high-priority task, and increase the computing resources allocated to task 1 to improve its parallel computing capability. At the same time, the resource scheduling device can synchronously adjust the key hyperparameters of task 1, for example, appropriately increase the batch size to improve the data throughput of single-step computation, and correspondingly increase the learning rate according to the linear scaling rule to match the increased effective batch size, maintaining stable training dynamics.

[0140] Furthermore, upon receiving a new training task, the resource scheduler can reassess the overall resource status of the computing cluster and the priority weights of all tasks. Based on the assessment results, it can scale down the resources of the currently running task (such as Task 1), reclaiming some GPU computing units to allocate to the new task. During this process, the resource scheduler can also readjust the hyperparameters of the affected task, correspondingly reducing the batch size and learning rate to ensure that the task maintains stable training convergence even after resource reduction.

[0141] In one possible implementation, the resource scheduling device can dynamically adjust the batch size based on real-time communication quality to reduce communication overhead. When an increase in inter-node communication latency or excessive bandwidth utilization is detected, the batch size is automatically reduced to decrease the amount of synchronous communication; conversely, when the communication link is idle, the batch size is appropriately increased to improve computational throughput.

[0142] For example, such as Figure 12 As shown, when Task 1 runs on GPUs 2 / 3 / 4 (compute nodes) of Server 1 and GPUs 2 / 3 / 4 of Server 2 (a total of 6 GPUs), while GPU 1 on both servers is idle, Task 1 can exclusively occupy all the network communication bandwidth between the two servers. Under these conditions of ample communication resources, the resource scheduling mechanism will adjust the batch size of Task 1 to a larger value (e.g., 16) to fully utilize computing resources and improve training throughput.

[0143] Then, when the new task 2 is scheduled to GPU 1 on both servers, task 1 and task 2 need to share the communication bandwidth between the servers. To balance the communication needs of the two tasks and achieve optimal overall training efficiency, the resource scheduling device will reduce the batch size of task 1 accordingly (e.g., from 16 to 12). By reducing the amount of data in a single communication, the communication efficiency of task 2 is ensured while maintaining stable training of task 1, thus achieving a reasonable allocation of communication resources among multiple tasks.

[0144] After Task 2 completes training and releases its resources, Task 1 regains its exclusive communication bandwidth. At this point, the resource scheduler will restore the batch size of Task 1 to a larger value (e.g., 16). If GPU 1 is found to be continuously idle, Task 1 can be further expanded to run on all GPUs, and hyperparameters such as the learning rate can be adjusted simultaneously to achieve collaborative optimization of resource allocation and parameter tuning.

[0145] In addition, the resource scheduling device can also be based on resource utilization and training stability.

[0146] Specifically, regarding resource utilization, the resource scheduling device can establish a linkage between batch size and resource load by monitoring the utilization metrics of graphics processors and other computing resources in real time. If the overall resource utilization consistently falls below a set threshold, the resource scheduling device can gradually increase the batch size in predetermined steps, thereby increasing the number of active threads in the computing unit and fully utilizing idle computing resources. Alternatively, when resource utilization approaches saturation or there is a risk of overload, the resource scheduling device can appropriately reduce the batch size to avoid performance degradation or system instability caused by resource contention, ensuring that computing tasks run within an efficient and stable load range.

[0147] Regarding training stability, during training, the resource scheduling mechanism continuously monitors key stability indicators such as fluctuations in the loss function curve and changes in the gradient norm. If abnormal fluctuations in the loss value, gradient explosion, or vanishing gradients are detected, the resource scheduling mechanism can activate a stability protection mechanism, appropriately reducing the batch size to provide more frequent gradient updates and a more stable optimization path, thereby improving the convergence and reliability of model training. Once the training state stabilizes, the batch size is gradually adjusted to the optimal level.

[0148] In one possible implementation, the resource scheduling device can adjust the learning rate in response to changes in resource allocation, depending on resource availability. When the training task is expanded, the learning rate is increased accordingly according to a linear scaling rule; when resources are reclaimed, causing a reduction in capacity, the learning rate is decreased simultaneously to maintain training stability.

[0149] For example, such as Figure 13As shown, when GPU 1 (compute node) on Server 1 and Server 2 is idle for an extended period, the training scheduler automatically scales up Task 1 (using all GPUs) and moderately increases the learning rate while scaling up to maintain an appropriate training dynamic range and accelerate convergence. Alternatively, when a new task arrives and uses GPU 1, Task 1 needs to be able to scale down and reduce the learning rate accordingly to ensure training stability.

[0150] In addition, the resource scheduling device can also adjust the hyperparameter learning rate based on preheating strategy, decay strategy, and adaptive strategy.

[0151] Specifically, the warm-up strategy is as follows: In the initial stage of training, the resource scheduling device adopts a gradual warming mechanism, starting with a small learning rate and gradually increasing it to a preset target value according to a preset growth strategy (such as linear growth or exponential growth). This alleviates problems such as gradient instability and loss value oscillation that may be caused by directly using a large learning rate for random parameter initialization in the early stage of training. It provides a smooth transition stage for model parameter updates and improves the stability and convergence efficiency in the early stage of training.

[0152] Decay Strategy: As training iterations progress and the model gradually approaches the optimal solution, the resource scheduling device can activate a learning rate decay mechanism. The resource scheduling device can gradually reduce the learning rate according to a pre-set schedule (e.g., phased decay), indicators such as accuracy improvement bottlenecks or the smoothness of loss changes, following specific rules (e.g., stepwise, cosine annealing). The decay strategy helps the model perform more refined parameter tuning in the later stages of convergence, avoiding excessive oscillations near the optimal solution, thereby improving the model's final generalization performance and convergence accuracy.

[0153] Adaptive adjustment strategy: The resource scheduling device can monitor dynamic indicators such as the convergence trend of the training loss curve and the changing characteristics of the gradient distribution in real time to adjust the learning rate. For example, if the loss descent stalls or the gradient norm shrinks significantly, the learning rate can be moderately increased to escape local optima or accelerate convergence; conversely, if there are signs of drastic loss fluctuations or gradient explosion, the learning rate can be decreased to restore training stability.

[0154] In another possible implementation, during actual training, there are complex correlations between different hyperparameters, and the resource scheduling device can jointly adjust the hyperparameters to improve the model training effect.

[0155] For example, joint tuning may include: joint tuning of learning rate and batch size, joint tuning of model capacity and regularization parameters, and joint tuning of parallelism strategy and communication configuration.

[0156] Joint adjustment of learning rate and batch size: The resource scheduler can jointly optimize the adjustment of batch size and learning rate, following the linear scaling rules in deep learning. When the learning rate needs to be adjusted, the resource scheduler can proportionally adjust the batch size to maintain a balance between the two. For example, when the learning rate increases by a factor of k, the batch size is increased by a factor of k accordingly to maintain the effectiveness of gradient updates and the stability of training dynamics.

[0157] Joint adjustment of model capacity and regularization parameters: When the model structure changes, the resource scheduling device can adjust regularization hyperparameters such as weight decay accordingly. By strengthening the regularization constraints, the risk of overfitting that may be caused by the increase in model complexity can be offset, thereby improving the model's expressive power while ensuring its generalization performance.

[0158] Joint adjustment of parallel strategies and communication configuration: For distributed training environments, the resource scheduling device can simultaneously optimize the corresponding communication configuration parameters while adjusting parallelization strategies such as data parallelism, model parallelism, or pipeline parallelism. For example, when using finer-grained model parallelism, the resource scheduling device can adjust the gradient synchronization communication algorithm or introduce activation recomputation strategies to achieve the best balance between computation and communication overhead.

[0159] In another possible implementation, the resource scheduling device can also dynamically adjust the priority of training tasks based on resource utilization efficiency, task completion progress, system feedback, and changes in the learning rate, ensuring the efficiency and fairness of resource allocation.

[0160] Specifically, for resource utilization efficiency-based tasks, the resource scheduling device can monitor the actual utilization efficiency of allocated computing resources (such as GPUs and memory) for each training task. For tasks that can efficiently utilize resources and achieve high computing throughput, the resource scheduling device can appropriately increase their scheduling priority, allowing them to receive more resource allocation; conversely, for tasks with severely idle resources or low utilization efficiency, the resource scheduling device can correspondingly decrease their priority, incentivizing task-level resource optimization and improving the overall efficiency of the cluster.

[0161] Regarding task completion progress, the resource scheduling device can assess the progress in real time by monitoring indicators such as the number of iterations and the loss reduction curve. Specifically, for tasks that are close to the training objective (such as achieving the expected accuracy or completing most iterations), the resource scheduling device can gradually reduce their priority to create conditions for them to smoothly end training and release resources; for tasks in the early or critical stages of training, the resource scheduling device can maintain or increase their priority to ensure that they receive a continuous and stable supply of resources.

[0162] For system performance feedback, the resource scheduling device can use cluster-level performance metrics (such as overall throughput and average task queuing latency) as feedback signals. When overall efficiency decreases or latency increases, the resource scheduling device can adjust task scheduling strategies, for example, prioritizing task types that bring higher system throughput; when the system is running smoothly and efficiently, the resource scheduling device can perform fair scheduling.

[0163] Regarding changes in the learning rate, the resource scheduler can identify the inherent relationship between the learning rate and resource requirements (e.g., an increase in the learning rate usually requires more computational resources) and use changes in the learning rate as a reference factor for priority adjustment. When a task enters a high learning rate phase, the resource scheduler can appropriately increase its priority to meet its increased computational needs; when the learning rate decays, the resource scheduler can adjust its resource allocation strategy to keep resource allocation synchronized with the dynamic rhythm of model training.

[0164] In another possible implementation, the resource scheduling device can schedule different training tasks based on their priorities to maximize resource utilization and ensure the efficiency and fairness of the training tasks.

[0165] Priority is related to the value of the training task, training cost, and cost-effectiveness. Specifically, the resource scheduling device can evaluate training tasks based on their business value, strategic importance, and expected benefits, classifying tasks into different levels, such as high-value and low-value. High-value tasks typically make a crucial contribution to improving model performance or are closely related to the organization's core business development, such as model optimization training for important product lines; while low-value tasks are mostly experimental research or auxiliary model training.

[0166] The resource scheduling mechanism can assess the cost of tasks based on their required computing resources (such as the number of GPUs) and expected training cycle, categorizing them into high-cost and low-cost tasks. High-cost tasks typically require a large amount of computing resources over a long period, while low-cost tasks have relatively smaller resource requirements. By classifying tasks by cost, the resource scheduling mechanism can optimize resource allocation strategies, preventing high-cost tasks from excessively consuming cluster resources and impacting overall system efficiency.

[0167] Resource scheduling devices can comprehensively consider the value and cost of tasks, calculating the cost-effectiveness index of training tasks. For example, training tasks can be divided into high-cost-effectiveness and low-cost-effectiveness categories. High-cost-effectiveness tasks can obtain significant value returns with relatively less resource input and should be given priority when allocating resources.

[0168] Furthermore, when there are few training tasks, the resource scheduling device can prioritize resource allocation based on value grading to ensure that high-value tasks are processed first. Alternatively, when there are few training tasks, it can consider both value grading and cost-effectiveness grading to balance resource allocation for high-value and cost-effective tasks.

[0169] In addition, the resource scheduling device can prioritize high-value tasks when resources are scarce, maximizing value with minimal cost. Alternatively, when resources are plentiful, it can allocate more resources to high-value tasks, even if their cost-effectiveness is relatively low.

[0170] Based on the above technical solution, this application can obtain the communication overhead and computing cluster resource allocation of the model training task in real time, and adjust the training hyperparameters based on the communication overhead and computing cluster resource allocation. It can self-optimize the training process according to the actual cluster load and communication efficiency, thereby effectively reducing the training delay caused by resource competition or communication bottlenecks while ensuring the model convergence quality, and achieving efficient execution of the training task.

[0171] In some embodiments, when allocating resources, this application may scale up or down the computing nodes in the first computing cluster or the second computing cluster in the following ways.

[0172] In one possible implementation, such as Figure 14 As shown, when the first computing cluster needs to allocate resources, the resource scheduling device can synchronize key state information of distributed training during the replica synchronization phase, including model parameters, current training progress (such as iteration rounds, optimizer status, etc.), and all relevant hyperparameter settings. After synchronization is complete, all computing node replicas uniformly load the complete state information to resume the training task.

[0173] Figure 14 In this context, when computing resources change, related technology A adopts a complete process of stopping, saving the state, destroying the container, reallocating, initializing, and reloading, resulting in excessively long restart times. This is especially true for large-scale models, where saving and reloading checkpoints and reinitializing the distributed training environment incurs significant time overhead, causing resources to remain idle during the switching period and resulting in low overall resource utilization.

[0174] Figure 14In this approach, when computing resources change, related technique B only initializes the newly added computing nodes and allows training on existing computing nodes to continue, finally synchronizing parameters. Although the switching speed is fast, related technique B can only synchronize model parameters, ignoring key training contexts such as hyperparameters and optimizer states. This disrupts the training dynamics, causing inconsistencies in optimizer momentum and other states, leading to oscillations during training and potentially prolonging the total time required for model convergence.

[0175] In some embodiments, such as Figure 15 As shown, Figure 15 This is a schematic diagram of a resource scheduling method for a training and promotion platform provided in an embodiment of this application.

[0176] Figure 15 In this context, resource scheduling methods in the training and inference platform can include scheduling optimization and model optimization. Scheduling optimization can include joint inference and training scheduling optimization, inference scheduling optimization, and training scheduling optimization. Model optimization can include model inference optimization and model training optimization.

[0177] This application embodiment can, based on the above method, exemplarily divide the resource scheduling device or electronic device of the training and promotion platform into functional modules. For example, the resource scheduling device or electronic device of the training and promotion platform may include various functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0178] Figure 16 This is a block diagram illustrating a resource scheduling device for a training platform according to an exemplary embodiment. (Refer to...) Figure 16 The resource scheduling device of the training and promotion platform includes: an acquisition unit 1601 and a scheduling unit 1602; In one possible implementation, the acquisition unit 1601 is used to acquire the first operating state of the first computing cluster and the second operating state of the second computing cluster; wherein, one of the first computing cluster and the second computing cluster is used to run model inference tasks, and the other is used to run model training tasks; the operating state of the computing cluster running model inference tasks is the operating state after a preset time period estimated based on the current operating state and historical operating states; the operating state of the computing cluster running model training tasks is the current operating state; the scheduling unit 1602 is used to schedule idle nodes to the first computing cluster when the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are idle nodes in the second computing cluster, so that the idle nodes execute the tasks of the first computing cluster.

[0179] In one possible implementation, the scheduling unit 1602 is specifically used to: determine the number of nodes required based on the first operating state; if the number of idle nodes in the second computing cluster is greater than the number of nodes required, select the number of target idle nodes in descending order of idle duration; wherein the idle duration is used to characterize the duration during which the idle node has not performed inference tasks and / or training tasks before the current time point; and schedule the target idle nodes to the first computing cluster.

[0180] In one possible implementation, the running state of the computing cluster for the running model inference task is obtained by inputting the current running state of the computing cluster for the running model inference task into the running state prediction model; wherein, the running state prediction model is trained in the following way: the historical running states are divided into multiple sample running states by using a preset sliding window; the size of the preset sliding window is related to the sample running states; based on the multiple sample running states, the initial running state prediction model is trained until the iteration stopping condition is met, and the running state prediction model is obtained.

[0181] In one possible implementation, when the first computing cluster is used to run model inference tasks, the scheduling unit 1602 is further configured to: determine whether a target node exists in the second computing cluster when a first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and a second operating state indicates that there are no idle nodes in the second computing cluster; wherein the target node is used to characterize a node scheduled from the first computing cluster to the second computing cluster; when it is determined that a target node exists, estimate the estimated impact of returning the target node to the first computing cluster on the model training task; and schedule the target node, whose estimated impact is less than a preset impact threshold, to the first computing cluster.

[0182] In one possible implementation, the scheduling unit 1602 is further configured to: when there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are nodes in a protected state in the computing cluster running the model inference task, select a target running node from the nodes in the protected state in descending order of the number of times the inference task has been executed, and assign the inference tasks of the unassigned nodes to the target running node; wherein, the protected state is used to characterize that the node is within a preset time interval after the completion of the inference task; or, when there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are no nodes in the protected state in the computing cluster running the model inference task, select a target running node from the idle nodes in ascending order of idle duration, and assign the inference tasks of the unassigned nodes to the target running node; wherein, the idle duration is used to characterize the duration for which the idle node has not executed an inference task and / or a training task before the current time point.

[0183] In one possible implementation, the preset time interval corresponding to the node is positively correlated with the number of times the node continuously transitions from the protected state to the running state; the running state is the state of running inference tasks.

[0184] In one possible implementation, the apparatus further includes: a processing unit; the processing unit is configured to: divide the model inference task into at least one computation stage and at least one communication stage; wherein the computation stage is a computation stage performed by a computation node for the model inference task; the communication stage is a stage in which different communication nodes exchange data for the model inference task; and when running the model inference task, overlap the at least one computation stage and the at least one communication stage in terms of runtime to reduce the communication overhead of the model inference task.

[0185] In one possible implementation, the acquisition unit 1601 is further configured to acquire the real-time communication overhead of the model training task and the resource allocation of the computing cluster; the processing unit is further configured to adjust the hyperparameters of the training task based on the communication overhead and resource allocation; wherein the hyperparameters include batch size and / or learning rate.

[0186] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0187] Figure 17 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 17 As shown, the electronic device includes, but is not limited to, a processor 1701 and a memory 1702.

[0188] The memory 1702 described above is used to store the executable instructions of the processor 1701. It is understood that the processor 1701 is configured to execute instructions to implement the testing method in the above embodiments.

[0189] It should be noted that those skilled in the art will understand that Figure 17 The electronic device structure shown does not constitute a limitation on the electronic device; the electronic device may include, but is not limited to, other electronic devices. Figure 17 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0190] Processor 1701 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in memory 1702, and by calling data stored in memory 1702, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Processor 1701 may include one or more processing units. Optionally, processor 1701 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 1701.

[0191] The memory 1702 can be used to store software programs and various data. The memory 1702 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 1702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0192] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1702 including instructions, which can be executed by a processor 1701 of an electronic device to implement the methods in the above embodiments.

[0193] In actual implementation, Figure 16 The functions in the acquisition unit 1601 and the scheduling unit 1602 can both be provided by Figure 17 The processor 1701 calls the computer program stored in the memory 1702 to implement the process. The specific execution process can be found in the description of the method section in the previous embodiment, and will not be repeated here.

[0194] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, and an optical data storage device.

[0195] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by a processor 1701 of an electronic device to perform the methods described above.

[0196] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of an electronic device, they implement the various processes of the above method embodiments and achieve the same technical effect as the above method. To avoid repetition, they will not be described again here.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0202] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the test method described in the above method embodiments.

[0203] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the test method in the method flow shown in the above method embodiments.

[0204] The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, a register, a hard disk, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC). In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0205] Since the testing apparatus, computer-readable storage medium, and computer program product in the embodiments of this application can be applied to the above methods, the technical effects that can be obtained can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0206] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A resource scheduling method for a training and promotion platform, characterized in that, include: The first operating state of the first computing cluster and the second operating state of the second computing cluster are obtained; wherein, one of the first computing cluster and the second computing cluster is used to run a model inference task, and the other is used to run a model training task; the operating state of the computing cluster running the model inference task is the operating state after a preset time period estimated based on the current operating state and the historical operating state; the operating state of the computing cluster running the model training task is the current operating state; When the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are idle nodes in the second computing cluster, the idle nodes are scheduled to the first computing cluster, so that the idle nodes execute the tasks of the first computing cluster.

2. The method according to claim 1, characterized in that, The step of scheduling the idle node to the first computing cluster, so that the idle node executes the tasks of the first computing cluster, includes: Based on the first operating state, determine the required number of nodes; If the number of idle nodes in the second computing cluster is greater than the required number of nodes, the required number of target idle nodes are selected in descending order of idle duration; wherein, the idle duration is used to characterize the duration during which the idle node has not performed inference tasks and / or training tasks before the current time point; The target idle node is scheduled to the first computing cluster.

3. The method according to claim 1, characterized in that, The running state of the computing cluster running the model inference task is obtained by inputting the current running state of the computing cluster running the model inference task into the running state prediction model. The operating state prediction model is trained in the following manner: The historical running states are divided into multiple sample running states by using a preset sliding window; the size of the preset sliding window is related to the sample running states. Based on the running states of multiple samples, the initial running state prediction model is trained until the iteration stopping condition is met, thus obtaining the running state prediction model.

4. The method according to claim 1, characterized in that, When the first computing cluster is used to run the model inference task, the method further includes: When the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there are no idle nodes in the second computing cluster, it is determined whether there is a target node in the second computing cluster; wherein, the target node is used to characterize a node scheduled from the first computing cluster to the second computing cluster; If the existence of the target node is confirmed, estimate the estimated impact of returning the target node to the first computing cluster on the model training task; The target nodes whose estimated impact is less than the preset impact threshold are scheduled to the first computing cluster.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: If there are inference tasks with unassigned nodes in the computing cluster running the model inference task, and there are nodes in a protected state in the computing cluster running the model inference task, a target running node is selected from the nodes in the protected state in descending order of the number of times the inference task has been executed, and the inference tasks of the unassigned nodes are assigned to the target running node; wherein, the protected state is used to indicate that the node is within a preset time interval after the completion of the inference task; Alternatively, if there are inference tasks on unassigned nodes in the computing cluster running the model inference task, and there are no protected nodes in the computing cluster running the model inference task, a target running node is selected from the idle nodes in ascending order of idle duration, and the inference tasks on the unassigned nodes are assigned to the target running node; the idle duration is used to characterize the duration for which the idle node has not executed inference tasks and / or training tasks before the current time point.

6. The method according to claim 5, characterized in that, The preset time interval corresponding to the node is positively correlated with the number of times the node continuously transitions from the protected state to the running state; the running state is the state of running inference tasks.

7. The method according to claim 5, characterized in that, The method further includes: The model inference task is divided into at least one computation stage and at least one communication stage; wherein, the computation stage is the operation stage performed by the computing node for the model inference task; and the communication stage is the stage in which different communication nodes exchange data for the model inference task. When running the model inference task, at least one of the computation phases and at least one of the communication phases are executed in parallel over time.

8. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the real-time communication overhead and resource allocation of the computing cluster for the model training task; Based on the communication overhead and resource allocation, the hyperparameters of the training task are adjusted; wherein the hyperparameters include batch size and / or learning rate.

9. A resource scheduling device for a training and promotion platform, characterized in that, The device includes: an acquisition unit and a scheduling unit; The acquisition unit is used to acquire the first operating state of the first computing cluster and the second operating state of the second computing cluster; wherein, one of the first computing cluster and the second computing cluster is used to run a model inference task, and the other is used to run a model training task; the operating state of the computing cluster running the model inference task is the operating state after a preset time period estimated based on the current operating state and the historical operating state; the operating state of the computing cluster running the model training task is the current operating state; The scheduling unit is configured to schedule the idle node to the first computing cluster when the first operating state indicates that the load of the first computing cluster is greater than or equal to a preset load threshold, and the second operating state indicates that there is an idle node in the second computing cluster, so that the idle node executes the task of the first computing cluster.

10. An electronic device, characterized in that, include: Processor; memory for storing instructions executable by the processor; The processor is configured to execute instructions to implement the resource scheduling method of the training and push platform according to any one of claims 1-8.

Images