Resource scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a resource scheduling method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining resource use information corresponding to a model training task from target equipment in a process of executing the model training task by utilizing equipment resources of the target equipment; processing the resource use information and historical training behavior information of a target user initiating the model training task by using a performance evaluation model to obtain a performance evaluation result corresponding to the model training task, the performance evaluation result is used for representing the performance of the model training task in execution and / or the performance of the equipment resource; and according to the performance evaluation result, generating a resource scheduling strategy for the model training task, so as to schedule equipment resources for the model training task according to the resource scheduling strategy. According to the embodiment of the invention, accurate resource scheduling can be carried out on the equipment resources.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a resource scheduling method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the continuous development of computer technology and neural network technology, in various practical application scenarios, a trained neural network model can be used to process specific tasks, thereby meeting actual needs.

[0003] In the related art, in the process of executing a model training task for a neural network model by using device resources, due to the high uncertainty of the model training process, the device resources and the model training task can be mismatched, and thus the model training process can be at risk. Therefore, how to accurately perform resource scheduling in the process of executing the model training task by using the device resources has become a technical problem to be solved. SUMMARY

[0004] The present disclosure provides a resource scheduling method and device, electronic equipment, computer readable storage medium and computer program product.

[0005] In a first aspect, the present disclosure provides a resource scheduling method, which comprises: obtaining, in the process of executing a model training task by using device resources of a target device, resource usage information corresponding to the model training task from the target device; processing, by using a performance evaluation model, the resource usage information and historical training behavior information of a target user who initiates the model training task, to obtain a performance evaluation result corresponding to the model training task; the performance evaluation result is used to represent the performance of the model training task in execution and / or the performance of the device resources; generating, according to the performance evaluation result, a resource scheduling strategy for the model training task, to schedule device resources for the model training task according to the resource scheduling strategy.

[0006] In a second aspect, the present disclosure provides a resource scheduling device, which comprises: an information obtaining module, a performance evaluation module and a strategy generating module.

[0007] The information obtaining module is configured to obtain, in the process of executing a model training task by using device resources of a target device, resource usage information corresponding to the model training task from the target device.

[0008] The performance evaluation module is configured to process the resource usage information and historical training behavior information of a target user initiating the model training task by using a performance evaluation model, to obtain a performance evaluation result corresponding to the model training task, wherein the performance evaluation result is used to represent the performance of the model training task in execution and / or the performance of the device resources.

[0009] The policy generation module is configured to generate a resource scheduling policy for the model training task according to the performance evaluation result, so as to schedule device resources for the model training task according to the resource scheduling policy.

[0010] In a third aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned resource scheduling method.

[0011] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned resource scheduling method.

[0012] In a fifth aspect, the present disclosure provides a computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-mentioned resource scheduling method.

[0013] The resource scheduling method provided by the embodiments of the present disclosure can obtain accurate resource usage information from a target device in the process of executing a model training task by using device resources of the target device, and process the resource usage information and historical training behavior information of a target user initiating the model training task by using a trained performance evaluation model, so that the performance evaluation model can comprehensively evaluate the model training task according to device resources and historical training behavior of a user and other data, thereby obtaining an accurate performance evaluation result. Then, according to the performance evaluation result, a resource scheduling policy for the model training task can be accurately generated, so as to accurately schedule device resources according to the resource scheduling policy.

[0014] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 This is a flowchart of a resource scheduling method provided in an embodiment of the present disclosure.

[0017] Figure 2 This is a schematic diagram of the structure of a resource awareness system in a resource scheduling method provided in an embodiment of this disclosure.

[0018] Figure 3 This is a schematic diagram illustrating a resource scheduling method provided in an embodiment of this disclosure.

[0019] Figure 4 This is a block diagram of a resource scheduling device provided in an embodiment of the present disclosure.

[0020] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0023] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0025] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0026] The resource scheduling method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0027] The technical data in the embodiments of this disclosure will be explained below.

[0028] Profiling tools, also known as performance analysis tools or profiling tools, are software tools used to detect, measure, and analyze the runtime behavior and resource consumption of a program. They help developers understand "what operations the program performed, how much time it spent, and how many resources it used" during execution, thereby identifying performance bottlenecks and optimizing system efficiency. The profiling tool disclosed here focuses on collecting GPU status information and performance metrics at the kernel level.

[0029] TensorBoard: A profiling tool for performance analysis and optimization, which can be used for daily training monitoring, experiment management, and result visualization.

[0030] PyTorch Profiler is a profiling tool for performance analysis and optimization. It can be used to locate slow operations, analyze CPU and GPU collaboration efficiency, and identify memory bottlenecks.

[0031] Nsight is a profiling tool for performance analysis and optimization. It can be used to analyze detailed performance metrics and thoroughly troubleshoot problems.

[0032] KUAE refers to the KUAE training platform. KUAE is a model training platform that uses a full-featured GPU as its core and provides a full-stack AI computing power solution from the underlying chip to the upper-layer application.

[0033] SLURM (Simple Linux Utility for Resource Management) is a job scheduling platform for Linux cluster systems, primarily used to manage and allocate computing resources, ensuring efficient utilization and avoiding resource contention.

[0034] Kubernetes (K8s for short) is a container orchestration platform used to automate the deployment, scaling, and management of containerized applications.

[0035] Pod: The smallest deployable computing unit created and managed in Kubernetes. It consists of one or more containers sharing storage, network, and runtime environment, and serves as the basic unit for application scheduling in the cluster.

[0036] OOM (Out Of Memory) refers to the phenomenon where the program crashes due to the exhaustion of video memory or system memory during model training.

[0037] Performance bottlenecks can refer to key factors that limit the speed and efficiency of model training, including: computational bottlenecks, I / O bottlenecks, and communication bottlenecks.

[0038] Resource abuse: This can refer to the inefficient or excessive use of resources such as computing, storage, and networks, which may lead to soaring costs or service unavailability.

[0039] Platform interference: This can refer to the negative impact of the external environment or shared resources on the training task, including resource contention and platform instability.

[0040] Pod affinity: Makes the current Pod run in the same topology domain (such as the same node, the same rack, the same availability zone, etc.) as certain specific Pods.

[0041] Pod Anti-Affinity: Prevents the current Pod from running in the same topology as certain specific Pods.

[0042] Webhook: A lightweight communication mechanism based on the HTTP protocol that enables real-time event notifications between applications using a publish / subscribe pattern. When a specified event occurs in a system, it automatically sends a request to an external service, transmitting event information to trigger a response.

[0043] API: Application Programming Interface, used for data synchronization or function calls between different systems.

[0044] Slurm scheduler: It is a job scheduling system used to manage Linux cluster resources. It can allocate compute nodes, monitor job execution, and handle resource contention issues.

[0045] The Kubernetes scheduler (kube-scheduler) is part of the cluster control plane and is responsible for scheduling Pods to run on appropriate Nodes.

[0046] Hint: This is a hint message used by the Kubernetes scheduler when allocating Pods to nodes, to optimize scheduling decisions.

[0047] Prometheus is a monitoring system based on a time-series database that periodically scrapes component metrics data via the HTTP protocol.

[0048] Grafana is a data visualization and monitoring analysis platform that supports multiple data sources and displays monitoring data intuitively through a visual interface.

[0049] With the continuous development of computer technology and neural network technology, trained neural network models can be used to handle specific tasks in various practical application scenarios, thereby meeting practical needs. However, in the process of utilizing device resources to perform model training tasks for neural network models, the high uncertainty of the model training process may lead to a mismatch between device resources and the model training task, resulting in risks in the model training process. For example, AI (artificial intelligence) cluster platforms widely use profiling tools such as TensorBoard, PyTorch Profiler, and Nsight to analyze the GPU and CPU utilization efficiency and bottlenecks of models in a single training session. However, these tools are mostly offline tools, focusing on model performance analysis, and are difficult to play a real-time role in platform-level scheduling. At the same time, scheduling systems (such as Kubernetes) often only schedule based on Pod resource requests, without considering user history, task model characteristics, or other information.

[0050] One performance analysis scheme for deep learning models in related technologies can visualize performance bottlenecks and generate optimization suggestions by analyzing key indicators such as execution time, memory bandwidth, and communication latency during model runtime. This scheme focuses on diagnosing performance problems at the operator level within the model, but it does not consider the impact of the multi-tenant environment of the platform and user behavior on performance bottlenecks, nor does it integrate with the scheduling system. Another performance analysis scheme in related technologies proposes an end-to-end automated testing framework, supporting the entire process from configuration generation and test execution to performance visualization analysis. This scheme focuses on self-service and low-barrier-to-use tools, lacks tenant behavior modeling capabilities, and cannot form an optimization feedback mechanism in platform-level task resource scheduling.

[0051] Based on the above, it can be seen that current AI training platforms, especially in large-scale cluster environments such as Kuae with multi-tenant sharing, have the following key technical defects and bottlenecks.

[0052] 1. Profiling information is fragmented and has a single perspective.

[0053] Existing profiling systems typically perform local sampling only during single-task operation, focusing on metrics such as memory usage and computation time for a single task. They lack cross-task analysis and longitudinal evolution modeling of user task execution behavior, and cannot detect trend problems such as resource waste, unreasonable configuration, and scheduling conflicts that exist in long-term user operation.

[0054] 2. The scheduler has a "blind perception" of user behavior.

[0055] Currently, Kubernetes, Slurm, and other platform schedulers only schedule tasks based on static resource request information at the Pod level. They cannot identify platform interference caused by user behavior preferences (such as fixed large batch configurations, frequent OOM reruns, long-term preemption, etc.), resulting in inaccurate scheduling results, low efficiency, and a lack of resource fairness and risk control mechanisms from the user's perspective.

[0056] 3. Uncontrollable resource interference and lack of platform governance measures.

[0057] When multiple users run concurrently on the same node, it may cause problems such as GPU temperature spikes, bandwidth conflicts, or full video memory. However, because the scheduler does not recognize profiling data and does not have a tenant-level attribution mechanism, the platform cannot proactively identify which user groups are causing excessive consumption and impact on resources, nor can it limit resources or migrate tasks based on users' historical performance.

[0058] 4. The existing solution does not establish a seamless "analysis-optimization-scheduling" process.

[0059] Based on the above solutions, it can be seen that they mainly focus on model-level performance bottleneck diagnosis. Although they can provide optimization suggestions, the analysis results are not fed back to the scheduling system to form resource decisions. Although some solutions support automated analysis processes, their focus is still on tool-based processes and result display, and they have not achieved a closed-loop mechanism for user behavior modeling and resource feedback.

[0060] 5. Lack of user profile modeling and continuous optimization capabilities.

[0061] Existing technologies lack tagging and structuring of user tasks, making it impossible to perform layered governance based on user attributes (team affiliation, model type, running frequency, etc.) and to dynamically update user behavior patterns. Consequently, they cannot achieve refined and intelligent evolution of platform-level resource governance.

[0062] In summary, current technologies have systemic gaps in areas such as profiling granularity, user modeling, scheduling coordination, and resource governance.

[0063] Based on this, the present disclosure provides a resource scheduling method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product; for details, please refer to the specific content in the following embodiments.

[0064] Figure 1 A flowchart illustrating a resource scheduling method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes steps S11-S13.

[0065] Step S11: During the process of performing the model training task using the device resources of the target device, obtain the resource usage information corresponding to the model training task from the target device.

[0066] The target device can be understood as a computing device used for model training. For example, the target device can be a computer, a server, a node in a distributed cluster, a training node in a training platform, etc.

[0067] Device resources can be understood as the hardware resources of the target device, including but not limited to: CPU, GPU, memory, video memory, bandwidth, etc.

[0068] The model training task can be understood as the task of training the model to be trained; the model to be trained can be understood as the model that needs to be trained, including but not limited to large models, transformer models, convolutional neural network models, etc.

[0069] Resource usage information can be understood as the information on the use of device resources during the execution of model training tasks, including but not limited to: CPU utilization, GPU utilization, memory usage, and video memory usage.

[0070] Specifically, the resource scheduling method in this embodiment can allocate device resources of a target device to a model training task and use the device resources to execute the model training task. During the execution of the model training task using the device resources of the target device, resource usage information corresponding to the model training task can be obtained from the target device at a preset period; or the target device can be instructed to report the resource usage information corresponding to the model training task at irregular intervals.

[0071] In some embodiments, obtaining the resource usage information corresponding to the model training task from the target device includes: receiving the resource usage information sent by an information acquisition module running on the target device; wherein, the information acquisition module is used to collect the resource usage information of the model training task for the device resources during the process of executing the model training task using the device resources, and the resource usage information is a portion of the resource usage information selected from all the resource usage information of the device resources according to a preset information identifier.

[0072] The information collection module can be understood as a module used to collect resource usage information. This information collection module can be deployed in the target device and can be a profiling tool or a monitoring tool.

[0073] The preset information identifier can be understood as an identifier used to select a portion of resource usage information from all resource usage information; the preset information identifier can be the identifier of the portion of resource usage information; or the preset information identifier can correspond to the identifier of the portion of resource usage information; the preset information identifier can be information such as name or number; for example, a Profiling Agent (i.e., information collection module) is deployed in the training node, which can collect all performance indicators (i.e., all resource usage information) during the operation of the task (i.e., model training task) in real time; in order to reduce the computational pressure and data storage pressure of the subsequent resource awareness system, the Profiling Agent in this method can select a portion of key performance indicators (i.e., partial resource usage information) from all performance indicators according to the identifier of the preset key performance indicators (i.e., preset information identifier) ​​and report it to the resource awareness system; the key performance indicators include GPU utilization, video memory usage, operator execution time, communication latency, etc.

[0074] Taking the application of the resource scheduling method provided in this embodiment in a training platform (i.e., an AI training cluster) as an example, the resource scheduling method is explained below. This resource scheduling method provides a resource awareness system. This resource awareness system can run on the control node of the training platform, which also includes multiple training nodes for executing model training tasks. The execution process of the resource awareness system includes five stages: data acquisition, behavior modeling, performance attribution, scheduling feedback, and model evolution. Based on this, in the data acquisition stage, the training platform can deploy a lightweight Profiling Agent (i.e., an information acquisition module) on each training node to collect key performance indicators (i.e., resource usage information) during the task (i.e., model training task) execution process in real time, including GPU utilization, memory usage, operator execution time, communication latency, etc. These key performance indicators are sent to the resource awareness system according to a preset period. Alternatively, the training platform can deploy monitoring tools such as Prometheus and Grafana on each training node to collect key performance indicators during task execution in real time and send them to the resource awareness system according to a preset period.

[0075] As can be seen from the above embodiments, this method improves the accuracy of subsequent performance evaluation results by collecting accurate resource usage information in real time through an information collection module configured in the target device.

[0076] In some embodiments, before obtaining the resource usage information corresponding to the model training task from the target device during the process of executing the model training task using the device resources of the target device, the method further includes: receiving a model training task sent by the target user through a user terminal, wherein the model training task has a corresponding model to be trained; determining the resource information to be allocated corresponding to the model training task based on the model type of the model to be trained and the historical training behavior information of the target user; selecting a target device from multiple candidate devices based on the resource information to be allocated, wherein the target device is used to execute the model training task; and allocating the idle device resources of the target device as the device resources allocated to the model training task.

[0077] Among them, historical training behavior information can be understood as information generated during the execution of model training tasks submitted by the target user in the past. This historical training behavior information can be the target user's user profile or user tag.

[0078] The information on resources to be allocated can be understood as information about the device resources that need to be allocated, including but not limited to the number of GPUs, memory area information, and video memory area information.

[0079] The model type can be the type of model to be trained, including but not limited to large model type, transformer type, convolutional neural network type, deep learning type, etc.

[0080] The user terminal includes, but is not limited to, smart terminals, client terminals, etc., and this disclosure does not impose specific restrictions.

[0081] Continuing with the previous example, when the training platform receives a model training task for a large model uploaded by a user through the user terminal, it will perform resource prediction based on the model type (i.e., the large model type) and the user profile (for example, resource prediction can be performed through a trained resource prediction model) to obtain the resource information to be allocated corresponding to the model training task. Then, based on the resource information to be allocated, it selects a training node from multiple training nodes whose idle device resources match the resource information to be allocated, and uses it as the target training node (i.e., the target device) to execute the model training task. Finally, the idle device resources in the target training node are used as the device resources to execute the model training task.

[0082] As can be seen from the above embodiments, this disclosure achieves intelligent resource scheduling for model training tasks by combining model type and historical training behavior information for resource allocation, and intelligently allocating device resources from multiple training nodes based on the resource information to be allocated, thereby improving resource allocation efficiency, accuracy and response speed.

[0083] Step S12: Process the resource usage information and the historical training behavior information of the target user who initiated the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task. The performance evaluation result is used to represent the performance of the model training task in execution and / or the performance of the device resources.

[0084] The performance evaluation model can be understood as a model used to evaluate the performance of the model training task. This model can be a large model, a deep learning model, a graph neural network model, etc., without specific limitations. The performance evaluation result can be understood as the result obtained after the performance evaluation model evaluates and processes resource usage information and historical training behavior information. This result can represent performance information such as the stability of the executing model training task and the probability of task failure (e.g., high or low probability of failure); it can also represent the load level of the device resources executing the model training task (e.g., excessive or insufficient load) and the risk level of device resource failure (e.g., device interruption, device damage). The performance evaluation result can include various types, including but not limited to task performance scores and resource risk levels.

[0085] In some embodiments, the historical training behavior information is generated based on the user attribute information of the target user and the historical task information of the historical model training task corresponding to the target user, wherein the historical task information includes at least one of resource usage information, task execution results, task failure information, and device resource information of historically allocated device resources; or the historical training behavior information is selected from multiple preset historical training behavior information based on the user attribute information of the target user.

[0086] User attribute information can be understood as the characteristic information of the target user during the model training process. This user attribute information belongs to the user dimension and includes, but is not limited to, historical submission records, task configuration preferences, task model types, and device resource configurations for historical model training tasks. Historical task information of historical model training tasks can be understood as the characteristic information during the execution of historical model training tasks. This historical task information belongs to the task dimension and includes, but is not limited to, task failure rate, resource request history, parameter tuning frequency, and stability (such as the number of retries and the frequency of OutOfMemoryErrors).

[0087] Among them, resource usage information can be understood as the device resource information used during the execution of historical model training tasks, such as resource request history; task execution results can be understood as the execution results of historical model training tasks, such as task failure rate and parameter tuning frequency; task fault information can be understood as fault information during the execution of historical model training tasks, such as stability (including but not limited to the number of retries and the frequency of OOM).

[0088] Using the previous example, during the model training process, the training platform can record users' historical submission records, resource configurations, and task failures, providing input for subsequent user profile modeling.

[0089] During the user profile modeling phase, the system (i.e., the resource awareness system) can extract multi-dimensional features such as user resource request preferences, parameter tuning behavior, and stability (e.g., number of retries, frequency of OutOfMemoryErrors) by abstractly modeling the user's historical task behavior characteristics, thereby constructing a dynamic user profile and forming a structured "tenant behavior vector". This profile can be iteratively updated periodically and supplemented in real time according to new tasks.

[0090] As can be seen from the above embodiments, this disclosure provides a method for generating and dynamically evolving user profiles, including multi-dimensional feature extraction of user behavior, structured encoding (such as the construction of "tenant behavior vectors"), and a periodic profile update mechanism, which enables the platform to have adaptive recognition capabilities and is the foundation for supporting intelligent strategies.

[0091] In some embodiments, processing the resource usage information and the historical training behavior information of the target user initiating the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task includes: analyzing the resource usage information to obtain the device operating status of the target device, and, if there is an anomaly in the device operating status, processing the resource usage information and the historical training behavior information of the target user initiating the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task; or processing the resource usage information and the historical training behavior information of the target user initiating the model training task using a performance evaluation model according to a preset time period to obtain the performance evaluation result corresponding to the model training task.

[0092] Among them, the device operating status can be understood as the operating status of various resource devices during the model training process. The device operating status includes, but is not limited to, stable and abnormal states. For example, when the CPU or GPU temperature is too high, the device operating status of the CPU or GPU may be abnormal. When the data transmission time delay is too high, the device operating status of bandwidth, memory or video memory may be abnormal.

[0093] The preset time period can be set according to the actual application scenario. For example, the preset time period can be a time period with an interval of 1 minute.

[0094] Specifically, the resource scheduling method provided in this disclosure can analyze the operating status of device resources based on the obtained resource usage information to obtain the operating status of the target device. For example, the resource usage information can be input into an analysis model (a trained neural network model) for analysis to obtain the operating status of the device.

[0095] If an anomaly is detected in the device's operating status, the following steps can be immediately executed: using a performance evaluation model to process the resource usage information and the historical training behavior information of the target user who initiated the model training task, and obtaining the performance evaluation result corresponding to the model training task. This ensures timely performance evaluation and resource scheduling in the event of device malfunction, thereby improving the operational stability of the target device.

[0096] Alternatively, the resource scheduling method provided in this disclosure can also trigger the execution of a step that uses a performance evaluation model to process resource usage information and historical training behavior information of the target user who initiated the model training task according to a preset time period (e.g., once every minute) to obtain the performance evaluation result corresponding to the model training task. This allows for real-time detection of device resources and model training tasks during the execution of the model training task by the device resources, ensuring the stability of task execution and the operational stability of the target device.

[0097] In some embodiments, the performance evaluation result includes a task performance score and / or a resource risk level; the step of processing the resource usage information and the historical training behavior information of the target user initiating the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task includes: inputting the resource usage information and the historical training behavior information of the target user initiating the model training task into the performance evaluation model for evaluation processing to obtain the task performance score and / or resource risk level corresponding to the model training task; wherein, the task performance score is used to represent the performance health of the model training task execution, and the resource risk level is used to characterize the load risk of the device resources.

[0098] Following the previous example, after obtaining the profiling metrics (i.e., resource usage information) collected by the profiling tool and the user profiles of the target users (e.g., tenants) (e.g., tenant profiles), performance attribution analysis can be performed based on this data. In the performance attribution stage: the system inputs the profiling metrics and tenant profiles into the fusion model (i.e., the performance evaluation model) to assess whether the current task has performance bottlenecks, resource abuse, or platform interference risks, and generates quantitative "performance health scores" (i.e., task performance scores) and / or "resource risk levels" for the model training task, which can be used as the basis for subsequent resource scheduling judgments to improve the accuracy of resource scheduling.

[0099] As can be seen from the above embodiments, this disclosure provides a mechanism for fusing profiling information with tenant behavior profiles. Specifically, this disclosure fuses profiling data (GPU, memory, operators, etc.) during task (i.e. model training task) with tenant behavior history (parameter tuning habits, failure rate, resource preferences, etc.) to train a performance attribution model with cross-task and multi-dimensional features and perform performance attribution analysis to identify performance bottlenecks and risk behaviors at the tenant level.

[0100] In some embodiments, the performance evaluation model is a graph neural network model; the step of processing the resource usage information and the historical training behavior information of the target user who initiated the model training task using the performance evaluation model to obtain the performance evaluation result corresponding to the model training task includes: constructing a task behavior graph using the resource usage information and the historical training behavior information of the target user who initiated the model training task; and analyzing the task behavior graph using the graph neural network model to obtain the performance evaluation result corresponding to the model training task.

[0101] Specifically, the task behavior graph is constructed using the resource usage information and the historical training behavior information of the target user who initiated the model training task. This includes: using the target user, target device, and model training task as graph nodes, the relationships between the target user, target device, and model training task as edges, and the resource usage information and historical training behavior information as attribute information of the edges; and constructing the task behavior graph based on the graph nodes, edges, and edge attribute information.

[0102] The task behavior graph can be understood as a graph used for resource attribution analysis of equipment resources, and it can also be a topology graph used for resource scheduling.

[0103] Following the previous example, this disclosure provides cross-task analysis capabilities supported by graph structure data. Specifically, a task behavior graph or scheduling topology graph (such as a triple graph consisting of user-task-node) is constructed based on user profiles and profiling metrics. Then, a graph neural network is used to perform resource attribution based on the task behavior graph or scheduling topology graph to obtain "performance health score" and "resource risk level". It should be noted that in the graph neural network, the process of nodes passing information through edges can be a feature propagation path. Each node will aggregate the feature information of its neighboring nodes and update its own features through a message passing mechanism.

[0104] Step S13: Based on the performance evaluation results, generate a resource scheduling strategy for the model training task, and schedule device resources for the model training task according to the resource scheduling strategy.

[0105] The resource scheduling strategy can be understood as a strategy for scheduling resources for the model training task; the resource scheduling strategy can be used to schedule the device resources allocated to the model training task. This resource scheduling strategy can be a resource reallocation strategy, a task level adjustment strategy for the model training task, etc. For example, the resource scheduling strategy can be a scheduling suggestion, a scheduling instruction, etc.

[0106] In some embodiments, generating a resource scheduling strategy for the model training task based on the performance evaluation result includes: performing strategy generation processing using a strategy generation model based on the task performance score and the resource risk level to obtain a resource scheduling strategy for the model training task; or matching the task performance score and the resource risk level with multiple preset scheduling strategies and determining the matched preset scheduling strategy as the resource scheduling strategy for the model training task.

[0107] Among them, the strategy generation model can be understood as a model used to generate resource scheduling strategies; the preset scheduling strategy can be understood as a pre-set strategy used to schedule resources for model training tasks; the preset scheduling strategy can be a summary and accumulation of allocated device resources during the execution of historical model training tasks.

[0108] Continuing with the previous example, if the "performance health score" and "resource risk level" corresponding to the model training task are generated during the performance attribution phase, scheduling feedback processing can be performed. During the scheduling feedback phase, the system generates scheduling suggestions based on the attribution analysis results (i.e., the "performance health score" and "resource risk level"). Specifically, the generation of these scheduling suggestions can be achieved by: inputting the "performance health score" and "resource risk level" into the large model for analysis to obtain scheduling suggestions corresponding to the model training task; or by matching the "performance health score" and "resource risk level" with preset scheduling suggestions (i.e., preset scheduling strategies) stored in the database, selecting the preset scheduling suggestions corresponding to the given "performance health score" and "resource risk level," and using these as the scheduling suggestions for the model training task.

[0109] The scheduling recommendations include, but are not limited to, node preferences, GPU quota adjustments, task priority adjustments, and Pod affinity / anti-affinity configurations.

[0110] As can be seen from the above embodiments, this disclosure can generate scheduling suggestions based on the attribution analysis results, realize the quantitative evaluation and feedback control of the running status of the training task, and generate scheduling strategies in advance when the model training task is identified as having insufficient resources or configuration imbalance risks in computing power, memory, communication, etc., thereby dynamically scheduling the device resources corresponding to the model training task and ensuring the smooth execution of the model training task.

[0111] In some embodiments, after generating a resource scheduling strategy for the model training task based on the performance evaluation result, the method further includes: sending the resource scheduling strategy to a resource scheduling unit, so that the resource scheduling unit performs scheduling processing on the device resources allocated to the model training task according to the resource scheduling strategy; wherein, the scheduling processing includes: reallocating corresponding device resources for the model training task, reselecting a device to execute the model training task from multiple candidate devices, and adjusting the priority of the model training task.

[0112] The resource scheduling unit can be understood as a unit used to schedule equipment resources. This resource scheduling unit can be the scheduler of the training platform, the cluster scheduling system, etc.

[0113] In this context, priority can be understood as the priority assigned to the model training task. Different priorities correspond to different target devices and device resources. For example, when the priority is low, the probability of the model training task failing is high. In this case, target devices with high load, low computing power, or poor performance can be assigned to the model training task to avoid wasting device resources. Conversely, when the priority is high, the probability of the model training task succeeding is high. Target devices with low load, high computing power, or better performance can be assigned to the model training task to avoid wasting device resources.

[0114] Following the previous example, after generating scheduling suggestions, these suggestions can be submitted to the cluster scheduling system or scheduler via a Webhook / API interface to ensure that the analysis results are implemented. The cluster scheduling system or scheduler can then perform operations such as adjusting task priorities, setting resource limits, and recommending nodes based on the scheduling suggestions.

[0115] As can be seen from the above embodiments, the embodiments of this disclosure realize a closed-loop feedback path of performance attribution and scheduling suggestions. This solution goes beyond performance evaluation and directly generates executable scheduling suggestions (priority, resource limits, target nodes, etc.), which are fed back to the scheduling system through Webhook / API interfaces, forming a closed-loop linkage of "analysis-optimization-scheduling-feedback". Furthermore, the embodiments of this disclosure provide a task priority adjustment and resource limit strategy driven by user behavior. By introducing mechanisms such as resource interference index, high-risk user identification, and model complexity classification, it achieves fine-grained control of different tasks, which is a key technical means to achieve multi-tenant fairness and high utilization collaboration.

[0116] Furthermore, it should be noted that the embodiments disclosed herein provide a non-intrusive interface design for the scheduling system, injecting scheduling suggestions in the form of a standardized interface, compatible with training platforms such as Kuae and Kubernetes, and possessing module decoupling and system-level portability, which is conducive to rapid engineering deployment and commercialization.

[0117] In some embodiments, after generating a resource scheduling strategy for the model training task based on the performance evaluation result, the method further includes: updating the historical training behavior information according to the task information corresponding to the model training task when the model training task is completed, to obtain updated historical training behavior information; and / or generating model training suggestion information according to the task information corresponding to the model training task when the model training task is completed, and displaying the model training suggestion information to the target user; wherein the task information includes at least one of: device resource information, resource usage information, task execution result, and task fault information.

[0118] The task information can be understood as the information generated during the execution of the model training task using device resources; the model training suggestion information can be understood as improvement suggestions for the model training task, used to optimize the device resource allocation scheme, model parameters of the model to be trained, model training method, etc. corresponding to the model training task.

[0119] Following the previous example, after the model training task is completed, this embodiment can also perform profile evolution and policy learning processing. During the profile evolution and policy learning phase, after the model training task is completed, the system collects the actual task execution trajectory information (including scheduling response, resource usage curves, failure status, etc.), updates the user profile based on this information, and achieves continuous accumulation of platform-level experience and dynamic optimization capabilities. Furthermore, it can also generate a model update strategy for the performance evaluation model based on task information such as scheduling response, resource usage curves, and failure status, and update the model parameters of the performance evaluation model according to this model update strategy.

[0120] This embodiment of the disclosure can also input information such as scheduling response, resource usage curve, and failure status during the execution of the model training task into the suggestion model for analysis, generate improvement suggestions for the model training task, and display the improvement suggestions for the model training task to the user.

[0121] As can be seen from the above embodiments, this disclosure supports a self-learning and continuously evolving behavior modeling strategy. With the dynamic updating of user profiles and feedback of strategy results, the system can continuously adjust evaluation weights and scheduling tendencies, and has the adaptability and evolutionary capability in long-term operation scenarios.

[0122] The resource scheduling method provided in this disclosure can obtain accurate resource usage information from the target device during the execution of a model training task using the device resources of the target device. It then uses a trained performance evaluation model to process the resource usage information and the historical training behavior information of the target user initiating the model training task. This allows the performance evaluation model to comprehensively evaluate the model training task based on multiple data points, including device resources and the user's historical training behavior, thereby obtaining accurate performance evaluation results. Based on the performance evaluation results, a resource scheduling strategy for the model training task can be accurately generated, enabling accurate resource scheduling of device resources according to this strategy.

[0123] Taking the application of the resource scheduling method provided in this disclosure in a training platform as an example, the resource scheduling method will be explained and illustrated. To achieve the fusion of profiling data analysis and tenant behavior modeling, as well as resource optimization scheduling feedback, this disclosure proposes a resource awareness system that can be deployed in an AI training cluster. Figure 2 This is a schematic diagram of the structure of a resource awareness system in a resource scheduling method provided in this embodiment of the disclosure, based on... Figure 2 It can be seen that the resource scheduling method provides a resource awareness system with a simple structure and complete data closed loop. The resource awareness system includes five modules: 1. Task acquisition module: a lightweight Profiling Agent (i.e., Profiling acquisition module) deployed on each training node, which collects profiling data (GPU utilization, memory peak, I / O bandwidth, execution time, etc.) in real time during model runtime. The profiling data aggregator can aggregate the collected profiling data and send it to the tenant behavior modeling module and the performance attribution analysis module.

[0124] 2. Tenant Behavior Modeling Module: Collects user-dimensional information, such as task failure rate, resource request history, task configuration preferences, parameter tuning frequency, task model type, etc., to form a multi-dimensional tenant profile.

[0125] 3. Performance Attribution Analysis Module: Integrates Profiling results with user behavior characteristics (i.e., tenant profiles) to train an attribution model (i.e., performance evaluation model). The trained attribution model can identify resource anomalies, performance bottlenecks, and potential risk factors.

[0126] 4. Scheduling suggestion generation module (also known as scheduling feedback module): Generates resource scheduling suggestions based on performance attribution results, and injects these resource suggestions into the Kuae scheduler via API to perform operations such as task priority adjustment, resource limit setting, and node recommendation.

[0127] 5. The profile update module provides a profile update mechanism that can update tenant profiles periodically or after a task is completed, forming a continuous learning capability.

[0128] Figure 3 This is a schematic diagram illustrating a resource scheduling method provided in an embodiment of this disclosure, based on... Figure 3 As can be seen, the resource scheduling method in this embodiment specifically includes steps S31-S38.

[0129] Step S31: Receive the training task.

[0130] Specifically, the training platform can receive model training tasks for large models uploaded by users. In this case, the training platform will perform resource prediction based on the type of the large model and the user profile (for example, by performing resource prediction through a trained resource prediction model) to obtain the resource information to be allocated corresponding to the model training task.

[0131] Then, based on the resource information to be allocated, a training node whose idle device resources match the resource information to be allocated is selected from multiple training nodes, and this node is used as the target training node for executing the model training task; the idle device resources in the target training node are used as the device resources for executing the model training task.

[0132] Step S32: Profiling data acquisition.

[0133] Specifically, during the process of executing the model training task using the device resources of the training nodes, the training platform deploys a lightweight Profiling Agent on each training node to collect key performance indicators in real time during the task (i.e., model training task) execution, including GPU utilization, video memory usage, operator execution time, communication latency, etc.

[0134] Step S33: Build user profiles.

[0135] Specifically, during the model training process, the training platform can record users' historical submission records, resource configurations, and task failures, providing input for subsequent user profile modeling.

[0136] In the process of building user profiles, the resource awareness system can extract multi-dimensional features such as user resource request preferences, parameter tuning behavior, and stability (e.g., number of retries, frequency of OutOfMemoryErrors) by abstractly modeling the characteristics of users' historical task behavior, thereby constructing a dynamic user profile and forming a structured "tenant behavior vector". This profile can be iteratively updated periodically and supplemented in real time according to new tasks.

[0137] Step S34: Performance attribution analysis.

[0138] Specifically, in the performance attribution phase, the system inputs profiling metrics and tenant profiles into the fusion model (i.e., the performance attribution model) to assess whether the current task has performance bottlenecks, resource abuse, or platform interference risks. It also generates quantitative "performance health scores" and "resource risk levels" for the model training task, which are used as the basis for subsequent resource scheduling judgments to improve the accuracy of resource scheduling.

[0139] Step S35: Generate scheduling recommendations.

[0140] Specifically, the "performance health score" and "resource risk level" are input into the large model for analysis to obtain scheduling suggestions for the model training tasks.

[0141] Alternatively, the "performance health score" and "resource risk level" can be matched with preset scheduling suggestions stored in the database to select the preset scheduling suggestions corresponding to the "performance health score" and "resource risk level" and use them as the scheduling suggestions for the model training task.

[0142] Step S36: Inject the scheduler execution.

[0143] Specifically, after generating scheduling suggestions, the suggestions can be submitted to the cluster scheduling system or scheduler via Webhook / API interface to ensure that the analysis results are implemented. The cluster scheduling system or scheduler can then perform operations such as adjusting task priorities, setting resource limits, and recommending nodes based on the scheduling suggestions.

[0144] Step S37: Collect the task execution results.

[0145] Specifically, after a task is completed, the system retrieves the actual running trajectory of the task, including information such as scheduling response, resource usage curve, and whether it failed.

[0146] Step S38: Update user profile.

[0147] Specifically, after the model training task is completed, the system collects the actual running trajectory of the task, including scheduling response, resource usage curve, failure and other information, and updates the user profile based on this information, so as to realize the platform-level experience continuous accumulation and dynamic optimization capabilities.

[0148] Furthermore, it can generate a model update strategy for the performance evaluation model based on task information such as scheduling response, resource usage curve, and whether the task failed or not, and update the model parameters of the performance evaluation model according to the model update strategy.

[0149] Based on the above steps, it can be seen that the resource scheduling method provided in this disclosure has the characteristics of module decoupling, distributed deployment, and strong platform compatibility. It can be flexibly embedded in schedulers such as Kuae, Kubernetes, or Slurm, and is suitable for large-scale heterogeneous computing environments with multiple tasks and multiple tenants.

[0150] To more clearly illustrate the technical solutions provided in the embodiments of this disclosure, several embodiments are given below in conjunction with actual deployment scenarios and task operation processes, based on a typical Kuae cluster training environment.

[0151] Example 1: Task scheduling priority adjustment example.

[0152] Step 1: User A and User B submit their respective deep learning model training tasks through the platform.

[0153] Step 2: The Profiling Agent deployed on the node collects task execution data in real time, including video memory usage, GPU utilization, execution time, etc.

[0154] Step 3: The platform scheduling center extracts user profiles from the task history database. User A had a 35% task failure rate, fluctuating GPU utilization, and frequent configuration changes over the past two weeks; while User B had stable tasks, a failure rate of less than 5%, and stable resource usage.

[0155] Step 4: The performance attribution module determines that User A's task is high-risk and its resource usage behavior is extreme, and generates a "low priority" scheduling suggestion; User B's task is marked as "scheduled priority protection".

[0156] Step 5: The scheduling feedback module injects the suggestion into the Kuae scheduler. User B's task is prioritized for scheduling to redundant resource nodes, while User A's task is restricted to scheduling only to low-load nodes (performance is limited), and the maximum video memory is limited to 28GB.

[0157] Example 2: Task resource interference perception and suppression example.

[0158] Step 1: The platform runs multiple user tasks simultaneously. During the execution of User C's task, the temperature of the GPU node where it resides increases by 40%, causing a decrease in the computing performance of adjacent tasks.

[0159] Step 2: The Profiling Agent detects significant resource consumption anomalies in user C's task on this node and records the GPU temperature curve and bandwidth saturation.

[0160] Step 3: The performance attribution module combines user C's historical task behavior (i.e., user profile) (which has caused resource bursts multiple times) with the current profiling data to determine that the user is a "platform interference risk user".

[0161] Step 4: The system automatically generates a scheduling strategy adjustment suggestion, requiring user C's task to be scheduled to a dedicated isolated resource pool and marked as a "user requiring rate limiting".

[0162] Step 5: The platform updates the user profile and pushes the interference event report to the platform administrator.

[0163] Example 3: User profile-driven node recommendation example.

[0164] Step 1: User D continuously submits Transformer model training tasks. Profiling records show that this type of task has high communication bandwidth requirements.

[0165] Step 2: After the system identifies the user's model type and configuration preferences, it dynamically matches the three most suitable nodes (NVLink interconnect bandwidth > 80% available) by referring to the current bandwidth usage of each node in the cluster.

[0166] Step 3: The scheduling feedback module constructs a scheduling hint, suggesting that Kuae prioritize binding user D task to one of the above nodes to improve training efficiency.

[0167] The above embodiments all constitute a complete data flow closed loop from data collection, user profile generation, attribution judgment, scheduling feedback to behavior update, demonstrating the operability and deployment practicality of this disclosure.

[0168] As can be seen from the above embodiments, the resource scheduling method provided in this disclosure relates to the fields of artificial intelligence platforms, distributed computing, performance analysis and resource scheduling. It provides a performance attribution analysis method that integrates runtime profiling information and multi-tenant behavior profiles, and is applicable to resource optimization and scheduling of multi-user mixed-distribution tasks in AI training platforms (such as Kuae, Kubernetes, Slurm, etc.).

[0169] This method proposes a mechanism that integrates profiling analysis with tenant behavior feature modeling, connecting performance attribution analysis with resource scheduling optimization feedback, making it more suitable for large-scale training platforms with multiple users and concurrent tasks. By integrating profiling and tenant behavior profiles, a resource optimization method for AI training platforms that is attributable, predictable, and schedulable is constructed. Furthermore, by incorporating profiling data, performance attribution, resource prediction, and scheduling suggestion generation are achieved, providing AI platforms with an intelligent resource governance capability.

[0170] The resource scheduling method provided in this disclosure aims to address the problems of uneven resource utilization, scheduling blind spots, and difficulty in performance attribution caused by the mixed distribution of multi-tenant tasks in AI training platforms. It proposes a performance analysis and resource optimization method that integrates profiling information and tenant behavior profiles, specifically solving the following problems.

[0171] 1. To address the issue of how to move from single-task profiling data collection to platform-level integration and combine it with long-term user behavior for attribution analysis, rather than being limited to static analysis of a single task, this disclosure's embodiments achieve an upgrade of profiling capabilities from point to surface. It improves from traditional single-task profiling to cluster-level data collection and unified modeling, supports cross-node and cross-user resource bottleneck identification and horizontal comparison, and enhances the platform's observability of the overall operating status.

[0172] 2. Regarding the question of how to structurally model user behavior profiles (such as historical task stability, GPU usage patterns, parameter tuning habits, resource request preferences, etc.) and incorporate them into the task scheduling system; the embodiments of this disclosure realize the ability to digitally model tenant behavior; through the construction of behavior profiles, the platform can identify user resource usage habits, stability, risk levels, etc., thereby promoting the formation of resource governance units "based on users", making scheduling strategies more refined and personalized.

[0173] 3. Regarding the issue of how to identify the degree of interference or risk level of users on the resource platform and dynamically adjust their task execution priority and resource allocation strategy accordingly; the embodiments of this disclosure realize that performance attribution decision is traceable and interpretable; the performance labels and resource health scores output by the system are interpretable and can be traced back to profiling indicators and behavioral characteristics, thereby improving the transparency of platform problem diagnosis and the credibility of decision-making.

[0174] 4. Regarding the question of how to extract scheduling suggestions from the fusion results of profiling data and profiles, and to convey them to schedulers such as Kuae and Kubernetes through the platform API interface to form a closed-loop feedback, the scheduling feedback mechanism of this disclosure realizes an automated closed loop from analysis to execution. This disclosure also designs a Webhook / API mechanism to directly interface with the scheduling system, so that the analysis results can be used for actual task arrangement and resource allocation, and for the first time opens up an execution path that links performance analysis and scheduling strategies.

[0175] The resource scheduling method provided in this disclosure achieves the following six significant technical effects through a structured system architecture and a closed-loop operation process.

[0176] 1. Achieve deep integration of profiling and user behavior profiling.

[0177] Existing technologies mostly focus on profiling analysis at the model level or for single tasks. However, this disclosure correlates profiling data with long-term user behavior profiles, establishing a three-dimensional relationship between "users, tasks, and resources" to provide tenant-oriented attribution analysis and governance suggestions. This integrated mechanism enables the platform to have "human-centered" scheduling and optimization capabilities, solving the fundamental limitation of existing systems that "only know the task, not the user."

[0178] 2. Provides a complete analysis-scheduling feedback closed loop.

[0179] Compared to solutions that only output performance evaluation reports and provide task configuration suggestions, this disclosure goes a step further in terms of result output by building a Webhook interface or scheduling plugin to enable attribution analysis results to directly drive task scheduling behavior (such as priority adjustment, node selection, and resource quota settings) in scheduling systems such as Kubernetes / Kuae, thereby achieving automated execution.

[0180] 3. Introduce tenant risk identification and platform governance capabilities.

[0181] This disclosure identifies and restricts "high-risk user behavior" tasks through methods such as resource abuse identification and GPU interference index assessment, thereby achieving dynamic protection of platform stability in a multi-tenant environment; while most existing technologies lack a cross-user perspective and do not have resource isolation and governance methods.

[0182] 4. Supports dynamic evolution of user profiles and strategy learning.

[0183] The user profiles constructed in this public disclosure possess continuous updating and dynamic evolution capabilities. Leveraging behavioral feedback after task execution, they form an evolutionary closed loop of "task—profile—optimization—feedback," exhibiting self-learning ability and adapting to different training tasks and long-term changes in user behavior. This capability far surpasses the static and passive characteristics of traditional profiling analysis.

[0184] 5. Widely compatible with various platforms and flexible deployment methods.

[0185] This solution adopts a non-intrusive deployment structure. The Profiling Agent is a lightweight module, and the scheduling feedback is a standard interface. It does not depend on specific framework modifications and can be directly integrated into mainstream platforms (such as K8s, Slurm, and Kuae), demonstrating strong engineering feasibility and scalability.

[0186] 6. Improve system resource utilization and platform governance capabilities. By identifying inefficient behaviors and users at risk of wasting resources, the system can provide early warnings or restrict resource allocation, thereby improving the overall utilization of the GPU cluster, reducing operating energy consumption and downtime risk, and has strong engineering implementation value.

[0187] It should be noted that, although the Profiling and tenant behavior profile fusion scheme proposed in this embodiment has the advantages of comprehensive data perception and feedback loop, there are also more optimized implementation paths, specifically including the following four aspects.

[0188] 1. Rule-based scheduling strategy: A set of fixed scheduling rule templates can be built (such as "if the GPU utilization is continuously lower than 30%, then reduce the task priority"), which can achieve partial scheduling control without dynamic behavior modeling.

[0189] 2. Static Profiling Data-Driven Analysis: The system can use only the current task profiling information (such as peak memory usage and execution time) to determine resource usage and generate scheduling suggestions, omitting historical analysis of user behavior.

[0190] 3. Manual attribution mechanism based on monitoring system: The platform can use monitoring tools such as Prometheus and Grafana to display resource indicators, allowing administrators to manually judge task anomalies and user behavior risks, and to intervene in scheduling.

[0191] 4. Use a simple tagging and classification mechanism to replace the user profile model: The platform can tag users (such as "stable" or "risky") and adjust resource allocation based on experience-based classification strategies.

[0192] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0193] In addition, this disclosure also provides a resource scheduling device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the resource scheduling methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0194] Figure 4 This is a block diagram of a resource scheduling device provided in an embodiment of the present disclosure.

[0195] Reference Figure 4 This disclosure provides a resource scheduling device, which includes an information acquisition module 401, a performance evaluation module 402, and a strategy generation module 403.

[0196] The information acquisition module 401 is used to acquire resource usage information corresponding to the model training task from the target device during the process of performing the model training task using the device resources of the target device.

[0197] The performance evaluation module 402 is used to process the resource usage information and the historical training behavior information of the target user who initiated the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task; the performance evaluation result is used to represent the performance of the model training task being executed and / or the performance of the device resources.

[0198] The strategy generation module 403 is used to generate a resource scheduling strategy for the model training task based on the performance evaluation results, so as to schedule device resources for the model training task according to the resource scheduling strategy.

[0199] In some embodiments, the performance evaluation result includes a task performance score and / or a resource risk level; the performance evaluation module 402 is configured to: input the resource usage information and the historical training behavior information of the target user who initiated the model training task into the performance evaluation model for evaluation processing, and obtain the task performance score and / or resource risk level corresponding to the model training task; wherein, the task performance score is used to represent the performance health of the model training task execution, and the resource risk level is used to characterize the load risk of the device resources.

[0200] In some embodiments, the strategy generation module 403 is configured to: perform strategy generation processing using a strategy generation model based on the task performance score and the resource risk level to obtain a resource scheduling strategy for the model training task; or match the task performance score and the resource risk level with multiple preset scheduling strategies, and determine the matched preset scheduling strategy as the resource scheduling strategy for the model training task.

[0201] In some embodiments, the performance evaluation module 402 is configured to: analyze the resource usage information to obtain the device operating status of the target device, and, if there is an anomaly in the device operating status, process the resource usage information and the historical training behavior information of the target user who initiated the model training task using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task; or process the resource usage information and the historical training behavior information of the target user who initiated the model training task using a performance evaluation model according to a preset time period to obtain the performance evaluation result corresponding to the model training task.

[0202] In some embodiments, the resource scheduling device further includes a task execution module, configured to: receive a model training task sent by the target user through a user terminal, wherein the model training task carries a model to be trained; determine the resource information to be allocated corresponding to the model training task based on the model type of the model to be trained and the target user's historical training behavior information; select a target device from multiple candidate devices based on the resource information to be allocated, wherein the target device is used to execute the model training task; and allocate the idle device resources of the target device as the device resources allocated to the model training task.

[0203] In some embodiments, the resource scheduling device further includes a resource scheduling module, configured to: send the resource scheduling policy to the resource scheduling unit, so that the resource scheduling unit performs scheduling processing on the device resources allocated to the model training task according to the resource scheduling policy; wherein the scheduling processing includes: reallocating the corresponding device resources for the model training task, reselecting the device to execute the model training task from multiple candidate devices, and adjusting the priority of the model training task.

[0204] In some embodiments, the resource scheduling device further includes an information generation module, configured to: update the historical training behavior information according to the task information corresponding to the model training task when the model training task is completed, to obtain updated historical training behavior information; and / or generate model training suggestion information according to the task information corresponding to the model training task when the model training task is completed, and display the model training suggestion information to the target user; wherein the task information includes at least one of: device resource information, resource usage information, task execution result, and task fault information.

[0205] In some embodiments, the information acquisition module 401 is further configured to: receive the resource usage information sent by the information collection module running in the target device; wherein, the information collection module is configured to collect the resource usage information of the model training task for the device resources during the process of performing the model training task using the device resources, and the resource usage information is a portion of the resource usage information selected from all the resource usage information of the device resources according to a preset information identifier.

[0206] In some embodiments, the historical training behavior information is generated based on the user attribute information of the target user and the historical task information of the historical model training task corresponding to the target user, wherein the historical task information includes at least one of resource usage information, task execution results, task failure information, and device resource information of historically allocated device resources; or the historical training behavior information is selected from multiple preset historical training behavior information based on the user attribute information of the target user.

[0207] The resource scheduling apparatus provided in this embodiment can obtain accurate resource usage information from the target device during the process of executing a model training task using the device resources of the target device. It then uses a trained performance evaluation model to process the resource usage information and the historical training behavior information of the target user initiating the model training task. This allows the performance evaluation model to comprehensively evaluate the model training task based on multiple data points, including device resources and the user's historical training behavior, thereby obtaining accurate performance evaluation results. Based on the performance evaluation results, a resource scheduling strategy for the model training task can be accurately generated, enabling accurate resource scheduling of device resources according to this strategy.

[0208] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0209] Reference Figure 5 This disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to enable the at least one processor 501 to perform the resource scheduling method described above.

[0210] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the resource scheduling method described above. The computer-readable storage medium may be volatile or non-volatile.

[0211] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described resource scheduling method.

[0212] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0213] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0214] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0215] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0216] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0217] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0218] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0219] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0220] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0221] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A resource scheduling method, characterized in that, include: During the process of performing a model training task using the device resources of the target device, resource usage information corresponding to the model training task is obtained from the target device; The resource usage information and the historical training behavior information of the target user who initiated the model training task are processed using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task. The performance evaluation result is used to represent the performance of the model training task in execution and / or the performance of the device resources. Based on the performance evaluation results, a resource scheduling strategy is generated for the model training task, so as to schedule device resources for the model training task according to the resource scheduling strategy.

2. The method according to claim 1, characterized in that, The performance evaluation results include task performance scores and / or resource risk levels; The process of using a performance evaluation model to process the resource usage information and the historical training behavior information of the target user initiating the model training task to obtain the performance evaluation result corresponding to the model training task includes: The resource usage information and the historical training behavior information of the target user who initiated the model training task are input into the performance evaluation model for evaluation processing to obtain the task performance score and / or resource risk level corresponding to the model training task. The task performance score is used to represent the performance health of the model training task, and the resource risk level is used to characterize the load risk of the device resources.

3. The method according to claim 2, characterized in that, The step of generating a resource scheduling strategy for the model training task based on the performance evaluation results includes: Based on the task performance score and the resource risk level, a policy generation model is used to perform policy generation processing to obtain a resource scheduling policy for the model training task; or The task performance score and the resource risk level are matched with multiple preset scheduling strategies, and the matched preset scheduling strategy is determined as the resource scheduling strategy for the model training task.

4. The method according to claim 2, characterized in that, The process of using a performance evaluation model to process the resource usage information and the historical training behavior information of the target user initiating the model training task to obtain the performance evaluation result corresponding to the model training task includes: The resource usage information is analyzed to obtain the device operating status of the target device. If any anomalies are found in the device operating status, a performance evaluation model is used to process the resource usage information and the historical training behavior information of the target user initiating the model training task to obtain the performance evaluation result corresponding to the model training task; or According to a preset time period, the resource usage information and the historical training behavior information of the target user who initiated the model training task are processed using a performance evaluation model to obtain the performance evaluation result corresponding to the model training task.

5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the resource usage information corresponding to the model training task from the target device during the process of performing the model training task using the device resources of the target device, the method further includes: Receive a model training task sent by the target user through a user terminal, wherein the model training task has a corresponding model to be trained; Based on the model type of the model to be trained and the historical training behavior information of the target user, determine the resource information to be allocated for the model training task; Based on the resource information to be allocated, a target device is selected from multiple candidate devices, wherein the target device is used to perform the model training task; The idle device resources of the target device are used as the device resources allocated to the model training task.

6. The method according to any one of claims 1 to 4, characterized in that, After generating a resource scheduling strategy for the model training task based on the performance evaluation results, the method further includes: The resource scheduling strategy is sent to the resource scheduling unit so that the resource scheduling unit can schedule the device resources allocated to the model training task according to the resource scheduling strategy. The scheduling process includes: reallocating corresponding device resources for the model training task, reselecting a device from multiple candidate devices to execute the model training task, and adjusting the priority of the model training task.

7. The method according to any one of claims 1 to 4, characterized in that, After generating a resource scheduling strategy for the model training task based on the performance evaluation results, the method further includes: Upon completion of the model training task, the historical training behavior information is updated based on the task information corresponding to the model training task to obtain the updated historical training behavior information; and / or Upon completion of the model training task, model training suggestion information is generated based on the task information corresponding to the model training task, and the model training suggestion information is displayed to the target user. The task information includes at least one of the following: equipment resource information, resource usage information, task execution results, and task fault information.

8. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the resource usage information corresponding to the model training task from the target device includes: Receive the resource usage information sent by the information acquisition module running in the target device; The information acquisition module is used to collect resource usage information of the model training task for the device resources during the process of executing the model training task using the device resources. The resource usage information is a portion of the resource usage information selected from all the resource usage information of the device resources according to a preset information identifier.

9. The method according to any one of claims 1 to 4, characterized in that, The historical training behavior information is generated based on the user attribute information of the target user and the historical task information of the historical model training tasks corresponding to the target user. The historical task information includes at least one of the following: resource usage information, task execution results, task failure information, and historically allocated device resource information; or The historical training behavior information is selected from multiple preset historical training behavior information based on the target user's user attribute information.

10. A resource scheduling device, characterized in that, include: The information acquisition module is used to acquire resource usage information corresponding to the model training task from the target device during the process of performing the model training task using the device resources of the target device; The performance evaluation module is used to process the resource usage information and the historical training behavior information of the target user who initiated the model training task using a performance evaluation model, and obtain the performance evaluation result corresponding to the model training task; the performance evaluation result is used to represent the performance of the model training task in execution and / or the performance of the device resources. The strategy generation module is used to generate a resource scheduling strategy for the model training task based on the performance evaluation results, so as to schedule device resources for the model training task according to the resource scheduling strategy.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-9.

Images