Large model training and pushing task scheduling method, device and equipment and storage medium

Abstract

The invention discloses a large model training and pushing task scheduling method and device, equipment and a storage medium, and relates to the technical field of information processing, and the method comprises the steps: obtaining reasoning task information corresponding to a model reasoning task and training task information corresponding to a model training task; based on the reasoning task information and the training task information, determining a first task execution position and first task scheduling time corresponding to the model reasoning task and a second task execution position and second task scheduling time corresponding to the model training task information; and performing task scheduling on the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time. According to the method and the device, flexible, accurate and timely large model training and pushing task scheduling can be realized, so that the problem that the switching from the training task to the reasoning task cannot be quickly completed due to relatively long time consumption caused by the condition that the large model reasoning task needs to load an oversized model file and the like is solved.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a large-model training and push task scheduling method, device, equipment and storage medium. Background Art

[0002] The current GPU (Graphics Processing Unit) mixed deployment (GPU mixed deployment refers to running multiple different types of tasks on the same GPU hardware resources) solution can be divided into two types based on the GPU occupation mode: a shared GPU core mixed deployment solution and a time-sharing exclusive GPU core mixed deployment solution. The shared GPU core mixed deployment solution mainly implements priority scheduling of GPU core resources between tasks by allocating GPU time slices to each task; the time-sharing exclusive GPU core mixed deployment solution implements time-sharing reuse by controlling the overall release and recovery of GPU core resources and video memory resources, and running different tasks in different time periods.

[0003] However, for large model services, large model inference services and large model training tasks will continue to occupy a large amount of GPU memory resources during operation, and GPU memory resources are incompressible resources. The co-location solution of shared GPU cores mainly co-locates GPU core resources, so it is not suitable for the co-location of large model inference services and training tasks. At the same time, large model services need to load hundreds of GB of model files when they are started, which takes several minutes to dozens of minutes. Traditional time-sharing co-location cannot quickly complete the switch from training tasks to inference tasks when traffic increases, resulting in excessive traffic on a single instance and resulting in service unavailability. Summary of the invention

[0004] The main purpose of this application is to provide a large model training and inference task scheduling method, device, equipment and storage medium, aiming to solve the problem that large model inference tasks take a long time and cannot quickly complete the switching from training tasks to inference tasks due to the need to load super-large model files.

[0005] To achieve the above objectives, the present application proposes a large model training task scheduling method, the method comprising:

[0006] Obtain the reasoning task information corresponding to the model reasoning task and the training task information corresponding to the model training task;

[0007] Based on the reasoning task information and the training task information, determine a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information;

[0008] Based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time, the model reasoning task and the model training task are scheduled.

[0009] In one embodiment, determining the first task execution position and the first task scheduling time corresponding to the model reasoning task and the second task execution position and the second task scheduling time corresponding to the model training task information based on the reasoning task information and the training task information includes:

[0010] Matching the reasoning task information and the training task information with a preset task execution resource library to determine a first task execution position corresponding to the model reasoning task and a second task execution position corresponding to the model training task information;

[0011] Performing correlation monitoring on the model reasoning task and the model training task;

[0012] If the triggering of the task scheduling request is monitored, the reasoning task information and the training task information are input into the task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

[0013] In one embodiment, the associating monitoring of the model reasoning task and the model training task includes:

[0014] Determine a first identifier corresponding to the model reasoning task and a second identifier corresponding to the model training task;

[0015] Based on the first identifier and the second identifier, dynamically associating the model reasoning task and the model training task;

[0016] Monitor whether the model inference tasks and model training tasks after dynamic management trigger task scheduling requests.

[0017] In one embodiment, the scheduling of the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time includes:

[0018] Obtaining the elastic control configuration and the first task scheduling type corresponding to the model reasoning task, and obtaining the second task scheduling type corresponding to the model training task;

[0019] Performing task scheduling on the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type;

[0020] The model training task is scheduled based on the second task execution location, the second task scheduling time, and the second task scheduling type.

[0021] In one embodiment, the elastic control configuration includes a timing control configuration and an indicator control configuration;

[0022] The scheduling of the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type includes:

[0023] If the flexible control configuration is a timing control configuration, obtaining a third task scheduling time in the timing control configuration;

[0024] Performing task scheduling on the model reasoning task based on the first task execution position, the third task scheduling time, and the first task scheduling type;

[0025] If the elastic control configuration is an indicator control configuration, determining whether the control indicator value in the indicator control configuration reaches a preset control threshold range;

[0026] If so, the model reasoning task is scheduled based on the first task execution location, the first task scheduling time, and the first task scheduling type.

[0027] In one embodiment, after scheduling the model training task based on the second task execution location, the second task scheduling time, and the second task scheduling type, the method further includes:

[0028] If the second task scheduling type is a task avoidance type, obtaining a task pause snapshot and a task avoidance end time of the model training task;

[0029] Based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot.

[0030] In one embodiment, after scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time, the method further includes:

[0031] Monitor the task scheduling of the model reasoning task and the model training task;

[0032] Visualizing the task scheduling situation;

[0033] The visualized task scheduling situation is pushed to the security administrator for the security administrator to review.

[0034] In addition, to achieve the above purpose, the present application also proposes a large model training and pushing task scheduling device, the large model training and pushing task scheduling device comprising:

[0035] An acquisition module is used to obtain the reasoning task information corresponding to the model reasoning task and the training task information corresponding to the model training task;

[0036] A determination module, used to determine, based on the reasoning task information and the training task information, a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information;

[0037] A scheduling module is used to schedule the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a large model training and pushing task scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model training and pushing task scheduling method as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large model training and pushing task scheduling method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the large model training and push task scheduling method as described above.

[0041] The present application provides a large model training and pushing task scheduling method, apparatus, equipment and storage medium. The large model training and pushing task scheduling method obtains reasoning task information corresponding to the model reasoning task and training task information corresponding to the model training task, and then determines the first task execution position and the first task scheduling time corresponding to the model reasoning task and the second task execution position and the second task scheduling time corresponding to the model training task information based on the reasoning task information and the training task information, thereby scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time, thereby ensuring that task execution resources are more efficiently allocated and utilized between reasoning and training tasks, and realizing rapid response to the resource requirements of reasoning tasks, thereby effectively reducing the waiting time of reasoning tasks, reducing the frequency of task migration and preemption, making task execution smoother, and thus improving the overall task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart diagram of the first embodiment of the large model training and pushing task scheduling method of the present application is provided;

[0045] Figure 2 A flowchart diagram of the second embodiment of the large model training and pushing task scheduling method of the present application is provided;

[0046] Figure 3 A brief flow chart of timing control configuration and indicator control configuration provided for the large model training and push task scheduling method of this application;

[0047] Figure 4 A brief system architecture diagram provided for the large model training and push task scheduling method of this application;

[0048] Figure 5 This is a schematic diagram of the module structure of a large model training and pushing task scheduling device according to an embodiment of the present application;

[0049] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the large model training and pushing task scheduling method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a big data service platform, a task scheduling system, etc. The following takes the task scheduling system as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the embodiment of the present application provides a large model training task scheduling method, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the large model training and push task scheduling method of the present application is provided.

[0055] In this embodiment, the large model training task scheduling method includes steps S11 to S13:

[0056] Step S11, obtaining reasoning task information corresponding to the model reasoning task and training task information corresponding to the model training task;

[0057] It should be noted that the model reasoning task refers to a task that uses a trained model to process new input data and generate prediction results. The reasoning task information refers to a detailed description and attributes of the reasoning task, including but not limited to the task ID, reasoning task identifier, input data (input data type and format of the reasoning task), task execution requirements (computing resources required to execute the reasoning task (such as GPU, CPU), memory size, training framework, image size, acceptable card type (GPU type) and storage medium, etc.), priority and expected response time, etc.

[0058] It should be further explained that the model training task refers to the task of using a large amount of labeled data to train or adjust the parameters of the machine learning model, with the goal of enabling the model to learn the patterns and regularities in the data. The training task information refers to the detailed description and attributes of the training task, including the task ID, training task identifier, training data set, resource requirements (such as the computing resources required to perform the training task (such as GPU, CPU), memory size, storage requirements), expected training time, and task priority, etc.

[0059] Specifically, the model inference task and the model training task are submitted according to the preset configuration template, and the model inference task and the training framework, image size, acceptable card type, storage medium and other information used by the model inference task are automatically identified through the preset configuration template, and the training task information corresponding to the model training task is identified through the preset configuration template, and then the identified information is associated and combined with the task corresponding to the information, so as to obtain the reasoning task information corresponding to the model inference task and the training task information corresponding to the model training task.

[0060] Step S12, based on the reasoning task information and the training task information, determining a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information;

[0061] It should be noted that the first task execution location refers to the specific execution location of the model reasoning task in the system, which can be a specific server, GPU device or container instance, and is not limited here. The first task scheduling time refers to the specific time point or time window when the model reasoning task is scheduled for execution.

[0062] It should be further explained that the second task execution location refers to the specific execution location of the model training task in the system, which can be a specific server, GPU device or container instance, without limitation here, and the second task scheduling time refers to the specific time point or time window when the model training task is scheduled for execution, which can be determined based on the task deadline, resource availability or other scheduling strategies.

[0063] Specifically, the reasoning task information and the training task information are matched with a preset task execution resource library to determine the first task execution position corresponding to the model reasoning task and the second task execution position corresponding to the model training task information, and then the model reasoning task and the model training task are associated with monitoring. If the triggering of a task scheduling request is monitored, the reasoning task information and the training task information are input into a task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

[0064] Step S13, scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time.

[0065] It should be noted that job scheduling is very important in fields such as parallel computing, distributed systems, operating systems, and cloud computing. It refers to the process of managing and optimizing the execution order of tasks (which can be programs, processes, or jobs) in the system, thereby improving the utilization of system resources, reducing task completion time (i.e., turnaround time), improving system throughput, and optimizing user experience.

[0066] Specifically, the elastic control configuration and the first task scheduling type corresponding to the model reasoning task are obtained, and the second task scheduling type corresponding to the model training task is obtained, and then based on the first task execution position, the first task scheduling time, the elastic control configuration and the first task scheduling type, the model reasoning task is scheduled, and then based on the second task execution position, the second task scheduling time and the second task scheduling type, the model training task is scheduled.

[0067] This embodiment obtains reasoning task information corresponding to the model reasoning task and training task information corresponding to the model training task, and then determines the first task execution position and the first task scheduling time corresponding to the model reasoning task and the second task execution position and the second task scheduling time corresponding to the model training task information based on the reasoning task information and the training task information, thereby scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time, thereby ensuring that task execution resources are more efficiently allocated and utilized between reasoning and training tasks, and realizing rapid response to the resource requirements of reasoning tasks, thereby effectively reducing the waiting time of reasoning tasks, reducing the frequency of task migration and preemption, making task execution smoother, and thus improving the overall task execution efficiency.

[0068] Based on this, the embodiment of the present application provides a large model training task scheduling method, referring to Figure 2 , Figure 2 A flowchart diagram is provided for the second embodiment of the large model training and push task scheduling method of this application.

[0069] In a feasible implementation, the determining, based on the reasoning task information and the training task information, a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information includes:

[0070] Step S21, matching the reasoning task information and the training task information with a preset task execution resource library, and determining a first task execution position corresponding to the model reasoning task and a second task execution position corresponding to the model training task information;

[0071] It should be noted that the preset task execution resource library refers to a database or resource pool, which contains detailed information on computing resources that can be used to execute tasks. These resources may include physical servers, virtual machines, container instances, specific hardware (such as GPU, CPU), storage devices, etc., as well as information about each resource's technical specifications, performance indicators, current status (idle or busy), location (where in the data center), etc., so as to provide the most suitable execution resource location for each task. There is no restriction on the type of storage information in the preset task execution resource library, and it can be set according to actual conditions.

[0072] Step S22, performing association monitoring on the model reasoning task and the model training task;

[0073] Specifically, determine the first identifier corresponding to the model reasoning task and the second identifier corresponding to the model training task, and then dynamically associate the model reasoning task and the model training task based on the first identifier and the second identifier, so as to monitor whether the model reasoning task and the model training task after dynamic management trigger a task scheduling request.

[0074] Step S23, if the triggering of the task scheduling request is monitored, the reasoning task information and the training task information are input into the task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

[0075] It should be noted that the task scheduling request is used to indicate that the system needs to schedule one or more tasks. The task scheduling request can be triggered by various events, such as the submission of a new task, the completion of an existing task, changes in system load, etc., and usually includes task identification, task type, resource requirements (computing resource specifications required to execute the task (such as the number of CPU cores, GPU model, memory size)), task priority, task dependencies, and task scheduling time, etc.

[0076] It should be further explained that the task scheduling time prediction model is used to predict the scheduling time of the task. It can predict the optimal scheduling time of the task based on factors such as the characteristics of the task, the status of the resources, historical scheduling data, etc., thereby improving the efficiency and accuracy of the scheduling, reducing the waiting time of the task, and further reducing the GPU resource overhead, leaving more idle resources for executing higher priority tasks.

[0077] Specifically, if the triggering of a task scheduling request is monitored, such as detecting a significant increase in user usage corresponding to a model reasoning task, the reasoning task information and the training task information are input into a task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

[0078] This embodiment matches the reasoning task information and the training task information with a preset task execution resource library to determine the first task execution position corresponding to the model reasoning task and the second task execution position corresponding to the model training task information, and then monitors the model reasoning task and the model training task in association. If the triggering of a task scheduling request is detected, the reasoning task information and the training task information are input into a task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model, thereby accurately matching tasks and resources, ensuring that each task can be executed on the most appropriate resources to avoid resource waste, and by predicting the scheduling time of tasks, reducing task waiting and execution time, thereby improving the overall task execution efficiency. At the same time, dynamic scheduling can better distribute the load in the system to prevent some resources from being overloaded while other resources are idle, thereby improving the flexibility and adaptability of task scheduling.

[0079] In a feasible implementation manner, the associating monitoring of the model reasoning task and the model training task includes:

[0080] Step S31, determining a first identifier corresponding to the model reasoning task and a second identifier corresponding to the model training task;

[0081] It should be noted that the first identifier refers to the unique identifier assigned to the model reasoning task, which may include the task ID, model version, task submission time, and user ID (the identifier of the user or team submitting the task), etc. There is no restriction here and it can be set according to actual conditions.

[0082] It should be further explained that the second identifier refers to a unique identifier assigned to the model training task, which may include a task ID, a model name, a training data set identifier, and a training configuration (configuration parameters used by the training task, such as a learning rate, a batch size, etc.). There is no restriction here and it can be set according to actual conditions.

[0083] Specifically, the first identifier corresponding to the model reasoning task and the second identifier corresponding to the model training task can be determined by reading the identifiers in the reasoning task information and the training task information, and no limitation is made here.

[0084] Step S32: dynamically associating the model reasoning task and the model training task based on the first identifier and the second identifier;

[0085] Specifically, it is determined which model reasoning tasks and which model training tasks should be associated according to preset association rules, wherein the preset association rules can be set based on the size of occupied resources, model training framework, task priority, user priority or other business logic. For example, if the reasoning task and the training task have similar resource requirements (such as the same GPU model or memory size), they can be associated so as to optimize when allocating resources. Further, after determining which model reasoning tasks and which model training tasks should be associated, the model reasoning tasks and model training tasks that need to be dynamically bound are queried according to the first identifier and the second identifier, and then the model reasoning tasks and the reasoning task information and training task information corresponding to the model training tasks are dynamically associated to achieve an overall monitoring of multiple parties. For example, based on the dynamic association relationship, when switching tasks or increasing traffic, fixed training tasks can be specified as reasoning tasks to avoid, thereby improving the avoidance speed while being able to clarify and reduce the impact of avoidance on training tasks in advance, thereby improving the user experience.

[0086] Step S33, monitoring whether the model reasoning tasks and model training tasks after dynamic management trigger task scheduling requests.

[0087] Specifically, the system monitoring tool is used to monitor whether the model inference tasks and model training tasks after dynamic management trigger task scheduling requests, such as monitoring whether the current GPU resource usage is reasonable, whether the current inference task volume has increased significantly, etc., which can be set according to actual conditions.

[0088] This embodiment determines the first identifier corresponding to the model reasoning task and the second identifier corresponding to the model training task, and then dynamically associates the model reasoning task and the model training task based on the first identifier and the second identifier, so as to monitor whether the model reasoning task and the model training task after dynamic management trigger task scheduling requests, and then identifies and resolves potential conflicts between tasks through dynamic association, such as resource contention, thereby reducing interruptions and avoiding failures in task execution, and at the same time accurately identifies and tracks each reasoning and training task, thereby achieving more accurate scheduling decisions, strengthening task management, and responding to task scheduling requests more quickly.

[0089] In a feasible implementation manner, the scheduling of the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time includes:

[0090] Step S41, obtaining the elastic control configuration and the first task scheduling type corresponding to the model reasoning task, and obtaining the second task scheduling type corresponding to the model training task;

[0091] It should be noted that the elastic control configuration refers to a set of parameters and settings used to control the scaling behavior of tasks in terms of resource usage and performance to achieve dynamic allocation and release of resources. The elastic control configuration includes timing control configuration and indicator control configuration.

[0092] It should be further explained that the first task scheduling type refers to a scheduling strategy or mode for reasoning tasks, including a pause type, a postponement type, and a task migration type, etc., which are not limited here. The second task scheduling type refers to a scheduling strategy or mode for training tasks.

[0093] Specifically, the elastic control configuration corresponding to the model inference task is obtained by reading the user configuration, and the first task scheduling type and the second task scheduling type corresponding to the model training task are determined by the current user access volume and GPU occupancy, without any restriction here.

[0094] Step S42, scheduling the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type;

[0095] Specifically, if the elastic control configuration is a timing control configuration, the third task scheduling time in the timing control configuration is obtained, and then the model reasoning task is scheduled based on the first task execution position, the third task scheduling time and the first task scheduling type. In addition, if the elastic control configuration is an indicator control configuration, it is determined whether the control indicator value in the indicator control configuration reaches the preset control threshold range, and if so, the model reasoning task is scheduled based on the first task execution position, the first task scheduling time and the first task scheduling type.

[0096] Step S43: Schedule the model training task based on the second task execution location, the second task scheduling time, and the second task scheduling type.

[0097] Specifically, the model training task is scheduled according to the second task execution location, the second task scheduling time and the second task scheduling type. For example, a deep learning training task is scheduled on the GPU of a specific server and starts execution at 10 pm. Batch scheduling is used to optimize resource utilization while allowing high-priority tasks to be preempted when resources are tight.

[0098] This embodiment obtains the elastic control configuration and the first task scheduling type corresponding to the model reasoning task, and obtains the second task scheduling type corresponding to the model training task, and then schedules the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration and the first task scheduling type, and then schedules the model training task based on the second task execution position, the second task scheduling time and the second task scheduling type, thereby managing resources more finely, ensuring the reasonable allocation of resources among different tasks, reducing GPU resource overhead, and leaving more idle resources for tasks to be executed, thereby optimizing task execution efficiency and improving the punctuality and accuracy of task scheduling.

[0099] In a feasible implementation manner, the elastic control configuration includes a timing control configuration and an indicator control configuration; the scheduling of the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration and the first task scheduling type includes:

[0100] Step S51, if the flexible control configuration is a timing control configuration, obtaining a third task scheduling time in the timing control configuration;

[0101] It should be noted that the timing control configuration refers to a preset scheduling strategy that adjusts resource allocation based on a specific time plan and is often used to predict and respond to periodic load changes, for example, increasing the computing resources of model training tasks at night to process batch jobs, or expanding the computing resources of model reasoning tasks before business peaks. The third task scheduling time refers to a time point or time period determined according to the timing control configuration. At this time point or time period, a specific task (such as a model training task) will be scheduled for execution first. For example, it supports the configuration of single execution, weekly and monthly cyclic execution and other timing rules.

[0102] Step S52, scheduling the model reasoning task based on the first task execution position, the third task scheduling time and the first task scheduling type;

[0103] Specifically, the model inference task is scheduled according to the first task execution location, the third task scheduling time and the first task scheduling type. For example, on a machine learning platform, the model training task is scheduled to be executed on a specific GPU. According to the timing control configuration, this task will be automatically started during the off-peak period of each day, that is, at 10 pm, to optimize resource utilization and reduce the impact on user interaction tasks.

[0104] Step S53, if the elastic control configuration is an index control configuration, determining whether the control index value in the index control configuration reaches a preset control threshold range;

[0105] It should be noted that the indicator control configuration refers to a dynamic resource adjustment strategy based on performance indicators, which involves monitoring specific performance indicators (such as CPU utilization, requests per second, GPU occupancy, user access, core utilization, memory usage, queue length, etc.), and automatically adjusting resource allocation according to changes in these indicators. For example, if the CPU utilization exceeds 80%, it triggers a control operation to reduce the computing resources of the model training task.

[0106] It should be further explained that the control index value refers to the specific value used to trigger resource adjustment in the index control configuration. The preset control threshold range refers to a series of values ​​set for the control index value. When the actual monitored index value reaches or exceeds this range, a specific resource allocation operation will be triggered. For example, the preset control threshold range is a CPU usage rate between 70% and 90%. When the actual usage rate enters this range, the system will expand or reduce resources according to the index control configuration.

[0107] Specifically, if the elastic control configuration is an indicator control configuration, then it is determined whether the control indicator value in the indicator control configuration reaches the preset control threshold range. Furthermore, if the control indicator value in the indicator control configuration does not reach the preset control threshold range, the corresponding task will continue to be executed according to the current task execution status.

[0108] Step S54: If yes, scheduling the model reasoning task based on the first task execution location, the first task scheduling time and the first task scheduling type.

[0109] Specifically, if yes, then based on the first task execution location, the first task scheduling time and the first task scheduling type, the model reasoning task is scheduled. For example, if the GPU usage rate meets the preset control threshold range, that is, the indicator control configuration is triggered, then the system will automatically adjust the resources and schedule the model reasoning task to the currently idle GPU and execute it at the first task scheduling time to ensure that the reasoning task can respond in time and optimize the overall resource utilization. Figure 3 , Figure 3 A brief flow chart of timing control configuration and indicator control configuration provided for the large model training and push task scheduling method of this application.

[0110] In this embodiment, if the elastic control configuration is a timing control configuration, the third task scheduling time in the timing control configuration is obtained, and then the model reasoning task is scheduled based on the first task execution position, the third task scheduling time and the first task scheduling type. If the elastic control configuration is an indicator control configuration, it is determined whether the control indicator value in the indicator control configuration reaches a preset control threshold range. If so, the model reasoning task is scheduled based on the first task execution position, the first task scheduling time and the first task scheduling type. The task scheduling situation is automatically adjusted according to real-time performance indicators and predetermined thresholds, the adaptability to changing workloads is improved, and it is helpful to balance the load among multiple computing resources to prevent overload of certain resources during some peak trial periods, thereby flexibly responding to the variability and real-time nature of task scheduling.

[0111] In a feasible implementation manner, after scheduling the model training task based on the second task execution location, the second task scheduling time, and the second task scheduling type, the method further includes:

[0112] Step S61: if the second task scheduling type is a task avoidance type, obtaining a task pause snapshot and a task avoidance end time of the model training task;

[0113] It should be noted that the task avoidance type is a scheduling strategy in which low-priority tasks are actively suspended or migrated when high-priority tasks need resources in order to make way for high-priority tasks. This type of scheduling can ensure that critical tasks can obtain the necessary computing resources in a timely manner, thereby meeting their requirements for response time and resource availability.

[0114] It should be further explained that the task pause snapshot refers to a complete record of the task status captured by the system when the task (such as the model training task) is paused, so that the task can be resumed from the exact same state. It includes task status, memory status, computing progress (the progress of task completion, such as the number of iterations of the training model), input and output data, configuration parameters (such as model parameters, etc.), and system resource usage.

[0115] Furthermore, the task avoidance end time refers to the time point at which the task avoidance operation is expected to end. After this time point, the originally avoided task can be resumed. The task avoidance end time can be based on the prediction of the scheduling system or can be manually set by the system administrator according to resource requirements and task priorities.

[0116] Specifically, if the second task scheduling type is a task avoidance type, then the task pause snapshot of the model training task and the task avoidance end time are obtained, wherein the task avoidance end time can be determined by the estimated completion time of high-priority tasks, the availability of system resources, and business needs and policies, and is not limited here.

[0117] Step S62: Based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot.

[0118] Specifically, based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot. For example, when a high-priority model inference task needs to use GPU resources, an ongoing model training task will be paused and its progress snapshot will be saved; once the high-priority task is completed, the system will resume the model training task according to the task pause snapshot at the task avoidance settlement time, and continue execution from the precise point of the last pause to ensure that the training process is seamless and reduce waste of computing resources.

[0119] In this embodiment, if the second task scheduling type is a task avoidance type, the task pause snapshot and the task avoidance end time of the model training task are obtained, and then based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot, thereby ensuring that the model training task can be seamlessly resumed after the avoidance ends without starting from the beginning, saving time and computing resources, that is, allowing temporary avoidance when high-priority tasks require resources, and then quickly resuming when resources are available, so as to maximize resource utilization efficiency, while reducing training interruptions caused by avoidance, maintaining the continuity of the training process, and facilitating model convergence and performance optimization, thereby adapting to dynamically changing resource requirements and priority adjustments.

[0120] In a feasible implementation manner, after scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time, the further step further includes:

[0121] Step S71, monitoring the task scheduling of the model reasoning task and the model training task;

[0122] It should be noted that the task scheduling situation refers to a comprehensive description of the task execution and resource allocation status in the system, including task status: the current status of each task, such as queued, running, paused, completed or failed; resource allocation: details of the resources allocated to each task, including CPU, memory, storage and GPU, etc.; scheduling history: the scheduling history of the task, including the submission time, start time and end time of the task; priority information: the priority setting of the task and how it affects the scheduling order of the tasks; task dependency: the dependency relationship between tasks, which tasks depend on the output of other tasks; performance indicators: performance indicators of task execution, such as processing speed, resource utilization and response time; scheduling delay Delay: the length of time a task waits for execution, including waiting time and execution delay; Errors and exceptions: any errors or exceptions encountered during task execution; User and project information: information about the user or team that submitted the task, and the project or workflow to which the task belongs; Scheduling policy application: the scheduling policies and algorithms applied to the task, and how they affect the scheduling of the task; Resource usage: resource usage, including resource utilization, idle time, and resource contention; Task queue status: the length of the task queue and the dynamic changes of the task waiting queue; Scheduling system status: the health status and performance of the scheduling system itself, such as processing power, latency, and throughput, which can be set according to actual conditions and are not restricted here.

[0123] Step S72, visualizing the task scheduling situation;

[0124] Specifically, the task scheduling situation can be visualized by developing a monitoring dashboard, which uses real-time data streams and historical logs to display key information such as the current status of the task, resource allocation, execution progress, performance indicators, etc., and presents it intuitively through graphical elements such as charts, timelines, and heat maps. At the same time, it provides interactive functions, allowing users to filter and drill down to detailed scheduling records of specific tasks based on conditions such as task name, time, and priority, thereby helping administrators quickly grasp the overall scheduling situation of the system and make corresponding management decisions.

[0125] Step S73, pushing the visualized task scheduling situation to the security administrator for the security administrator to view.

[0126] Specifically, the visualized task scheduling situation is pushed to the security administrator, where it can be pushed through computer display screens, APP pushes, mini-programs, web pages, etc., without any restrictions, for the security administrator to view.

[0127] This embodiment monitors the task scheduling of the model inference task and the model training task, and then visualizes the task scheduling, thereby pushing the visualized task scheduling to the security administrator for the security administrator to view, thereby more intuitively monitoring the scheduling status of the task, supporting the security administrator to make faster and more accurate scheduling decisions based on real-time information, timely discover potential problems or anomalies, and improve user experience.

[0128] For example, to help understand the implementation process of the large model training task scheduling method, please refer to Figure 4 , Figure 4 A brief system architecture diagram is provided for the large model training and push task scheduling method of this application.

[0129] It should be noted that the examples in the figure are only used to understand the present application and do not constitute a limitation on the large model training and push task scheduling method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0131] This application also provides a large model training task scheduling device, please refer to Figure 5 , the large model training and pushing task scheduling device comprises:

[0132] An acquisition module 51 is used to acquire reasoning task information corresponding to the model reasoning task and training task information corresponding to the model training task;

[0133] A determination module 52 is used to determine, based on the reasoning task information and the training task information, a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information;

[0134] The scheduling module 53 is used to schedule the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time.

[0135] The large model training and pushing task scheduling device is also used for:

[0136] Matching the reasoning task information and the training task information with a preset task execution resource library to determine a first task execution position corresponding to the model reasoning task and a second task execution position corresponding to the model training task information;

[0137] Performing correlation monitoring on the model reasoning task and the model training task;

[0138] If the triggering of the task scheduling request is monitored, the reasoning task information and the training task information are input into the task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

[0139] The large model training and pushing task scheduling device is also used for:

[0140] Determine a first identifier corresponding to the model reasoning task and a second identifier corresponding to the model training task;

[0141] Based on the first identifier and the second identifier, dynamically associating the model reasoning task and the model training task;

[0142] Monitor whether the model inference tasks and model training tasks after dynamic management trigger task scheduling requests.

[0143] The large model training and pushing task scheduling device is also used for:

[0144] Obtaining the elastic control configuration and the first task scheduling type corresponding to the model reasoning task, and obtaining the second task scheduling type corresponding to the model training task;

[0145] Performing task scheduling on the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type;

[0146] The model training task is scheduled based on the second task execution location, the second task scheduling time, and the second task scheduling type.

[0147] The large model training and pushing task scheduling device is also used for:

[0148] If the flexible control configuration is a timing control configuration, obtaining a third task scheduling time in the timing control configuration;

[0149] Performing task scheduling on the model reasoning task based on the first task execution position, the third task scheduling time, and the first task scheduling type;

[0150] If the elastic control configuration is an indicator control configuration, determining whether the control indicator value in the indicator control configuration reaches a preset control threshold range;

[0151] If so, the model reasoning task is scheduled based on the first task execution location, the first task scheduling time, and the first task scheduling type.

[0152] The large model training and pushing task scheduling device is also used for:

[0153] If the second task scheduling type is a task avoidance type, obtaining a task pause snapshot and a task avoidance end time of the model training task;

[0154] Based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot.

[0155] The large model training and pushing task scheduling device is also used for:

[0156] Monitor the task scheduling of the model reasoning task and the model training task;

[0157] Visualizing the task scheduling situation;

[0158] The visualized task scheduling situation is pushed to the security administrator for the security administrator to review.

[0159] The large model training and pushing task scheduling device provided by the present application adopts the large model training and pushing task scheduling method in the above embodiment, which can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the large model training and pushing task scheduling device provided by the present application are the same as the beneficial effects of the large model training and pushing task scheduling method provided by the above embodiment, and other technical features in the large model training and pushing task scheduling device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0160] The present application provides a large model training and pushing task scheduling device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model training and pushing task scheduling method in the above-mentioned embodiment 1.

[0161] Reference below Figure 6 , which shows a schematic diagram of the structure of a large model training and pushing task scheduling device suitable for implementing the embodiment of the present application. The large model training and pushing task scheduling device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The large model training and push task scheduling device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] like Figure 6As shown, the large model training and pushing task scheduling device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the large model training and pushing task scheduling device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the large model training and pushing task scheduling device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a large model training and pushing task scheduling device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0163] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0164] The large model training and pushing task scheduling device provided by the present application adopts the large model training and pushing task scheduling method in the above embodiment, which can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the large model training and pushing task scheduling device provided by the present application are the same as the beneficial effects of the large model training and pushing task scheduling method provided by the above embodiment, and the other technical features in the large model training and pushing task scheduling device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0165] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0167] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the large model training and push task scheduling method in the above-mentioned embodiment.

[0168] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0169] The above-mentioned computer-readable storage medium may be included in the large model training and pushing task scheduling device; or it may exist independently without being assembled into the large model training and pushing task scheduling device.

[0170] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the large model training and pushing task scheduling device, the large model training and pushing task scheduling device:

[0171] Obtain the reasoning task information corresponding to the model reasoning task and the training task information corresponding to the model training task;

[0172] Based on the reasoning task information and the training task information, determine a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information;

[0173] Based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time, the model reasoning task and the model training task are scheduled.

[0174] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0177] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large model training and pushing task scheduling method, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the large model training and pushing task scheduling method provided in the above-mentioned embodiment, and will not be repeated here.

[0178] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the large model training and push task scheduling method as described above.

[0179] The computer program product provided in this application can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the large model training and pushing task scheduling method provided in the above embodiment, which will not be repeated here.

[0180] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A large model training task scheduling method, characterized in that: include: Obtain the reasoning task information corresponding to the model reasoning task and the training task information corresponding to the model training task; Based on the reasoning task information and the training task information, determine a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information; Based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time, the model reasoning task and the model training task are scheduled.

2. The large model training task scheduling method according to claim 1, characterized in that: The determining, based on the reasoning task information and the training task information, a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information, includes: Matching the reasoning task information and the training task information with a preset task execution resource library to determine a first task execution position corresponding to the model reasoning task and a second task execution position corresponding to the model training task information; Performing correlation monitoring on the model reasoning task and the model training task; If the triggering of the task scheduling request is monitored, the reasoning task information and the training task information are input into the task scheduling time prediction model to obtain the first task scheduling time and the second task scheduling time output by the task scheduling time prediction model.

3. The large model training task scheduling method according to claim 2, characterized in that: The associating monitoring of the model reasoning task and the model training task includes: Determine a first identifier corresponding to the model reasoning task and a second identifier corresponding to the model training task; Based on the first identifier and the second identifier, dynamically associating the model reasoning task and the model training task; Monitor whether the model inference tasks and model training tasks after dynamic management trigger task scheduling requests.

4. The large model training task scheduling method according to claim 1, characterized in that: The scheduling of the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time includes: Obtaining the elastic control configuration and the first task scheduling type corresponding to the model reasoning task, and obtaining the second task scheduling type corresponding to the model training task; Performing task scheduling on the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type; The model training task is scheduled based on the second task execution location, the second task scheduling time, and the second task scheduling type.

5. The large model training task scheduling method according to claim 4, characterized in that: The elastic control configuration includes a timing control configuration and an indicator control configuration; The scheduling of the model reasoning task based on the first task execution position, the first task scheduling time, the elastic control configuration, and the first task scheduling type includes: If the flexible control configuration is a timing control configuration, obtaining a third task scheduling time in the timing control configuration; Performing task scheduling on the model reasoning task based on the first task execution position, the third task scheduling time, and the first task scheduling type; If the elastic control configuration is an indicator control configuration, determining whether the control indicator value in the indicator control configuration reaches a preset control threshold range; If so, the model reasoning task is scheduled based on the first task execution location, the first task scheduling time, and the first task scheduling type.

6. The large model training task scheduling method according to claim 4, characterized in that: After scheduling the model training task based on the second task execution position, the second task scheduling time and the second task scheduling type, the method further includes: If the second task scheduling type is a task avoidance type, obtaining a task pause snapshot and a task avoidance end time of the model training task; Based on the task avoidance settlement time, the model training task is executed according to the task progress in the task pause snapshot.

7. The large model training task scheduling method according to claim 1, characterized in that: After scheduling the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position, and the second task scheduling time, the method further includes: Monitor the task scheduling of the model reasoning task and the model training task; Visualizing the task scheduling situation; The visualized task scheduling situation is pushed to the security administrator for the security administrator to review.

8. A large model training task scheduling device, characterized in that: include: An acquisition module is used to obtain the reasoning task information corresponding to the model reasoning task and the training task information corresponding to the model training task; A determination module, used to determine, based on the reasoning task information and the training task information, a first task execution position and a first task scheduling time corresponding to the model reasoning task and a second task execution position and a second task scheduling time corresponding to the model training task information; A scheduling module is used to schedule the model reasoning task and the model training task based on the first task execution position, the first task scheduling time, the second task execution position and the second task scheduling time.

9. A large model training and pushing task scheduling device, characterized in that: The large model training and pushing task scheduling device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large model training and pushing task scheduling method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large model training and pushing task scheduling method as described in any one of claims 1 to 7 are implemented.

Images