Task scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a task scheduling method and device, electronic equipment, a storage medium and a program product, and relates to the field of big data, and the method comprises the steps: obtaining a task environment state constructed by a to-be-scheduled task and a current data resource environment; calling a preset prompt generator, determining a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment; and inputting the cue word into a preset large model to obtain a scheduling scheme of the target scheduling task, and scheduling the target scheduling task based on the scheduling scheme. The technical problem that task scheduling is not efficient enough is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data, and in particular to a task scheduling method, device, electronic device, storage medium, and program product. Background Art

[0002] In the field of big data, task scheduling is a crucial technology that directly affects the overall performance and resource utilization of the system where the task is located. Currently, commonly used task scheduling technologies include those based on classical optimization methods, which rely on predefined rules and search iteration methods. They are difficult to effectively cope with complex and dynamic environmental changes and are prone to falling into local optimality. Falling into local optimality means that resources are not fully utilized. For example, some resources may be over-allocated, while others may be idle. This unbalanced resource allocation will reduce the overall efficiency of the system. Therefore, there is currently a technical problem that task scheduling is not efficient enough.

[0003] The above contents are only used to assist in understanding the technical solutions of the embodiments of the present application, and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a task scheduling method, device, electronic device, storage medium and program product, aiming to solve the technical problem that task scheduling is not efficient enough.

[0005] To achieve the above object, the present application provides a task scheduling method, which includes:

[0006] Get the task environment status constructed by the task to be scheduled and the current data resource environment;

[0007] Calling a preset prompt generator to determine a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment;

[0008] The prompt word is input into a preset large model to obtain a scheduling plan for the target scheduling task, so as to schedule the target scheduling task based on the scheduling plan.

[0009] In one embodiment, the method further comprises:

[0010] Determining a scheduling score for the scheduling solution;

[0011] Based on the scheduling score, the preset prompt generator is optimized.

[0012] In one embodiment, the step of determining the scheduling score of the scheduling solution includes:

[0013] Executing the scheduling plan to obtain a scheduling result of executing the scheduling plan;

[0014] Determine the execution result parameters corresponding to each preset quantitative scoring indicator from the scheduling result;

[0015] A weighted sum is performed on each of the execution result parameters to obtain a scheduling score for the scheduling result.

[0016] In one embodiment, the step of obtaining the task environment state constructed by the task to be scheduled and the current data resource environment includes:

[0017] Based on the acquired task to be scheduled and the dependency relationship of each subtask in the task to be scheduled, a task state vector of the task to be scheduled is generated, wherein the task state vector includes a dependency vector corresponding to each subtask in the task to be scheduled;

[0018] Generate a resource utilization state vector of the data resource environment based on the resource utilization rate of each data center in the current data resource environment;

[0019] The task state vector and the resource utilization state vector are concatenated to obtain the task environment state.

[0020] In one embodiment, the step of calling a preset prompt generator, determining a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment comprises:

[0021] Calling a preset prompt generator to determine a target scheduling task, a target dependency vector of each target subtask in the target scheduling task, and a remaining available resource vector in the data resource environment from the task environment state, and generating a task description vector for each target subtask in the target scheduling task;

[0022] The available resource vector, each target dependency vector and each task description vector are concatenated to obtain a prompt word.

[0023] In one embodiment, the step of inputting the prompt word into a preset large model to obtain a scheduling solution for the target scheduling task includes:

[0024] Inputting the prompt word into a preset large model, encoding the prompt word to obtain a text vector of the prompt word, and scheduling and allocating the target scheduling task based on the text vector to obtain the scheduling plan;

[0025] The scheduling scheme includes the execution order and node resources of each target subtask in the target scheduling task.

[0026] In one embodiment, the method comprises:

[0027] Obtaining a training sample of a preset model to be trained, and dividing the training sample into a plurality of sub-training samples;

[0028] Allocating each of the sub-training samples to each preset computing node, and configuring a preset model to be trained on each of the preset computing nodes;

[0029] Iteratively training the preset model to be trained in each of the preset computing nodes through the corresponding sub-training samples, and periodically synchronizing the preset models to be trained corresponding to each of the preset computing nodes;

[0030] Based on the loss function of each preset computing node, calculate the training loss value of the preset to-be-trained model corresponding to each preset computing node;

[0031] Calculate the loss average of the training loss values ​​corresponding to each of the preset computing nodes, and update the preset to-be-trained models on each of the preset computing nodes based on the loss average;

[0032] When the preset model to be trained corresponding to any of the preset computing nodes meets the preset training end condition, the preset model to be trained is used as the preset training completion model;

[0033] Among them, when the preset model to be trained is a preset large model to be trained, the preset training completion model is the preset large model; when the preset model to be trained is a preset prompter to be trained, the preset training completion model is a preset prompt generator.

[0034] In one embodiment, the method further comprises:

[0035] The target scheduling task and the target dependency vector corresponding to the target scheduling task are removed from the task environment state, and based on the node resources of each target subtask in the target scheduling task in the scheduling scheme, the resource utilization state vector in the task environment state is adjusted to update the task environment state.

[0036] In addition, to achieve the above-mentioned purpose, an embodiment of the present application provides a task scheduling device, the device comprising:

[0037] An acquisition module is used to obtain the task environment state constructed by the task to be scheduled and the current data resource environment;

[0038] A prompt word generation module, used to call a preset prompt generator, determine a target scheduling task from the task environment state, and generate a prompt word corresponding to the target scheduling task and the data resource environment;

[0039] The scheduling scheme generating module is used to input the prompt word into a preset large model to obtain a scheduling scheme for the target scheduling task, so as to schedule the target scheduling task based on the scheduling scheme.

[0040] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a program of the task scheduling method stored in the memory and executable on the processor. When the program of the task scheduling method is executed by the processor, the steps of the task scheduling method as described above can be implemented.

[0041] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a computer-readable storage medium, on which a program for implementing the task scheduling method is stored. When the program of the task scheduling method is executed by a processor, the steps of the task scheduling method as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the task scheduling method as described above when executed by a processor.

[0043] One or more technical solutions proposed in the embodiments of the present application have at least the following technical effects: the present application can obtain the task environment status of the task to be scheduled and the current data resource environment, and in the present application, the preset prompt generator can determine the target scheduling task from the task environment status, so that the target scheduling task that is currently suitable for scheduling allocation can be determined in the task environment status, thereby facilitating the improvement of the efficiency of scheduling allocation. Furthermore, the preset prompt generator can also generate prompt words corresponding to the target scheduling task and the data resource environment, so that the prompt words can comprehensively represent the current state of the data resource environment and the state of the target scheduling task, and the preset large model itself has powerful semantic understanding and reasoning capabilities, so that the preset large model can capture the complex relationship between the target scheduling task and the data resource environment through the prompt words, thereby facilitating the preset large model to generate a globally optimal scheduling plan, rather than performing task scheduling based on established rules, avoiding the scheduling plan from falling into local optimality, and making task scheduling more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the embodiments of the present application, and together with the description are used to explain the principles of the embodiments of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 This is a flowchart of an embodiment of a task scheduling method according to an embodiment of the present application;

[0047] Figure 2 A flowchart of another embodiment of the task scheduling method of the present application embodiment;

[0048] Figure 3 This is a flowchart of an example of a task scheduling method according to an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the module structure of the task scheduling device according to an embodiment of the present application;

[0050] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the task scheduling method in the embodiment of the present application.

[0051] The purpose, features and advantages of the embodiments of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the embodiments of the present application and are not used to limit the embodiments of the present application.

[0053] In order to better understand the technical solutions of the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] In the field of big data, task scheduling is a crucial technology that directly affects the overall performance and resource utilization of the system. However, traditional scheduling methods often show low efficiency when facing complex and changing environments, and are difficult to adapt to changing task requirements and data resource environments.

[0055] Currently, commonly used task scheduling technologies mainly include scheduling technologies based on classical optimization methods and scheduling technologies based on reinforcement learning methods. Among them, scheduling technologies based on classical optimization methods often use rule-based heuristic methods and meta-heuristic methods based on iterative search for scheduling. Although the method is simple and the calculation speed is fast, it is based on predefined rules and search iterative methods for scheduling, so the flexibility and adaptability are not high, and it is difficult to effectively respond to complex and dynamic environmental changes, and it is easy to fall into local optimality.

[0056] The scheduling technology based on reinforcement learning methods uses machine learning and deep learning to learn historical scheduling data and perform prediction and optimization, which can improve the global optimal performance to a certain extent. However, when faced with complex data resource environments and dynamic and changing task requirements, there are situations such as heavy computing load and difficulty in convergence, which not only reduces the scheduling efficiency of tasks, but also easily reduces the global optimal performance of task scheduling.

[0057] Therefore, from the perspective of task scheduling quality, scheduling technologies based on classical optimization methods (such as heuristic and meta-heuristic algorithms) lack flexibility and adaptability, and are difficult to cope with complex and dynamic environments, resulting in scheduling solutions often being limited to local optimal solutions. Although reinforcement learning-based methods can schedule tasks by learning historical scheduling data, it is still difficult to ensure the global optimal solution quality of the scheduling solution when faced with complex data resource environments, dynamic and changing task requirements, and complex reward functions corresponding to reinforcement learning.

[0058] From the perspective of task scheduling efficiency, the computational efficiency of metaheuristic methods in classical optimization methods will decrease significantly as the scale of the problem increases, because metaheuristic methods require multiple rounds of iterative searches, which is particularly evident in complex scenarios such as cross-domain and dynamic changes in data resource environments. Scheduling methods based on reinforcement learning often require a large amount of training data and computing resources in high-dimensional, multi-constrained task scheduling problems, and have slow convergence speeds, making it difficult to quickly respond to dynamically changing data resource environments, so there is currently a problem of low scheduling efficiency.

[0059] Finally, from the perspective of the versatility of task scheduling, scheduling technology based on classical optimization methods is difficult to effectively capture environmental changes, resulting in scheduling delays and reduced system performance. Methods based on reinforcement learning lack sufficient flexibility and adaptability when faced with changing task requirements and data resource environments, and are difficult to adapt to a variety of task scheduling scenarios.

[0060] To this end, the embodiment of the present application proposes a task scheduling method, which generates prompt words for target scheduling tasks and real-time data resource environments through a preset prompt generator, and inputs the prompt words into a preset large model to generate a scheduling plan, thereby realizing efficient task scheduling in a complex environment. Specifically, the preset prompt generator can generate prompt words based on the real-time data resource environment and the target scheduling task, so that the prompt words can comprehensively represent the state of the current data resource environment and the state of the target scheduling task, and the preset large model itself has strong semantic understanding and reasoning capabilities, so that the preset large model can capture the complex relationship between the target scheduling task and the data resource environment through the prompt words, thereby facilitating the preset large model to generate the optimal scheduling plan, rather than performing task scheduling based on established rules, avoiding the scheduling plan from falling into a local optimum, and making task scheduling more efficient.

[0061] Furthermore, since the embodiments of the present application can generate prompt words based on the real-time data resource environment and the corresponding target scheduling tasks, it is possible to quickly adapt to changes in the data resource environment and changes in the target scheduling plan, and then guide the preset large model to generate a scheduling plan through the prompt words, rather than directly generating a scheduling plan based on the data resource environment and the target scheduling task. This embodiment first generates prompt words and then guides the preset large model to generate a scheduling plan through the prompt words, thereby avoiding the problem of dimensionality explosion faced in traditional reinforcement learning (for example, the more complex the data resource environment and / or the more complex the tasks to be scheduled, the more complex the dimensions corresponding to the data resource environment and the tasks to be scheduled that need to be described by traditional reinforcement learning).

[0062] In this embodiment, the preset large model is guided by prompt words to generate a scheduling plan, which is essentially a fine-tuning of the preset large model, and the large model does not need to be retrained, so that the preset large model can be guided by prompt words to generate a scheduling plan with a small amount of training time, thereby reducing the computing cost and improving the efficiency of task scheduling. For example, the APTU method (Adaptive Prompt Tuning and Utilization) can be used for fine-tuning.

[0063] In addition, since the preset prompt generator can generate highly adaptable prompt words based on different task scenarios (e.g., different tasks to be deployed) and resource states (e.g., different data resource environments), the scheduling scheme generated subsequently can flexibly respond to various complex task requirements and resource changes. This facilitates the realization of efficient and intelligent task scheduling in different application scenarios, effectively making up for the lack of versatility in task scheduling.

[0064] Based on this, the present application embodiment provides a task scheduling method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the task scheduling method of the present application. The task scheduling method includes steps S10 to S30:

[0065] Step S10, obtaining a task environment state constructed by the task to be scheduled and the current data resource environment;

[0066] It should be noted that the task to be scheduled is a task that needs to be scheduled. The task to be scheduled may include multiple subtasks. Each subtask in the task to be scheduled is a subtask that needs to be scheduled. The data resource environment is used to characterize the resource utilization of each data center. The task environment state is used to comprehensively characterize the task to be scheduled and the current data resource environment.

[0067] This embodiment can be applied to a task scheduling system, which includes a task queue manager and a status monitor. The task queue manager can be used to receive tasks that need to be scheduled. The tasks to be scheduled can be stored in the task queue manager. The storage form of the tasks to be scheduled in the task queue manager can be a task queue, which is a collection of tasks that need to be scheduled.

[0068] The state monitor can be used to monitor the data resource environment corresponding to the task scheduling system. Exemplarily, the tasks to be scheduled can be obtained from the task queue manager, the current data resource environment can be obtained from the state monitor, and the task environment state can be constructed based on the tasks to be scheduled and the data resource environment.

[0069] In a feasible embodiment, step S10 further includes steps S11 to S13:

[0070] Step S11, based on the acquired task to be scheduled and the dependency relationship of each subtask in the task to be scheduled, generating a task state vector of the task to be scheduled, wherein the task state vector includes a dependency vector corresponding to each subtask in the task to be scheduled;

[0071] It should be noted that the tasks to be scheduled can be obtained from the task queue manager. Each subtask in the tasks to be scheduled has its own corresponding dependency relationship. The subtask is the task that needs to be scheduled, and the dependency relationship is used to describe the execution order between two subtasks. The task state vector includes each subtask and the corresponding dependency vector, and the dependency vector is used to describe the dependency relationship between subtasks. The task state vector can also be determined based on the directed acyclic graph DAG (Directed Acyclic Graph) corresponding to the tasks to be scheduled. There are multiple nodes and multiple edges in the directed acyclic graph. The nodes in the directed acyclic graph represent subtasks, and the edges in the directed acyclic graph represent the dependency relationship between two subtasks.

[0072] For example, each task in the to-be-scheduled task can be represented as T={t 1 ,t 2 ,…,t n}, where T represents the task to be scheduled, t 1 is the first subtask, t 2 For the second subtask, t n It is represented as the nth subtask, where n is the total number of subtasks in the task to be scheduled. The dependency vector can be expressed as E = {(t i ,t j )|t i →t j}, E is the dependency vector, representing subtask t i Must be in subtask t j Before completion, ti is the i-th subtask, t j For the jth subtask, the task state vector can be expressed as G = (T, E). This embodiment describes the subtasks and corresponding dependencies in the task to be scheduled through G = (T, E), thereby achieving an accurate description of the subtasks and dependencies.

[0073] Step S12, generating a resource utilization state vector of the data resource environment based on the resource utilization rate of each data center in the current data resource environment;

[0074] Step S13, concatenating the task state vector and the resource utilization state vector to obtain the task environment state.

[0075] It should be noted that the data resource environment includes multiple data centers, each of which has its own corresponding resource utilization rate, which represents the proportion of the resources used in the data center to the total resources of the data center. The resource utilization state vector is used to describe the resource utilization of each data center in the data resource environment.

[0076] For example, the resource utilization state vector can be expressed as R = {r 1 ,r 2 ,…,r m}, where R is the resource utilization state vector, r i represents the resource utilization of the ith data center, and m represents m data centers. Resource utilization can also include CPU utilization, memory utilization, and network bandwidth utilization. For example, suppose the resource utilization of the ith data center can be represented as r i ={cpu i ,mem i ,bw i}, where cpu i Indicates the CPU utilization of the i-th data center, mem i represents the memory utilization of the ith data center, bw i Represents the network bandwidth utilization of the i-th data center.

[0077] After determining the task state vector and the resource utilization state vector, the task state vector and the resource utilization state vector can be concatenated to obtain the task environment state. Exemplarily, the task environment state can be expressed as S = (G, R), where S is the task environment state, G is the task state vector, and R is the resource utilization state vector.

[0078] This embodiment can accurately describe subtasks, dependencies between subtasks, and resource utilization of a data center through the task environment state, thereby facilitating subsequent accurate use of the data resource environment for scheduling and allocating target scheduling tasks.

[0079] Step S20, calling a preset prompt generator, determining a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment;

[0080] It should be noted that the preset prompt generator is used to generate prompt words (Prompt), which can be represented as multi-dimensional vectors, and the prompt words can comprehensively reflect the status of the target scheduling task and the data resource environment. The preset prompt generator can be obtained by pre-training APG-EDQN (adaptive Prompt Generation with Enhanced Double Deep Q-Networks).

[0081] The target scheduling task is a set of subtasks in the task to be scheduled that need to be included in the prompt word. The target scheduling task may include multiple target subtasks. Each target subtask in the target scheduling task is a subtask that needs to be included in the prompt word. The data resource environment is used to characterize the resource utilization of each data center. The target scheduling task can be a task to be scheduled or a part of the subtasks in the task to be scheduled. This embodiment does not specifically limit this. For example, when the number of tasks in the task to be scheduled is greater than the preset task number threshold, the target scheduling task can be a part of the subtasks in the task to be scheduled.

[0082] Among them, the preset task number threshold can be set based on the actual situation. When the number of tasks in the task to be scheduled is greater than the preset task number threshold, it means that the number of subtasks in the task to be scheduled is too large. If the task to be scheduled is used as the target scheduling task, it is easy to increase the pressure of task scheduling, resulting in a decrease in the efficiency of task scheduling. Therefore, in the present embodiment, when there are too many tasks in the task to be scheduled, the target scheduling tasks whose target subtask number is less than the preset task number threshold can be determined from the tasks to be scheduled in sequence, so as to schedule and allocate the target scheduling tasks in sequence, thereby improving the efficiency of task scheduling. In other embodiments, when the number of subtasks in the task to be scheduled is greater than the preset task number threshold, the task to be scheduled can also be directly split into multiple task subsets, the task subset includes multiple subtasks, and the task subset is used as the target scheduling task, etc. This embodiment does not make specific limitations on this.

[0083] Exemplarily, a preset prompt generator may be called to determine a target scheduling task from the task environment state, and generate prompt words corresponding to both the target scheduling task and the data resource environment.

[0084] In a feasible embodiment, step S20 further includes steps S21 to S22:

[0085] Step S21, calling a preset prompt generator, determining a target scheduling task, a target dependency vector of each target subtask in the target scheduling task, and a remaining available resource vector in the resource environment from the task environment state, and generating a task description vector for each target subtask in the target scheduling task;

[0086] It should be noted that the task environment state is used to characterize all tasks in the task queue manager and the data resource environment corresponding to the task scheduling system. All subtasks in the task queue manager can be represented by vectors, and the data resource environment can also be represented by vectors. For example, the task environment state can be obtained by concatenating the vector corresponding to the task to be scheduled and the vector corresponding to the data resource environment.

[0087] In a feasible embodiment, step S21 further includes steps S211 to S213:

[0088] Step S22, calling a preset prompt generator, determining the target scheduling task, the target dependency vector of each target subtask in the target scheduling task, and the remaining available resource vector in the data resource environment from the task environment state, and generating a task description vector for each target subtask in the target scheduling task;

[0089] Step S23, concatenating the available resource vector, each target dependency vector and each task description vector to obtain a prompt word.

[0090] It should be noted that the preset prompt generator can generate prompt words based on the task environment state. The prompt words are a high-dimensional representation vector that can be represented in the form of a matrix. The target dependency vector is the dependency vector of the target subtask. The available resource vector is used to describe the remaining available resources in the data resource environment. The available resource vector includes the remaining resources of each data center in the data resource environment. The preset prompt generator can be trained in advance. The task description vector is used to characterize the resources required for the target subtask and the execution parameters of the target subtask, such as the execution priority and execution duration of the target subtask.

[0091] Exemplarily, after determining the task environment state, the preset prompt generator can select a target scheduling task in the task environment state. Specifically, the target scheduling task can be obtained from the task state vector in the task environment state, and the target dependency vector of each target subtask in the target scheduling task can be obtained from the task state vector. The available resource vector can also be obtained from the resource utilization state vector in the task environment state. The preset prompt generator can also generate a task description vector for each target subtask in the target scheduling task.

[0092] Among them, after the target scheduling task is determined in the task environment state, the target scheduling task can be removed from the task environment state, and the target scheduling task can be marked as a scheduled task. The remaining tasks in the task environment state are unscheduled tasks. Furthermore, each target dependency vector corresponding to the target scheduling task can be removed from the task environment state to ensure the accuracy of the task environment state.

[0093] For example, the preset prompt generator can select multiple subtasks as target subtasks in sequence. When the preset prompt generator selects subtask t i When the prompt word is included, the target scheduling task can be expressed as T x_selected =T selected +{t i}, T x_selected For the selected subtask t i The target scheduling task after selected To select t i The task description vector corresponding to the target scheduling task can be expressed as: D x_task =D task +description(t i ), where D x_task is the set of task description vectors corresponding to the target scheduling task, description(t i ) is subtask t i The task description vector, D task The task scheduled for the target does not contain subtask t i The set of task description vectors at the time.

[0094] In this embodiment, the prompt word can be expressed as P = [P task ,P resource ,P constraint ], P task is the representation of the task description vector corresponding to each target subtask in the target scheduling task in the prompt word, P resource is the representation of the available resource vector in the data resource environment in the prompt word, P constraint It is the representation of the target dependency vector corresponding to each target subtask in the target scheduling task in the prompt word.

[0095] P task Contains the resource requirements of each target subtask. Assume that each task t i The task description vector is d t (t i ), then P task =[d t (t 1 ),d t (t 2 ),…,dt (t n )]. Among them, d t (t i ) can be further refined into d t =[Feature 1 ,Feature 2 ,…,Feature k ], Feature 1 For the first description object, Feature 2 For the second description object, Feature k is the kth description object, k is a positive integer. For example, the description objects included in the task description vector of each task can be the type of target subtask, execution priority, task execution time, CPU requirement, memory requirement, network bandwidth requirement, etc.

[0096] P resource It can be used to describe the total resource situation of the data resource environment and the resource situation of each data center in the data resource environment. j The available resource vector is d r (r j ), then P resource =[d r (r 1 ),d r (r 2 ),…,d r (r m )], where m is the number of data centers, d r (r j ) can also include multi-dimensional information such as CPU, memory, bandwidth, etc., for example, d r (r j )=[CPU,Memory,Bandwidth,…], where CPU is the resource j CPU utilization, Memory as resource j The memory utilization of the resource is j bandwidth utilization.

[0097] P constraint Contains all target dependency vectors corresponding to the target scheduling task. Assume that the target dependency vector e ij Denoted as the target subtask t i Must be in the target subtask t j If completed before, then P constraint =[e 12 ,e 23 ,…,e ij ]. Where i and j are both positive integers, e ijIt can also be expressed as e ij =[t i ,t j ].

[0098] Therefore, the prompt word can be characterized as:

[0099]

[0100] Among them, P is the prompt word, Task 1 ~Task n It is represented by the task description vector of the first target subtask to the task description vector of the nth target subtask, where n is a positive integer. Each task description vector includes Feature 1 ~Feature k , Feature 1 ~Feature k Represents the first description object to the kth description object, where k is a positive integer and is the number of description objects corresponding to each target subtask. all Resource is the available resource vector corresponding to the total resources of the data resource environment. host-1 ~Resource host-l is the available resource vector from the host-1 data center to the host-1 data center. The available resource vector of each data center can include resources such as CPU, Memory, and Bandwidth. l is a positive integer, and host is the host. Constraint is the set of target dependency vectors, [t 1 ,t 2 ] is the target subtask t 1 The target dependency vector of , or the target subtask t 2 The target dependency vector, [t 2 ,t 3 ] is the target subtask t 3 The target dependency vector of , or the target subtask t 2 The target dependency vector, [t i ,t j ] is the target subtask t i The target dependency vector of , or the target subtask t j For the same target subtask, it can include multiple target dependency vectors or correspond to one target dependency vector.

[0101] This embodiment uses high-dimensional vectors to represent prompt words, thereby facilitating a more comprehensive representation of the target scheduling task and the current data resource environment, thereby facilitating the subsequent scheduling and allocation of the target scheduling task by a preset large model, thereby improving the efficiency of task scheduling.

[0102] Step S30, inputting the prompt word into the preset large model to obtain a scheduling plan for the target scheduling task, so as to schedule the target scheduling task based on the scheduling plan.

[0103] It should be noted that the preset large model is used to generate a scheduling plan based on the prompt word. The scheduling plan is used to describe the scheduling allocation of each target subtask in the target scheduling task. The scheduling plan includes a sub-scheduling plan for each target subtask. The sub-scheduling plan includes the execution time and node resources of the target subtask. The execution time refers to the time when the target subtask starts to execute. The node resources are represented by the data center allocated to the target subtask and the resources that the target subtask can use in the data center.

[0104] For example, the prompt word is input to the preset large model, and the scheduling plan is output. The scheduling plan can be expressed as in, Represents task t i The sub-scheduling scheme, It can also be expressed as Among them, start i is the execution time of the i-th target subtask, center i is the node resource of the i-th target subtask.

[0105] In a feasible embodiment, step S30 further includes step S31:

[0106] Step S31, inputting the prompt word into a preset large model, encoding the prompt word, obtaining a text vector of the prompt word, and scheduling and allocating the target scheduling task based on the text vector to obtain the scheduling plan;

[0107] The scheduling scheme includes the execution order and node resources of each target subtask in the target scheduling task.

[0108] It should be noted that the text vector is a vector that can be understood by the preset large model. The preset large model includes an encoder, which is used to encode the prompt word into a text vector. For example, each vector in the prompt word can be encoded separately to obtain a sub-text vector corresponding to each vector in the prompt word, and each sub-text vector is concatenated to obtain a text vector. For example, the text vector obtained by encoding the prompt word can be expressed as:

[0109] E p =Encoder(P)

[0110] Among them, E pis a text vector, P is a prompt word, and Encoder is an encoder. The preset large model can schedule and allocate the target scheduling task based on the text vector. For example, based on the built-in Transformer structure of the preset large model, the self-attention mechanism of the Transformer structure can be used to calculate the correlation between the sub-text vectors in the text vector, so as to generate a scheduling plan for the target scheduling task. In this embodiment, the preset large model is used to encode the prompt word to obtain the text vector of the prompt word, and is also used to schedule and allocate the target scheduling task based on the text vector to obtain a scheduling plan.

[0111] In this embodiment, by inputting prompt words into a preset large model, the preset large model can capture the complex relationship between the target scheduling task and the data resource environment through the prompt words, thereby facilitating the scheduling allocation for the target scheduling task, improving the global optimality of the scheduling scheme, and avoiding the scheduling scheme from falling into the local optimality. In this embodiment, the preset large model can be an LLM (Large Language Model) model. There is no need to retrain the large model. The LLM model can be fine-tuned to obtain the preset large model in this embodiment. Fine-tuning training refers to training without requiring a large amount of training data.

[0112] Further, see Figure 2 In a feasible embodiment, the task scheduling method further includes steps A10 to A20:

[0113] Step A10, determining a scheduling score of the scheduling solution;

[0114] It should be noted that the scheduling score can be used to evaluate the execution effect of the scheduling plan. The higher the scheduling score, the better the execution effect of the scheduling plan, and the lower the scheduling score, the worse the execution effect of the task scheduling plan. Exemplarily, the scheduling score can be determined based on the result after the scheduling plan is executed.

[0115] In a feasible embodiment, step A10 further includes steps A11 to A13:

[0116] Step A11, executing the scheduling plan and obtaining a scheduling result of executing the scheduling plan;

[0117] Step A12, determining the execution result parameters corresponding to each preset quantitative scoring indicator from the scheduling result;

[0118] Step A13, performing weighted summation on each execution result parameter to obtain a scheduling score of the scheduling effect.

[0119] It should be noted that the scheduling result is the result generated after the scheduling plan is executed. The scheduling result includes multiple result parameters, and the result parameters are used to describe the execution status of the scheduling plan. For example, each result parameter can be the actual execution end time of each target subtask in the scheduling plan, the resource utilization rate of each data center in the data resource environment, the total task completion time corresponding to the scheduling plan (the total task completion time is the actual completion time of the last target subtask in the scheduling plan), etc. This embodiment does not make specific limitations on this.

[0120] The preset quantitative scoring indicators can be set based on actual conditions. For example, the preset quantitative scoring indicators can be set based on the scheduling target of the preset target scheduling task, or the preset quantitative scoring indicators can be set based on the data resource environment, etc. This embodiment does not specifically limit this. The preset quantitative scoring indicators are preset indicators for evaluating the execution effect of the scheduling scheme. For example, each preset quantitative scoring indicator can be a task delay indicator, a resource utilization indicator, a total task completion time indicator, and a resource utilization balance indicator.

[0121] The execution result parameters can be determined based on the scheduling results and the preset quantitative scoring indicators. Different preset quantitative scoring indicators correspond to different execution result parameters. For example, the result parameters corresponding to the preset quantitative scoring indicators can be directly obtained from the scheduling results as the execution result parameters, or the result parameters corresponding to the preset quantitative scoring indicators can be obtained from the scheduling results, and the result parameters are calculated to obtain the execution result parameters.

[0122] For example, when the preset quantitative scoring indicator is the total task completion time indicator, the total task completion time in the scheduling scheme can be obtained from the scheduling result, and the total task completion time is used as the execution result parameter corresponding to the total task completion time indicator. For example, the total task completion time can be expressed as Makespan = max i finish i , where Makespan is the total task completion time, max i finish i The completion time of the last task in the schedule.

[0123] For example, when the preset quantitative scoring indicator is the task delay indicator, the actual completion time of each target subtask in the scheduling plan can be obtained from the scheduling result. For each target subtask, the difference between the actual completion time of the target subtask and the preset target completion time of the obtained target subtask can be calculated to obtain the task delay corresponding to each target subtask. The average value of the task delay of each target subtask can be used as the execution result parameter of the task time indicator.

[0124] Exemplarily, the formula for calculating the task delay of each target subtask may be:

[0125] Latency=max(0,finish i -due i )

[0126] Among them, Latency is the task delay of the i-th target subtask, finish i is the actual end time of the i-th target subtask, due i is the preset target completion time of the i-th target subtask; max(0, finish i -due i ) represents when finish i -due i When it is less than 0, the delay of the target subtask is 0. i -due i When it is greater than 0, the delay of the target subtask is finish i -due i .

[0127] For example, when the preset quantitative scoring indicator is a resource utilization indicator, the resource utilization of each data center after the scheduling scheme is executed can be obtained from the scheduling result, and the average resource utilization of each data center can be used as the execution result parameter corresponding to the resource utilization indicator. Exemplarily, the calculation formula for the average resource utilization can be:

[0128]

[0129] Among them, Resource Util is the average resource utilization, n is the number of data centers, r i is the resource utilization of the ith data center, r i Including the CPU utilization, memory utilization, and bandwidth utilization corresponding to the i-th data center.

[0130] For example, when the preset quantitative scoring indicator is a resource utilization balance indicator, the resource utilization of each data center after the scheduling scheme is executed can be obtained from the scheduling result, and the resource balance of the resources used in each data center can be calculated based on each resource utilization, and the resource balance can be used as the execution result parameter corresponding to the resource utilization balance indicator. Exemplarily, the formula for calculating resource balance can be:

[0131]

[0132] Among them, is the resource balance, r i is the resource utilization of the jth data center, is the average resource utilization of each data center in the data resource environment, and m is the number of data centers.

[0133] In addition, it should be noted that the weights corresponding to the execution result parameters can be set based on the actual situation, and this embodiment does not specifically limit this. The execution result parameters can be weighted and summed to obtain the scheduling score of the scheduling result. For example, when the preset quantitative scoring indicators include the task delay indicator, the resource utilization indicator, the total task completion time indicator, and the resource utilization balance indicator, the total task completion time, the average value of the task delay, the average resource utilization, and the resource balance can be weighted and summed to obtain the scheduling score.

[0134] This embodiment evaluates the scheduling result of the scheduling scheme to obtain a scheduling score, thereby facilitating subsequent optimization of the preset prompt generator based on the scheduling score, so as to improve the accuracy of the prompt words generated by the preset prompt generator.

[0135] Step A20, optimizing the preset prompt generator based on the scheduling score.

[0136] It should be noted that the scheduling score can be used as a reward value feedback value preset prompt generator to optimize the preset prompt generator. For example, the prompt generation strategy of the preset prompt generator can be optimized by the scheduling score, and the prompt generation strategy is used to guide the preset prompt generator to generate prompt words.

[0137] Exemplarily, the formula for optimizing the preset prompt generator may be:

[0138]

[0139] Among them, π new The optimized prompt generation strategy for the preset prompt generator, π old is the prompt generation strategy before optimization of the preset prompt generator, Reward is the scheduling score. α is the learning rate of the preset prompt generator, Generate the gradient corresponding to the policy for the prompt.

[0140] This embodiment optimizes the preset prompt generator by scheduling the scoring, thereby facilitating improving the accuracy of the prompt words generated by the preset prompt generator.

[0141] In a feasible embodiment, the task scheduling method further includes steps B10 to B60:

[0142] Step B10, obtaining a training sample of a preset model to be trained, and dividing the training sample into a plurality of sub-training samples;

[0143] Step B20, allocating each sub-training sample to each preset computing node, and configuring a preset to-be-trained model on each preset computing node;

[0144] It should be noted that, in this embodiment, both the preset large model and the preset prompt generator can be obtained by pre-training, and the corresponding training processes can be parallel training. Parallel training refers to training the same model on multiple preset computing nodes. The preset model to be trained can be a preset large model to be trained, or a preset prompter to be trained. The preset large model to be trained is the model to be trained corresponding to the preset large model, and the preset prompter to be trained is the model to be trained corresponding to the preset prompt generator. When the preset model to be trained is the model to be trained corresponding to the preset large model, the preset large model is obtained by training the preset model to be trained, and when the preset prompt generator is the model to be trained corresponding to the preset prompt generator, the preset model to be trained is obtained by training the preset model to be trained.

[0145] In this embodiment, the training process of the preset large model to be trained can be a fine-tuning process. For example, the APTU method can be used to perform parallel fine-tuning on the preset large model to be trained on multiple preset computing nodes. The fine-tuning process corresponding to the APTU method also trains the preset large model to be trained through sub-training samples.

[0146] The preset computing node can be a server, and the preset model to be trained can be trained on each preset computing node. When the preset model to be trained is a preset large model to be trained, the training sample includes: the task to be scheduled, the data resource environment where the task is located, and the corresponding scheduling scheme label and scheduling score label, and each sub-training sample also includes the task to be scheduled, the data resource environment where the task is located, and the corresponding scheduling scheme label and scheduling score label.

[0147] When the preset model to be trained is a preset prompter to be trained, the training sample includes: the task environment state and the prompt word label for training; each preset computing node has its own corresponding sub-training sample. The number of preset computing nodes can be set based on actual conditions, and this embodiment does not specifically limit this. In this embodiment, each sub-training sample can be subjected to data cleaning, standardization, etc., so as to ensure the accuracy of training.

[0148] Step B30, iteratively training the preset model to be trained in each preset computing node through the corresponding sub-training samples, and periodically synchronizing the preset model to be trained corresponding to each preset computing node;

[0149] It should be noted that each preset computing node can iteratively train the preset model to be trained. In order to ensure the consistency of the preset models to be trained on each preset computing node, the preset models to be trained corresponding to each preset computing node can be periodically synchronized to improve the efficiency of training the preset models to be trained.

[0150] Exemplarily, the average value of the model parameters of the preset model to be trained corresponding to each preset computing node is periodically calculated, and the model parameters of the preset model to be trained on each preset computing node are updated based on the average value. The model parameters refer to the internal variables that need to be learned and adjusted during the training process of the preset model to be trained, such as weights and biases, etc. This embodiment does not make any specific limitations on this.

[0151] For example, the formula for synchronizing the model parameters of the preset model to be trained on the preset computing node may be:

[0152]

[0153] Among them, θ i are the model parameters of the preset model to be trained on the i-th preset computing node, is the average value of the model parameters of each preset model to be trained, and N is the number of preset models to be trained. and i are both positive integers.

[0154] Step B40, based on the loss function of each preset computing node, calculating the training loss value of the preset to-be-trained model corresponding to each preset computing node;

[0155] Step B50, calculating the loss average of the training loss values ​​corresponding to each preset computing node, and updating the preset to-be-trained models on each preset computing node based on the loss average;

[0156] It should be noted that the loss function on each preset computing node can be the same or different. The training loss value of the preset training model corresponding to each preset computing node can be calculated, and the preset model to be trained on each preset computing node can be updated based on the average loss value. Specifically, the training parameters of the preset model to be trained can be updated based on the average loss value.

[0157] Exemplarily, the formula for updating the preset to-be-trained model on the preset computing node based on the loss average value may be:

[0158]

[0159] Among them, θ x is the model parameter of the preset model to be trained after updating based on the average loss value, θ is the model parameter of the preset model to be trained before updating based on the average loss value, α is the learning rate of the model to be trained, is the average loss, is the loss function of the i-th preset computing node. is the gradient corresponding to the model parameters, N is the number of preset computing nodes, and the number of preset computing nodes can be the same as the number of preset models to be trained.

[0160] Step B60, when the preset to-be-trained model corresponding to any preset computing node meets the preset training end condition, the preset to-be-trained model is used as the preset training completion model;

[0161] It should be noted that the preset training end condition indicates that the training of the preset model to be trained is completed, and the preset training end condition includes: the average loss value is less than or equal to the preset loss threshold, the number of iterations of the preset model to be trained is greater than the preset number threshold, or the training time of the preset model to be trained is greater than the preset time threshold. The preset loss threshold, the preset number threshold, and the preset time threshold can be set based on actual conditions, and this embodiment does not specifically limit this.

[0162] When the preset model to be trained is a preset large model to be trained, the preset training completed model is the preset large model, and when the preset model to be trained is a preset prompter to be trained, the preset training completed model is a preset prompt generator. The preset large model to be trained can be an LLM or other large language model, etc., which is not specifically limited in this embodiment, and the preset prompter to be trained can be APG-EDQN.

[0163] Exemplarily, when the preset model to be trained is the preset large model to be trained, the training samples of the preset large model to be trained are obtained, the training samples are divided into multiple sub-training samples, each sub-training sample is respectively assigned to each preset computing node, the preset large model to be trained is configured at each preset computing node, the preset large model to be trained is iteratively trained through the corresponding sub-training samples in each preset computing node, and the preset large models to be trained corresponding to each preset computing node are periodically synchronized; based on the respective loss functions of each preset computing node, the training loss value of the preset large model to be trained corresponding to each preset computing node is calculated; the loss average of the training loss values ​​corresponding to each preset computing node is calculated, and the respective preset large models to be trained on each preset computing node are updated based on the loss average; when the preset large model to be trained corresponding to any preset computing node meets the preset training end condition, the preset large model to be trained is used as the preset large model.

[0164] When the preset model to be trained is a preset prompt to be trained, obtain training samples of the preset prompt to be trained, divide the training samples into multiple sub-training samples, assign each sub-training sample to each preset computing node, configure the preset prompt to be trained at each preset computing node, iteratively train the preset prompt to be trained through the corresponding sub-training samples in each preset computing node, and periodically synchronize the preset prompt to be trained corresponding to each preset computing node; based on the loss function of each preset computing node, calculate the training loss value of the preset prompt to be trained corresponding to each preset computing node; calculate the loss average of the training loss values ​​corresponding to each preset computing node, and update the preset prompt to be trained on each preset computing node based on the loss average; when the preset prompt to be trained corresponding to any preset computing node meets the preset training end condition, use the preset prompt to be trained as the preset prompt generator.

[0165] This embodiment improves the efficiency of model training by performing parallel training on preset models to be trained.

[0166] In a feasible embodiment, the task scheduling method also includes step C10: removing the target scheduling task and the target dependency vector corresponding to the target scheduling task from the task environment state, and adjusting the resource utilization state vector in the task environment state based on the node resources of each target subtask in the target scheduling task in the scheduling scheme to update the task environment state.

[0167] It should be noted that the task environment state can reflect the set of tasks that need to be scheduled and the state of the data resource environment. In order to ensure the accuracy of the task environment state, after determining the target scheduling task and the target dependency vector corresponding to the target scheduling task, this embodiment can remove the target scheduling task and the corresponding target dependency vector from the task environment state to ensure the accuracy of the task environment state.

[0168] After the scheduling plan is determined, the resource utilization state vector corresponding to the data resource environment can be updated based on the corresponding node resources in the scheduling plan, thereby ensuring the accuracy of the task environment state, and facilitating the preset prompt generator and the preset large model to adapt to changes in the data resource environment and tasks, thereby improving the efficiency of task scheduling.

[0169] For example, if there is a target subtask t j If the corresponding part of the resources in the jth data is used, there exists r t+1,j =f(r t,j ,t j ), where r t+1,j is the updated resource utilization of the data center, f(r t,j ,t j ) is a function used toi After using the resources of the jth data center, the remaining resources of the jth data center are adjusted, thereby facilitating updating the resource utilization of the jth data center.

[0170] For a better understanding of this embodiment, please refer to Figure 3 , briefly describe the process of task scheduling in this embodiment: this embodiment can be applied to a task scheduling system, and the task scheduling system includes a task queue manager, a state monitor, a preset large model, a preset prompt generator, a scoring module, and an iterative optimization module. Among them, the task queue manager is used to receive and manage subtasks that need to be scheduled, for example, J1~J7 can represent the first subtask to the seventh subtask, etc., and the task queue manager can also include a DAG corresponding to each subtask, for example, J1~J4 are connected in sequence to form a ring, and J5~J7 are connected in sequence. The state monitor can monitor the data resource environment, and the preset prompt generator can generate corresponding prompt words based on the data resource environment and the tasks to be scheduled in the task queue manager, and then the prompt words can be input into the preset large model, and the preset large model can generate a scheduling plan based on the prompt words. The scheduling plan can be input into the scoring module and the state monitor, and the scoring module can score the scheduling plan to obtain a scheduling score, and the scheduling score can be fed back to the preset prompt generator, and the preset prompt generator can be optimized based on the scheduling score. The state monitor can update the data resource environment based on the scheduling plan.

[0171] This embodiment determines the scheduling scheme of the target scheduling task by presetting the prompt generator and the preset large model, thereby providing a more efficient and flexible scheduling scheme in a dynamically changing data resource environment, and improving the scheduling efficiency and system performance of the task scheduling system. Moreover, this application determines the corresponding preset quantitative scoring indicators based on the actual situation in order to calculate the scheduling score of the scheduling scheme, so as to optimize the preset prompt generator through the scheduling score, thereby optimizing the preset prompt generator, facilitating the improvement of the global optimization capability of the generated scheduling scheme, and improving the scheduling efficiency. Moreover, in the embodiment of the present application, by optimizing the task scheduling, the task waiting time and the scheduling decision time are reduced, the energy consumption and hardware loss of the task scheduling system are reduced, and the preset large model does not need to be retrained, reducing the waste of training data and computing resources, thereby facilitating the reduction of the operating costs of the data center.

[0172] The present application also provides a task scheduling device, please refer to Figure 5 , the device comprises:

[0173] An acquisition module 10 is used to acquire a task environment state constructed by the task to be scheduled and the current data resource environment;

[0174] The prompt word generating module 20 is used to call a preset prompt generator, determine the target scheduling task from the task environment state, and generate a prompt word corresponding to the target scheduling task and the data resource environment;

[0175] The scheduling scheme generating module 30 is used to input the prompt word into a preset large model to obtain a scheduling scheme for the target scheduling task, so as to schedule the target scheduling task based on the scheduling scheme.

[0176] The task scheduling device provided in the embodiment of the present application adopts the task scheduling method in the above embodiment, aiming to solve the technical problem that task scheduling is not efficient enough. Compared with the prior art, the beneficial effects of the task scheduling method provided in the embodiment of the present application are the same as the beneficial effects of the task scheduling method provided in the above embodiment, and other technical features in the task scheduling device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0177] An embodiment of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the task scheduling method in the above embodiment.

[0178] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0179] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus.

[0180] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0181] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0182] The electronic device provided in the embodiment of the present application adopts the task scheduling method in the above embodiment 1 to solve the technical problem that task scheduling is not efficient enough. Compared with the prior art, the beneficial effect of product flow data allocation provided in the embodiment of the present application is the same as the beneficial effect of the task scheduling method provided in the above embodiment, and the other technical features in the task scheduling device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0183] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0184] The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be based on the protection scope of the claims.

[0185] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the task scheduling method in the above-mentioned embodiment 1.

[0186] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, equipment or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable EPROM (Electrical Programmable Read Only Memory, read-only memory) or flash memory, an optical fiber, a portable compact disk CD-ROM (compact disc read-only memory, read-only memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution device, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

[0187] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0188] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains the task environment state constructed by the task to be scheduled and the current data resource environment; calls a preset prompt generator, determines the target scheduling task from the task environment state, and generates a prompt word corresponding to the target scheduling task and the data resource environment; inputs the prompt word into a preset large model, schedules and allocates the target scheduling task, and obtains a scheduling plan for the target scheduling task.

[0189] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0190] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0191] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0192] The computer-readable storage medium provided in the embodiment of the present application stores computer-readable program instructions for executing the above-mentioned task scheduling method, aiming to solve the technical problem that task scheduling is not efficient enough. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the task scheduling method provided in the above-mentioned embodiment, and will not be described in detail here.

[0193] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the task scheduling method as described above when the computer program is executed by a processor.

[0194] The computer program product provided in the embodiment of the present application is intended to solve the technical problem that task scheduling is not efficient enough. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of the present application are the same as the beneficial effects of the task scheduling method provided in the above embodiment, which will not be repeated here.

[0195] The above are only preferred embodiments of the embodiments of the present application, and are not intended to limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the embodiments of the present application.

Claims

1. A task scheduling method, characterized in that: The method includes: Get the task environment status constructed by the task to be scheduled and the current data resource environment; Calling a preset prompt generator to determine a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment; The prompt word is input into a preset large model to obtain a scheduling plan for the target scheduling task, so as to schedule the target scheduling task based on the scheduling plan.

2. The method according to claim 1, characterized in that The method further comprises: Determining a scheduling score for the scheduling solution; Based on the scheduling score, the preset prompt generator is optimized.

3. The method according to claim 2, characterized in that The step of determining the scheduling score of the scheduling solution comprises: Executing the scheduling plan to obtain a scheduling result of executing the scheduling plan; Determine the execution result parameters corresponding to each preset quantitative scoring indicator from the scheduling result; A weighted sum is performed on each of the execution result parameters to obtain a scheduling score for the scheduling result.

4. The method according to claim 1, characterized in that The step of obtaining the task environment state constructed by the task to be scheduled and the current data resource environment includes: Based on the acquired task to be scheduled and the dependency relationship of each subtask in the task to be scheduled, a task state vector of the task to be scheduled is generated, wherein the task state vector includes a dependency vector corresponding to each subtask in the task to be scheduled; Generate a resource utilization state vector of the data resource environment based on the resource utilization rate of each data center in the current data resource environment; The task state vector and the resource utilization state vector are concatenated to obtain the task environment state.

5. The method according to claim 1, characterized in that The step of calling a preset prompt generator, determining a target scheduling task from the task environment state, and generating a prompt word corresponding to the target scheduling task and the data resource environment comprises: Calling a preset prompt generator to determine a target scheduling task, a target dependency vector of each target subtask in the target scheduling task, and a remaining available resource vector in the data resource environment from the task environment state, and generating a task description vector for each target subtask in the target scheduling task; The available resource vector, each target dependency vector and each task description vector are concatenated to obtain a prompt word.

6. The method according to claim 1, characterized in that The step of inputting the prompt word into a preset large model to obtain a scheduling solution for the target scheduling task includes: Inputting the prompt word into a preset large model, encoding the prompt word to obtain a text vector of the prompt word, and scheduling and allocating the target scheduling task based on the text vector to obtain the scheduling plan; The scheduling scheme includes the execution order and node resources of each target subtask in the target scheduling task.

7. The method according to claim 1, characterized in that The method includes: Obtaining a training sample of a preset model to be trained, and dividing the training sample into a plurality of sub-training samples; Allocating each of the sub-training samples to each preset computing node, and configuring a preset model to be trained on each of the preset computing nodes; Iteratively training the preset model to be trained in each of the preset computing nodes through the corresponding sub-training samples, and periodically synchronizing the preset models to be trained corresponding to each of the preset computing nodes; Based on the loss function of each preset computing node, calculate the training loss value of the preset to-be-trained model corresponding to each preset computing node; Calculate the loss average of the training loss values ​​corresponding to each of the preset computing nodes, and update the preset to-be-trained models on each of the preset computing nodes based on the loss average; When the preset model to be trained corresponding to any of the preset computing nodes meets the preset training end condition, the preset model to be trained is used as the preset training completion model; Among them, when the preset model to be trained is a preset large model to be trained, the preset training completion model is the preset large model; when the preset model to be trained is a preset prompter to be trained, the preset training completion model is a preset prompt generator.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The target scheduling task and the target dependency vector corresponding to the target scheduling task are removed from the task environment state, and based on the node resources of each target subtask in the target scheduling task in the scheduling scheme, the resource utilization state vector in the task environment state is adjusted to update the task environment state.

9. A task scheduling device, characterized in that: The device comprises: An acquisition module is used to obtain the task environment state constructed by the task to be scheduled and the current data resource environment; A prompt word generation module, used to call a preset prompt generator, determine a target scheduling task from the task environment state, and generate a prompt word corresponding to the target scheduling task and the data resource environment; The scheduling scheme generating module is used to input the prompt word into a preset large model to obtain a scheduling scheme for the target scheduling task, so as to schedule the target scheduling task based on the scheduling scheme.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the task scheduling method described in any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which is stored a program for implementing the task scheduling method. The program for implementing the task scheduling method is executed by a processor to implement the steps of the task scheduling method as described in any one of claims 1 to 8.

12. A program product, characterized in that The program product is a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the task scheduling method according to any one of claims 1 to 8 are implemented.

Images