Task scheduling method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a task scheduling method, which is applied to a task processing platform comprising a plurality of nodes, and comprises the following steps: predicting an initial resource demand and an initial running time length of each operator task belonging to a plurality of task flows by using a first prediction model; obtaining an initial optimal resource allocation mode by adopting an improved genetic algorithm, and carrying out resource allocation according to the initial optimal resource allocation mode; after the plurality of operator tasks run, continuously using a second prediction model to predict the resource demand of the next time period based on the resource consumption of each operator task in the current time period, and adopting an improved genetic algorithm to obtain the optimal resource allocation mode of the next time period. And resource allocation is carried out on each operator task again according to the optimal resource allocation mode of the next time period in the next time period. Reallocation of resources is realized by predicting future resource requirements in real time, resource waste is reduced, and efficient parallel scheduling of operator tasks in a plurality of task flows with complex topological structures is realized by improving a genetic algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method, device, electronic device and computer-readable storage medium. Background Art

[0002] Task scheduling is to determine the optimal node allocation and execution order for the multiple steps that make up a task based on the node resource situation.

[0003] In related technologies, task scheduling adopts a static scheduling method, that is, node resources are allocated to operator tasks before the task is run, and the resources are not changed after the task is run. This can easily lead to inflexible task allocation and waste of resources. Although some related technologies adopt a dynamic scheduling method, they do not perform dynamic scheduling based on the real-time and accurate resource requirements of operator tasks, which can easily lead to inaccurate resource allocation and waste of resources.

[0004] In addition, most of the current task scheduling is aimed at the scheduling of multiple operator tasks in a single task flow, and it is difficult to achieve parallel scheduling of multiple task flows. There have also been explorations on the parallel scheduling of multiple task flows in related technologies, but they are only applicable to multiple task flows with operator tasks in a serial structure. There is no more effective means for the parallel scheduling of multiple task flows with complex topological structures. Summary of the invention

[0005] In a first aspect, an embodiment of the present application provides a task scheduling method, which is applied to a task processing platform, wherein the task processing platform includes a plurality of nodes, and the method includes:

[0006] S1: Based on recent operation records of multiple operator tasks, use a first prediction model to respectively predict initial resource requirements and initial operation durations of each of the operator tasks; wherein the multiple operator tasks belong to multiple task flows;

[0007] S2: Based on the initial resource requirements and the initial running time of each of the operator tasks, an improved genetic algorithm is used to obtain an initial optimal resource allocation method;

[0008] S3: Allocate resources to each of the operator tasks according to the initial optimal resource allocation method;

[0009] S4: After the plurality of operator tasks are run, the second prediction model is continuously used to predict the resource demand of each operator task in the next period based on the resource consumption of each operator task in the current period, and based on the resource demand of all operator tasks in the next period, an improved genetic algorithm is used to obtain the optimal resource allocation method for the next period, and resources are reallocated to each operator task in the next period according to the optimal resource allocation method for the next period.

[0010] In a second aspect, an embodiment of the present application provides a task scheduling device, which is applied to a task processing platform, wherein the task processing platform includes a plurality of nodes, and is characterized in that the device includes:

[0011] An initial prediction module, used to predict the initial resource requirements and initial running time of each of the operator tasks based on the recent running records of the multiple operator tasks using the first prediction model; wherein the multiple operator tasks belong to multiple task flows;

[0012] A scheduling algorithm module, used to obtain an initial optimal resource allocation method using an improved genetic algorithm based on the initial resource requirements and the initial running time of each of the operator tasks;

[0013] A scheduling execution module, used for allocating resources to each of the operator tasks according to the initial optimal resource allocation method;

[0014] The real-time prediction and scheduling module is used to continuously use the second prediction model to predict the resource requirements of each operator task in the next period based on the resource consumption of each operator task in the current period after the multiple operator tasks are run, and based on the resource requirements of all operator tasks in the next period, use an improved genetic algorithm to obtain the optimal resource allocation method for the next period, and reallocate resources to each operator task in the next period according to the optimal resource allocation method for the next period.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it can execute the method of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method of the first aspect can be executed.

[0017] The beneficial effects of the present invention include at least:

[0018] 1. The present invention uses the first prediction model to accurately predict the initial resource requirements and initial running time of each operator task, and can generate an initial optimal resource allocation method based on the accurate resource requirements, thereby ensuring the accuracy of resource allocation to each operator task before the task runs;

[0019] 2. After the task is run, the present invention uses the second prediction model to predict the resource demand of the next period in real time based on the resource consumption of the operator task in the current period, and then obtains the corresponding optimal resource allocation method based on the resource demand of the next period, and uses the optimal resource allocation method to reallocate resources, continuously and dynamically cycle, thereby realizing accurate resource allocation and reducing resource waste;

[0020] 3. In obtaining the initial optimal resource allocation method and the optimal resource allocation method for each next time period, the present invention realizes parallel scheduling of operator tasks in multiple task flows with complex topological structures by using an improved genetic algorithm. It can quickly find the optimal resource allocation method under high complexity, greatly reducing the computing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of the task scheduling method of the present invention;

[0023] Figure 2 It is a schematic diagram of the first prediction model framework of the present invention;

[0024] Figure 3 A schematic diagram of the topological structure of multiple task flows of the present invention;

[0025] Figure 4 A schematic diagram of a crossover operation process of the present invention;

[0026] Figure 5 It is a schematic diagram of using the second prediction model of the present invention to predict resource demand in the next time period based on resource consumption in the current time period;

[0027] Figure 6 It is a structural schematic diagram of the task scheduling device of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0029] Figure 1A flowchart of a task scheduling method provided by the present invention is shown. The method is applied to a task processing platform. The task processing platform includes executing multiple nodes, such as Figure 1 As shown, the method includes:

[0030] S1: Based on recent operation records of the plurality of operator tasks, using a first prediction model, respectively predicting initial resource requirements and initial operation durations of the plurality of operator tasks; wherein the plurality of operator tasks belong to a plurality of task flows;

[0031] S2: Based on the initial resource requirements and the initial running time of each of the operator tasks, an improved genetic algorithm is used to obtain an initial optimal resource allocation method;

[0032] S3: Allocate resources to each of the operator tasks according to the initial optimal resource allocation method;

[0033] S4: After the plurality of operator tasks are run, the second prediction model is continuously used to predict the resource demand of each operator task in the next period based on the resource consumption of each operator task in the current period, and based on the resource demand of all operator tasks in the next period, an improved genetic algorithm is used to obtain the optimal resource allocation method for the next period, and resources are reallocated to each operator task in the next period according to the optimal resource allocation method for the next period.

[0034] Based on an optional implementation, the task flow in the present invention may be various services for processing remote sensing images, including but not limited to image change detection, image quality inspection, image segmentation and model training services. Operator tasks may be specific job operations included in the task flow, such as image feature extraction, image feature enhancement, knowledge image search, etc. included in image segmentation services. The task flow of the present invention may be a task flow with a complex topological structure, including but not limited to a tree topological structure, that is, at least some operator tasks have the same preceding or succeeding operator tasks. The task processing platform in the present invention may be a system platform for the creation, release and other processing of operator tasks. The node in the present invention is a machine node that can run operator tasks, such as a server or a processor.

[0035] According to an embodiment of the present invention, S1: based on recent operation records of the plurality of operator tasks, using a first prediction model to respectively predict the initial resource requirements and initial operation duration of each of the operator tasks; wherein the plurality of operator tasks belong to a plurality of task flows, including:

[0036] Obtain the recent running records, input file size and operator ID of the operator task;

[0037] The multiple resource consumptions in the recent operation records constitute a recent resource consumption sequence, and the multiple operation durations constitute a recent operation duration sequence;

[0038] The recent resource consumption sequence, the recent running time sequence, the input file size, and the operator ID are input into the first prediction model to obtain the initial resource requirement and the initial running time of the operator task.

[0039] Specifically, recent operation records are operation records of operator tasks in the recent period of time. Recent operation records can be obtained from historical operation records, which record the resource consumption and operation time of each operator task in the past. Resource consumption refers to the hardware resources required for the execution of operator tasks, including CPU usage, GPU usage, and memory usage. Operator ID is used to uniquely identify operator tasks, and input file size refers to the file size required to be input when the operator task is executed.

[0040] As an optional implementation, the first prediction model is obtained by pre-training iTransformer using the first data set. Figure 2 The first prediction model framework schematic diagram shown in the figure shows that the first prediction model uses an inverted Transformer, including an embedding layer, L stacked Transformer blocks, and a projection layer. The Transformer module includes multivariate attention, layer normalization, and feed-forward neural network. The first data set is obtained by collecting historical data of operator task execution, including resource consumption, execution time, and input file size of historical execution.

[0041] As an optional implementation, the recent multiple running times of each operator task are combined into a recent running time sequence , recent resource consumption constitutes a recent resource consumption sequence ; Input the recent running time sequence, recent resource consumption sequence, operator ID and the input file size Md of each operator task into the first prediction model, and the first prediction model performs feature extraction and prediction to obtain the running time prediction sequence corresponding to each operator task , Resource Demand Forecast Sequence Among them, the recent resource consumption sequence Includes CPU usage sequence , GPU occupancy sequence And the memory usage sequence A multidimensional sequence of, where g is an integer, indicating taking the most recent g historical data, for example, 5. Correspondingly, the resource demand prediction sequence includes the CPU occupancy prediction sequence , GPU occupancy prediction sequence And memory usage prediction sequence multidimensional sequence.

[0042] Next, for each operator task, the corresponding initial runtime and initial resource requirement are obtained based on the runtime prediction sequence and resource requirement prediction sequence. Optionally, the maximum value of the runtime prediction sequence is taken as the initial runtime. ; Take the maximum values ​​of the CPU occupancy prediction sequence, GPU occupancy prediction sequence, and memory occupancy prediction sequence as the corresponding initial CPU demand , Initial GPU requirements and initial memory requirements , thus obtaining the initial resource requirements .

[0043] The present invention uses the first prediction model to predict the initial resource requirements and initial running time according to the previous running information of the operator task, so as to accurately predict the initial running time and initial resource requirements of each operator task before the task runs, which is beneficial to the subsequent accurate scheduling of each task and the effective use of node resources.

[0044] Next, step S2 is described in detail in conjunction with a specific embodiment.

[0045] According to an embodiment of the present invention, S2: based on the initial resource requirements and the initial running time of each of the operator tasks, an improved genetic algorithm is used to obtain an initial optimal resource allocation method, including:

[0046] S21: Perform chromosome encoding, where one chromosome includes a pair of operator sequence and node sequence, and one chromosome maps to one resource allocation mode.

[0047] Specifically, a chromosome includes two sequences, one is an operator sequence (Operator Series, OS) representing the order in which each operator task is executed, and the other is a node sequence (MachineSeries, MS) representing the node that executes each operator task. A chromosome maps a resource allocation method, that is, a resource allocation method can be uniquely determined using the operator sequence OS and the node sequence MS. Assume there are n task flows , each task flow contains multiple operator tasks, and there are m nodes distributed on the system , each node has corresponding hardware resources.

[0048] Encoding the operator sequence OS includes:

[0049] First, determine the order between task flows. Specifically, randomly assign multiple task flows The order between them is used to obtain the task flow sequence JS. The number i represents the i-th task flow, with task flow The numbers 1, 2, ..., n are arranged to represent the order between the task flows, and the task flow sequence JS is obtained.

[0050] Attached Figure 3 A schematic diagram of the topology of multiple task flows, for example, 3 task flows Each task flow includes 5 operator tasks. The order between task flows is randomly assigned to obtain the task flow sequence .

[0051] Next, determine the order of all operator tasks in each task flow. Based on the topological structure, the task flow The operator tasks in the random distribution order are obtained to obtain the sub-operator sequence ,in, In the sub-operator sequence In the task flow The number i and the operator task in the task flow The number sequence in j represents the operator ,in, , that is, j does not exceed the task flow For example, randomly generate task flows while ensuring topological order The sub-operator sequence of , and its corresponding operator order is ; Similarly, the sub-operator sequence of the randomly generated task flow J2 is ; The sub-operator sequence of the randomly generated task flow J3 is .

[0052] Then, determine the operator sequence that represents the order of all operator tasks. According to the task flow sequence JS and the sub-operator sequence of each task flow , , The operator sequence OS representing the total order of all operator tasks in all task flows can be determined. Specifically, the order between task flows is determined according to the task flow sequence JS, and then the order of operator tasks in each task flow is determined according to the sub-operator sequence to obtain the operator sequence OS. For example, according to the task flow sequence JS and the sub-operator sequence in the above example , , , get the operator sequence consisting of multiple operator tasks of the multi-task flow , the operator tasks corresponding to the operator sequence OS are sorted as follows: .

[0053] The operator sequence OS determines the execution order between all operators. The maximum duration of multiple task flows can be calculated based on the operator sequence OS.

[0054] The node sequence MS encoding process includes:

[0055] Random task flow Each operator task in is assigned to a node, represented by a node number, to obtain a node sequence MS. Assume there are 3 nodes , using node numbers 1, 2, and 3 as the values ​​of the node sequence, indicating the nodes that will execute the corresponding operator tasks, for example, for the task flow Each operator task

[0056] Randomly assign nodes to get node sequence .

[0057] The total length of the operator sequence OS is the same as the total length of the node sequence MS, which is the sum N of all operators in all task flows.

[0058] S22: Generate an initial population, randomly generate multiple chromosomes, including multiple non-identical operator sequences and multiple non-identical node sequences, and initialize the number of iterations to 1.

[0059] Under the premise of following the dependency of operator tasks, S1 chromosomes are randomly generated according to the above encoding rules, including S1 non-completely identical operator sequences and S1 non-completely identical node sequences. For example, S1 is 400, and the number of initialization iterations Gen is 1.

[0060] S23: Use the fitness function to calculate the fitness of each chromosome of the current population.

[0061] According to one embodiment of the present invention, using the fitness function to calculate the fitness of each chromosome of the current population specifically includes:

[0062] Step a: According to the order of each operator task in the operator sequence, the predicted earliest start time, the credible earliest start time and the completion time of each operator task are calculated one by one.

[0063] Calculate the predicted earliest start time, credible earliest start time, and completion time of the operator task, including:

[0064] (1): Obtain operator tasks according to the operator sequence OS , calculate the operator task The predicted earliest start time is calculated as follows:

[0065] (1)

[0066] in, Indicates the earliest start time of the forecast. Indicates the completion time. Represents operator tasks Prior computation tasks based on topology, Represents operator tasks Any previous subtask The completion time, , , Representing task flow The number of operator tasks in .

[0067] The completion time of the operator task is calculated according to formula (2), which will be introduced later.

[0068] The predicted earliest start time of an operator task depends on the maximum completion time of all previous operator tasks based on the topology on the same task flow. In particular, the first operator task obtained is the starting operator of the task flow, which has no previous operator tasks based on the topology, so its predicted earliest start time is 0.

[0069] (2): Get operator tasks Node Real-time node resource graph , and in the real-time node resource graph On the search, the operator tasks are satisfied at the same time Initial runtime and initial resource requirements The available interval, based on which the credible earliest start time is obtained .

[0070] Specifically, the operator tasks are obtained according to the node sequence MS and the operator sequence OS query The node is ,in, ; Get the node Real-time node resource graph ; In the real-time node resource graph On, from the earliest start time of the forecast Starting from, the search also meets the initial running time and initial resource requirements Available range , and the start time of the available interval As an operator task The earliest credible start time .

[0071] The real-time node resource graph records the real-time resource usage of the node, including the usage of the CPU, GPU, and memory in the node.

[0072] (3): Based on the earliest credible start time and initial run time , get the completion time of the operator task , the calculation formula is as follows:

[0073] (2)

[0074] The completion time of an operator task depends on the earliest credible start time of the operator task and the initial running time of the operator task.

[0075] (4): Update the node Node resource graph .

[0076] Specifically, at the node Node resource graph In the example, the slave operator task The earliest credible start time To completion time In the interval, the size is The CPU resources and size are The GPU resources and size are The memory resources are marked as occupied.

[0077] Starting from the first operator task of the operator sequence OS, steps (1) to (4) are executed cyclically for each operator task, and the predicted earliest start time and completion time of each operator task are calculated one by one.

[0078] Step b: Based on the predicted earliest start time and completion time of all operator tasks, the fitness of the chromosome is obtained.

[0079] Specifically, the minimum value of the predicted earliest start time of all operator tasks is calculated to obtain the scheduling start time; the maximum value of the completion time of all operator tasks is calculated to obtain the scheduling completion time; the difference between the scheduling completion time and the scheduling start time is calculated to obtain the scheduling duration of the chromosome; the inverse of the scheduling duration is used as the fitness.

[0080] The fitness of a chromosome indicates how fast the resource allocation method corresponding to the chromosome is.

[0081] S24: Based on the fitness, the current population is subjected to selection, crossover and mutation operations to obtain the next generation population.

[0082] There are many methods for selection operation, including roulette selection, tournament selection and linear sorting selection. One or more of the above methods can be used, and the present invention is not limited to this.

[0083] The crossover operation includes the crossover operation of the operator sequence and the hybrid operation of the node sequence.

[0084] (1) Crossover operation of operator sequence

[0085] The crossover operation of operator sequences is divided into two steps, including task flow-based crossover and operator task-based crossover.

[0086] It should be noted that the crossover operation should not destroy the topological relationship between operator tasks. Specifically, the first operator sequence OS1 and the second operator sequence OS2 are randomly selected from the parent chromosome for the crossover operation. First, a task flow-based crossover is performed, including:

[0087] Randomly select a task flow Ji as the target task flow, retain all operator tasks belonging to the target task flow Ji in the first operator sequence OS1, and delete other operator tasks; take out all operator tasks of all task flows except the target task flow Ji in the second operator sequence OS2 according to the order in the second operator sequence OS2, and fill them into the vacant positions in the first operator sequence OS1 in turn, to obtain the first intermediate operator sequence OSM1;

[0088] All operator tasks belonging to the target task flow Ji in the second operator sequence OS2 are retained, and other operator tasks are deleted; all operator tasks of all task flows except the target task flow Ji in the first operator sequence OS1 are taken out in the order in the first operator sequence OS1, and are filled in the vacant positions in the second operator sequence OS2 in sequence to obtain the second intermediate operator sequence OSM2.

[0089] Figure 4 The process of crossover operation is shown as Figure 4 As shown, assuming OS1 and OS2 are:

[0090] OS1: [10,20,30,21,11,31,12,22,32,23,33,13,24,14,34]

[0091] OS2: [10,11,20,21,30,31,12,22,32,23,33,13,24,14,34]

[0092] Assume that task flow J2 is selected as the target task flow, and perform task flow-based crossover. For OS1, keep all operator tasks belonging to task flow J2 in OS1, delete other operator tasks, and take operator tasks from OS1 in order except for the operator tasks of task flow J2 to fill the vacant positions, and get OSM1 as follows:

[0093] OSM1: [10,20,11,21,30,31,12,22,32,23,33,13,24,14,34]

[0094] Process OS2 in the same way to obtain OSM2 as follows:

[0095] OSM2: [10,11,20,30,21,31,12,23,32,22,33,13,14,34,24]

[0096] Next, perform operator-based task crossover, including:

[0097] The truncation position was arbitrarily selected to truncate OSM1 and OSM2 into the front and back parts respectively;

[0098] Keep the first part of OSM1, delete the second part, and select operator tasks from OSM2 in sequence to fill the vacant positions of OSM1, and obtain the first crossover operator sequence OSC1;

[0099] The first part of OSM2 is retained, the second part is deleted, and operator tasks are sequentially selected from OSM1 to fill the vacant positions of OSM2 in turn, thereby obtaining the second crossover operator sequence OSC2.

[0100] Continue to refer to the attached Figure 4 , for OSM1 and OSM2 in the above example, perform crossover based on operator tasks. Assuming that the truncation is selected at the fourth operator task, for OSM1, OSM1 is truncated into the front and back parts after the fourth operator task, the front part of OSM1 is retained, the back part is deleted, and operator tasks are sequentially selected from OSM2 to fill the vacant positions of OSM1 in turn, and OSC1 is obtained:

[0101] OSC1: [10,20,11,21,30,31,12,23,32,22,33,13,14,34,24]

[0102] Process OSM2 in the same way to obtain OSC2 as follows:

[0103] OSC2: [10,11,20,30,21,31,12,22,32,23,33,13,24,14,34]

[0104] In the improved genetic algorithm adopted by the present invention, through two steps of first performing a crossover operation on the operator sequence based on the task flow and then performing a crossover operation based on the operator task, a full crossover operation on each operator task is achieved without destroying the topological sorting of the operator tasks, so that a high-quality and reliable new population can be obtained, and the optimal chromosome can be quickly obtained.

[0105] (2) Crossover operation of node sequence

[0106] The truncation position is arbitrarily selected, for example, at the position of the third value of the node sequence, the first node sequence MS1 and the second node sequence MS2 are truncated into the front part and the rear part respectively, and the front part or the rear part of the first node sequence MS1 and the second node sequence MS2 are interchanged to obtain the first cross node sequence MSC1 and the second cross node sequence MSC2.

[0107] (3) Mutation operation of crossover operator sequence

[0108] The mutation operation should not destroy the topological structure of the operator tasks in the task flow. The steps include:

[0109] A target mutation operator task is randomly selected from the first crossover operator sequence OSC1; starting from the target mutation operator task, a target subsequence is searched forward or backward in the first crossover operator sequence OSC1, and the target subsequence should satisfy that all operator tasks in the target subsequence belong neither to the previous operator task nor to the subsequent operator task of the operator task; the operator task is taken out from the original position and inserted into any position in the subsequence to obtain the first mutation operator sequence OST1.

[0110] Optionally, a mutation operation is performed on the second crossover operator sequence OSC2 in the same manner to obtain a second mutation operator sequence OST2.

[0111] (4) Mutation operation of crossover node sequence

[0112] A position is randomly selected in the first cross node sequence MSC1, and the node at the position is replaced with any other node to obtain a first mutation node sequence MST1. Optionally, a mutation operation is performed on the second cross node sequence MSC2 in the same manner to obtain a second mutation node sequence MST2.

[0113] It should be noted that in the embodiment of the present invention, a crossover operation and a mutation operation are performed on the chromosomes of each generation of the population according to the crossover probability and the mutation probability respectively.

[0114] S25: Determine whether the current number of iterations is less than the maximum number of iterations. If so, increase the number of iterations by one and proceed to S23; otherwise, proceed to S26.

[0115] The maximum number of iterations is preset, for example, 500. If the current number of iterations does not exceed the maximum number of iterations, steps S23 to S25 are executed in a loop to iterate continuously. If the current number of iterations exceeds the maximum number of iterations, the population iteration is terminated and step S26 is entered to determine the initial optimal resource allocation method.

[0116] S26: Using the resource allocation method mapped by the chromosome with the highest fitness in the current population as the initial optimal resource allocation method.

[0117] The resource allocation method mapped by the chromosome with the highest fitness in the current population is the fastest and most efficient resource allocation method. The initial optimal resource allocation method will allocate resources according to the operator sequence and node sequence of the chromosome with the highest fitness in the current population.

[0118] S3: Allocate resources to each of the operator tasks according to the initial optimal resource allocation method.

[0119] The system can perform initial resource allocation based on the initial optimal resource allocation method, and allocate multiple operator tasks of multiple task flows to corresponding multiple nodes. At this time, each operator task starts running on the corresponding node.

[0120] Since the resources allocated to each operator task on the node are fixed, and the resources consumed by the operator task in real time when running on the node are dynamically changing, in order to avoid wasting node resources, when the operator task starts running on each node, it is necessary to further dynamically allocate the resources within the node.

[0121] Next, step S4 is described in detail in conjunction with a specific embodiment.

[0122] According to an embodiment of the present invention, S4: after the plurality of operator tasks are run, the resource requirements of each operator task in the next period are predicted continuously using the second prediction model based on the resource consumption of each operator task in the current period, and based on the resource requirements of all operator tasks in the next period, an improved genetic algorithm is used to obtain the optimal resource allocation method for the next period, and in the next period, resources are re-allocated to each operator task according to the optimal resource allocation method for the next period, including:

[0123] S41: Collect multiple real-time resource consumptions of the operator task in the current period to obtain a resource consumption sequence.

[0124] Specifically, after each operator task runs, Collect the real-time resource consumption of each resource separately, including the real-time resource consumption of CPU, GPU, memory, and file storage. Set the time window with a width of K, and the time period in the time window is the current period.

[0125] Figure 5 FIG. 4 shows a schematic diagram of using the second prediction model to predict resource demand in the next period based on resource consumption in the current period, such as Figure 5 As shown in the figure, during the period t1~t1+K, each resource obtains p sample values ​​as multiple real-time resource consumptions, and the p sample values ​​form a resource consumption sequence in chronological order, including the CPU consumption sequence , GPU consumption sequence , memory consumption sequence .

[0126] S42: Based on the resource consumption sequence in the current time period, use a second prediction model to perform feature extraction and prediction to obtain a resource demand sequence in the next time period.

[0127] The second prediction model is pre-trained based on iTransformer using the second data set. Its main structure is the same as the first prediction model, including an embedding layer, multiple stacked Transformer modules and a projection layer, which will not be repeated here. The difference from the first prediction model lies in the setting of the number of model layers and the number of hidden layer dimensions, as well as the specific data set for training the model. Among them, the second data set constructs a real-time data sequence from multiple real-time data collected by the operator task at short intervals within a continuous time period, including real-time resource consumption and real-time input file size, and continuously collects multiple real-time sequences to obtain the second data set.

[0128] The second prediction model has the ability of real-time reasoning. When an operator task starts to execute, the resource consumption information of the operator task during operation is instantly transmitted to the second prediction model at fixed time intervals. The model instantly predicts the resource requirements of the operator in the future, realizing a dynamic and real-time update process.

[0129] Continue to refer to Figure 5 , the resource consumption sequence is input into the second prediction model, the second prediction model extracts the features of each resource consumption sequence in the time period t1~t1+K, and predicts multiple moments in the next time period, for example, the next time period is t2~t2+Z, and obtains the corresponding resource demand sequence. Provide a prediction value, and each resource will get a resource demand sequence consisting of q prediction values. The resource demand sequence includes the CPU demand sequence , GPU requirement sequence , memory requirement sequence . Wherein, q and p are integers, and their values ​​can be the same or different.

[0130] S43: Based on the resource requirement sequences of all operator tasks in the next period, the improved genetic algorithm is used to obtain the optimal resource allocation method in the next period, and in the next period, resources are reallocated to the tasks according to the optimal resource allocation method.

[0131] Specifically, for each resource demand sequence, the average or maximum value of q predicted values ​​in the future time period can be taken as the corresponding predicted resource demand. For each operator task, the remaining running time can be obtained based on the initial running time and the running time. The predicted resource demand and the remaining running time of each resource are input into the improved genetic algorithm to obtain the optimal resource allocation method in the next time period. The process of using the improved genetic algorithm to obtain the optimal resource allocation method in the next time period is the same as in step S2 and will not be repeated here. The optimal resource allocation method in the next time period includes the latest scheduling order of each operator task and the latest allocated node of each operator task.

[0132] Then, in the next time period, resources are reallocated for the tasks according to the optimal resource allocation method, and each operator task is allocated to the latest allocated node according to the latest scheduling order.

[0133] S44: During the execution of each operator task, the current time period and the next time period are continuously updated, and steps S41-S43 are executed cyclically until all the operator tasks are completed.

[0134] Specifically, as the operator task runs, each cycle moves the time sliding window of size K with a step size D, updates the current period and the next period, and obtains an optimal resource allocation method corresponding to the next period. The optimal resource allocation method is used to re-allocate resources for all operator tasks. The step size D can be set as needed, preferably, the step size D is equal to Z.

[0135] In particular, during the dynamic resource allocation process, excess resources are released in real time, and the released resources can be used to allocate to newly added operator tasks in real time.

[0136] In the embodiment of the present invention, step S4 can reasonably utilize node resources, improve node resource utilization, and reduce resource waste by dynamically allocating resources using the latest optimal resource allocation method at regular intervals.

[0137] In one embodiment of the present invention, a task scheduling device is provided, which is applied to a task processing platform. The task processing platform includes multiple nodes. Figure 6The task scheduling device 600 shown in FIG. 1 includes:

[0138] The initial prediction module 601 is used to predict the initial resource requirements and initial running time of each operator task based on the recent running records of multiple operator tasks using the first prediction model; wherein the multiple operator tasks belong to multiple task flows.

[0139] The scheduling algorithm module 602 is used to obtain the initial optimal resource allocation method using an improved genetic algorithm based on the initial resource requirements and initial running time of each operator task.

[0140] The scheduling execution module 603 is used to allocate resources to each of the operator tasks according to the initial optimal resource allocation method.

[0141] The real-time prediction and scheduling module 604 is used to continuously use the second prediction model to predict the resource requirements of each operator task in the next period based on the resource consumption of each operator task in the current period after the operator task is run, and based on the resource requirements of all operator tasks in the next period, use an improved genetic algorithm to obtain the optimal resource allocation method for the next period, and reallocate resources to each operator task in the next period according to the optimal resource allocation method for the next period.

[0142] An embodiment of the present invention further provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the methods in the aforementioned method embodiments when executing the program. Further, the electronic device further comprises: at least one input device; at least one output device. The above-mentioned memory, processor, input device and output device are connected via a bus. Among them, the input device may specifically be a camera, a touch panel, a physical button or a mouse, etc. The output device may specifically be a display screen. The memory may be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as a disk memory. The memory is used to store a set of executable program codes, and the processor is coupled to the memory.

[0143] The embodiment of the present invention also provides a computer-readable storage medium, which may be disposed in the electronic device in each of the above embodiments, and the computer-readable storage medium may be the electronic device in the aforementioned embodiments. The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the steps of the method described in any one of the aforementioned method embodiments are implemented. Furthermore, the computer-storable medium may also be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0144] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A task scheduling method, applied to a task processing platform, wherein the task processing platform includes a plurality of nodes, characterized in that: The method comprises: S1: Based on recent operation records of multiple operator tasks, use a first prediction model to respectively predict initial resource requirements and initial operation durations of each of the operator tasks; wherein the multiple operator tasks belong to multiple task flows; S2: Based on the initial resource requirements and the initial running time of each of the operator tasks, an improved genetic algorithm is used to obtain an initial optimal resource allocation method; S3: Allocate resources to each of the operator tasks according to the initial optimal resource allocation method; S4: After the plurality of operator tasks are run, the second prediction model is continuously used to predict the resource demand of each operator task in the next period based on the resource consumption of each operator task in the current period, and based on the resource demand of all operator tasks in the next period, an improved genetic algorithm is used to obtain the optimal resource allocation method for the next period, and in the next period, resources are reallocated to each operator task according to the optimal resource allocation method for the next period.

2. The method according to claim 1, characterized in that The first prediction model and the second prediction model are obtained based on iTransformer pre-training.

3. The method according to claim 2, characterized in that The S1 includes: Obtain the recent running records, input file size and operator ID of the operator task; The multiple resource consumptions in the recent operation records constitute a recent resource consumption sequence, and the multiple operation durations constitute a recent operation duration sequence; The recent resource consumption sequence, the recent running time sequence, the input file size, and the operator ID are input into the first prediction model to obtain the initial resource requirement and the initial running time of the operator task.

4. The method according to claim 1, characterized in that: The S2 includes: S21: performing chromosome encoding, wherein one chromosome includes a pair of operator sequences and node sequences, and one chromosome maps to one resource allocation mode; S22: Generate an initial population, randomly generate multiple chromosomes, including multiple non-identical operator sequences and multiple non-identical node sequences, and initialize the number of iterations to 1; S23: Calculate the fitness of each chromosome of the current population using the fitness function; S24: Based on the fitness, the current population is subjected to selection, crossover and mutation operations to obtain the next generation population; S25: Determine whether the current number of iterations is less than the maximum number of iterations. If so, increase the number of iterations by one and proceed to S23; otherwise, proceed to S26; S26: Using the resource allocation method mapped by the chromosome with the highest fitness in the current population as the initial optimal resource allocation method.

5. The method according to claim 4, characterized in that The S23 includes: Step a: Calculate the predicted earliest start time, credible earliest start time and completion time of each operator task one by one according to the order of each operator task in the operator sequence; Step b: Based on the predicted earliest start time and completion time of all operator tasks, the fitness of the chromosome is obtained.

6. The method according to claim 4, characterized in that The S24 includes: The crossover operation includes two crossovers: a crossover based on a task flow and a crossover based on an operator task.

7. The method according to claim 2, characterized in that The S4 includes: S41: Collect multiple real-time resource consumptions of operator tasks in the current period to obtain a resource consumption sequence; S42: Based on the resource consumption sequence in the current period, use a second prediction model to perform feature extraction and prediction to obtain a resource demand sequence in the next period; S43: based on the resource demand sequence of all operator tasks in the next period, using the improved genetic algorithm to obtain the optimal resource allocation method in the next period, and reallocating resources to the tasks according to the optimal scheduling strategy in the next period; S44: During the execution of each operator task, the current time period and the next time period are continuously updated, and steps S41-S43 are executed cyclically until all the operator tasks are completed.

8. A task scheduling device, applied to a task processing platform, the task processing platform comprising a plurality of nodes, characterized in that: The device includes: An initial prediction module, used to predict the initial resource requirements and initial running time of each of the operator tasks based on the recent running records of the multiple operator tasks using the first prediction model; wherein the multiple operator tasks belong to multiple task flows; A scheduling algorithm module, used to obtain an initial optimal resource allocation method using an improved genetic algorithm based on the initial resource requirements and the initial running time of each of the operator tasks; A scheduling execution module, used for allocating resources to each of the operator tasks according to the initial optimal resource allocation method; The real-time prediction and scheduling module is used to continuously use the second prediction model to predict the resource requirements of each operator task in the next period based on the resource consumption of each operator task in the current period after the multiple operator tasks are run, and based on the resource requirements of all operator tasks in the next period, use an improved genetic algorithm to obtain the optimal resource allocation method for the next period, and reallocate resources to each operator task in the next period according to the optimal resource allocation method for the next period.

9. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images