Intelligent scheduling method and system for resource-limited data operation platform

Abstract

The invention discloses an intelligent scheduling method and system for a resource-limited data operation platform, and belongs to the field of computers. The method comprises the following steps: S1, acquiring historical operation information of a data operation platform to obtain a historical operation set; s2, splitting the historical job set into historical job subsets to obtain historical job subset information; s3, inputting the global information and the historical job subset information of the data job platform into a job scheduling agent for training to obtain a trained job scheduling agent; s4, acquiring to-be-scheduled job information of the data job platform to obtain a to-be-scheduled job set; step S5, splitting the to-be-scheduled job set into to-be-scheduled job subsets to obtain to-be-scheduled job subset information; and S6, obtaining an optimal solution sequence of the to-be-scheduled job subset by using the trained job scheduling agent, and solving and updating global information. According to the method, the resource use upper limit can be remarkably compressed, and the overall scheduling performance and the platform resource utilization efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an intelligent scheduling method and system for a resource-constrained data operation platform. Background Technology

[0002] In a cloud computing environment, the data job platform, as a core component for resource scheduling and task execution, has a direct impact on the overall resource utilization and operating costs of the platform through its scheduling strategy. For cloud service providers, a reasonable scheduling strategy can fully tap the potential of existing resources and achieve the business goal of cost reduction and efficiency improvement without adding additional hardware resources.

[0003] Currently, common scheduling methods typically model the problem as a mixed-integer linear programming (MILP) model in order to obtain a theoretically optimal solution. However, in practical applications with complex job dependencies, multi-dimensional resource constraints, and large-scale task sets, MILP models are enormous, traditional solvers are costly to solve, and have slow response times, making it difficult to meet the high concurrency and real-time requirements of cloud platforms. Therefore, there is an urgent need for a scheduling method with high computational efficiency that can effectively optimize the upper limit of resource usage to improve scheduling performance, reduce resource waste, and lower overall operating costs.

[0004] To address large-scale scheduling problems, current mainstream methods typically employ a decomposition-solution strategy, dividing the original scheduling problem into several subproblems and processing them iteratively. This type of method belongs to iterative greedy algorithms, using heuristic sorting to solve the subproblems and improve scheduling efficiency. However, the order in which the subproblems are solved has a significant impact on the final scheduling quality. Relying solely on static sorting or human experience can easily lead to the scheduling solution getting trapped in local optima, even affecting the feasibility of the original problem. Therefore, determining a subproblem-solving order superior to traditional heuristics has become a crucial challenge for improving overall scheduling performance.

[0005] In summary, existing scheduling methods still suffer from low computational efficiency, unstable solution results, and a lack of flexibility in scheduling strategies under large-scale constrained environments. Therefore, providing a scheduling method that can identify the optimal solution order for subproblems and effectively optimize resource utilization limits is a problem that needs to be solved to improve the scheduling performance and system responsiveness of data job platforms in complex scenarios.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide an intelligent scheduling method and system for resource-constrained data operation platforms, which can significantly reduce the upper limit of resource usage, improve overall scheduling performance and platform resource utilization efficiency, thereby solving the above-mentioned technical problems existing in the prior art.

[0008] The objective of this invention is achieved through the following technical solution: An intelligent scheduling method for a resource-constrained data job platform includes: Step S1: Obtain historical job information from the data job platform to obtain a historical job set; Step S2: Based on the dependencies of historical task information in the historical task set, split the historical task set into non-overlapping historical task subsets, and aggregate the historical task information in the historical task subsets to obtain historical task subset information; Step S3: Obtain the global information of the data job platform, input the global information of the data job platform and the historical job subset information into the job scheduling agent, and use reinforcement learning to train it to obtain the trained job scheduling agent. Step S4: Obtain the set of jobs to be scheduled from the data job platform; Step S5: Based on the dependency relationship of the job information in the job set to be scheduled, split the job set to be scheduled into non-overlapping job subsets, and aggregate the job information in the job subsets to obtain job subset information. Step S6: Input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order to realize the scheduling of each job, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the contained jobs.

[0009] An intelligent scheduling system for a resource-constrained data job platform that implements the method described in this invention includes: The module comprises a data acquisition module, a job splitting module, a job scheduling agent module, an agent training module, and an iterative solution module; among which, The data acquisition module can acquire historical job information from the data job platform to obtain a historical job set, and acquire job information to be scheduled from the data job platform to obtain a job set to be scheduled. The job splitting module is communicatively connected to the data acquisition module. It can split the historical job set into non-overlapping historical job subsets based on the dependency relationship of historical job information in the historical job set, and aggregate the historical job information in the historical job subsets to obtain historical job subset information; and split the job set to be scheduled into non-overlapping job subsets based on the dependency relationship of job information in the job set to be scheduled, and aggregate the job information in the job subsets to be scheduled to obtain job subset information. The job scheduling agent module is communicatively connected to the job splitting module and is used to determine the solution order of the historical job subset based on the historical job subset information and the global information of the data job platform; and to determine the solution order of the job subset to be scheduled based on the job subset information and the global information of the data job platform. The agent training module is communicatively connected to the job scheduling agent module. It can solve the historical job subsets according to the solution order of the historical job subsets to obtain the scheduling baseline of the historical job set. The global information of the data job platform, the information of the historical job subsets and the scheduling baseline of the historical job set are input into the job scheduling agent module for training to obtain a trained job scheduling agent module. The iterative solution module is communicatively connected to the agent training module. It can input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent module for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the contained jobs.

[0010] Compared with existing technologies, the intelligent scheduling method and system for resource-constrained data work platforms provided by this invention have the following advantages: By leveraging reinforcement learning to determine the optimal solution order for a subset of jobs to be scheduled, scheduling efficiency can be effectively improved and resource costs reduced. This invention significantly outperforms traditional methods on large-scale instances, exhibiting the following advantages: High-efficiency scheduling: By optimizing scheduling strategies through reinforcement learning, high-quality scheduling schemes can be quickly found on large-scale instances, significantly improving scheduling efficiency. Resource optimization: Resources can be rationally allocated based on task resource requirements and time constraints, reducing resource waste and minimizing resource costs. Scalability: A modular design can be adopted, with modules interacting through data interfaces, facilitating the expansion and integration of new functional modules to meet the needs of different application scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 The overall flowchart of the intelligent scheduling method for the resource-constrained data job platform provided in the embodiments of the present invention is shown.

[0013] Figure 2This is a block diagram of an intelligent scheduling system for a resource-constrained data job platform provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0015] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0016] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0017] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0018] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0019] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0020] The solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0021] like Figure 1 As shown, this invention provides an intelligent scheduling method for a resource-constrained data job platform. It addresses the problems of low solution efficiency and suboptimal sub-problem solving order in existing methods in large-scale scenarios, proposing a scheduling method that integrates problem decomposition and sub-problem solving order optimization mechanisms. It includes: Step S1: Obtain historical job information from the data job platform to obtain a historical job set; Step S2: Based on the dependencies of historical task information in the historical task set, split the historical task set into non-overlapping historical task subsets, and aggregate the historical task information in the historical task subsets to obtain historical task subset information; Step S3: Obtain the global information of the data job platform, input the global information of the data job platform and the historical job subset information into the job scheduling agent, and use reinforcement learning to train it to obtain the trained job scheduling agent. Step S4: Obtain the set of jobs to be scheduled from the data job platform; Step S5: Based on the dependency relationship of the job information in the job set to be scheduled, split the job set to be scheduled into non-overlapping job subsets, and aggregate the job information in the job subsets to obtain job subset information. Step S6: Input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order to realize the scheduling of each job, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the contained jobs.

[0022] Preferably, in step S1 of the above method, the historical job information of the data job platform includes: Earliest start time of the task, latest start time of the task, task execution time, resources required for the task, task dependencies, and resource cost per unit of the task; In step S1, the historical job information further includes: a scheduling baseline, which refers to the average value of the sum of the upper limits of various resource usage of the data job platform obtained by solving the historical job set multiple times in history.

[0023] Preferably, in step S2 of the above method, the historical task set is split into disjoint historical task subsets according to the dependencies of historical task information in the historical task set, and the historical task information in the historical task subsets is aggregated to obtain historical task subset information, including: Step S21: Create each historical job node based on the historical job information, and use the historical job information corresponding to each historical job node as the information of each historical job node; the historical job information does not include historical job dependencies and scheduling baselines. Step S22: Construct directed edges between corresponding historical task nodes based on historical task dependencies to form a historical task graph; Step S23: Based on the connectivity of the historical task graph, split the historical task graph into associated historical task subgraphs, and restore the associated historical task subgraphs into a subset of associated historical tasks; Step S24: Calculate the average, minimum, maximum and variance of the historical job information based on the historical job information of each historical job in the associated historical job subset. Combine the average, minimum, maximum and variance of the historical job information with the number of historical jobs in the associated historical job subset to obtain the historical job subset information. The historical job information in each of the above steps does not include historical job dependencies and scheduling baselines.

[0024] Preferably, in step S2 of the above method, the historical job subset information includes: the average value of job information contained in the historical job subset, the minimum value of job information contained in the historical job subset, the maximum value of job information contained in the historical job subset, and the variance of job information contained in the historical job subset; wherein, the job information does not include job dependencies and scheduling baseline.

[0025] Preferably, in step S3 of the above method, the global information and historical job subset information of the data job platform are input into the job scheduling agent for training using reinforcement learning to obtain a trained job scheduling agent, including: Step S31: Input the global information and historical job subset information of the data job platform into the job scheduling agent for processing, obtain the solution order of the sub-models corresponding to the historical job subsets, and output the next sub-model to be solved according to the solution order. Step S32: Solve the sub-model to be solved output in step S31, update the global information of the data operation platform according to the solution results, and determine whether all sub-models have been solved. If not, return to step S31. If yes, obtain the sum of the upper limits of various resource usage by solving the sub-models corresponding to all historical operation subsets, and execute step S33. In step S33, if the sum of the various resource usage limits is greater than the scheduling baseline, the job scheduling agent reduces the joint probability of the solution order under the global information of the current data job platform by adjusting its own parameters; if the sum of the various resource usage limits is less than the scheduling baseline, the job scheduling agent increases the joint probability of the solution order under the global information of the current data job platform by adjusting its own parameters.

[0026] Preferably, in step S3 of the above method, the global information of the data job platform includes: a set of scheduled job subsets, the total amount of various resources in each time slot of the data job platform, and the usage of various resources in each time slot; The job scheduling agent adopts either an attention model or a graph neural network model, and adjusts its own parameters through short-term policy optimization or policy gradient.

[0027] Preferably, in step S4 of the above method, the set of jobs to be scheduled is obtained by acquiring the job information to be scheduled from the data job platform in the following manner: Obtain the set of jobs to be scheduled from the data job platform, and determine the earliest start time, latest start time, job execution time, resources required for the job, job dependencies and resource cost per unit of each job from the set of jobs to be scheduled. The job information to be scheduled does not include the scheduling baseline, which refers to the average value of the sum of the upper limits of various resources used by the data job platform obtained by solving the historical job set multiple times in history.

[0028] Preferably, step S5 of the above method involves splitting the set of jobs to be scheduled into disjoint subsets of jobs to be scheduled based on the dependencies of the job information in the set of jobs to be scheduled, and aggregating the job information in the subsets of jobs to be scheduled to obtain the job subset information, including: Step S51: Based on the information of the jobs to be scheduled, excluding the dependencies of the jobs to be scheduled, create each job node to be scheduled, and use the information of the jobs to be scheduled, excluding the dependencies of the jobs to be scheduled, corresponding to each job node to be scheduled as the information of each job node to be scheduled. Step S52: Based on the dependencies of the jobs to be scheduled, construct directed edges between the corresponding nodes of the jobs to be scheduled to form a graph of the jobs to be scheduled. Step S53: Based on the connectivity of the job graph to be scheduled, split the job graph to be scheduled into associated job subgraphs, and restore the associated job subgraphs to associated job subsets. Step S54: Based on the information of the scheduled jobs other than the scheduled job dependencies of each scheduled job in the associated scheduled job subset, calculate the average, minimum, maximum and variance of the scheduled job information. Combine the average, minimum, maximum and variance of the scheduled job information with the number of scheduled jobs in the associated scheduled job subset as the scheduled job subset information. Step S6 involves inputting the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent for processing in the following manner: obtaining the optimal solution order of the subset of jobs to be scheduled; iteratively solving the subset of jobs to be scheduled according to the optimal solution order; obtaining the start time of the jobs contained in the subset of jobs; and iteratively updating the global information of the data job platform based on the start time of the contained jobs, including: Step S61: Input the global information of the data job platform and the information of the subset of jobs to be scheduled into the trained job scheduling agent for processing, obtain the optimal solution order of the sub-models corresponding to the subset of jobs to be scheduled, and output the next sub-model to be solved according to the optimal solution order. Step S62: Solve the sub-model to be solved output in step S61 to obtain the start time of the job corresponding to the sub-model. Update the global information of the data job platform according to the start time, execution time and resource usage of the job, and determine whether all sub-models have been solved. If not, return to step S61. If yes, complete the solution of all sub-models.

[0029] Preferably, in step S31 of the above method, the sub-model corresponding to the subset of historical tasks is a mixed integer programming model, and its objective function is: ; The constraints are: ; In the above formula, The actual start time of historical task i in the historical task subset is used as the decision variable of the sub-model; A subset of historical assignments; The unit cost of resource k; This represents the upper limit for the use of resource k; The execution time of historical job i; Dependencies between historical tasks The resource requirement of historical task i for resource k; For all resource types; The earliest start time of historical assignment i This represents the latest start time of historical assignment i.

[0030] like Figure 2 As shown, embodiments of the present invention also provide an intelligent scheduling system for a resource-constrained data job platform that implements the above method, comprising: The module comprises a data acquisition module 21, a job splitting module 22, a job scheduling agent module 23, an agent training module 24, and an iterative solution module 25; among which, The data acquisition module 21 can acquire historical job information from the data job platform to obtain a historical job set, and acquire job information to be scheduled from the data job platform to obtain a job set to be scheduled. The job splitting module 22 is communicatively connected to the data acquisition module 21. It can split the historical job set into non-overlapping historical job subsets according to the dependency relationship of historical job information in the historical job set, and aggregate the historical job information in the historical job subsets to obtain historical job subset information; and split the job set to be scheduled into non-overlapping job subsets according to the dependency relationship of job information in the job set to be scheduled, and aggregate the job information in the job subsets to be scheduled to obtain job subset information. The job scheduling agent module 23 is communicatively connected to the job splitting module 22, and is used to determine the solution order of the historical job subset based on the historical job subset information and the global information of the data job platform; and to determine the solution order of the job subset to be scheduled based on the job subset information and the global information of the data job platform. The agent training module 24 is communicatively connected to the job scheduling agent module 23. It can solve the historical job subsets according to the solution order of the historical job subsets to obtain the scheduling baseline of the historical job set. The global information of the data job platform, the information of the historical job subsets and the scheduling baseline of the historical job set are input into the job scheduling agent module 23 for training to obtain the trained job scheduling agent module 23. The iterative solution module 25 is communicatively connected to the agent training module 24. It can input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent module for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the jobs contained therein.

[0031] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0032] Example 1 This embodiment provides an intelligent scheduling method for a resource-constrained data job platform, the overall process of which is as follows: Figure 1 As shown, it includes the following steps: Step S1: Obtain historical job information from the data job platform to obtain a historical job set; Step S2: Based on the dependencies in the historical job information in the historical job set, split the historical job set into non-overlapping historical job subsets, and aggregate the historical job information in the historical job subsets to obtain the historical job subset information; Step S3: Input the global information and historical job subset information of the data job platform into the job scheduling agent for training to obtain a trained job scheduling agent; Step S4: Obtain the set of jobs to be scheduled from the data job platform; Step S5: Based on the dependency relationship of the job information in the job set to be scheduled, split the job set to be scheduled into non-overlapping job subsets, and aggregate the job information in the job subsets to obtain job subset information. Step S6: Input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent to obtain the optimal solution order of the subset of jobs to be scheduled. Iterate the solution of the subset of jobs to be scheduled according to the optimal solution order to realize the scheduling of each job. Obtain the start time of the jobs contained in the subset of jobs to be scheduled. Iterate and update the global information of the data job platform according to the start time of the jobs contained therein.

[0033] The historical job information also includes the scheduling baseline, which is the average of the sum of the upper limits of various resources used by the job platform obtained by solving the historical job set multiple times in history.

[0034] The historical job subset information includes: the average value of job information (excluding job dependencies and scheduling baselines) of the jobs contained in the historical job subset, the minimum value of job information (excluding job dependencies and scheduling baselines), the maximum value of job information (excluding job dependencies and scheduling baselines), and the variance of job information (excluding job dependencies and scheduling baselines).

[0035] The global information of the data operation platform includes: a subset of scheduled jobs, the total amount of various resources in each time slot of the operation platform, and the usage of various resources in each time slot of the operation platform.

[0036] The information on jobs to be scheduled includes: the earliest start time of each job, the latest start time of each job, the execution time of each job, the resources required for each job, the dependencies of each job, and the resource cost per unit of each job; this information on jobs to be scheduled does not include the scheduling baseline.

[0037] For example, historical job information from the data job platform is obtained to obtain a historical job set. Data is collected from the historical job scheduling information. For a specific job set, the earliest start time, latest start time, execution time, resources required, dependencies, unit resource cost, and scheduling baseline of each job are determined. This generates multiple sets of training data to form a training database for the job scheduling agent.

[0038] It should be noted that the goal of the job scheduling agent is to determine the optimal solution order for a subset of jobs to be scheduled, thereby minimizing the sum of the upper limits of various resource usage.

[0039] In a preferred embodiment, step S2 specifically includes the following steps: Step S21: Based on historical job information (excluding historical job dependencies and scheduling baselines), create each historical job node and use the historical job information (excluding historical job dependencies and scheduling baselines) corresponding to each historical job node as the information of each historical job node. Step S22: Based on the dependencies of historical tasks, construct directed edges between corresponding historical task nodes to form a historical task graph; Step S23: Based on the connectivity of the historical task graph, split the historical task graph into related historical task subgraphs, and restore the related historical task subgraphs into related historical task subsets; Step S24: Calculate the average, minimum, maximum and variance of each historical job in the associated historical job subset (excluding historical job dependencies and scheduling baselines), and combine this with the number of historical jobs in the associated historical job subset as historical job subset information.

[0040] In a preferred embodiment, step S3 specifically includes the following steps: Step S31: Input the global information and historical job subset information of the data job platform into the scheduling agent to obtain the solution order of the sub-models corresponding to the historical job subsets, and output the next sub-model to be solved according to the solution order. Step S32: Solve the sub-model to be solved output in step S31, update the global information of the data operation platform according to the solution results, and determine whether all sub-models have been solved. If not, return to step S31. If yes, obtain the sum of the upper limits of various resource usage by solving the sub-models corresponding to all historical operation subsets, and execute step S33. In step S33, if the sum of the various resource usage limits is greater than the scheduling baseline, the job scheduling agent will adjust its own parameters to reduce the joint probability of the solution order under the global information of the current data job platform; if the sum of the various resource usage limits is less than the scheduling baseline, the job scheduling agent will adjust its own parameters to increase the joint probability of the solution order under the global information of the current data job platform.

[0041] By alternately executing steps S31 and S32, in each iteration, step S31 outputs the next sub-model to be solved, step S32 solves it and updates the global information of the data job platform based on the returned result, then returns to step S31 to select the next sub-model to be solved, until all sub-models are solved before executing step S33; the sub-models corresponding to the historical job subset are modeled and solved using methods such as mixed integer programming or neural combinatorial optimization; the job scheduling agent adopts architectures such as attention models and graph neural networks; the job scheduling agent adjusts its own parameters through methods such as short-term policy optimization and policy gradient.

[0042] For example, the sub-model corresponding to the subset of historical assignments is a mixed-integer programming model: Its objective function is: ; The constraints are: ; In the above formula, A subset of historical assignments; The actual start time of historical task i in the historical task subset is used as the decision variable of the sub-model; For all resource types; The unit cost of resource k; This represents the upper limit for the use of resource k; The execution time of historical job i; Dependencies between historical tasks The resource requirement of historical task i for resource k; The earliest start time of historical assignment i This represents the latest start time of historical assignment i.

[0043] In a preferred embodiment, step S4 described above specifically includes the following steps: Obtain the information of jobs to be scheduled from the data job platform, obtain the set of jobs to be scheduled, and determine the earliest start time, latest start time, job execution time, resources required for the job, job dependencies, and resource cost per unit of the job for each job to be scheduled.

[0044] In a preferred embodiment, step S5 specifically includes the following steps: Step S51: Based on the job information to be scheduled (excluding job dependencies), create each job node to be scheduled, and use the job information to be scheduled (excluding job dependencies) corresponding to each job node as the information of each job node to be scheduled. Step S52: Based on the dependencies of the jobs to be scheduled, construct directed edges between the corresponding nodes of the jobs to be scheduled to form a graph of the jobs to be scheduled. Step S53: Based on the connectivity of the job graph to be scheduled, split the job graph to be scheduled into associated job subgraphs, and restore the associated job subgraphs to associated job subsets. Step S54: Calculate the average, minimum, maximum and variance of each scheduled job in the associated scheduled job subset (excluding the job dependencies), and combine the number of scheduled jobs in the associated scheduled job subset as the scheduled job subset information.

[0045] In a preferred embodiment, step S6 specifically includes the following steps: Step S61: Input the global information of the data job platform and the information of the subset of jobs to be scheduled into the trained job scheduling agent for processing, obtain the optimal solution order of the sub-models corresponding to the subset of jobs to be scheduled, and output the next sub-model to be solved according to the optimal solution order. Step S62: Solve the sub-model to be solved output in step S61 to obtain the start time of the job corresponding to the sub-model. Update the global information of the data job platform according to the start time of the job, and determine whether all sub-models have been solved. If not, return to step S61. If yes, complete the solution of all sub-models.

[0046] By alternately executing steps S61 and S62, in each iteration, step S61 outputs the next sub-model to be solved, and step S62 solves it and updates the global information of the data operation platform based on the returned result, and then returns to step S61 to select the next sub-model to be solved, until all sub-models are solved.

[0047] Example 2 This embodiment provides an intelligent scheduling system for a resource-constrained data job platform, the overall architecture of which is as follows: Figure 2 As shown, it includes: The module comprises a data acquisition module 21, a job splitting module 22, a job scheduling agent module 23, an agent training module 24, and an iterative solution module 25; among which, The data acquisition module 21 can acquire historical job information from the data job platform to obtain a historical job set, and acquire job information to be scheduled from the data job platform to obtain a job set to be scheduled. The job splitting module 22 is communicatively connected to the data acquisition module 21. It can split the historical job set into non-overlapping historical job subsets according to the dependency relationship of historical job information in the historical job set, and aggregate the historical job information in the historical job subsets to obtain historical job subset information; and split the job set to be scheduled into non-overlapping job subsets according to the dependency relationship of job information in the job set to be scheduled, and aggregate the job information in the job subsets to be scheduled to obtain job subset information. The job scheduling agent module 23 is communicatively connected to the job splitting module 22, and is used to determine the solution order of the historical job subset based on the historical job subset information and the global information of the data job platform; and to determine the solution order of the job subset to be scheduled based on the job subset information and the global information of the data job platform. The agent training module 24 is communicatively connected to the job scheduling agent module 23. It can solve the historical job subsets according to the solution order of the historical job subsets to obtain the scheduling baseline of the historical job set. The global information of the data job platform, the information of the historical job subsets and the scheduling baseline of the historical job set are input into the job scheduling agent module 23 for training to obtain the trained job scheduling agent module 23. The iterative solution module 25 is communicatively connected to the agent training module 24. It can input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent module for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the jobs contained therein.

[0048] The job information includes the earliest start time, the latest start time, the execution time, the resources required, the dependencies, and the resource cost per unit of each job; the historical job information also includes the scheduling baseline; the job information to be scheduled does not include the scheduling baseline. The job subset information includes the average value of the job information (excluding job dependencies and scheduling baseline) of the jobs contained in the job subset, the minimum value of the job information (excluding job dependencies and scheduling baseline), the maximum value of the job information (excluding job dependencies and scheduling baseline), and the variance of the job information (excluding job dependencies and scheduling baseline). The scheduling baseline refers to the average of the sum of the upper limits of various resource usage of the job platform obtained by solving a specific set of historical jobs multiple times in history; The global information of the data operation platform includes a subset of scheduled jobs, the start time of the scheduled jobs, the total amount of various resources in each time slot of the operation platform, and the usage of various resources in each time slot of the operation platform.

[0049] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0050] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An intelligent scheduling method for a resource-constrained data operation platform, characterized in that, include: Step S1: Obtain historical job information from the data job platform to obtain a historical job set; Step S2: Based on the dependencies of historical task information in the historical task set, split the historical task set into non-overlapping historical task subsets, and aggregate the historical task information in the historical task subsets to obtain historical task subset information; Step S3: Obtain the global information of the data job platform, input the global information of the data job platform and the historical job subset information into the job scheduling agent, and use reinforcement learning to train it to obtain the trained job scheduling agent. Step S4: Obtain the set of jobs to be scheduled from the data job platform; Step S5: Based on the dependency relationship of the job information in the job set to be scheduled, split the job set to be scheduled into non-overlapping job subsets, and aggregate the job information in the job subsets to obtain job subset information. Step S6: Input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order to realize the scheduling of each job, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the contained jobs.

2. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1, characterized in that, In step S1, the historical job information of the data job platform includes: Earliest start time of the task, latest start time of the task, task execution time, resources required for the task, task dependencies, and resource cost per unit of the task; In step S1, the historical job information further includes: a scheduling baseline, which refers to the average value of the sum of the upper limits of various resource usage of the data job platform obtained by solving the historical job set multiple times in history.

3. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1, characterized in that, In step S2, the historical job set is divided into disjoint subsets of historical jobs based on the dependencies of historical job information in the historical job set, and the historical job information in the subsets is aggregated to obtain historical job subset information, including: Step S21: Create each historical job node based on the historical job information, and use the historical job information corresponding to each historical job node as the information of each historical job node; the historical job information does not include historical job dependencies and scheduling baselines. Step S22: Construct directed edges between corresponding historical task nodes based on historical task dependencies to form a historical task graph; Step S23: Based on the connectivity of the historical task graph, split the historical task graph into associated historical task subgraphs, and restore the associated historical task subgraphs into a subset of associated historical tasks; Step S24: Calculate the average, minimum, maximum and variance of the historical job information based on the historical job information of each historical job in the associated historical job subset. Combine the average, minimum, maximum and variance of the historical job information with the number of historical jobs in the associated historical job subset to obtain the historical job subset information. The historical job information in each of the above steps does not include historical job dependencies and scheduling baselines.

4. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1 or 3, characterized in that, In step S2, the historical job subset information includes: the average value of job information of the jobs contained in the historical job subset, the minimum value of job information of the jobs contained in the historical job subset, the maximum value of job information of the jobs contained in the historical job subset, and the variance of job information of the jobs contained in the historical job subset; wherein, the job information does not include job dependencies and scheduling baseline.

5. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1, characterized in that, In step S3, the global information and historical job subset information of the data job platform are input into the job scheduling agent for training using reinforcement learning, resulting in a trained job scheduling agent: Step S31: Input the global information and historical job subset information of the data job platform into the job scheduling agent for processing, obtain the solution order of the sub-models corresponding to the historical job subsets, and output the next sub-model to be solved according to the solution order. Step S32: Solve the sub-model to be solved output in step S31, update the global information of the data operation platform according to the solution results, and determine whether all sub-models have been solved. If not, return to step S31. If yes, obtain the sum of the upper limits of various resource usage by solving the sub-models corresponding to all historical operation subsets, and execute step S33. In step S33, if the sum of the various resource usage limits is greater than the scheduling baseline, the job scheduling agent reduces the joint probability of the solution order under the global information of the current data job platform by adjusting its own parameters; if the sum of the various resource usage limits is less than the scheduling baseline, the job scheduling agent increases the joint probability of the solution order under the global information of the current data job platform by adjusting its own parameters.

6. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1 or 5, characterized in that, In step S3, the global information of the data job platform includes: the set of scheduled job subsets, the total amount of various resources in each time slot of the data job platform, and the usage of various resources in each time slot. The job scheduling agent adopts either an attention model or a graph neural network model, and adjusts its own parameters through short-term policy optimization or policy gradient.

7. The intelligent scheduling method for a resource-constrained data operation platform according to claim 1, characterized in that, Place In step S4, the set of jobs to be scheduled is obtained by acquiring the job information to be scheduled from the data job platform in the following manner: Obtain the set of jobs to be scheduled from the data job platform, and determine the earliest start time, latest start time, job execution time, resources required for the job, job dependencies and resource cost per unit of each job from the set of jobs to be scheduled. The job information to be scheduled does not include the scheduling baseline, which refers to the average value of the sum of the upper limits of various resources used by the data job platform obtained by solving the historical job set multiple times in history.

8. The intelligent scheduling method for a resource-constrained data job platform according to claim 7, characterized in that, Step S5 involves splitting the set of jobs to be scheduled into disjoint subsets of jobs based on the dependencies of the job information in the set of jobs to be scheduled, and aggregating the job information in the subsets of jobs to be scheduled to obtain the job subset information, including: Step S51: Based on the information of the jobs to be scheduled, excluding the dependencies of the jobs to be scheduled, create each job node to be scheduled, and use the information of the jobs to be scheduled, excluding the dependencies of the jobs to be scheduled, corresponding to each job node to be scheduled as the information of each job node to be scheduled. Step S52: Based on the dependencies of the jobs to be scheduled, construct directed edges between the corresponding nodes of the jobs to be scheduled to form a graph of the jobs to be scheduled. Step S53: Based on the connectivity of the job graph to be scheduled, split the job graph to be scheduled into associated job subgraphs, and restore the associated job subgraphs to associated job subsets. Step S54: Based on the information of the scheduled jobs other than the scheduled job dependencies of each scheduled job in the associated scheduled job subset, calculate the average, minimum, maximum and variance of the scheduled job information. Combine the average, minimum, maximum and variance of the scheduled job information with the number of scheduled jobs in the associated scheduled job subset as the scheduled job subset information. Step S6 involves inputting the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent for processing in the following manner: obtaining the optimal solution order of the subset of jobs to be scheduled; iteratively solving the subset of jobs to be scheduled according to the optimal solution order; obtaining the start time of the jobs contained in the subset of jobs; and iteratively updating the global information of the data job platform based on the start time of the contained jobs, including: Step S61: Input the global information of the data job platform and the information of the subset of jobs to be scheduled into the trained job scheduling agent for processing, obtain the optimal solution order of the sub-models corresponding to the subset of jobs to be scheduled, and output the next sub-model to be solved according to the optimal solution order. Step S62: Solve the sub-model to be solved output in step S61 to obtain the start time of the job corresponding to the sub-model. Update the global information of the data job platform according to the start time, execution time and resource usage of the job, and determine whether all sub-models have been solved. If not, return to step S61. If yes, complete the solution of all sub-models.

9. The intelligent scheduling method for a resource-constrained data operation platform according to claim 5, characterized in that, In step S31, the sub-model corresponding to the subset of historical tasks is a mixed integer programming model, and its objective function is: ； The constraints are: ； In the above formula, The actual start time of historical task i in the historical task subset is used as the decision variable of the sub-model; A subset of historical assignments; The unit cost of resource k; This represents the upper limit for the use of resource k; The execution time of historical job i; Dependencies between historical tasks The resource requirement of historical task i for resource k; For all resource types; The earliest start time of historical assignment i This represents the latest start time of historical assignment i.

10. An intelligent scheduling system for a resource-constrained data job platform implementing the method of any one of claims 1-9, characterized in that, include: The system comprises a data acquisition module (21), a job splitting module (22), a job scheduling agent module (23), an agent training module (24), and an iterative solution module (25); among which, The data acquisition module (21) can acquire historical job information from the data job platform to obtain a historical job set, and acquire job information to be scheduled from the data job platform to obtain a job set to be scheduled. The job splitting module (22) is communicatively connected to the data acquisition module (21). It can split the historical job set into non-overlapping historical job subsets according to the dependency relationship of historical job information in the historical job set, and aggregate the historical job information in the historical job subsets to obtain historical job subset information; and split the job set to be scheduled into non-overlapping job subsets according to the dependency relationship of job information in the job set to be scheduled, and aggregate the job information in the job subsets to be scheduled to obtain job subset information. The job scheduling agent module (23) is communicatively connected to the job splitting module (22) and is used to determine the solution order of the historical job subset based on the historical job subset information and the global information of the data job platform; and to determine the solution order of the job subset to be scheduled based on the job subset information and the global information of the data job platform. The agent training module (24) is connected to the job scheduling agent module (23) and can solve the historical job subset according to the solution order of the historical job subset to obtain the scheduling baseline of the historical job set; input the global information of the data job platform, the historical job subset information and the scheduling baseline of the historical job set into the job scheduling agent module (23) for training to obtain the trained job scheduling agent module (23); The iterative solution module (25) is communicatively connected to the agent training module (24). It can input the information of the subset of jobs to be scheduled and the global information of the data job platform into the trained job scheduling agent module for processing, obtain the optimal solution order of the subset of jobs to be scheduled, iteratively solve the subset of jobs to be scheduled according to the optimal solution order, obtain the start time of the jobs contained in the subset of jobs to be scheduled, and iteratively update the global information of the data job platform according to the start time of the jobs contained therein.

Images