Task scheduling method, device and equipment, computer storage medium and program product

By building a mapping table and task scheduling model between data center and data files, and optimizing task allocation, the problem of low task scheduling efficiency in cross-data center environments is solved, and task execution efficiency and resource utilization are improved.

CN118981364APending Publication Date: 2024-11-19CHINA MOBILE INFORMATION TECHNOLOGY CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411046859.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing task scheduling methods fail to effectively consider the dependencies between tasks and data call locations in cross-data center environments, resulting in low task execution efficiency.

Method used

By obtaining job information, data file information and data center information, a mapping table between data center and data file is constructed, a task collection is generated, a task scheduling model with the optimization goal of minimizing the longest task completion time in the data center, and the optimization solution model is Determine the best task allocation plan.

Benefits of technology

It realizes the optimization of task scheduling, improves resource utilization and task execution efficiency, balances the data center load, and avoids resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981364A_ABST
    Figure CN118981364A_ABST
Patent Text Reader

Abstract

The invention discloses a task scheduling method, device and equipment, a computer storage medium and a program product. The method comprises the steps of obtaining task information, data file information and data center information; constructing a mapping table of a data center and a data file storage position based on the data file information and the data center information; generating a task set based on the task information; based on the task set, the mapping table, the data file information and the data center information, a target function is constructed with minimization of data center task completion time as an optimization target, constraint conditions are set, a task scheduling model is established, and decision variables of the task scheduling model are data centers distributed by tasks; calculating an optimal solution of the target function under a set constraint condition; and determining a task scheduling scheme according to the optimal solution of the objective function, and scheduling the tasks to each data center. According to the embodiment of the invention, the dependency relationship between the tasks and the data calling position are comprehensively considered during task scheduling, so that the task execution efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular relates to a task scheduling method, device, equipment, computer storage medium, and program product. Background Art

[0002] In recent years, due to the growing demand for data centers and storage analysis, cloud service providers have established many data centers around the world to meet the growing and diverse needs of users. By distributing data centers in various places, cloud providers can not only maximize the opportunity for customers to access data reliably, but also reduce the waiting time perceived by users for access.

[0003] Existing task scheduling and resource allocation methods often schedule the resources of a centralized cloud computing center. All resources are stored in the centralized cloud computing center and can be directly called, without involving resource allocation and scheduling across data centers. When it comes to cross-data center task scheduling, existing methods often assign the task to the data center where the data file is located, or after assigning the task execution data center, randomly select the data file where the data file is located to obtain the data file, without considering the dependency between tasks and the data call location, resulting in low task execution efficiency. Summary of the invention

[0004] The embodiments of the present application provide a task scheduling method, apparatus, device, computer storage medium, and program product to solve the problem that the prior art does not take into account the dependencies between tasks and the data call locations, resulting in low task execution efficiency.

[0005] In a first aspect, an embodiment of the present application provides a task scheduling method, including:

[0006] Obtaining job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes the data file storage location, the data file computing load size, and the data file size, and the data center information includes the bandwidth information between data centers and the computing capacity value of the data center;

[0007] A mapping table between data centers and data file storage locations is constructed based on the data file storage location and bandwidth information between data centers;

[0008] Generate task sets based on scheduled tasks and task dependencies;

[0009] Based on the task set, mapping table, data file calculation load size, data file size, data center computing capacity value and bandwidth information between data centers, the objective function is constructed with minimizing the longest task completion time of the data center as the optimization goal, and constraints are set to establish a task scheduling model. Among them, the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution;

[0010] Optimize and solve the task scheduling model to obtain the minimum value of the longest task completion time in the data center under the constraint conditions of the objective function;

[0011] The tasks are scheduled to each data center based on the data center to which the tasks are assigned corresponding to the minimum value of the longest task completion time in the data center.

[0012] In a second aspect, an embodiment of the present application provides a task scheduling device, the device comprising:

[0013] An acquisition module is used to acquire job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes data file storage location, data file computing load size, and data file size, and the data center information includes bandwidth information between data centers and data center computing capacity values;

[0014] A mapping module, used to construct a mapping table between data centers and data file storage locations based on data file storage locations and bandwidth information between data centers;

[0015] A generation module is used to generate a task set based on the scheduled tasks and task dependencies;

[0016] A construction module is used to calculate the load size, data file size, data center computing capacity value and bandwidth information between data centers based on the task set, mapping table and data file, and to construct an objective function and set constraints with minimizing the longest task completion time of the data center as the optimization goal, so as to establish a task scheduling model, wherein the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution;

[0017] The calculation module is used to optimize and solve the task scheduling model and obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function;

[0018] The scheduling module is used to schedule tasks to various data centers according to the data center to which the task is assigned corresponding to the minimum value of the longest task completion time in the data center.

[0019] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the task scheduling method of the first aspect is implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the task scheduling method of the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the task scheduling method of the first aspect.

[0022] The task scheduling method, apparatus, device, computer storage medium, and program product provided in the embodiments of the present application obtain job information, data file information, and data center information, wherein the job information includes the scheduling task and task dependency, the data file information includes the data file storage location, the data file computing load size, and the data file size, and the data center information includes the bandwidth information between data centers and the computing capacity of the data center; a mapping table between the data center and the data file storage location is constructed based on the data file storage location and the bandwidth information between the data centers, and the location of the data file called by each data center can be determined by constructing the mapping table; a task set is generated based on the scheduling task and the task dependency, and the tasks with dependencies are divided to ensure that the tasks are correctly executed according to the dependencies, thereby simplifying the complexity of task scheduling; based on The task set, mapping table, data file calculation load size, data file size, data center computing capacity value and bandwidth information between data centers are used to construct the objective function and set constraints with the optimization goal of minimizing the longest task completion time of the data center, establish a task scheduling model, and transform the scheduling problem into a mathematical problem; optimize and solve the task scheduling model to obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function, so as to find the best task scheduling plan, balance the load of the data center, and avoid resource waste; according to the data center where the task is assigned to the minimum value of the longest task completion time of the data center, the task is scheduled to each data center, realizing the optimization of task scheduling, and improving resource utilization and task execution efficiency by comprehensively considering the dependency relationship between tasks and the data call location. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 It is a schematic diagram of a system framework structure that can implement the task scheduling method provided in the embodiment of the present application;

[0025] Figure 2 It is a flowchart of a task scheduling method provided in an embodiment of the present application;

[0026] Figure 3 is a structural diagram of a task scheduling device provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0030] In the prior art, most of the research work on cross-data center resource scheduling methods focuses on the task scheduling of cloud computing data centers, and does not involve cross-regional geographical distribution. However, for the scenario of cross-data center collaborative computing in big data services, multiple data centers across geographical distribution store a large amount of data, and the physical distance between data centers is relatively far, and the network bandwidth resources connecting them are limited, so that the scheduling algorithms that perform well for centralized data centers cannot be effectively applied between geographically distributed data centers; due to the heterogeneity of geographically distributed data centers and the particularity of cross-geographically distributed data center scheduling, MapReduce, Spark and other architectures that perform large-scale data-intensive operations on clusters are not suitable for direct use, and even with targeted adjustments, performance will inevitably degrade severely. In addition, after cross-domain operations are decomposed into tasks assigned to different data centers, there are complex dependencies between tasks, which is manifested as a specific order of execution of tasks. For the mathematical abstraction of this problem, the constraint expression is the emergence of a large number of nonlinear constraints. Especially for big data business, cross-domain big data computing jobs often need to collect and process a large amount of data. These data are often collected from different data sources, stored in different data centers, have different formats and qualities, and require different preprocessing and aggregation operations. The constraints generated by the corresponding abstract mathematical problems are extremely complex. Therefore, how to reasonably and effectively allocate computing jobs to various data center sites to make full use of the locality or proximity of data, thereby improving the efficiency of job completion and effectively utilizing computing center resources has become a very important issue in the industry.

[0031] In order to solve the problems of the prior art, the embodiments of the present application provide a task scheduling method, apparatus, device, computer storage medium, and program product to obtain job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes the data file storage location, the data file computing load size, and the data file size, and the data center information includes the bandwidth information between data centers and the computing capacity of the data center; a mapping table between the data center and the data file storage location is constructed based on the data file storage location and the bandwidth information between the data centers, and the location of the data file called by each data center can be determined by constructing the mapping table; a task set is generated based on the scheduling tasks and task dependencies, and tasks with dependencies are divided to ensure that tasks are correctly executed according to the dependencies, thereby simplifying task scheduling. complexity; based on the task set, mapping table, data file calculation load size, data file size, data center computing capacity value and bandwidth information between data centers, the objective function is constructed with the minimization of the longest task completion time of the data center as the optimization goal and constraints are set to establish a task scheduling model, and the scheduling problem is converted into a mathematical problem; the task scheduling model is optimized and solved to obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function, so as to find the best task scheduling solution, balance the load of the data center and avoid resource waste; according to the data center to which the task is assigned corresponding to the minimum value of the longest task completion time of the data center, the task is scheduled to each data center, and the task scheduling is optimized. By comprehensively considering the dependency relationship between tasks and the data call location, the resource utilization and task execution efficiency are improved.

[0032] The following first provides an example of a system framework to which the task scheduling method provided in the embodiment of the present application can be applied.

[0033] like Figure 1 As shown, the system framework may include: a scheduling server and multiple data centers, the scheduling server and the data centers are communicatively connected, and each data center may include multiple sub-scheduling servers.

[0034] The scheduling receives jobs submitted by users based on the message queue, manages the job queue using the first-in-first-out strategy and job priority, decomposes the job into a series of subtasks, and assigns the tasks to each data center for execution through a preset algorithm. The data center stores the data files required to execute the tasks and is used to execute the tasks contained in the tasks, such as Figure 1 As shown in , the scheduling server allocates tasks A1 and B1, A2 and B2, and B3 to data center 1, data center 2, and data center 3 respectively.

[0035] Each data center has available computing resources, that is, computing processing power, and can store big data. Data is stored in data centers in various locations. Considering cost constraints, some data, especially frequently accessed data, has multiple copies in data centers in different locations, but not all data has multiple copies in different data centers. Data centers can communicate and access the data files required to perform tasks.

[0036] The following is an introduction to the task scheduling method provided in the embodiment of the present application.

[0037] Figure 2 FIG. 1 is a flow chart showing a task scheduling method provided by an embodiment of the present application. Figure 2 As shown, the method may include the following steps: S201 to S206.

[0038] S201. Obtain job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes the data file storage location, the data file computing load size, and the data file size, and the data center information includes the bandwidth information between data centers and the data center computing capacity value.

[0039] Among them, the data file calculation load size is the size of the processing load or calculation load of the data file during the calculation process, which depends on factors such as the amount of data contained in the data file, the structure of the data, the calculation operations that need to be performed, etc. The data file size is the space actually occupied by the data file content on the storage medium.

[0040] In some embodiments, when scheduling a job, the job may be decomposed into a group of independent tasks according to the required data and business nature, and then allocated to each data center node for execution.

[0041] S202: Construct a mapping table between data centers and data file storage locations based on the data file storage location and bandwidth information between data centers.

[0042] By constructing a mapping table between data centers and data file storage locations, data files stored in different locations can be called according to the bandwidth information between data centers, thereby maximizing bandwidth utilization and improving data transmission efficiency.

[0043] S203: Generate a task set based on the scheduled tasks and task dependencies.

[0044] Among them, when there are dependencies between scheduling tasks, the target task can only be executed after the predecessor task is completed. Therefore, the scheduling tasks with dependencies are combined into a task set. The dependent tasks can be scheduled separately, simplifying the complexity of the scheduling process.

[0045] S204. Based on the task set, mapping table, data file calculation load size, data file size, data center computing capacity value and bandwidth information between data centers, the objective function is constructed with minimizing the longest task completion time of the data center as the optimization goal, and constraints are set to establish a task scheduling model, wherein the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution.

[0046] In some embodiments, the objective function may also be set to minimize the duration of a single job, and the completion time of the job is determined by the end time of the last completed task belonging to the job executed on any data center.

[0047] When building a scheduling model, based on task sets and mapping tables, the dependencies between tasks and the locations of data calls are comprehensively considered, which can help maximize the resource utilization of the data center and improve the efficiency of task execution.

[0048] S205, optimizing and solving the task scheduling model to obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function.

[0049] In some embodiments, when optimizing and solving the task scheduling model, a genetic algorithm, a Bayesian optimization algorithm, a particle swarm optimization algorithm, etc. may be used to solve the optimal solution of the objective function.

[0050] S206. Schedule the tasks to the data centers according to the data centers to which the tasks are assigned corresponding to the minimum values ​​of the longest task completion time in the data centers.

[0051] In some embodiments, tasks assigned to the same data center are executed in a first-in-first-out order.

[0052] The task scheduling method of the embodiment of the present application constructs a mapping table between data centers and data file storage locations based on the data file storage location and the bandwidth information between data centers, so as to determine the location of each data center calling the data file; generates a task set based on the scheduling task and the task dependency relationship, and divides the tasks with dependencies, so as to ensure that the tasks are correctly executed according to the dependencies and simplify the complexity of task scheduling; by establishing a task scheduling model that constructs an objective function with minimizing the longest task completion time of the data center as the optimization goal, the scheduling problem is converted into a mathematical problem; the task scheduling model is optimized and solved to obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function, so as to obtain the task scheduling plan with the highest execution efficiency, balance the load of the data center, and avoid resource waste; according to the data center to which the task is assigned corresponding to the minimum value of the longest task completion time of the data center, the task is scheduled to each data center, so as to achieve the optimization of task scheduling, and improve the resource utilization and task execution efficiency by comprehensively considering the dependency relationship between tasks and the data call location.

[0053] In some embodiments, a mapping table between data centers and data file storage locations is constructed based on the storage locations of data files in each data center and bandwidth information between data centers, including:

[0054] Based on the storage location of the data files in each data center and the bandwidth information between the data centers, the storage data center is selected when the target data center obtains the data file, where the storage data center is the data center with the largest bandwidth from the target data center to store the required data files;

[0055] A mapping table is constructed according to the target data center and the storage data center, wherein each element represents the storage data center location of the data file required for the target data center to obtain the task.

[0056] By selecting the storage data center with the largest bandwidth to the target data center for storing the required data files, it can ensure that bandwidth resources are fully utilized during data transmission, maximize data transmission efficiency and speed, generate a mapping table to indicate the data acquisition location, and optimize the data access path.

[0057] In some embodiments, when there are three geographically distributed data center sites, ie, DC={DC1, DC2, DC3}, the bandwidth matrix between the three data centers DC1, DC2 and DC3 is shown in Table 1.

[0058] Table 1 Data center bandwidth table

[0059]

[0060]

[0061] In some embodiments, the storage location of the data file, that is, at which data center sites the data file is located, can be represented by a binary state variable L i,j k When the value is 1, it indicates that the jth subtask T of the i-th job j i The data file accessed is D j i , at the data center site DC k A copy exists on the data center site DC. k Data file D does not exist on j i A copy of .

[0062] When there are 10 data files stored in three data centers, the data file storage locations are shown in Table 2, where f i The number of the data file.

[0063] Table 2 Data file storage location table

[0064] DC1 DC2 DC3 f1,f2 f3,f4,f5 - f7 f6 f8,f9,f10 - f1 - f3 f8 f2 - - f5

[0065] In some embodiments, the task prioritizes accessing data from an available data center with large bandwidth. That is, if there are multiple copies of the data required for the task stored in multiple data centers with different geographical locations, the site with the largest network bandwidth (fastest data transmission speed) between the site and the data center where the task is executed is selected to read and access the required data files. If the required data files are located locally, that is, in the data center where the task is executed, that data center is directly selected.

[0066] When the data in Table 1 and Table 2 are used to calculate and generate a mapping table, the mapping table is as shown in Table 3.

[0067] Table 3 Mapping table

[0068]

[0069] In some embodiments, generating a task set based on the scheduled tasks and task dependencies includes:

[0070] When there are dependencies between tasks in the scheduled task, the dependent tasks are combined into unit tasks;

[0071] Unit tasks and other tasks that have no dependencies are organized into task sets.

[0072] Among them, the marked unit task is a combination of tasks with a dependency constraint relationship, and the tasks in the unit task are assigned to the same data center for execution.

[0073] Combining tasks with dependencies into unit tasks and decomposing the task scheduling problem into smaller units can effectively reduce the complexity of task scheduling and make the scheduling algorithm more efficient and easy to implement.

[0074] In some embodiments, when there are dependencies between tasks in the scheduled tasks, the tasks with dependencies are combined into a unit task, including:

[0075] Divide the tasks with dependencies into multiple layers according to the dependencies, where the completion of each layer of tasks is the prerequisite for the start of the next layer of tasks;

[0076] The multi-layer tasks are input into the task set as unit tasks according to the order of task hierarchy.

[0077] By dividing tasks with dependencies into multiple layers of tasks and executing them according to the hierarchical relationship, we can ensure that tasks are executed in the correct order and conditions, improve the reliability of task execution, reduce the waiting time between tasks, and improve the efficiency of task execution.

[0078] In some embodiments, the decision variables of the task scheduling model can be represented by Boolean variables, that is, Indicates whether the jth task of the ith job is assigned to the dth data center site. If it is assigned to the dth data center, it takes 1, and if it is not assigned to the data center, it takes 0. The state space formed by the values ​​of all decision variables is Among them, DC d represents the data center numbered d; T j i represents the jth subtask of the i-th job, and T is the set of all tasks to be scheduled.

[0079] In some embodiments, the decision variables of the task scheduling model can also be represented by integer variables, that is, S i,j ∈{1,2,…,N}, which indicates to which data center the jth task of the ith job is assigned, N is the number of sites in the data center, and the state space formed by the values ​​of all decision variables is

[0080] In some embodiments, the objective function is:

[0081]

[0082] Among them, E max is the maximum task completion time of the data center, is the jth subtask of the ith job, T is the set of all tasks to be scheduled, Assign the jth subtask of the i-th job to the d-th data center site, represents the computing time required to execute the jth subtask of the ith job in data center d, Calculate the load size for the jth task of the i-th job, C d Calculate the capacity value for the dth data center, represents the network transmission time of the data required by the jth subtask of the i-th job, B d is the network bandwidth between the target data center and the storage data center, is the size of the data file accessed by the jth subtask of the ith job, DC d is the data center numbered d, and DC is the data center set.

[0083] When the jth subtask of the ith job is assigned to data center d, its execution time includes the computational execution time of the task and the network transmission time for obtaining the data file. When the data file is in the local data center, the network transmission time is 0.

[0084] This objective function minimizes the maximum task completion time in the data center by considering the execution status of all tasks in all data centers, which helps to balance the load of each data center and avoid long waiting or idleness of the data center due to uneven resource allocation.

[0085] In some embodiments, the constraints include:

[0086] A task can only be assigned to one data center for execution, which is expressed as:

[0087]

[0088] in, Assign the jth subtask of the ith job to the dth data center site, T i i is the jth subtask of the ith job, T is the set of all tasks to be scheduled, DC d is the data center numbered d, and DC is the data center set.

[0089] By adding the constraint that a task can only be assigned to one data center for execution, the complexity of task scheduling is simplified, the additional network and computing overhead caused by task migration is reduced, and competition for the same task between different data centers is avoided, reducing resource competition conflicts.

[0090] In some embodiments, when calling data files from other data centers, the data needs to be copied from the data center that calls the data file to the local data center. After the corresponding data file is copied locally, the storage location of the data file in each data center and the data content in the mapping table are updated.

[0091] Figure 3 Schematic diagram of a task scheduling device provided in an embodiment of the present application. Figure 3 As shown, the device may include an acquisition module 301 , a mapping module 302 , a generation module 303 , a construction module 304 , a calculation module 305 and a scheduling module 306 .

[0092] The acquisition module 301 is used to acquire job information, data file information, and data center information, wherein the job information includes the scheduling tasks and task dependencies, the data file information includes the data file storage location, the data file computing load size, and the data file size, and the data center information includes the bandwidth information between data centers and the data center computing capacity value;

[0093] A mapping module 302, for constructing a mapping table between data centers and data file storage locations based on data file storage locations and bandwidth information between data centers;

[0094] A generation module 303, used to generate a task set based on the scheduled tasks and task dependencies;

[0095] A construction module 304 is used to calculate the load size, data file size, data center computing capacity value and bandwidth information between data centers based on the task set, mapping table and data file, construct an objective function with minimizing the longest task completion time of the data center as the optimization goal and set constraints to establish a task scheduling model, wherein the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution;

[0096] The calculation module 305 is used to optimize and solve the task scheduling model to obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function;

[0097] The scheduling module 306 is used to schedule tasks to each data center according to the data center to which the task is assigned corresponding to the minimum value of the longest task completion time of the data center.

[0098] In some embodiments, the acquisition module is further used to select a storage data center when the target data center acquires the data file based on the storage location of the data file in each data center and the bandwidth information between the data centers, wherein the storage data center is the data center with the largest bandwidth from the target data center for storing the required data file;

[0099] The construction module is also used to construct a mapping table according to the target data center and the storage data center, wherein each element represents the storage data center location of the data file required for the target data center to obtain the task.

[0100] In some embodiments, the construction module is further used to combine the tasks with dependencies into a unit task when there are dependencies among the tasks in the scheduled task;

[0101] The building module is also used to build unit tasks and other tasks without dependencies into task sets.

[0102] In some embodiments, the generation module is further used to divide the tasks with dependency relationships into multiple layers of tasks according to the dependency relationships, wherein the completion of each layer of tasks is a prerequisite for the start of the next layer of tasks;

[0103] The generation module is also used to input the multi-layer tasks into the task set as unit tasks in the order of task level division.

[0104] In some embodiments, the objective function set by the building block is:

[0105]

[0106] Among them, E max is the maximum task completion time of the data center, T i i is the jth subtask of the ith job, T is the set of all tasks to be scheduled, Assign the jth subtask of the i-th job to the d-th data center site, represents the computing time required to execute the jth subtask of the i-th job in data center d, and wji is the computing load of the jth task of the i-th job; represents the network transmission time of the data required by the jth subtask of the i-th job, B d is the network bandwidth between the target data center and the storage data center, is the size of the data file accessed by the jth subtask of the ith job, DC d is the data center numbered d, and DC is the data center set.

[0107] In some embodiments, the constraints set by the building blocks include:

[0108] A task can only be assigned to one data center for execution, which is expressed as:

[0109]

[0110] in, Assign the jth subtask of the ith job to the dth data center site, T i i is the jth subtask of the ith job, T is the set of all tasks to be scheduled, DC d is the data center numbered d, and DC is the data center set.

[0111] Figure 3 The various modules in the device shown can be implemented Figure 2 Each step in the above steps can achieve the corresponding technical effect, which will not be repeated here for the sake of brevity.

[0112] Figure 4 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application is shown.

[0113] The terminal device may include a processor 401 and a memory 402 storing computer program instructions.

[0114] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0115] The memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 402 may include a removable or non-removable (or fixed) medium, or the memory 402 is a non-volatile solid-state memory. The memory 402 may be inside or outside the integrated gateway disaster recovery device.

[0116] In one example, the memory 402 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present disclosure.

[0117] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement Figure 2 The operation log association method in the illustrated embodiment.

[0118] In one example, the terminal device may further include a communication interface 403 and a bus 404. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 404 and communicate with each other.

[0119] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0120] Bus 404 includes hardware, software or both, and the components of terminal equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), Hyper Transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 404 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0121] In addition, in combination with the task scheduling method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the task scheduling methods in the above embodiment is implemented.

[0122] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the task scheduling methods in the above embodiments when the computer program is processed and executed.

[0123] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0124] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0125] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0126] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0127] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A task scheduling method, characterized in that: include: Obtaining job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes data file storage location, data file computing load size, and data file size, and the data center information includes bandwidth information between data centers and data center computing capacity values; A mapping table between data centers and data file storage locations is constructed based on the data file storage location and bandwidth information between data centers; Generate task sets based on scheduled tasks and task dependencies; Based on the task set, mapping table, data file calculation load size, data file size, data center computing capacity value and bandwidth information between data centers, the objective function is constructed with minimizing the longest task completion time of the data center as the optimization goal, and constraints are set to establish a task scheduling model. Among them, the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution; Optimize and solve the task scheduling model to obtain the minimum value of the longest task completion time in the data center under the constraint conditions of the objective function; The tasks are dispatched to each data center according to the data center to which the tasks are assigned corresponding to the minimum value of the longest task completion time of the data center.

2. The task scheduling method according to claim 1, characterized in that: The mapping table between data centers and data file storage locations is constructed based on the storage locations of data files in each data center and the bandwidth information between data centers, including: Based on the storage location of the data files in each data center and the bandwidth information between the data centers, the storage data center is selected when the target data center obtains the data file, where the storage data center is the data center with the largest bandwidth from the target data center to store the required data files; A mapping table is constructed according to the target data center and the storage data center, wherein each element represents the storage data center location of the data file required for the target data center to obtain the task.

3. The task scheduling method according to claim 1, characterized in that: The generating of a task set based on the scheduled tasks and task dependencies includes: When there are dependencies between tasks in the scheduled task, the dependent tasks are combined into unit tasks; The unit task and other tasks that have no dependency relationship are constructed into a task set.

4. The task scheduling method according to claim 3, characterized in that: When there are dependencies between the tasks in the scheduling task, the tasks with dependencies are combined into a unit task, including: Divide the tasks with dependencies into multiple layers according to the dependencies, where the completion of each layer of tasks is the prerequisite for the start of the next layer of tasks; The multi-layer tasks are input into a task set as unit tasks according to the task level division order.

5. The task scheduling method according to claim 1, characterized in that: The objective function is: Among them, E max is the maximum task completion time of the data center, is the jth subtask of the ith job, T is the set of all tasks to be scheduled, Assign the jth subtask of the i-th job to the d-th data center site, represents the computing time required to execute the jth subtask of the ith job in data center d, Calculate the load size for the jth task of the i-th job, C d Calculate the capacity value for the dth data center, represents the network transmission time of the data required by the jth subtask of the i-th job, B d is the network bandwidth between the target data center and the storage data center, is the size of the data file accessed by the jth subtask of the ith job, DC d is the data center numbered d, and DC is the data center set.

6. The task scheduling method according to claim 1, characterized in that: The above-mentioned task can only be assigned to one data center for execution, including: in, Assign the jth subtask of the i-th job to the d-th data center site, is the jth subtask of the ith job, T is the set of all tasks to be scheduled, DC d is the data center numbered d, and DC is the data center set.

7. A task scheduling device, characterized in that: The device comprises: An acquisition module, used to acquire job information, data file information, and data center information, wherein the job information includes scheduling tasks and task dependencies, the data file information includes data file storage location, data file computing load size, and data file size, and the data center information includes bandwidth information between data centers and data center computing capacity values; A mapping module, used to construct a mapping table between data centers and data file storage locations based on data file storage locations and bandwidth information between data centers; A generation module is used to generate a task set based on the scheduled tasks and task dependencies; A construction module is used to calculate the load size, data file size, data center computing capacity value and bandwidth information between data centers based on the task set, mapping table and data file, and to construct an objective function and set constraints with minimizing the longest task completion time of the data center as the optimization goal, so as to establish a task scheduling model, wherein the decision variable of the task scheduling model is the data center to which the task is assigned, and the constraint condition is that a task can only be assigned to one data center for execution; The calculation module is used to optimize and solve the task scheduling model and obtain the minimum value of the longest task completion time of the data center under the constraint conditions of the objective function; The scheduling module is used to schedule tasks to each data center according to the data center to which the task is assigned corresponding to the minimum value of the longest task completion time of the data center.

8. A terminal device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the task scheduling method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the task scheduling method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the task scheduling method as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Computing task execution method and device, equipment and storage medium

    CN119322768A