Task scheduling methods, devices, equipment, and storage media

Abstract

This disclosure provides a task scheduling method, apparatus, device, and storage medium, relating to artificial intelligence technologies such as cloud computing, big data, and information flow. The task scheduling method includes: creating a first target task; identifying the first target task to obtain its dependent tasks; determining the scheduling time of the first target task; and executing the first target task according to the dependent tasks and the scheduling time. According to this task scheduling method, the dependent tasks of the first target task can be automatically obtained. Therefore, when the number of dependent tasks of the first target task is large, there is no need to manually add the dependent tasks, thus effectively improving the efficiency and accuracy of adding dependent tasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as cloud computing, big data and information flow, specifically to a task scheduling method, a task scheduling device, an electronic device, a computer-readable storage medium and a computer program product. Background Technology

[0002] In related technologies, when configuring and scheduling tasks, the main task is configured first, then each table or task that the main task depends on is manually added in turn, the scheduling time is manually configured, and finally the task is executed according to the scheduling time. However, when the number of tables or tasks that the main task depends on is large, the lookup process becomes cumbersome, resulting in low efficiency and accuracy. Summary of the Invention

[0003] This disclosure provides a task scheduling method, apparatus, device, and storage medium.

[0004] According to a first aspect of this disclosure, a task scheduling method is provided, comprising:

[0005] Create the first objective task;

[0006] Identify the primary objective task and obtain its dependent tasks;

[0007] Determine the scheduling time for the primary objective task;

[0008] Execute the first target task based on dependent tasks and scheduling time.

[0009] According to a second aspect of this disclosure, a task scheduling apparatus is provided, comprising:

[0010] Create a module to create the first target task;

[0011] The first identification module is used to identify the first target task and obtain the dependent tasks of the first target task;

[0012] The determination module is used to determine the scheduling time of the first target task;

[0013] The execution module is used to execute the first target task based on dependent tasks and scheduling time.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] A memory that is communicatively connected to at least one processor; wherein,

[0017] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method of any of the above-described embodiments.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform methods according to any of the embodiments of the above aspects.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method of any of the embodiments described above.

[0020] According to the technical solution disclosed herein, by automatically identifying the first target task, the dependent tasks of the first target task can be automatically obtained. Thus, when the number of dependent tasks of the first target task is large, there is no need to manually add the dependent tasks of the first target task, thereby effectively improving the efficiency and accuracy of adding dependent tasks.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart illustrating a task scheduling method according to an embodiment of the present disclosure;

[0024] Figure 2 This is a flowchart illustrating the process of obtaining the dependent tasks of the first target task according to an embodiment of this disclosure;

[0025] Figure 3 This is a flowchart illustrating the process of determining the scheduling time of the first target task according to an embodiment of this disclosure;

[0026] Figure 4 This is an application example diagram of the task scheduling method according to the embodiments of this disclosure;

[0027] Figure 5 This is an application example diagram illustrating the data input table for obtaining the first target task according to an embodiment of this disclosure;

[0028] Figure 6 This is a structural block diagram of a task scheduling apparatus according to an embodiment of the present disclosure;

[0029] Figure 7This is a block diagram of an electronic device used to implement the task scheduling method of the embodiments of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] like Figure 1 As shown, the task scheduling method according to the first aspect of this disclosure includes:

[0032] Step S110: Create the first target task.

[0033] Step S120: Identify the first target task and obtain the dependent tasks of the first target task.

[0034] Step S130: Determine the scheduling time for the first target task.

[0035] Step S140: Execute the first target task based on the dependent tasks and scheduling time.

[0036] In the fields of big data and cloud computing, the implementation of batch data processing systems and streaming data processing systems requires the automatic creation and execution of tasks. In these systems, there is a dependency relationship between the target task and its dependent tasks; that is, the dependent task is an upstream task of the target task, and the execution of the target task depends on its corresponding dependent task.

[0037] For example, Baidu Data Factory (Pingo) is an integrated batch and streaming data processing system provided on Baidu AI Cloud. The Long Scheduler in Pingo is responsible for task configuration and scheduling. When configuring a scheduled task in the Long Scheduler, a first target task is first created and added to the waiting queue. Then, the dependent tasks of the first target task are automatically identified. Finally, after the data output of the dependent tasks is ready, the first target task is executed according to the scheduled time.

[0038] The identification of the first target task may include performing data analysis and logical analysis on the first target task to obtain the dependency relationship of the first target task, and then obtain the corresponding dependent tasks.

[0039] According to the task scheduling method of this disclosure, by automatically identifying the first target task, the dependent tasks of the first target task can be automatically obtained. Thus, when the number of dependent tasks of the first target task is large, there is no need to manually add the dependent tasks of the first target task, thereby effectively improving the efficiency and accuracy of adding dependent tasks.

[0040] In one implementation, reference Figure 2 In step S120, the first target task is identified to obtain the dependent tasks of the first target task, which may include:

[0041] Step S201: Identify the first target task and obtain the data input table for the first target task.

[0042] Step S202: Obtain the dependent tasks based on the task identifier information corresponding to the data input table.

[0043] By identifying the data input table of the first target task, the task identification information of the data input table can be read from the corresponding metadata. The metadata of the data input table includes the task identification information of the task that produced the data input table. Therefore, the read task identification information is the task identification information of the dependent tasks.

[0044] The task identification information can be the task name or task number, etc.

[0045] For example, the metadata corresponding to the data input table can have a mapping relationship with the data input table and be stored in the metadata database, thereby obtaining the metadata corresponding to the data input table through the metadata database. The metadata corresponding to the data input table can also be directly read from the data input table. Alternatively, the metadata corresponding to the data input table can be partially read directly from the data input table and partially obtained from the metadata database according to the mapping relationship.

[0046] In related technologies, when manually adding dependencies, often only the tables that a task depends on can be found, but the specific task corresponding to that table cannot be determined. However, in some cases, such as when backtracking data and discovering that some partitions of a table are missing, or when a partition of a table exists but has not been fully generated due to multiple inserts, the specific task corresponding to that table can be determined through the metadata database and the pre-established mapping relationship between tables and tasks.

[0047] The above-described embodiments of this disclosure can automatically obtain the data input table that the first target task depends on by identifying the first target task, and then read the task identification information of the dependent tasks from the data input table to automatically obtain the dependent tasks. When there are a large number of data input tables that the first target task depends on, it is not necessary to manually add the data input table for the first target task, thereby improving the efficiency and accuracy of adding the data input table.

[0048] On the other hand, the tasks of the data input table and the output data input table can be automatically associated to establish a mapping relationship between the data input table and the tasks of the output data input table. Thus, the dependent tasks of the first target task can be quickly obtained through the task identification information of the data input table.

[0049] In one implementation, reference Figure 3 In step S130, determining the scheduling time of the first target task may include:

[0050] Step S301: Identify the first target task and obtain the data input table for the first target task.

[0051] Step S302: Determine the scheduling time based on the task execution time corresponding to the data input table.

[0052] By identifying the data input table of the first target task, the task execution time of the data input table can be read from the corresponding metadata. Based on this execution time, the scheduling time of the first target task can be determined. The metadata corresponding to the data input table includes the task execution time of the task that produced the data input table. Therefore, the read task execution time is the execution time of the dependent tasks.

[0053] In related technologies, task scheduling times typically need to be set manually. However, since the completion times of tables or tasks that the task depends on are unknown, manually set times may be unreasonable. For example, after the task scheduling begins, it may take a long time for the tasks that the task depends on to complete. During this time, the task occupies queue resources until the dependent tasks are completed, resulting in low resource utilization.

[0054] In addition to automatically obtaining the data input table dependent on the first target task by identifying the first target task to improve the efficiency and accuracy of adding the data input table, the above-described embodiments of this disclosure can also automatically associate the data input table with the task that produces the data input table, thereby establishing a mapping relationship between the data input table and the task that produces the data input table. Then, the task execution time of the dependent task can be quickly obtained through the task execution time of the data input table, thereby automatically generating a reasonable scheduling time and effectively improving production efficiency and resource utilization.

[0055] In one implementation, determining the scheduling time in step S302 based on the task execution time corresponding to the data input table may include: determining the scheduling time based on preset time logic conditions and the task execution time corresponding to the data input table, wherein the scheduling time is later than the task execution time.

[0056] For example, based on the data input table obtained from identifying the first target task, the execution time of the task that produced the data input table can be read from the metadata, which is the execution time of the dependent tasks of the first target task. Based on preset time logic conditions and the execution times of dependent tasks, the recommended start and end times for the first target task can be given, and the scheduling time can be automatically generated.

[0057] For example, if the task of reading the output data input table from the metadata completes at 9:03:40, and the dependent task of the first target task also completes at 9:03:40, it can be determined that the scheduling time of the first target task is later than 9:03:40 to ensure that the first target task is scheduled after the dependent task has completed its execution. Based on preset time logic conditions, recommended start and end times for the first target task can be given, such as a recommended start time of 9:03:41 or 9:03:45, etc., thereby automatically generating a scheduling time and executing the first target task according to the scheduling time.

[0058] Therefore, the above settings can further ensure the rationality of the start and end times of the first target task, avoid the first target task occupying queue resources before the dependent tasks are completed, and thus further improve resource utilization.

[0059] In one implementation, combined with Figure 5 In steps S201 and S301, the first target task is identified to obtain a data input table for the first target task. This may include: parsing the first target task using Structured Query Language (SQL) to obtain a corresponding logical plan; and parsing the logical plan to obtain a data input table for the first target task.

[0060] For example, in batch and streaming data processing systems, when running the computation engine (Spark), the primary target task is parsed into a syntax tree, generating a logical plan. The syntax tree includes multiple tables, and during syntax tree generation, these tables can be associated with the tasks that produce them. During the parsing of the logical plan, the corresponding data input tables can be obtained by editing code and extracting data from multiple tables according to predetermined strategies. For example, in... Figure 5In the example, after parsing the logical plan, two data input tables, a and b, are obtained.

[0061] Therefore, by parsing the SQL of the first target task, the data input table of the first target task can be automatically identified, eliminating the need for manual searching and further improving the efficiency of adding the data input table of the first target task.

[0062] In one implementation, reference Figure 4 The task scheduling method further includes: identifying a first target task and obtaining a data output table for the first target task, wherein the data output table for the first target task is a data input table for at least one second target task; mapping the task identification information of the first target task, the table identification information of the data output table, and the task execution time of the first target task to the data output table and storing them in a metadata database.

[0063] For example, when running the Spark computing engine, the first target task is parsed into a syntax tree, generating a logical plan. Parsing the logical plan yields an input table and an output table. The output table is the result table. By executing the first target task, the data in the input table is adjusted and processed to obtain the target data, and the final output table is the result table.

[0064] Therefore, the task identifier information of the first target task, the table identifier information of the data output table, and the task execution time of the first target task can be used as metadata to establish a mapping relationship with the data output table and stored in the metadata database. Thus, when the data output table serves as the data input table for the second target task, metadata information such as the dependent tasks of the second target task can be obtained by calling the metadata database.

[0065] The table identifier information of the data output table can be the name or number of the data output table.

[0066] The data output table of the first objective task can be used as the data input table of the second objective task; that is, the first objective task can be a dependent task of the second objective task. Therefore, referring to the task scheduling method for the first objective task described above, when creating the second objective task, by identifying the second objective task, the dependent task of the second objective task, i.e., the first objective task, can be obtained; simultaneously, the execution time of the first objective task can be obtained, and then the scheduling time of the second objective task can be determined based on the execution time of the first objective task; finally, the second objective task can be executed according to the scheduling times of the first and second objective tasks.

[0067] In this way, on the one hand, the data output table of the first target task can be automatically obtained by identifying the first target task; on the other hand, the task identification information of the first target task, the identification information of the data output table, and the task execution time of the first target task can be bound and stored. Thus, when scheduling the second target task, the data output table of the first target task can be used as the data input table of the second target task, which is convenient for providing the second target task with metadata information such as dependent tasks (first target task) and the task execution time of dependent tasks.

[0068] like Figure 6 As shown, the task scheduling apparatus 600 according to a second aspect embodiment of the present disclosure includes:

[0069] Create module 610 to create the first target task;

[0070] The first identification module 620 is used to identify the first target task and obtain the dependent tasks of the first target task;

[0071] Module 630 is used to determine the scheduling time of the first target task;

[0072] Execution module 640 is used to execute the first target task based on dependent tasks and scheduling time.

[0073] In one embodiment, the first identification module 620 includes:

[0074] The first identification submodule is used to identify the first target task and obtain the data input table of the first target task;

[0075] The generation submodule is used to generate dependent tasks based on the task identifier information corresponding to the data input table.

[0076] In one implementation, the determining module 630 includes:

[0077] The first identification submodule is used to identify the first target task and obtain the data input table of the first target task;

[0078] The determination submodule is used to determine the scheduling time based on the task execution time corresponding to the data input table.

[0079] In one implementation, the determining submodule is further configured to:

[0080] The scheduling time is determined based on the preset time logic conditions and the task execution time corresponding to the data input table, wherein the scheduling time is later than the task execution time.

[0081] In one implementation, the first identification submodule is further configured to:

[0082] The first objective task is parsed using Structured Query Language (SQL) to obtain the corresponding logical plan;

[0083] The logical plan is analyzed to obtain the data input table for the first objective task.

[0084] In one embodiment, the task scheduling device 600 further includes:

[0085] The second identification module is used to identify the first target task and obtain a data output table of the first target task, wherein the data output table of the first target task is a data input table of at least one second target task.

[0086] The corresponding module is used to map the task identification information of the first target task, the table identification information of the data output table, and the task execution time of the first target task to the data output table, and store them in the metadata database.

[0087] The functions and effects of each module or sub-module in the various devices of this disclosure embodiment can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0088] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0089] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0090] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0091] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0092] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as task scheduling methods. For example, in some embodiments, the task scheduling method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform task scheduling methods by any other suitable means (e.g., by means of firmware).

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0099] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0100] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task scheduling method, comprising: In batch data processing systems and streaming data processing systems, create the first target task; The first target task is identified to obtain the data input table for the first target task; The data input table is obtained based on the syntax tree of the SQL statement of the first target task; Based on the task identifier information corresponding to the data input table, the dependent tasks of the first target task are obtained from the metadata; Determine the scheduling time for the first target task; The scheduling time is later than the execution completion time of the dependent task; The first target task is executed based on the dependent tasks and the scheduling time.

2. The method according to claim 1, wherein, Determining the scheduling time of the first target task includes: The first target task is identified to obtain the data input table for the first target task; The scheduling time is determined based on the task execution time corresponding to the data input table.

3. The method according to claim 2, wherein, The scheduling time is determined based on the task execution time corresponding to the data input table, including: The scheduling time is determined based on preset time logic conditions and the task execution time corresponding to the data input table, wherein the scheduling time is later than the task execution time.

4. The method according to any one of claims 1-3, wherein, The first target task is identified to obtain a data input table for the first target task, including: The first target task is parsed using Structured Query Language (SQL) to obtain the corresponding logical plan; The logical plan is parsed to obtain the data input table for the first target task.

5. The method according to claim 1, further comprising: The first target task is identified to obtain a data output table for the first target task, wherein the data output table for the first target task is a data input table for at least one second target task; The task identifier information of the first target task, the table identifier information of the data output table, and the task execution time of the first target task are mapped to the data output table and stored in the metadata database.

6. A task scheduling device, comprising: Create a module to create the first target task in batch data processing systems and streaming data processing systems; The first identification module includes a first identification submodule, used to identify the first target task and obtain a data input table for the first target task; The data input table is obtained based on the syntax tree of the SQL statement of the first target task; The generation submodule is used to generate dependent tasks based on the task identifier information corresponding to the data input table; The determination module is used to determine the scheduling time of the first target task; The scheduling time is later than the execution completion time of the dependent task; An execution module is used to execute the first target task based on the dependent tasks and the scheduling time.

7. The apparatus according to claim 6, wherein, The determining module includes: The first identification submodule is used to identify the first target task and obtain the data input table of the first target task; The determination submodule is used to determine the scheduling time based on the task execution time corresponding to the data input table.

8. The apparatus according to claim 7, wherein, The Determine submodule is also used for: The scheduling time is determined based on preset time logic conditions and the task execution time corresponding to the data input table, wherein the scheduling time is later than the task execution time.

9. The apparatus according to any one of claims 6-8, wherein, The first identification submodule is also used for: The first target task is parsed using Structured Query Language (SQL) to obtain the corresponding logical plan; The logical plan is parsed to obtain the data input table for the first target task.

10. The apparatus according to claim 6, further comprising: The second identification module is used to identify the first target task and obtain a data output table of the first target task, wherein the data output table of the first target task is a data input table of at least one second target task. The corresponding module is used to map the task identification information of the first target task, the table identification information of the data output table, and the task execution time of the first target task to the data output table, and store them in the metadata database.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein In this context, the computer instructions are used to cause the computer to perform any of the following actions according to claims 1-5. One method of the method.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images