Task scheduling method and device, computing device, storage medium and program product

Abstract

The application discloses a task scheduling method, comprising: obtaining a target task and current resource information, wherein the target task comprises a plurality of category sub-tasks, the plurality of category sub-tasks have an association relationship, each category sub-task has a corresponding task priority, a maximum concurrency number and an environment component, and the current resource information indicates a current resource used for executing the target task; determining a data flow direction between categories in the plurality of category sub-tasks according to the association relationship between the category sub-tasks; determining an execution sequence of each sub-task according to the data flow direction between categories in the plurality of category sub-tasks, the task priority of each category sub-task and the maximum concurrency number; and executing each sub-task by using an execution environment of the corresponding category sub-task according to the execution sequence of each sub-task and the current resource information. The method is beneficial to resource scheduling and parallel processing of a server cluster.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to task scheduling methods and apparatus, computing devices, computer-readable storage media, and computer program products. Background Technology

[0002] In the field of computer technology, it is often necessary to process multiple tasks in a short period of time. These tasks may be similar or different, and they may also have logical relationships regarding their execution sequence and different priorities. How to schedule these tasks under limited resource conditions is a pressing problem that needs to be solved.

[0003] Existing task scheduling technologies are based on the Hadoop distributed framework, which means that task scheduling needs to be accomplished by relying on various components of the Hadoop distributed framework. The use of these components is very complex, and the description and scheduling of tasks are also very complex, resulting in high usage costs. Moreover, under the Hadoop distributed framework, task nodes are difficult to dynamically scale up and down, resulting in low resource utilization. Summary of the Invention

[0004] In view of this, this application provides a task scheduling method and apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are intended to alleviate or overcome some or all of the defects mentioned above, as well as other possible defects.

[0005] According to a first aspect of this application, a task scheduling method is provided, comprising: acquiring a target task and current resource information, wherein the target task includes multiple categories of sub-tasks, the multiple categories of sub-tasks are related to each other, each category of sub-tasks has a corresponding task priority, maximum concurrency, and environment components, the maximum concurrency indicates the number of sub-tasks of the corresponding category to be executed, the environment components represent the execution environment of the sub-tasks of the corresponding category, and the current resource information indicates the current resources used to execute the target task; determining the data flow direction between the categories of sub-tasks according to the relationship between the categories of sub-tasks, the data flow direction indicating the source of input data and the destination of output data of each category of sub-tasks; determining the execution order of each sub-task according to the data flow direction between the categories of sub-tasks, the task priority of each category of sub-tasks, and the maximum concurrency; and executing each sub-task using the execution environment of the corresponding category of sub-tasks according to the execution order of each sub-task and the current resource information, thereby completing the target task.

[0006] In some embodiments of the task scheduling method according to this application, determining the data flow direction between categories of sub-tasks based on the association between sub-tasks of various categories includes: for every two categories of sub-tasks that have a direct association, in response to the fact that the sub-task of the first category is the parent task of the sub-task of the second category, determining the destination of the output data of the sub-task of the first category as the sub-task of the second category, and determining the source of the input data of the sub-task of the second category as the sub-task of the first category.

[0007] In embodiments of some task scheduling methods according to this application, determining the execution order of each sub-task based on the data flow direction between categories of sub-tasks, the task priority of each category of sub-tasks, and the maximum concurrency includes: establishing a basic directed acyclic graph (DAG) based on the multiple categories of sub-tasks and the data flow direction between each category of sub-tasks, wherein the nodes of the basic DAG represent the categories of sub-tasks, and the edges of the basic DAG represent the data flow direction between two connected categories of sub-tasks; determining an execution directed acyclic graph (DAG) based on the basic DAG and the maximum concurrency of each category of sub-tasks, wherein the nodes of the execution DAG represent sub-tasks, and the edges of the execution DAG represent the data flow direction between two connected sub-tasks; and determining the execution order of each sub-task in the execution DAG based on the task priority of each category of sub-tasks.

[0008] In some embodiments of the task scheduling method according to this application, determining the execution order of each sub-task in the directed acyclic graph based on the task priority of each category of sub-tasks includes: traversing the directed acyclic graph in a depth-first manner based on the task priority of each category of sub-tasks; and determining the execution order of each sub-task based on the order in which each node is traversed.

[0009] In some embodiments of the task scheduling method according to this application, determining the execution order of each sub-task in the directed acyclic graph based on the task priority of each category of sub-tasks includes: traversing the directed acyclic graph in a breadth-first manner based on the task priority of each category of sub-tasks; and determining the execution order of each sub-task based on the order in which each node is traversed.

[0010] In some embodiments of the task scheduling method according to this application, the edges of the directed acyclic graph have data storage paths, wherein the data storage paths store the output data of the parent task in one of the two subtasks connected by the edge.

[0011] In some embodiments of the task scheduling method according to this application, the edges of the directed acyclic graph also have a log table indicating the execution status of the parent task in the two sub-tasks connected by the edge.

[0012] In some embodiments of the task scheduling method according to this application, the execution of each sub-task is carried out according to the execution order of each sub-task and the current resource information, using the execution environment of the corresponding category of sub-task, including: for each sub-task to be executed: allocating resources to the sub-task according to the current resource information; establishing a corresponding execution environment according to the corresponding environment components of the sub-task; obtaining the input data of the sub-task to be executed according to the execution directed acyclic graph; and processing the input data in the corresponding execution environment to obtain output data.

[0013] In some embodiments of the task scheduling method according to this application, the input data includes at least one object to be processed, each object to be processed having a processing priority; and the step of processing the input data in the corresponding execution environment includes: processing the at least one object to be processed according to the processing priority of the at least one object to be processed.

[0014] In some embodiments of the task scheduling method according to this application, obtaining a target task includes: obtaining a task schedule, the task schedule indicating the execution time of at least one task; and in response to the arrival of the execution time of one of the at least one tasks, obtaining the one task as the target task.

[0015] In some embodiments of the task scheduling method according to this application, each target task also has a task priority, and the method further includes: in response to the arrival of the execution time of multiple tasks, determining the task to be acquired according to the task priority of each task.

[0016] According to a second aspect of this application, a task scheduling apparatus is provided, comprising: an acquisition module configured to acquire a target task and current resource information, wherein the target task includes multiple categories of sub-tasks, the multiple categories of sub-tasks are associated with each other, each category of sub-task has a corresponding task priority, maximum concurrency, and environment components, the maximum concurrency indicates the number of sub-tasks of the corresponding category to be executed, the environment components represent the execution environment of the sub-tasks of the corresponding category, and the current resource information indicates the current resources used to execute the target task; and a first determination module configured to determine based on each category. The first module determines the data flow direction between the sub-tasks based on the relationships between them, indicating the source of input data and the destination of output data for each sub-task category. The second module is configured to determine the execution order of each sub-task based on the data flow direction between the sub-task categories, the task priority of each sub-task category, and the maximum concurrency. The third module is configured to execute each sub-task based on the execution order of each sub-task and the current resource information, utilizing the execution environment of the corresponding sub-task category, thereby completing the target task.

[0017] According to a third aspect of this application, a computing device is provided, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform steps of a task scheduling method according to some embodiments of this application.

[0018] According to a fourth aspect of this application, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed, implement a task scheduling method according to some embodiments of this application.

[0019] According to a fifth aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement a task scheduling method according to some embodiments of this application.

[0020] In the task scheduling method and apparatus according to some embodiments of this application, firstly, complete decoupling from the Hadoop distributed framework is achieved, eliminating the need for cooperation with any Hadoop distributed framework components. Therefore, it is only necessary to clarify the relationships between the sub-tasks of the target task, the maximum concurrency of the sub-tasks, and the priority of the sub-tasks, without configuring other information, resulting in extremely low usage costs. Secondly, each category of sub-tasks has corresponding environment components, which means that execution environments can be temporarily established to execute sub-tasks and deleted after execution, thus exhibiting high dynamic scaling performance and reducing resource costs. Finally, by first determining the data flow direction between categories and then determining the specific execution order of each sub-task, it is possible to schedule each task by focusing only on the data input and output between sub-tasks. This can significantly reduce task scheduling costs while ensuring the efficient execution of a large number of tasks.

[0021] These and other advantages of this application will become clear from the embodiments described below, and will be illustrated with reference to the embodiments described below. Attached Figure Description

[0022] Embodiments of this application will now be described in more detail with reference to the accompanying drawings, wherein: Figure 1 Exemplary application scenarios of the task scheduling method according to some embodiments of this application are shown; Figure 2 This is a flowchart of a task scheduling method according to some embodiments of this application; Figure 3 This is a flowchart illustrating the determination of the execution order of various sub-tasks according to some embodiments of this application; Figure 4 This is a schematic diagram of the underlying directed acyclic graph depicting tasks based on some embodiments of this application; Figure 5 This is a schematic diagram of an execution directed acyclic graph depicting tasks based on some embodiments of this application; Figure 6 This is a schematic diagram of another execution directed acyclic graph depicting tasks based on some embodiments of this application; Figure 7 This is a schematic diagram illustrating the execution of a sub-task according to some embodiments of this application; Figure 8 This is a schematic diagram of an exemplary architecture of a task scheduling method according to some embodiments of this application; Figure 9 It is based on Figure 8 The task scheduling flowchart is an exemplary architecture of the task scheduling method shown. Figure 10This is a schematic diagram of a partial architecture according to an embodiment of this application; Figure 11 It is based on Figure 8 The scheduling flowchart of the scheduler in the architecture of the task scheduling method shown; Figure 12 This is a schematic diagram of the storage structure in data storage according to some embodiments of this application; Figure 13 It is a basic directed acyclic graph according to some embodiments of this application; Figure 14 yes Figure 13 A schematic diagram of the log table in an embodiment of the task scheduling method shown; Figure 15 This is a schematic diagram illustrating the execution of sub-tasks according to some embodiments of this application; Figure 16 This is an exemplary structural block diagram of a task scheduling apparatus according to other embodiments of this application; Figure 17 An example system is shown, which includes an example computing device representing one or more systems and / or devices that can implement the various methods described herein. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the terms "and / or" and similar terms include all combinations of any, multiple, and all of the associated listed items.

[0028] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0029] Before detailing the embodiments of this application, some related concepts will be explained for clarity.

[0030] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, and can be used on demand, expanded at any time, and paid for based on usage. As a provider of basic cloud computing capabilities, a cloud computing resource pool platform is established, deploying various types of virtual resources within the pool for external customers to choose from. The cloud computing resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0031] Containers: A container is a standard software unit that packages code and all its dependencies, enabling applications to run quickly and reliably from one computing environment to another. Most containers are lightweight, standalone, executable software packages that include everything needed to run the application: code, runtime, system tools, system libraries, and settings. Container technology can load both the operating system image and the application into memory simultaneously, and can also load from network disks, because starting dozens of images at the same time does not place a significant load on the network and storage. Subsequent image creation only requires pointing to a generic image, greatly reducing the memory required.

[0032] K8S (Kubernetes): is an open-source system for automating the deployment, scaling, and management of containerized applications.

[0033] Tencent Cloud Container Service (Tencent Kubernetes Engine, TKE): Provides highly scalable, high-performance container management services centered around containers, based on native Kubernetes.

[0034] Pod: A Pod is a composite unit of multiple containers running on the same machine, and it's a fundamental unit in Kubernetes. A Pod consists of a main container and several auxiliary containers that work together to perform a specific function. A Pod is formed by packaging multiple processes (containers are also a type of isolated process) into a single namespace.

[0035] Garbage Collection (GC) Task: In computer science, garbage collection is a form of automatic memory management. This application also specifically refers to data cleanup related to the file system; for example, in some file systems, file deletion is marked for deletion, while garbage collection tasks asynchronously and concurrently delete files.

[0036] Cloud Object Storage (COS): Cloud Object Storage is a distributed storage service launched by Tencent Cloud that has no directory hierarchy, no data format restrictions, can accommodate massive amounts of data, and supports access via HTTP / HTTPS protocols.

[0037] Lifecycle tasks: Object storage supports configuration based on object lifecycles, which allows objects that meet certain rules to automatically perform certain operations under specified conditions. For example, cooling down or deleting objects that haven't been accessed recently.

[0038] Inventory Tasks: Inventory tasks are a feature that helps users manage objects. Based on the user's inventory task configuration, object storage can periodically scan the user's specified objects or objects with the same prefix within the user's storage space daily or weekly, and output an inventory report. This report is stored as a CSV file in the user-specified bucket. The file lists the stored objects and their corresponding metadata, and records the object attribute information required by the user based on the user's configuration.

[0039] A Directed Acyclic Graph (DAG) is a graph in which it is impossible to return to a given vertex by any number of edges. Any directed tree is a DAG.

[0040] JavaScript Object Notation (JSON) is a lightweight data-interchange format that includes six constructor characters, strings, numbers, and three literals. It is easy for humans to read and write, and can exchange data between multiple languages. It is also easy for machines to parse and generate.

[0041] Raft Protocol: A protocol for achieving distributed consensus, enabling the overall state of a cluster in a distributed architecture to reach a consensus. The core problem solved by the Raft protocol is distributed consensus, and it is commonly used in distributed architectures. This protocol allows the cluster to continue functioning normally even if a portion of the storage nodes fail, previously passed resolutions remain accessible, and the cluster state remains consistent.

[0042] Figure 1 An exemplary application scenario 100 of a task scheduling method according to some embodiments of this application is illustrated. Application scenario 100 may include a client 101, a network 102, a service unit 103, a task schedule 104, and a server cluster 105. The service unit 103 is communicatively coupled to the client 101 and the task schedule 104 via the network 102 and can communicate with the server cluster 105. In this embodiment, the task schedule 104 is shown separately, but this is not limiting; it may also exist in the service unit 103 or in the client 101 as a local file.

[0043] In this embodiment, the service unit 103 is the execution entity of the task scheduling method of this application, and the server cluster 105 is the execution entity that executes the target task and is managed by the service unit 103.

[0044] In this embodiment, service unit 103 acquires target task and current resource information. The target task includes multiple categories of sub-tasks, which are related to each other. Each category of sub-task has a corresponding task priority, maximum concurrency, and environment components. The maximum concurrency indicates the number of sub-tasks of the corresponding category that are executed. The environment components represent the execution environment of the corresponding sub-tasks. The current resource information indicates the current resources used to execute the target task. Specifically, client 101 acquires or modifies task schedule 104 through network 102, for example, by adding a new task to task schedule 104. Service unit 103 can acquire the target task by acquiring task schedule 104 through network 102. The current resource information reflects the resource information in server cluster 105. Network 102 can be, for example, a wide area network (WAN), local area network (LAN), wireless network, public telephone network, intranet, or any other type of network well known to those skilled in the art.

[0045] Then, service unit 103 determines the data flow direction between the various categories of sub-tasks based on the relationships between them. This data flow direction indicates the source of input data and the destination of output data for each category of sub-task. This data flow direction is a crucial basis for service unit 103 to subsequently determine the data storage address of each sub-task in server cluster 105.

[0046] Next, service unit 103 determines the execution order of each sub-task based on the data flow direction between categories, the task priority of each category, and the maximum concurrency. After determining the execution order of each sub-task, service unit 103 can specify the storage address of the input data and the storage address of the output data for each sub-task in the server cluster 105. Service unit 103 does not need to concern itself with the configuration of any other components.

[0047] Finally, service unit 103 executes each sub-task according to its execution order and current resource information, utilizing the execution environment of the corresponding sub-task category, thereby completing the target task. When executing each sub-task, service unit 103 can allocate the sub-task to be executed to an idle server in server cluster 105 based on current resource information, instead of executing it on service unit 103 itself. For example, it can establish a corresponding execution environment on an idle server to execute the sub-task.

[0048] exist Figure 1 In the application scenarios shown, the task scheduling methods according to some embodiments of this application are implemented on the service unit 103. However, this is merely illustrative and not restrictive. The task scheduling methods according to some embodiments of this application can also be implemented on other entities with sufficient computing resources and capabilities, such as on the client 101 with sufficient computing resources and capabilities. Of course, they can also be implemented partially on the service unit 103 and partially on the client 101; this is not restrictive.

[0049] As understood by those skilled in the art, instances of service unit 103 and server cluster 105 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Servers can be connected directly or indirectly via wired or wireless communication, and this application does not impose any limitations on this.

[0050] Client 101 can be any type of mobile computing device, including mobile computers (e.g., personal digital assistants (PDAs), laptops, notebooks, tablets, netbooks, etc.), mobile phones (e.g., cellular phones, smartphones, etc.), wearable computing devices (e.g., smartwatches, head-mounted devices, including smart glasses, etc.) or other types of mobile devices. In some embodiments, client 101 can also be a fixed computing device, such as a desktop computer, game console, smart TV, etc. Furthermore, in the case where application scenario 100 includes multiple clients 101, the multiple clients 101 can be the same or different types of computing devices.

[0051] like Figure 1 As shown, client 101 may include a display screen and a terminal application that can be interacted with by the end user via the display screen. The terminal application can be a local application, a web application, or a lightweight app (such as a mobile app or WeChat app). If the terminal application is a local application that requires installation, it can be installed on client 101. If the terminal application is a web application, it can be accessed through a browser. If the terminal application is a mini-program, it can be opened directly on client 101 without installation by searching for relevant information (such as the application's name) or scanning its graphic code (such as a barcode or QR code).

[0052] Figure 2 This is a flowchart of a task scheduling method according to some embodiments of this application. The method 200 shown can be implemented in a service unit (e.g., at...). Figure 1 The service unit 103 shown is implemented on the client 101. In some embodiments, when the client 101 has sufficient computing resources and computing power, the task scheduling method according to some embodiments of this application can be executed directly on the client 101. In other embodiments, the task scheduling method according to some embodiments of this application can also be executed by a combination of the service unit 103 and the client 101. Figure 2 As shown, the task scheduling method according to some embodiments of this application may include steps S201-S204.

[0053] In step S201, target task and current resource information are obtained. The target task includes multiple categories of sub-tasks, which are related to each other. Each category of sub-task has a corresponding task priority, maximum concurrency, and environment components. The maximum concurrency indicates the number of sub-tasks of the corresponding category that are executed. The environment components represent the execution environment of the corresponding sub-tasks. The current resource information indicates the current resources used to execute the target task. It should be noted that the target task and current resource information can be obtained simultaneously or at different times, and they are independent of each other. This application does not limit this. Furthermore, in some embodiments, the target task is obtained first, followed by the current resource information.

[0054] Common types of target tasks include garbage collection tasks, lifecycle tasks, and inventory tasks. In various embodiments of this application, garbage collection tasks are used as examples for illustration. In this embodiment, the target task can be obtained directly from the client or from a pre-defined task schedule. The target task can include (i.e., be divided into) multiple categories of sub-tasks, and these sub-tasks are related to each other. These relationships are predetermined in the task description of the target task. The task description refers to the detailed information describing the target task. For example, the relationships between different categories of sub-tasks can be represented in the task description using an array data structure. Each category of sub-tasks includes a predetermined number of sub-tasks, which is equal to the corresponding maximum concurrency of the sub-tasks in that category. For example, if the maximum concurrency of the delete-inode category of sub-tasks is 5, this means that a total of 5 delete-inode category sub-tasks need to be executed. Furthermore, because the execution environments for each category of sub-tasks are temporarily established in this embodiment, each category of sub-task needs to define its own corresponding environment components, for example, by specifying the index of the environment components. This way, when executing a sub-task within that category, the corresponding execution environment can be established based on the index of the environment component, thus executing the sub-task within that category. As an example, the index of the environment component can be a mirror address of the Kubernetes execution environment. Regarding the current resource information, these resources refer to those available for executing sub-tasks, such as the size of free memory and the number of available processors.

[0055] In step S202, based on the relationship between the sub-tasks of each category, the data flow direction between the categories of sub-tasks is determined. The data flow direction indicates the source of the input data and the destination of the output data of each category of sub-task.

[0056] Since different categories of sub-tasks are used to accomplish different goals, there are definite relationships between the sub-tasks of each category. These relationships need to be clearly defined to achieve logically sound task scheduling; otherwise, task scheduling may get stuck in a deadlock, preventing the target task from being completed. A deadlock here refers to a situation where, for example, the first sub-task requires the output of the second sub-task to execute, but the second sub-task's execution order is lower than the first sub-task, causing both sub-tasks to fail and resulting in a scheduling error. In step S202, the relationships between the sub-tasks of each category are clearly defined by determining the data flow direction between the categories, providing the necessary logical support for the complete implementation of subsequent task scheduling.

[0057] In step S203, the execution order of each sub-task is determined based on the data flow direction between categories, the task priority of each category, and the maximum concurrency. Based on the data flow direction between categories, the correct execution order of each category's sub-tasks can be ensured. Then, the specific execution order of each sub-task can be determined based on two other factors related to the execution order—task priority and maximum concurrency.

[0058] In step S204, based on the execution order of each sub-task and the current resource information, each sub-task is executed using the execution environment of the corresponding category of sub-tasks, thereby completing the target task. The execution environment used here can be established according to the corresponding environment components of the sub-task category. In some embodiments, the execution environment can be a Kubernetes-based Pod execution environment established according to the index of the environment components and based on the container management platform TKE. This Pod execution environment can be implemented using one or more containers.

[0059] In the task scheduling method according to the embodiments of this disclosure, firstly, it achieves complete decoupling from the Hadoop distributed framework, without requiring cooperation from any Hadoop distributed framework components. Therefore, it only needs to clarify the relationships between the sub-tasks of the target task, the maximum concurrency of the sub-tasks, and the priority of the sub-tasks, without configuring other information, resulting in extremely low usage costs. Secondly, each category of sub-tasks has corresponding environment components, which means that execution environments can be temporarily established to execute sub-tasks and deleted after execution, thus achieving high dynamic scaling performance and reducing resource costs. Finally, by first determining the data flow direction between categories and then determining the specific execution order of each sub-task, it is possible to schedule each task by only focusing on the data input and output between sub-tasks. This can significantly reduce task scheduling costs while ensuring the efficient execution of a large number of tasks.

[0060] Regarding the acquisition of the target task, in some embodiments, the process specifically includes: first, acquiring a task schedule indicating the execution time of at least one task; then, in response to the arrival of the execution time of one of the at least one task, acquiring that task as the target task. The task schedule is... Figure 1 The task schedule 104 shown can be modified, added, deleted, and viewed by the client. This task schedule can be stored in a database, with each task having a task name and an index for its description. To ensure timely execution of tasks, the task schedule in the database can be monitored in real time. After determining the target task to be executed based on the task schedule, the task description for that target task is downloaded based on the index of its description.

[0061] In some embodiments, each target task also has a task priority. When selecting a target task to acquire, the task to be acquired needs to be determined based on the task priority of each task in response to the arrival of the execution times of multiple tasks. This ensures that tasks with higher task priorities are completed first.

[0062] Table 1 shows the information configuration for each task in the task schedule according to some embodiments of this application: Table 1 serial number name type describe Notes 1 id bigint(20) Task identifier / 2 type tinyint(10) Task type Garbage collection tasks, etc. 3 cron_str varchar(511) Trigger time / 4 priority int(11) Task Priority / .

[0063] Where id is the task identifier, type is used to indicate the task type, cron_str is used to determine the time to trigger the task, such as 8 am every day, and priority is used to set the task priority.

[0064] Based on the information configuration shown in Table 1, the following is an example of a task description of the target task written into a JSON file. "graph":{ "name":"gc", "priority":100, "cron-str":"8:00", "jobs":[{ "name":"list-fs", "level":0, "concurrent-max":1, "image":"csighub.tencentyun / list-fs-worker:v1", }, { "name":"list-inode", "level":1, "concurrent-max":2, "image":"csighub.tencentyun / list-inode-worker:v1", "cluster":{ "cpu-limit":"100m", "mem-limit":"64Mi" } }, { "name":"delete-inode", "level":2, "concurrent-max":4, "image":"csighub.tencentyun / delete-inode-worker", } ], "dependence":{ "list-fs":["list-inode"], "list-inode:["delete-inode] } } 。

[0065] The task description in the JSON file is parsed as follows: `graph` represents the target task object, the target task type is "gc" (garbage collection task), the task priority is 100, and the task execution time is 8:00 AM. The "jobs" structure lists the sub-tasks of each category included in the target task and their relationships. This garbage collection task includes three categories of sub-tasks: `list-fs`, `list-inode`, and `delete-inode`, with priorities of 0, 1, and 2 respectively; maximum concurrency (`concurrent-max`) of 1, 2, and 4 respectively; and the `image` field of each category of sub-task specifies the index of its respective environment component. In the garbage collection task, the `list-fs` category sub-task retrieves the target file, the `list-inode` category sub-task retrieves the files marked for deletion from the target file, and the `delete-inode` category sub-task deletes the underlying block data based on the files marked for deletion. The "dependence" structure uses an array to illustrate the relationships between the subtasks of each category. Subtasks of the list-fs category are parent tasks of subtasks of the list-inode category, and subtasks of the list-inode category are parent tasks of subtasks of the delete-inode category. Furthermore, subtasks of the list-inode category have resource constraints: a minimum of 100 MB of memory and a minimum of 64 MB of storage space. This means that executing a subtask of the list-inode category requires at least 100 MB of memory and 64 MB of storage space. This concise task description provides a complete picture of the target task, resulting in low usage costs. This concise task description is partly due to the decoupling of the task scheduling method in this application from the Hadoop distributed framework.

[0066] In some embodiments, determining the data flow direction between multiple categories of sub-tasks based on the relationships between them includes: for every two categories of sub-tasks with a direct relationship, in response to the first category being the parent task of the second category, the destination of the output data of the first category is determined to be the second category, and the source of the input data of the second category is determined to be the first category. Each sub-task has either a direct or indirect relationship. A direct relationship means there is a parent-child relationship between two sub-tasks, reflecting the logical relationship in their execution, which is a rule that must be determined and followed. An indirect relationship means that two sub-tasks share a common parent task, and their execution does not depend on each other's output data, thus allowing them to execute relatively independently or concurrently. In this embodiment, determining the direct relationship between the categories of sub-tasks is sufficient to determine the source of the input data and the destination of the output data for each category of sub-tasks.

[0067] Once the direction of data flow between categories in multiple sub-tasks is determined, the execution order of each sub-task can be easily determined. Figure 3 This is a flowchart illustrating the determination of the execution order of various sub-tasks according to some embodiments of this application. In this embodiment, the execution order of each sub-task is determined based on the data flow direction between categories of sub-tasks, the task priority of each category of sub-tasks, and the maximum concurrency, including steps S301-303.

[0068] In step S301, a basic directed acyclic graph (DAG) is established based on the multiple categories of sub-tasks and the data flow direction between the sub-tasks of each category. The nodes of the basic DAG represent the categories of sub-tasks, and the edges of the basic DAG represent the data flow direction between two connected sub-task categories. It is worth noting that the nodes of this basic DAG are not the final sub-tasks to be executed, but rather indicate the category of the sub-tasks. The specific number of executions depends on the maximum concurrency of the sub-tasks in that category.

[0069] Figure 4This is a schematic diagram of the basic directed acyclic graph (DAG) depicting the task based on some embodiments of this application. In the aforementioned embodiment depicting the target task using a JSON file, the target task is a garbage collection task, and the three sub-tasks are list-fs, list-inode, and delete-inode, respectively. Therefore, the basic DAG includes three nodes. Furthermore, the sub-task of the list-fs category is the parent task of the sub-task of the list-inode category, and the sub-task of the list-inode category is the parent task of the sub-task of the delete-inode category. Therefore, the edge between the list-fs node and the list-inode node points to the list-inode node, and the edge between the list-inode node and the delete-inode node points to the delete-inode node.

[0070] In step S302, an execution directed acyclic graph (DAG) is determined based on the basic DAG and the maximum concurrency of each category's sub-tasks. Nodes in the execution DAG represent sub-tasks, and edges represent the data flow direction between connected sub-tasks. After determining the basic DAG indicating the data flow direction between categories, the complete execution DAG can be directly determined based on the maximum concurrency of each category's sub-tasks. When the maximum concurrency is greater than 1, although multiple sub-tasks in that category need to be executed, the input data for each sub-task in that category is the same. Therefore, when determining the execution DAG, the source of the input data for each sub-task does not necessarily have to be the same parent task; it only needs to be from the same category's parent task. Thus, there are multiple ways to determine the execution DAG.

[0071] Figure 5 This is a schematic diagram of an execution directed acyclic graph depicting tasks based on some embodiments of this application. For example... Figure 5 As shown, based on Figure 4 The basic directed acyclic graph shown has the following sub-tasks: the list-fs category with a maximum concurrency of 1 is executed once, serving as the root node 501 of the directed acyclic graph; the list-inode category with a maximum concurrency of 2 is executed twice, generating two nodes 502 and 503, both with node 501 as their parent; and the delete-inode category with a maximum concurrency of 4 is executed four times, generating four nodes 504-507. Nodes 504-505 are all parented by node 502, and nodes 506-507 are all parented by node 503.

[0072] Figure 6 This is a schematic diagram of another execution directed acyclic graph depicting tasks based on some embodiments of this application. For example... Figure 6 As shown, based on Figure 4 The basic directed acyclic graph shown has the following sub-tasks: the list-fs category with a maximum concurrency of 1 is executed once, serving as the root node 601 of the directed acyclic graph; the list-inode category with a maximum concurrency of 2 is executed twice, generating two nodes 602 and 603, both with node 601 as their parent; and the delete-inode category with a maximum concurrency of 4 is executed four times, generating two nodes 604-607, where nodes 604-606 are both with node 602 as their parent, and node 607 is with node 603 as its parent.

[0073] In step S302, Figure 5 and Figure 6 All execution directed acyclic graphs are potentially finalized.

[0074] In step S303, the execution order of each sub-task in the directed acyclic graph (DAG) is determined based on the task priority of each category of sub-tasks. When determining the DAG, the task priority of each category of sub-tasks can be combined to traverse the DAG, and the execution order of each task is determined according to the order in which each node is traversed. Since this DAG is based on the underlying DAG, task scheduling can be guaranteed, and deadlock can be avoided.

[0075] Regarding step S303, different execution orders can be obtained by using different traversal methods. In some embodiments, the execution order of each sub-task in the directed acyclic graph is determined based on the task priority of each category of sub-tasks, including: traversing the directed acyclic graph according to a depth-first traversal method based on the task priority of each category of sub-tasks; and determining the execution order of each sub-task according to the order in which each node is traversed. Figure 5 In the illustrated embodiment, the execution order of each sub-task is as follows: list-fs sub-task 501 → list-inode sub-task 502 → delete-inode sub-task 504 → delete-inode sub-task 505 → list-inode sub-task 503 → delete-inode sub-task 506 → delete-inode sub-task 507.

[0076] In other embodiments, the execution order of each sub-task in the directed acyclic graph is determined based on the task priority of each category of sub-tasks, including: traversing the directed acyclic graph using a breadth-first traversal method based on the task priority of each category of sub-tasks; and determining the execution order of each sub-task based on the order in which each node is traversed. Figure 5In the illustrated embodiment, the execution order of each sub-task is as follows: list-fs sub-task 501 → list-inode sub-task 502 → list-inode sub-task 503 → delete-inode sub-task 504 → delete-inode sub-task 505 → delete-inode sub-task 506 → delete-inode sub-task 507.

[0077] The task scheduling method embodiments of this application primarily consider the input and output of each sub-task. Therefore, after determining the execution of the directed acyclic graph, the data storage method for each sub-task can be further determined. In some embodiments, the edges of the executed directed acyclic graph have data storage paths, wherein the data storage path stores the output data of the parent task (i.e., the source task based on the direction of the edge) of the two sub-tasks connected by the edge. In this embodiment, for two sub-tasks connected by an edge, when the parent task is completed, the output data is directly stored in the data storage path of the edge. When preparing to execute the sub-task, the data is directly obtained from the data storage path of the edge as input data. For the root node sub-task of the entire executed directed acyclic graph, the input data needs to be obtained from the external environment or local cache.

[0078] For ease of scheduling, in some embodiments, the edges of the directed acyclic graph also have a log table, which indicates the execution status of the parent task in the two sub-tasks connected by the edge. This allows for real-time monitoring of the execution status of each sub-task, improving the user experience.

[0079] The task scheduling method of this application can also intervene at the execution level, for example, by providing instructions required to execute each sub-task. Figure 7 This is a schematic diagram illustrating the execution of a sub-task according to some embodiments of this application. Figure 7 In the embodiment shown, the sub-task is executed according to the execution order of the sub-task and the current resource information, using the execution environment of the corresponding category of sub-task, including: for each sub-task to be executed, it is executed according to steps S701-S704.

[0080] In step S701, resources are allocated to the sub-task based on the current resource information. Only with the resources available to execute the sub-task can its execution be initiated. These resources include various computing resources, such as memory, storage space, and the number of processors.

[0081] In step S702, a corresponding execution environment is established based on the environment components corresponding to the sub-task. This execution environment can be a container or a Pod execution environment based on at least one container. Leveraging the lightweight and low-memory characteristics of containers, the execution environment is dynamically established whenever a sub-task needs to be executed. This improves the performance of dynamic scaling and greatly enhances resource utilization efficiency. Since containers can be loaded from network disks, an index of the environment components can be provided in the task description of the target task, and then the corresponding container is loaded from the network disk based on that index.

[0082] In step S703, the input data of the sub-tasks to be executed is obtained according to the directed acyclic graph. The input data of each sub-task is stored in a specified storage path. In the embodiments of the task scheduling method of this application, only the direction of data flow needs to be considered to achieve the orderly execution of tasks, without calling other components, such as components of various Hadoop distributed frameworks. Therefore, the start of the execution of a sub-task only requires obtaining the input data of that sub-task.

[0083] In step S704, input data is processed in the corresponding execution environment to obtain output data. Different processing instructions are provided to process the input data according to the category of the sub-task; these processing instructions are all adapted to the execution environment and can run within it.

[0084] Regarding step S704, in some embodiments, the input data includes at least one object to be processed, each object to be processed having a processing priority; and in the corresponding execution environment, processing the input data includes: processing the at least one object to be processed according to the processing priority of the at least one object to be processed.

[0085] "Object" is a general term used in this application to refer to the basic unit to be processed in each sub-task. Depending on the category of the target task, an "object" can be a specific file or an instance of a software program, etc. In a garbage collection task, an object refers to a file; therefore, the input data for each sub-task can be at least one object to be processed. These objects can be obtained from the external environment or obtained in advance from the external environment and then cached locally. These objects can be pre-set with different processing priorities by the external environment. According to various embodiments of the task scheduling method of this application, these objects to be processed can be combined with the processing priorities of each object to meet the needs of the external environment and improve the user experience. As an example, when executing a sub-task of the delete-inode category, the input data obtained can be 100 files processed by its parent task in the directed acyclic graph, of which file number 50 has the highest priority. Therefore, when executing this sub-task of the delete-inode category, the underlying data of file number 50 is deleted first.

[0086] Figure 8 This is a schematic diagram of an exemplary architecture of a task scheduling method according to some embodiments of this application. In this embodiment, the architecture of the task scheduling method according to this application can be designed as follows: Gateways 802-804 provide clients 801 with an interface to database 805, allowing clients 801 to access database 805, which stores task schedules, enabling CRUD operations on these schedules. Gateways 802-804 also provide clients 801 with an interface to task management terminals 806-808, allowing clients to directly configure these terminals. Task management terminals 806-808 perform master-slave switching and backup according to the Raft protocol, thereby improving their disaster recovery capabilities.

[0087] The task management terminal 806 includes a trigger 809 and a scheduler 810. The trigger 809 retrieves the task schedule from the database 805 and determines the target task. Then, based on the index of the target task in the task schedule, it downloads the corresponding task description from the COS and parses it, generating a basic directed acyclic graph (DAG). The trigger 809 transmits this basic DAG to the scheduler 810. The scheduler 810, based on current resource information, determines the execution order of each sub-task according to the task scheduling method according to some embodiments of this application, obtaining an execution queue 811 for the sub-tasks. Figure 8As shown, the execution queue 811 includes sub-tasks 812-814. This current resource information is obtained from TKE 819 by the worker server cluster 815 and transmitted to the scheduler 810. In addition, the scheduler 810 also needs to retrieve objects from the object store 825 as input data for the first sub-task to be executed (i.e., the root node sub-task of the underlying directed acyclic graph or the execution directed acyclic graph).

[0088] The execution of the sub-tasks is handled by the worker server cluster 815. Worker server cluster 815 includes worker servers 816-818, with worker server 816 being the master worker server. Worker servers 816-818 use the Raft protocol for master-slave switching and backup, thereby improving the disaster recovery capabilities of the task management end. Since TKE 819 contains available resources, after obtaining sub-task 812, worker server 816 can establish an execution environment in TKE 819 based on the corresponding environment components, and encode the execution instructions for sub-task 820 through the interface of TKE 819. These execution instructions depend on the category and execution environment of sub-task 812. Then, in the established execution environment, TKE 819 retrieves input data from COS824 according to the data storage path of the parent task of the sub-task. It then executes sub-task 812 according to its corresponding execution instructions, storing the execution status (e.g., success, failure, pause) of the sub-task in the COS824 log table of the edge corresponding to the parent task. If the sub-task executes successfully, it also stores the output data in COS824 according to the data storage path of the edge corresponding to the parent task. Following the execution order of each sub-task, after sub-task 820 is executed, sub-tasks 813-814 are executed concurrently or sequentially based on the current resource information. During the execution of each sub-task, the task management terminal 806 can store the execution status of each sub-task in database 805 for user review.

[0089] Figure 9 It is based on Figure 8 The illustrated task scheduling method is an exemplary architecture task scheduling flowchart, which includes a common process for the task management terminal 806 and the worker server cluster 815.

[0090] After starting at point 901, the task management terminal 806 first retrieves the task schedule 902 from the database 903 and obtains the target task from it. Current resource information can be retrieved later. Then, at point 904, the task description of the target task is parsed, and a basic directed acyclic graph (DAG) is generated at point 905 based on the relationships between the sub-tasks of various categories. At this point, current resource information 906 can be obtained first, and the pending objects 907 of the root node sub-tasks can be obtained based on the root node sub-tasks of the basic DAG. Then, the scheduler 810 generates an execution DAG at point 908 and records the execution DAG in the database 903 after successful generation. Next, the directed acyclic graph (DAG) is traversed at point 909. During the traversal, sub-tasks are executed and checked at point 910. To fully utilize resources, as many sub-tasks as possible are executed concurrently, within the current resource limits. If a sub-task is completed, its execution result is recorded in database 903 for user review. At point 911, it is determined whether all sub-tasks in the DAG have been successfully recorded in the database. If the traversal is not complete, the execution and checking of sub-task 910 continues. If the determination at point 911 regarding successful recording in the database is negative (i.e., N), the traversal status of the DAG at point 909 is checked and the traversal continues. If the determination at point 911 regarding successful recording in the database is positive (i.e., Y), the task scheduling process ends at point 912.

[0091] The following will explain Figure 8 The diagram shows details of the partial architecture of the task scheduling method. Figure 10 This is a schematic diagram of a partial architecture according to an embodiment of this application. User 1001 can write a task description 1004 and store it in COS 1005, while updating the task schedule in database 1002 accordingly through an interface. Trigger 1003 retrieves the target task that should be executed at this time from the task schedule in database 1002, then retrieves the corresponding task description from COS 1005 according to the index of the target task, and determines the data flow direction between the sub-tasks of each category according to the association relationship between the sub-tasks of each category in the task description, and determines the basic directed acyclic graph of the sub-tasks of each category according to the data flow direction between categories. Finally, the basic directed acyclic graph is transmitted to scheduler 1006 to generate an execution directed acyclic graph.

[0092] Figure 11 It is based on Figure 8 The diagram shows the scheduling flowchart of the scheduler in the architecture of the task scheduling method. This scheduling process occurs after the underlying directed acyclic graph of the target task is obtained. Figure 11As shown, the scheduler 1101 retrieves the objects to be processed for the root node sub-task from the object memory 1103 (such as COS) based on the root node sub-task in the underlying directed acyclic graph, and retrieves the current resource information 1107 from the worker server 1102. The retrieved target task can be a garbage collection task 1104, a list task 1105, a lifecycle task 1106, etc. Taking the garbage collection task 1104 as an example, the scheduler can determine the resources allocated to each category of sub-task based on the task priority of each category of sub-task in the underlying directed acyclic graph. For example, in the garbage collection task 1104 which includes three categories of sub-tasks, 50% of the current resources can be allocated to the list-fs category sub-task with the highest task priority, 20% of the current resources can be allocated to the delete-inode category sub-task with the lowest task priority, and the remaining 30% of the current resources can be allocated to the list-inode category sub-task with the medium task priority. Then, the highest concurrency 1108 of each category of sub-task is obtained, and the directed acyclic graph 1109 is determined based on this. The first sub-task executed is the root node sub-task in the directed acyclic graph 1109. This involves obtaining the input data 1110 for the root node sub-task to drive the execution of other sub-tasks. For example, if the maximum concurrency of the root node sub-task 1112 in the basic directed acyclic graph is 2, then two root node sub-tasks in the directed acyclic graph 1109 will be executed: concurrent sub-tasks 1113 and 1114. The resources used to execute these two sub-tasks are the allocated resources 1115. Finally, based on the current resource information, the directed graph is traversed at point 1111 to execute other sub-tasks. The task scheduling process ends when the traversal of the directed graph at point 1111 is completed.

[0093] In some embodiments, the data structure of the nodes performing a directed acyclic graph can be configured according to the configuration information in Table 2: Table 2 serial number name type describe 1 id bigint(20) Identifier of this node's task 2 pid varchar(255) Parent node task identifier 3 level tinyint(4) The level of the node 4 priority tinyint(4) Task Priority 5 status tinyint(3) 0: Creating, 1: Instantiating, 2: Running, 3: Completed 6 result tinyint(3) 0: Success, 1: Failure, 2: Cancel 7 container_name varchar(255) Container name

[0094] The data structure for edges in a directed acyclic graph can be configured according to the information in Table 3: Table 3 serial number name type describe 1 id bigint(20) Side marker 2 pid bigint(20) Source task identifier 3 cid bigint(20) Purpose / Task Identifier 4 middle_path varchar(255) Data storage path

[0095] In Table 3, the source task refers to the parent task among the two sub-tasks connected by the edge, and the destination task refers to the child task among the two sub-tasks connected by the edge. The output data of the source task is stored in the data storage path `middle_path`.

[0096] Figure 12 This is a schematic diagram of the storage structure in data storage according to some embodiments of this application. For example... Figure 12As shown, the data storage path 1201 consists of two parts. The first part is the file directory 1202, which records the storage index of each data file. Based on this file directory 1202, the corresponding data file can be quickly located. The second part contains the individual data files, including data files 1203-1204. These data files store object information; for example, in a garbage collection task, one data file stores one file processed each time. By setting the file directory 1202, the efficiency of data retrieval can be improved, and the execution time of sub-tasks can be reduced.

[0097] As described in the previous examples, the edges of a directed acyclic graph can have corresponding log tables. Figure 13 It is a basic directed acyclic graph according to some embodiments of this application. For example... Figure 13 As shown, the data flow direction between the sub-tasks of each category is: root node sub-task 1301 -> sub-task A 1302 and sub-task B 1304, sub-task A 1302 -> sub-task C 1303. The maximum concurrency of sub-task A 1302 and sub-task C 1303 is 2, and the maximum concurrency of the root node sub-task and sub-task B 1304 is 1.

[0098] Figure 14 yes Figure 13 The diagram illustrates the log tables of an embodiment of the task scheduling method. First, a directed acyclic graph (DAG) is determined based on the task priority, maximum concurrency, and data flow direction between categories for each sub-task. The node layer 1401 of the DAG includes a root node sub-task. The DAG also includes an intermediate layer 1402, which corresponds to the edges involving the root node and includes log tables 1-3, all used to record the execution status of the root node sub-task. For example, for a garbage collection task, log tables 1-3 record how many objects the root node sub-task has processed and whether it has successfully completed execution, among other execution-related information. Similarly, log tables 4-5 in the intermediate layer 1404 record the execution status of two sub-tasks A in node layer 1403. Based on this execution status, it can be determined whether to drive the execution of two sub-tasks C in node layer 1405. For example, if log table 4 shows that the corresponding sub-task A has completed execution, then sub-task C above node layer 1405 can begin execution.

[0099] Figure 15 This is a schematic diagram illustrating the execution of sub-tasks according to some embodiments of this application, and is a specific implementation of step S204, mainly consisting of... Figure 8The work server cluster 815 shown is used to complete this task. After determining the execution order of each sub-task, a sub-task concurrency pool 1501 is established. This sub-task concurrency pool 1501 stores all sub-tasks that can be executed concurrently as allowed by the current resources. During execution, a separate thread can be allocated to each sub-task for execution. This thread uses the container management platform interface 1508 to establish the execution environment and determine the execution instructions for the corresponding sub-task, so that sub-tasks 1502-1504 can be executed in the container management platform in executors 1505-1507. Each sub-task corresponds to an executor, which can be written in any coding language.

[0100] Figure 16 This is an exemplary structural block diagram of a task scheduling apparatus 1600 according to some embodiments of this application. The apparatus 1600 includes: an acquisition module 1601, a first determination module 1602, a second determination module 1603, and an execution module 1604. The acquisition module 1601 is configured to acquire a target task and current resource information, wherein the target task includes multiple categories of sub-tasks, the multiple categories of sub-tasks are associated with each other, each category of sub-task has a corresponding task priority, maximum concurrency, and environment components, the maximum concurrency indicates the number of sub-tasks of the corresponding category to be executed, the environment components represent the execution environment of the corresponding category of sub-tasks, and the current resource information indicates the current resources used to execute the target task. The first determination module 1602 is configured to determine the data flow direction between the categories of sub-tasks based on the association between the categories of sub-tasks, the data flow direction indicating the source of input data and the destination of output data for each category of sub-task. The second determining module 1603 is configured to determine the execution order of each sub-task based on the data flow direction between categories, the task priority of each category of sub-task, and the maximum concurrency. The execution module 1604 is configured to execute each sub-task based on the execution order of each sub-task and the current resource information, using the execution environment of the corresponding category of sub-task, thereby completing the target task.

[0101] It should be noted that the various modules described above can be implemented in software, hardware, or a combination of both. Multiple different modules can be implemented within the same software or hardware architecture, or a single module can be implemented by multiple different software or hardware architectures.

[0102] The task scheduling apparatus according to some embodiments of this application firstly achieves complete decoupling from the Hadoop distributed framework, eliminating the need for cooperation with any Hadoop distributed framework components. Therefore, it only requires clarifying the relationships between the sub-tasks of the target task, the maximum concurrency of the sub-tasks, and the priority of the sub-tasks, without configuring other information, resulting in extremely low usage costs. Secondly, each category of sub-tasks has corresponding environment components, which means that execution environments can be temporarily established to execute sub-tasks and deleted after execution, thus exhibiting high dynamic scaling performance and reducing resource costs. Finally, by first determining the data flow direction between categories and then determining the specific execution order of each sub-task, it is possible to schedule each task by focusing only on the data input and output between sub-tasks. This can significantly reduce task scheduling costs while ensuring the efficient execution of a large number of tasks.

[0103] Figure 17 The illustration depicts an example system 1700, which includes an example computing device 1710 representing one or more systems and / or devices that can implement the various methods described herein. The computing device 1710 may be, for example, a server of a service provider, a device associated with a server, a system-on-a-chip, and / or any other suitable computing device or computing system. (Refer to above) Figure 16 The described task scheduling device 1600 can take the form of a computing device 1710. Alternatively, the task scheduling device 1600 can be implemented as a computer program as an application 1716.

[0104] The example computing device 1710 shown includes a processing system 1711 communicatively coupled to each other, one or more computer-readable media 1712, and one or more I / O interfaces 1713. Although not shown, the computing device 1710 may also include a system bus or other data and command transfer system that couples the various components to each other. The system bus may include any or a combination of different bus architectures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of the various bus architectures. Various other examples, such as control and data lines, are also conceived.

[0105] Processing system 1711 represents the functionality of performing one or more operations using hardware. Therefore, processing system 1711 is illustrated as including hardware elements 1714 that can be configured as processors, function blocks, etc. This may include other logic devices implemented in hardware as application-specific integrated circuits (ASICs) or formed using one or more semiconductors. Hardware element 1714 is not limited by the materials in which it is formed or the processing mechanism employed therein. For example, a processor may consist of semiconductors and / or transistors (e.g., integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically executable instructions.

[0106] Computer-readable medium 1712 is illustrated as including memory / storage device 1715. Memory / storage device 1715 represents a memory / storage capacity associated with one or more computer-readable media. Memory / storage device 1715 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage device 1715 may include fixed media (e.g., RAM, ROM, fixed hard disk drive, etc.) and removable media (e.g., flash memory, removable hard disk drive, optical disk, etc.). Computer-readable medium 1712 may be configured in various other ways as further described below.

[0107] One or more I / O interfaces 1713 represent the functionality to allow users to input commands and information to the computing device 1710 using various input devices and optionally also to present information to the user and / or other components or devices using various output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones (e.g., for voice input), scanners, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., capable of detecting non-touch-related motion as gestures using visible or invisible wavelengths (such as infrared frequencies), etc. Examples of output devices include display devices, speakers, printers, network interface cards, haptic-responsive devices, etc. Therefore, the computing device 1710 can be configured to support user interaction in various ways as further described below.

[0108] The computing device 1710 also includes an application 1716. The application 1716 may be, for example, a software instance of a task scheduling device 1600, and may be used in combination with other elements in the computing device 1710 to implement the techniques described herein.

[0109] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform the task scheduling methods provided in the various optional implementations described above.

[0110] This document describes various technologies within the general context of software and hardware components or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc., that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," and "component" generally refer to software, firmware, hardware, or a combination thereof. The technologies described herein are characterized as platform-independent, meaning that these technologies can be implemented on a variety of computing platforms with various processors.

[0111] Implementations of the described modules and technologies may be stored on or transmitted across some form of computer-readable medium. Computer-readable medium may include a variety of media accessible by the computing device 1710. By way of example and not limitation, computer-readable medium may include "computer-readable storage medium" and "computer-readable signal medium".

[0112] In contrast to simple signal transmission, carrier waves, or signals themselves, a "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented using methods or techniques suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, hard disks, cassette tapes, magnetic tapes, disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of art suitable for storing desired information and accessible by a computer.

[0113] "Computer-readable signal medium" refers to a signal-bearing medium configured to transmit instructions, such as via a network, to hardware of computing device 1710. A signal medium typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transmission mechanism. Signal media also include any information transmission medium. The term "modulated data signal" refers to a signal in which one or more of its characteristics are set or altered to encode information. By way of example and not limitation, communication media include wired media such as wired networks or direct connections, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0114] As previously stated, hardware element 1714 and computer-readable medium 1712 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware, which in some embodiments can be used to implement at least some aspects of the techniques described herein. Hardware elements may include components of integrated circuits or systems-on-a-chip, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations or other hardware devices in silicon. In this context, hardware elements can serve as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by the hardware element, and as hardware devices for storing instructions for execution, such as the previously described computer-readable storage medium.

[0115] The foregoing combinations can also be used to implement the various techniques and modules described herein. Therefore, software, hardware, or program modules and other program modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1714. The computing device 1710 can be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Thus, for example, by using the computer-readable storage medium and / or hardware element 1714 of the processing system, modules can be implemented at least partially in hardware as modules executable as software by the computing device 1710. Instructions and / or functions can be executable / operable by one or more articles of art (e.g., one or more computing devices 1710 and / or processing system 1011) to implement the techniques, modules, and examples described herein.

[0116] In various embodiments, the computing device 1710 can be configured in various ways. For example, the computing device 1710 can be implemented as a computer-type device, including personal computers, desktop computers, multi-screen computers, laptop computers, netbooks, etc. The computing device 1710 can also be implemented as a mobile device-type device, including mobile devices such as mobile phones, portable music players, portable gaming devices, tablet computers, multi-screen computers, etc. The computing device 1710 can also be implemented as a television-type device, including devices with or connected to a generally large screen in a leisure viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0117] The techniques described herein can be supported by these various configurations of computing device 1710, and are not limited to specific examples of the techniques described herein. Functionality can also be implemented, wholly or partially, on the “cloud” 1720 using distributed systems, such as through platform 1722 as described below.

[0118] Cloud 1720 includes and / or represents platform 1722 for resource 1724. Platform 1722 abstracts the underlying functionality of the hardware (e.g., server) and software resources of cloud 1720. Resource 1724 may include applications and / or data that can be used when performing computer processing on a server located remotely from computing device 1710. Resource 1724 may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks.

[0119] Platform 1722 can abstract resources and functions to connect computing device 1710 to other computing devices. Platform 1722 can also be used to abstract resource hierarchy to provide a corresponding level of hierarchy for any encountered needs for resource 1724 implemented via platform 1722. Therefore, in interconnect device embodiments, the implementation of the functions described herein can be distributed throughout system 1700. For example, functions can be implemented partly on computing device 1710 and partly through platform 1722, which abstracts the functions of cloud 1720.

[0120] It should be understood that, for clarity, embodiments of this application have been described with reference to different functional units. However, it will be apparent that, without departing from this application, the functionality of each functional unit may be implemented in a single unit, in multiple units, or as part of other functional units. For example, functionality described as being performed by a single unit may be performed by multiple different units. Therefore, references to specific functional units are considered merely as references to the appropriate units used to provide the described functionality, and not as indicating a strict logical or physical structure or organization. Thus, this application may be implemented in a single unit, or may be physically and functionally distributed among different units and circuits.

[0121] Although this application has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Rather, the scope of this application is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may be advantageously combined, and inclusion in different claims does not imply that such a combination of features is not feasible and / or advantageous. The order of features in the claims does not imply that the features must be in any particular order of their operation. Furthermore, in the claims, the word "comprising" does not exclude other elements, and the terms "a" or "an" do not exclude a plurality. Reference numerals in the claims are provided only as explicit examples and should not be construed as limiting the scope of the claims in any way.

Claims

1. A task scheduling method, comprising: Obtain target task and current resource information, wherein the target task includes multiple categories of sub-tasks, the multiple categories of sub-tasks are related to each other, each category of sub-tasks has a corresponding task priority, maximum concurrency, and environment components, the maximum concurrency indicates the number of sub-tasks of the corresponding category are executed, the environment components represent the execution environment of the sub-tasks of the corresponding category, the execution environment is a temporary Pod container based on K8s, which is deleted after the execution of the corresponding sub-task is completed, and the current resource information indicates the current resources used to execute the target task; Based on the relationships between the sub-tasks of each category, the data flow direction between the categories of sub-tasks is determined, and the data flow direction indicates the source of the input data and the destination of the output data of each category of sub-task. Based on the multiple categories of sub-tasks and the data flow direction between the sub-tasks of each category, a basic directed acyclic graph is established, wherein the nodes of the basic directed acyclic graph represent the categories of sub-tasks, and the edges of the basic directed acyclic graph represent the data flow direction between two connected sub-tasks of different categories. The basic directed acyclic graph is split according to the maximum concurrency of each category of sub-tasks to obtain the execution directed acyclic graph, wherein the nodes of the execution directed acyclic graph represent sub-tasks, and the edges of the execution directed acyclic graph represent the data flow direction between two connected sub-tasks. Based on the task priority of each category of sub-tasks, the execution directed acyclic graph is traversed, and the execution order of each sub-task in the execution directed acyclic graph is determined according to the order in which each node is traversed. Based on the execution order of each sub-task and the current resource information, the execution environment of the corresponding sub-task category is used to execute each sub-task, thereby completing the target task.

2. The method according to claim 1, wherein determining the data flow direction between categories of sub-tasks based on the association between sub-tasks of each category includes: For every two subtasks in the multiple categories that have a direct relationship, in response to the fact that the subtask of the first category is the parent task of the subtask of the second category, the destination of the output data of the subtask of the first category is determined to be the subtask of the second category, and the source of the input data of the subtask of the second category is determined to be the subtask of the first category.

3. The method according to claim 1, wherein traversing the directed acyclic graph of execution based on the task priority of each category of sub-tasks, and determining the execution order of each sub-task in the directed acyclic graph of execution according to the order in which each node is traversed, includes: Based on the task priority of each category of sub-tasks, the execution directed acyclic graph is traversed in a depth-first traversal manner; The execution order of each sub-task is determined by the order in which each node is traversed.

4. The method according to claim 1, wherein traversing the directed acyclic graph of execution based on the task priority of each category of sub-tasks, and determining the execution order of each sub-task in the directed acyclic graph of execution according to the order in which each node is traversed, includes: Based on the task priority of each category of sub-tasks, the execution directed acyclic graph is traversed in a breadth-first traversal manner; The execution order of each sub-task is determined by the order in which each node is traversed.

5. The method of claim 1, wherein the edges of the directed acyclic graph have data storage paths, wherein the data storage paths store the output data of the parent task in one of the two subtasks connected by the edges.

6. The method of claim 5, wherein the execution of the edge of the directed acyclic graph further comprises a log table indicating the execution status of the parent task in one of the two sub-tasks connected by the edge.

7. The method according to claim 1, wherein executing each sub-task according to the execution order of each sub-task and the current resource information, using the execution environment of the corresponding category of sub-task, comprises: For each sub-task that is executed: Allocate resources to the sub-tasks based on the current resource information; Establish the corresponding execution environment based on the corresponding environment components of the sub-task; Based on the directed acyclic graph, obtain the input data of the executed sub-task; In the corresponding execution environment, the input data is processed to obtain output data.

8. The method of claim 7, wherein the input data includes at least one object to be processed, each object to be processed having a processing priority; Furthermore, processing the input data in the corresponding execution environment includes: The at least one object to be processed shall be processed according to the processing priority of the at least one object to be processed.

9. The method according to claim 1, wherein obtaining the target task includes: Obtain the task schedule, which indicates the execution time of at least one task; In response to the arrival of the execution time of one of the at least one tasks, the one task is obtained as the target task.

10. The method of claim 9, wherein each target task further has a task priority, and the method further comprises: In response to the arrival of the execution time of multiple tasks, the task to be acquired is determined according to the task priority of each task.

11. A task scheduling device, comprising: The acquisition module is configured to acquire target task and current resource information. The target task includes multiple categories of sub-tasks, which are related to each other. Each category of sub-task has a corresponding task priority, maximum concurrency, and environment components. The maximum concurrency indicates the number of sub-tasks of the corresponding category that are executed. The environment components represent the execution environment of the sub-tasks of the corresponding category. The execution environment is a temporary Pod container based on K8s, which is deleted after the execution of the corresponding sub-task. The current resource information indicates the current resources used to execute the target task. The first determining module is configured to determine the data flow direction between the sub-tasks of the multiple categories based on the association between the sub-tasks of each category, wherein the data flow direction indicates the source of the input data and the destination of the output data of each sub-task. The second determining module is configured to: establish a basic directed acyclic graph (DAG) based on the multiple categories of sub-tasks and the data flow direction between the sub-tasks of each category, wherein the nodes of the basic DAG represent the categories of sub-tasks, and the edges of the basic DAG represent the data flow direction between two connected sub-tasks of different categories; split the basic DAG based on the maximum concurrency of each category of sub-tasks to obtain an execution DAG, wherein the nodes of the execution DAG represent sub-tasks, and the edges of the execution DAG represent the data flow direction between two connected sub-tasks; traverse the execution DAG based on the task priority of each category of sub-tasks, and determine the execution order of each sub-task in the execution DAG according to the order in which each node is traversed. The execution module is configured to execute each sub-task according to the execution order of the sub-tasks and the current resource information, using the execution environment of the corresponding category of sub-tasks, thereby completing the target task.

12. A computing device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-10.

13. A computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images