Task scheduling system, method, device and electronic equipment

Abstract

The embodiment of the present invention provides a task scheduling system, method, device and electronic device, which relates to the field of computer technology. The system includes: an airflow platform and a container cloud platform; the airflow platform is used to load a specified image file containing the data content of each task to be executed, parse and process the specified image file, and obtain the data content of each task to be executed; determine the specified scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling order; the container cloud platform is used to execute the code content corresponding to the task to be executed for each task to be executed currently received, using the specified data content of the task to be executed whenever the data content of at least one task to be executed sent by the airflow platform is received. This solution can realize the scheduling and scheduling of tasks with various scheduling relationships.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling system, method, device and electronic equipment. Background Art

[0002] With the development of the information society, whether it is Internet applications or enterprise-level applications, there are a large number of tasks that need to be executed every day in the background, and there are various complex dependencies between these tasks. For example, in fund evaluation services, the entire process involves hundreds of tasks, and there are complex calling relationships between each task. Therefore, it is necessary to schedule each task to ensure that each task is executed smoothly in the order of scheduling. The so-called scheduling means arranging the execution order of each task that has a calling relationship and executing the tasks in the execution order.

[0003] In the related technology, a task scheduling platform is used to schedule and orchestrate each task, such as the xxl-job lightweight distributed task scheduling and orchestration platform; wherein each task is independently configured, and in the content of any task configuration, the name of the task to be executed subsequently needs to be stated, and after the task is completed, the execution of the subsequent task is triggered.

[0004] The related technology can only cope with the task scheduling scenario in a serial manner, for example, executing each task in sequence in a serial manner; but it cannot realize other task scheduling scenarios other than serial, for example, it is necessary to execute tasks A, B, and C first, then execute task D, and then execute task G after tasks D, E, and F are all executed. The task scheduling method of the related technology seems to be incapable of meeting the requirements. It can be seen that a new task scheduling system is urgently needed to realize the scheduling and scheduling of tasks with various scheduling relationships. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a task scheduling system, method, device and electronic device to schedule tasks with various scheduling relationships. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a task scheduling system, the task scheduling system comprising: an airflow platform and a container cloud platform;

[0007] The airflow platform is used to load a specified image file containing the data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed, and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0008] The airflow platform is also used to determine the specified scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling order; wherein the specified scheduling order is used to characterize the execution order of each task to be executed;

[0009] The container cloud platform is used to start a target container for processing the task to be executed, using the container start command in the specified data content of the task to be executed, whenever receiving the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, allocate resources to the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed.

[0010] Optionally, the designated image file also records the calling relationship of each task to be executed;

[0011] The airflow platform is also used to parse the specified image file to obtain the calling relationship of each task to be executed;

[0012] The system also includes: an orchestration and scheduling platform;

[0013] The orchestration and scheduling platform is used to obtain the calling relationship of each task to be executed from the airflow platform, and use the calling relationship to generate the specified orchestration and scheduling order, and send it to the airflow platform.

[0014] Optionally, the airflow platform is further used to generate the specified orchestration scheduling order based on the parsed calling relationship of each task to be executed and using a specified scheduling component.

[0015] Optionally, the system further includes: a devops platform and a code management platform;

[0016] The code management platform is used to receive code files containing data contents of various tasks to be executed;

[0017] The devops platform is used to pull the code file from the code management platform, package the code file into the specified image file, and move the specified image file to a specified storage path;

[0018] The airflow platform is also used to scan the specified storage path, and if the specified image file exists in the specified storage path, obtain the specified image file from the specified storage path.

[0019] Optionally, the airflow platform is also used to display the execution trace of each task to be executed after sending the data content of the task to be executed to the container cloud platform; wherein the execution trace of each task to be executed includes the execution status information of the task to be executed.

[0020] In a second aspect, an embodiment of the present invention provides a task scheduling method, which is applied to the airflow platform, and the method includes:

[0021] Loading a specified image file containing the data content of each task to be executed, parsing the specified image file to obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0022] Determine the designated orchestration and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the designated orchestration and scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, use the container start command in the designated data content of the task to be executed to start the target container for processing the task to be executed, allocate resources for the target container according to the resource parameters recorded in the pod template file in the designated data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the designated data content of the task to be executed; wherein the designated orchestration and scheduling order is used to characterize the execution order of the tasks to be executed.

[0023] In a third aspect, an embodiment of the present invention provides a task scheduling method, which is applied to a container cloud platform, and the method includes:

[0024] Whenever data content of at least one task to be executed sent by the airflow platform is received, for each task to be executed currently received, a container start command in the specified data content of the task to be executed is used to start a target container for processing the task to be executed;

[0025] Allocate resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed;

[0026] Based on the environment variables in the specified data content of the task to be executed, using the target container to execute the code content corresponding to the task to be executed;

[0027] Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of the tasks to be executed.

[0028] In a fourth aspect, an embodiment of the present invention provides a task scheduling device, which is applied to an airflow platform, and the device includes:

[0029] A parsing module, used to load a specified image file containing the data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0030] A sending module is used to determine the specified orchestration and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified orchestration and scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, the container startup command in the specified data content of the task to be executed is used to start the target container for processing the task to be executed, according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, resources are allocated to the target container, and based on the environment variables in the specified data content of the task to be executed, the code content corresponding to the task to be executed is executed using the target container; wherein the specified orchestration and scheduling order is used to characterize the execution order of the tasks to be executed.

[0031] In a fifth aspect, an embodiment of the present invention provides a task scheduling device, which is applied to a container cloud platform, and the device includes:

[0032] A startup module, for starting a target container for processing each task to be executed by using a container startup command in the specified data content of the task to be executed, whenever receiving data content of at least one task to be executed sent by the airflow platform;

[0033] An allocation module, used to allocate resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed;

[0034] An execution module, configured to execute the code content corresponding to the task to be executed using the target container based on the environment variables in the specified data content of the task to be executed;

[0035] Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of the tasks to be executed.

[0036] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0037] Memory, used to store computer programs;

[0038] The processor is used to implement any of the task scheduling methods when executing the program stored in the memory.

[0039] In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the task scheduling methods described above is implemented.

[0040] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned task scheduling methods.

[0041] Beneficial effects of the embodiments of the present invention:

[0042] The task scheduling and scheduling system provided by the embodiment of the present invention, the airflow platform can load and parse the specified image file, obtain the data content of each task to be executed, and determine the specified scheduling and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling and scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, for each task to be executed currently received, it can use the container startup command to start the target container for processing the task to be executed, and allocate resources for the target container according to the resource parameters recorded in the pod template file, and based on the environment variables, use the target container to execute the code content corresponding to the task to be executed. It can be seen that the scheme provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling and scheduling order, and the container cloud platform can use the data content of the task to be executed to execute the code content of at least one task to be executed whenever it receives the data content of at least one task to be executed. Therefore, this scheme provides a new type of task scheduling and scheduling system, which can realize the scheduling and scheduling of tasks of various scheduling relationships.

[0043] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0045] Figure 1 A schematic diagram of a task scheduling system provided by an embodiment of the present invention;

[0046] Figure 2 An interactive schematic diagram of a task scheduling system provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of an embodiment of a task scheduling system provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a dag file provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a task calling relationship provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a task execution trace provided by an embodiment of the present invention;

[0051] Figure 7 A flowchart of a task scheduling method provided by an embodiment of the present invention;

[0052] Figure 8 Another schematic diagram of a task scheduling method provided by an embodiment of the present invention;

[0053] Fig. 9 A schematic diagram of the structure of a task scheduling device provided by an embodiment of the present invention;

[0054] Fig.10 A schematic diagram of the structure of another task scheduling device provided by an embodiment of the invention;

[0055] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0057] The related technologies are only capable of simple scheduling scenarios where a single task is executed in sequence, but cannot be used for other task scheduling scenarios other than serial scheduling. In addition, the scheduling methods of the related technologies cannot meet the needs of the business system in terms of visualization of task scheduling, automation of deployment scheduling process, and task traceability of business system execution. Therefore, a new type of task scheduling system is urgently needed to realize the scheduling of tasks with various scheduling relationships.

[0058] Based on this, the embodiments of the present invention provide a task scheduling system, method, device and electronic device to schedule tasks with various scheduling relationships.

[0059] The following first describes a task scheduling system provided by an embodiment of the present invention.

[0060] Among them, the task scheduling system is suitable for scenarios where there is a need to schedule the execution order of each task to be executed, and execute each task to be executed in the order of scheduling. In addition, the scheduling order can be the scheduling order of each task to be executed for any requirement, for example: the scheduling order of executing each task to be executed in sequence in a serial manner, or at least including a scheduling order for executing multiple tasks to be executed in parallel, such as: first executing tasks A, B, C, then executing task D, and then executing task G after tasks D, E, and F are all executed. The present invention does not limit the specific application scenario and scheduling order of the system.

[0061] A task scheduling system provided by an embodiment of the present invention includes: an airflow platform and a container cloud platform.

[0062] Among them, the above-mentioned airflow platform can be a platform that includes multiple components and arranges the execution order of scheduled tasks according to the dependencies of the tasks. The multiple components included in the airflow platform may include: a metadata library component for storing task status, a scheduler component for arranging the execution order of scheduled tasks, and an executor component for determining the planned work process for executing each task, etc.

[0063] The above-mentioned container cloud platform can include multiple containers for executing various tasks to be executed; the container cloud platform can quickly build and configure a data center cluster, monitor the resource usage of the container, configure the threshold of container resource usage, and automatically expand the resources used by the container to execute the task according to the actual resource usage of the task being executed.

[0064] Among them, in a task scheduling system provided by an embodiment of the present invention,

[0065] The airflow platform is used to load a specified image file containing the data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed, and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0066] The airflow platform is also used to determine the specified scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling order; wherein the specified scheduling order is used to characterize the execution order of each task to be executed;

[0067] The container cloud platform is used to start a target container for processing the task to be executed, using the container start command in the specified data content of the task to be executed, whenever receiving the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, allocate resources to the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed.

[0068] The task scheduling and scheduling system provided by the embodiment of the present invention, the airflow platform can load and parse the specified image file, obtain the data content of each task to be executed, and determine the specified scheduling and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling and scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, for each task to be executed currently received, it can use the container startup command to start the target container for processing the task to be executed, and allocate resources for the target container according to the resource parameters recorded in the pod template file, and based on the environment variables, use the target container to execute the code content corresponding to the task to be executed. It can be seen that the scheme provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling and scheduling order, and the container cloud platform can use the data content of the task to be executed to execute the code content of at least one task to be executed whenever it receives the data content of at least one task to be executed. Therefore, this scheme provides a new type of task scheduling and scheduling system, which can realize the scheduling and scheduling of tasks of various scheduling relationships.

[0069] In the following, a task scheduling system provided by an embodiment of the present invention is described in detail with reference to the accompanying drawings.

[0070] Figure 1 A schematic diagram of a task scheduling system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system may include an airflow platform 100 and a container cloud platform 200 .

[0071] The airflow platform 100 is used to load a specified image file containing data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed, and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0072] The airflow platform 100 is also used to determine the specified scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform 200 in sequence according to the specified scheduling order; wherein the specified scheduling order is used to characterize the execution order of each task to be executed;

[0073] The container cloud platform 200 is used to start a target container for processing each task to be executed, using the container start command in the specified data content of the task to be executed, whenever it receives the data content of at least one task to be executed sent by the airflow platform 100, allocate resources to the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed.

[0074] Next, in combination with the interaction process between the airflow platform 100 and the container cloud platform 200 in the above task scheduling system, Figure 1 The task scheduling system provided by the embodiment of the present invention is specifically described.

[0075] Figure 2 A schematic diagram of an interactive relationship of a task scheduling system provided by an embodiment of the present invention, such as Figure 2 As shown, the interaction process between the airflow platform 100 and the container cloud platform 200 in the above task scheduling system may include the following steps:

[0076] S201: the airflow platform 100 loads a specified image file containing the data content of each task to be executed, and parses the specified image file to obtain the data content of each task to be executed;

[0077] The data content of each task to be executed includes the code content of the task to be executed and the specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0078] Usually, the tasks to be executed are multiple tasks with complex calling relationships. Before executing each task to be executed, the airflow platform 100 can first load a specified image file containing the data content of each task to be executed, and parse the specified image file to obtain the data content of each task to be executed. That is, the data content of each task to be executed can be contained in a specified image file, and the airflow platform 100 can parse the specified image file into the data content of each task to be executed, so that through subsequent steps, the data content of each task to be executed is sent to the container cloud platform 200 used to execute each task to be executed, so as to realize the execution of each task to be executed.

[0079] It should be noted that the designated image file may be any image file containing the data content of each task to be executed. The method for obtaining the designated image file will be described in detail later and will not be described here.

[0080] S202: The airflow platform 100 determines the specified scheduling order corresponding to each task to be executed, and sends the data content of each task to be executed to the container cloud platform 200 in sequence according to the specified scheduling order;

[0081] The specified scheduling order is used to represent the execution order of each task to be executed;

[0082] It should be noted that the above-mentioned designated image file also records the calling relationship of each task to be executed, and the airflow platform 100 is also used to parse the calling relationship of each task to be executed from the designated image file. The so-called calling relationship of each task to be executed is the dependency relationship between each task to be executed, for example: Tasks A and B need to be executed first, and then Task C can be executed.

[0083] After obtaining the calling relationship of each task to be executed, a specified orchestration scheduling order representing the execution order of each task to be executed can be generated based on the calling relationship. The airflow platform 100 can use the calling relationship to generate the above-mentioned specified orchestration scheduling order, or obtain the specified orchestration scheduling order by sending the calling relationship of each task to other platforms in the task orchestration scheduling system for generating the specified orchestration scheduling order.

[0084] Exemplarily, in one implementation, the task scheduling system further includes: a scheduling platform;

[0085] The orchestration and scheduling platform is used to obtain the calling relationship of each task to be executed from the airflow platform 100, and use the calling relationship to generate the specified orchestration and scheduling order, and send it to the airflow platform 100.

[0086] The airflow platform 100 can receive the specified orchestration and scheduling sequence sent by the orchestration and scheduling platform. At this time, the airflow platform 100 does not need to use the capabilities of the airflow platform 100 to generate the specified orchestration and scheduling sequence, and can focus on the above-mentioned steps of parsing the specified image file and / or sending the data content of each task to be executed to the container cloud platform 200 in sequence according to the specified orchestration and scheduling sequence, thereby improving the execution efficiency of the task orchestration and scheduling system for each task to be executed.

[0087] Exemplarily, in another implementation, the airflow platform 100 has a component for orchestrating and scheduling the execution order of each task to be executed. The airflow platform 100 is also used to generate the specified orchestration and scheduling order based on the parsed calling relationship of each task to be executed using a specified scheduling component.

[0088] The airflow platform 100 can also generate a specified scheduling order using the components it contains. At this time, the airflow platform 100 does not need to interact with the scheduling platform, reducing the complexity of the task scheduling system.

[0089] In addition, the airflow platform 100 does not provide an execution environment for the code content of each task to be executed, that is, the airflow platform 100 cannot directly execute the code content of each task to be executed obtained by parsing. Therefore, the airflow platform 100 can send the data content of each task to be executed to the container cloud platform 200 in sequence according to the specified scheduling order of each task to be executed, so as to execute each task to be executed. In addition, the specified scheduling order can also be called a message queue, and the message queue contains the execution order of each task to be executed in sequence. For example, the execution order of each task to be executed represented in the message queue can be A, B, and C, that is, task A needs to be executed first, and after task A is completed, task B is executed, and after task B is completed, task C is executed.

[0090] In addition, the airflow platform 100 is also used to display the execution trace of each task to be executed after sending the data content of the task to be executed to the container cloud platform 200; wherein the execution trace of each task to be executed includes the execution status information of the task to be executed.

[0091] After sending the data content of each task to be executed to the container cloud platform 200, the airflow platform 100 can also display the execution traces of each task to be executed sent to the container cloud platform 200, such as: executing, successful execution, failed execution and other execution status, and the tree view control can be used to view the execution traces of each task to be executed. In addition, the specified scheduling order determined by the airflow platform 100 can also be displayed; that is, the task scheduling system provided by the embodiment of the present invention can realize the visualization of task scheduling, the automation of deployment scheduling process, and the task traces of business system execution.

[0092] S203: Whenever the container cloud platform 200 receives data content of at least one task to be executed sent by the airflow platform 100, for each task to be executed currently received, the container start command in the specified data content of the task to be executed is used to start a target container for processing the task to be executed;

[0093] The container cloud platform 200 can receive the data content of at least one task to be executed sent by the airflow platform 100. Whenever the data content of at least one task to be executed is received, since the container cloud platform 200 uses containers to execute various tasks, and the data content of the task to be executed includes a container startup command for starting the container, for each task to be executed currently received, the container startup command in the specified data content of the task to be executed can be used to start the target container for processing the task to be executed, and the task to be executed is executed through subsequent steps.

[0094] It should be noted that the container cloud platform 200 can receive any number of pending tasks sent by the airflow platform 100 in a specified orchestration order, for example: the data content of pending task A, or the data content of pending task A, the data content of pending task B, and the data content of pending task C that need to be executed simultaneously, which are all reasonable.

[0095] In addition, in order to realize the execution of the code content of each task to be executed in accordance with the specified orchestration and scheduling order, the container cloud platform 200 can also receive the data content of the subsequent task to be executed in the specified orchestration sequence after the currently executed task to be executed is completed. For example: the execution order of each task to be executed in the specified orchestration and scheduling order is to execute task A first, then execute task B and task C, and finally execute task D. The container cloud platform 200 can first receive the data content of task A, and after task A is completed, receive the data content of task B and the data content of task C. After task B and task C are completed, receive the data content of task D, so that tasks A, B, C and D are executed in the specified orchestration and scheduling order.

[0096] S204: The container cloud platform 200 allocates resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed;

[0097] It is understandable that when each task is executed, some resources of the container cloud platform 200 need to be occupied, and these resources may be computing resources, CPU resources, memory resources, etc. Therefore, after starting the target container for processing the task to be executed, resources can be allocated to the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, so that the task to be executed can be executed smoothly.

[0098] In addition, in some scenarios, the resources allocated to the target container will change. For example, using the resource parameters recorded in the pod template file, the target container is allocated 20G of memory resources. However, in the actual task operation, the memory resources occupied by the task may be greater than 20G, for example, 25G, 30G, etc. At this time, the container cloud platform 200 can also monitor the actual operation of the task, and adjust the resource parameters allocated to the target container according to the actual resources used, so that each task can run normally.

[0099] S205: The container cloud platform 200 uses the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed;

[0100] The specified task content also contains environment variables for representing the execution environment of the task to be executed. The execution environment of the task to be executed can be: test environment, production environment, etc. In addition, the environment variables of the task to be executed can also represent the database or interface that needs to be connected when executing the task to be executed. For example, if an environment variable of the task to be executed represents that the task to be executed is a production environment, then when executing the task to be executed, the database of the generation environment corresponding to the task to be executed or the interface of the production environment can be connected. The environment variables of the task to be executed can also represent the execution entry of the task to be executed when executing the task to be executed. The so-called execution entry is the execution path for executing the task to be executed. For example, the execution entry of the task to be executed can be different disks, such as Disk 1 and Disk 2 both store the task to be executed.

[0101] Therefore, the container cloud platform 200 can also use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed. For the code content of each task to be executed received, the code content of each task to be executed can be executed in sequence according to the sending order of the specified orchestration and scheduling order of the airflow platform 100, so that each task to be executed can be completed in accordance with the specified orchestration and scheduling order.

[0102] The task scheduling and scheduling system provided by the embodiment of the present invention, the airflow platform can load and parse the specified image file, obtain the data content of each task to be executed, and determine the specified scheduling and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling and scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, for each task to be executed currently received, it can use the container startup command to start the target container for processing the task to be executed, and allocate resources for the target container according to the resource parameters recorded in the pod template file, and based on the environment variables, use the target container to execute the code content corresponding to the task to be executed. It can be seen that the scheme provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling and scheduling order, and the container cloud platform can use the data content of the task to be executed to execute the code content of at least one task to be executed whenever it receives the data content of at least one task to be executed. Therefore, this scheme provides a new type of task scheduling and scheduling system, which can realize the scheduling and scheduling of tasks of various scheduling relationships.

[0103] Optionally, in another embodiment of the present invention, the system further comprises: a devops platform and a code management platform;

[0104] The code management platform is used to receive code files containing data contents of various tasks to be executed;

[0105] The devops platform is used to pull the code file from the code management platform, package the code file into the specified image file, and move the specified image file to a specified storage path;

[0106] The airflow platform is also used to scan the specified storage path, and if the specified image file exists in the specified storage path, obtain the specified image file from the specified storage path.

[0107] It should be noted that the code file containing the data content of each task to be executed can be a code file generated according to the task execution requirements. The code file can be a dag (Directed Acyclic Graph) file, and the code management platform can be a gitlab platform, which is used to temporarily store the code file so that other platforms can pull the code file from the code management platform when necessary. Among them, devops (a combination of Development and Operations) is a general term for a set of processes, methods and systems. It is used to promote communication, collaboration and integration between development (applications or software engineering), technical operations and quality assurance departments, and can achieve close cooperation between development and operations to deliver software products and services on time.

[0108] The devops platform can pull the code file from the code management platform and package the code file into the above-mentioned designated image file, which can be a docker image of the code file, and move it to the designated storage path via the designated image file.

[0109] The airflow platform 100 can scan the specified storage path, which can be a scheduled scan or a scan according to any scanning condition. If the specified image file exists in the specified storage path, the specified image file can be obtained from the specified storage path.

[0110] It should be noted that the designated storage path may be a database or storage path agreed upon by the devops platform and the airflow platform 100, which is reasonable, and the present invention does not limit the designated storage path.

[0111] By obtaining the specified image file through the above-mentioned devops platform and code management platform, it is possible to realize the orchestration of code files for multiple tasks, or add the code content of new tasks to the code file, so that the specified image file of the code file containing the data content of each task to be executed can be delivered to the users or enterprises that need to execute each task to be executed, etc., thereby realizing the continuous delivery of complex task orchestration.

[0112] The task scheduling system provided by the embodiment of the present invention is introduced in detail below in conjunction with a specific embodiment.

[0113] like Figure 3 As shown, the task scheduling system includes: an automated deployment platform and a task scheduling platform, the automated deployment platform includes: a gitlab platform and a devops platform, the task scheduling platform includes: an airflow platform and a tke platform, and also includes a dags file existing in the interaction process. Among them, the dags file corresponds to the above-mentioned specified image file.

[0114] The automated deployment platform is used to automatically deploy the dags file containing the data content of each task to be executed through the interaction of the gitlab platform and the devops platform. The task orchestration platform is used to use the dags file to orchestrate and schedule each task to be executed in the dags file, and execute each task to be executed in the order of the orchestration and scheduling.

[0115] First, according to the needs of the task, a dag file can be generated, which corresponds to the above-mentioned code file containing the data content of each task to be executed, and the dag file can be submitted to the gitlab platform, which corresponds to the above-mentioned code management platform and can also be called a code hosting platform. This is reasonable. Figure 4 As shown, the dag file contains two parts. One part is the parameter definition of each task, including the image version (the image version of the code content of each task), pod template file, container startup command, environment variables, etc. The pod template file, container startup command, and environment variables correspond to the parameters contained in the above-specified data content. Of course, each task can also correspond to a task name: task fund_etf_lof_pool_reducer, and task fund_etf_lof_nav_reducer, which is convenient for recording and using the data content under the task name; the other part is the scheduling relationship of the task: fund_etf_lof_pool_reducer>>fund_etf_lof_nav_reducer, corresponding to the calling relationship of the above-mentioned tasks to be executed. For example: if task B is executed after task A is completed, the scheduling relationship between tasks A and B can be described as A>>B. If task B needs to be executed after both tasks A and C are completed, the scheduling relationship between tasks A, B and C can be described as two independent lines A>>B, C>>B.

[0116] The devops platform can pull the dag files saved by the gitlab platform, package the dag files into mirrored dags files, corresponding to the specified mirror files mentioned above, and move the dags files to the agreed shared cloud storage path, corresponding to the specified storage path mentioned above.

[0117] The task scheduling platform schedules the execution order of each task to be executed through the interaction of the airflow platform and the tke platform, and executes each task to be executed according to the execution order.

[0118] The airflow platform can periodically scan the agreed shared cloud storage path. When the shared cloud storage path contains a dags file, it can load and parse the dags file to obtain the data content of each task to be executed. And it can generate a call relationship diagram through the scheduling relationship of the task, such as Figure 5 As shown, the call relationship diagram represents the execution order of tasks: first execute the task fund_etf_lof_pool_reducer, then execute the task fund_etf_lof_nav_reducer. And determine the execution order of each task to be executed, corresponding to the above-mentioned specified scheduling order. The airflow platform can send task messages to the tke platform in sequence through the message queue according to the determined execution order of each task, corresponding to the above and according to the specified scheduling order, send the data content of each task to be executed to the container cloud platform in sequence; the tke platform is a container cloud platform, corresponding to the above-mentioned container cloud platform, whenever the tke platform receives the data content of at least one task, for each task currently received, it can use the container startup command in the data content of the task to start the tke container used to execute the task, corresponding to the above-mentioned target container, and according to the pod template file in the data content of the task, determine the resource parameters of the tke container, and give the environment variables in the data content of the task, and use the tke container to execute the code content of the task.

[0119] The tasks executed by the airflow platform all contain execution traces. The airflow platform can display the execution traces of each task. Corresponding to the above display of the execution traces of the tasks to be executed, the execution traces of each task can be viewed through the tree view control. Figure 6As shown in the figure, for the tasks fund_etf_lof_pool_reducer and fund_etf_lof_nav_reducer contained in the dag file, their execution traces are shown in the figure, where the horizontal axis represents the time of task execution, for example: the nth, n+1, n+2...n+5th day, and the vertical axis represents the scheduling relationship of the tasks. The dag file can be regarded as a whole task, which includes the tasks fund_etf_lof_pool_reducer and fund_etf_lof_nav_reducer. The scheduling relationship represented by the vertical axis can be that the task fund_etf_lof_pool_reducer is executed first, and then the task fund_etf_lof_nav_reducer is executed. The symbols in the figure mean: ① indicates that the task has not been received, ② indicates that the task is in execution, ③ indicates that the task is in successful execution, ④ indicates that the task is in failed execution, and ⑤ indicates that there is an error in the execution condition of the task. For example, for task B, its execution condition is that task A is completed. When an error occurs in the execution of task A, the execution status of task B will be represented as the execution status of ⑤; blank indicates that the dag file does not contain the task.

[0120] The task scheduling and dispatching system provided by the present invention uses the task scheduling function of the airflow platform to schedule each task, and the airflow platform does not provide the execution environment of the task, and can send each task to the container cloud platform in sequence by means of a message queue containing the execution order of each task. The container cloud platform can start the corresponding container, determine the resources allocated to the container and the environment variables for task execution through the data content of each task, and execute the code content corresponding to the task. Through the capabilities of the airflow platform, it is possible to schedule and dispatch tasks for various scheduling relationships, visualize the task scheduling, and leave traces of tasks executed by the business system. In order to solve the automation of the deployment and scheduling process, the present invention adopts a combination of the devops platform and the gitlab platform to achieve continuous delivery of tasks with various scheduling relationships.

[0121] Based on the above task scheduling system, the present invention also provides a task scheduling method, which is applied to the airflow platform, such as Figure 7 As shown, the method includes the following steps S701-S702:

[0122] S701: Loading a designated image file containing data contents of each task to be executed, parsing the designated image file to obtain data contents of each task to be executed;

[0123] Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed.

[0124] S702: Determine the designated orchestration and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the designated orchestration and scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, use the container start command in the designated data content of the task to be executed to start the target container for processing the task to be executed, allocate resources for the target container according to the resource parameters recorded in the pod template file in the designated data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the designated data content of the task to be executed;

[0125] The specified scheduling order is used to represent the execution order of the tasks to be executed.

[0126] It can be seen that in the solution provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, it can use the data content of the task to be executed to execute the code content of at least one task to be executed. Therefore, this solution provides a new task scheduling method, which can realize the scheduling of tasks with various scheduling relationships.

[0127] Optionally, the designated image file also records the calling relationship of each task to be executed, and the method further includes:

[0128] Parsing the specified image file to obtain the calling relationship of each task to be executed;

[0129] The calling relationship of each task to be executed is sent to the orchestration and scheduling platform, so that the orchestration and scheduling platform uses the calling relationship to generate the specified orchestration and scheduling order and sends it to the airflow platform.

[0130] Optionally, the method further comprises:

[0131] Based on the call relationship of each task to be executed obtained by parsing, the specified scheduling component is used to generate the specified orchestration scheduling sequence.

[0132] Optionally, the designated image file is a file that is pulled by the devops platform from the code management platform, packaged with code files containing data contents of each task to be executed, and saved in a designated storage path. The method further includes:

[0133] The designated storage path is scanned, and if the designated storage path has a designated image file, the designated image file is obtained from the designated storage path.

[0134] Optionally, after sending the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling order, the method further includes:

[0135] Display the execution traces of the tasks to be executed; wherein each execution trace of the tasks to be executed includes the execution status information of the tasks to be executed.

[0136] Based on the above task scheduling system, the present invention also provides a task scheduling method, which is applied to the container cloud platform, such as Figure 8 As shown, the method includes the following steps S801-S803:

[0137] S801: Whenever data content of at least one task to be executed sent by the airflow platform is received, for each task to be executed currently received, a container start command in the specified data content of the task to be executed is used to start a target container for processing the task to be executed;

[0138] S802: Allocate resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed;

[0139] S803: Based on the environment variables in the specified data content of the task to be executed, using the target container to execute the code content corresponding to the task to be executed;

[0140] Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of the tasks to be executed.

[0141] It can be seen that in the solution provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, it can use the data content of the task to be executed to execute the code content of at least one task to be executed. Therefore, this solution provides a new task scheduling method, which can realize the scheduling of tasks with various scheduling relationships.

[0142] Among them, the designated image file also records the calling relationship of each task to be executed, and the designated orchestration and scheduling order is that the airflow platform parses the calling relationship of each task to be executed from the designated image file, and sends the calling relationship of each task to be executed to the orchestration and scheduling platform. The orchestration and scheduling platform uses the calling relationship to generate and send the designated orchestration and scheduling order to the airflow platform; or, the airflow platform generates the designated orchestration and scheduling order based on the parsed calling relationship of each task to be executed using the designated scheduling component; the designated image file is the file obtained from the designated storage path when the airflow platform scans the designated storage path, and if the designated image file exists in the designated storage path.

[0143] Based on the above task scheduling method, the present invention also provides a scheduling device, which is applied to the airflow platform, such as Fig. 9 As shown, the device comprises:

[0144] The parsing module 910 is used to load the specified image file containing the data content of each task to be executed, and parse the specified image file to obtain the data content of each task to be executed;

[0145] The data content of each task to be executed includes the code content of the task to be executed and the specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed;

[0146] The sending module 920 is used to determine the designated scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the designated scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, the container startup command in the designated data content of the task to be executed is used to start the target container for processing the task to be executed, and according to the resource parameters recorded in the pod template file in the designated data content of the task to be executed, resources are allocated to the target container, and based on the environment variables in the designated data content of the task to be executed, the code content corresponding to the task to be executed is executed by using the target container;

[0147] The specified scheduling order is used to represent the execution order of the tasks to be executed.

[0148] It can be seen that in the solution provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, it can use the data content of the task to be executed to execute the code content of at least one task to be executed. Therefore, this solution provides a new task scheduling method, which can realize the scheduling of tasks with various scheduling relationships.

[0149] Optionally, the designated image file also records the calling relationship of each task to be executed, and the device further includes:

[0150] The first determination module is used to parse the calling relationship of each task to be executed from the specified image file; send the calling relationship of each task to be executed to the orchestration and scheduling platform, so that the orchestration and scheduling platform uses the calling relationship to generate the specified orchestration and scheduling order, and send it to the airflow platform.

[0151] Optionally, the device further comprises:

[0152] The second determining module is used to generate the specified arrangement scheduling order based on the call relationship of each task to be executed obtained by parsing and using the specified scheduling component.

[0153] Optionally, the designated image file is a file that is pulled by the devops platform from the code management platform, packaged with code files containing data contents of each task to be executed, and stored in a designated storage path, and the device further includes:

[0154] The acquisition module is used to scan the specified storage path, and if the specified image file exists in the specified storage path, acquire the specified image file from the specified storage path.

[0155] Optionally, the device further comprises:

[0156] The display module is used to display the execution traces of the tasks to be executed; wherein each execution trace of the tasks to be executed includes the execution status information of the tasks to be executed.

[0157] Based on the above task scheduling method, the present invention also provides a task scheduling device, which is applied to a container cloud platform, such as Fig.10 As shown, the device comprises:

[0158] The startup module 1010 is used for starting a target container for processing each task to be executed by using a container startup command in the specified data content of the task to be executed, whenever the data content of at least one task to be executed sent by the airflow platform is received;

[0159] An allocation module 1020, configured to allocate resources for the target container according to resource parameters recorded in the pod template file in the specified data content of the task to be executed;

[0160] An execution module 1030 is used to execute the code content corresponding to the task to be executed using the target container based on the environment variables in the specified data content of the task to be executed;

[0161] Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of the tasks to be executed.

[0162] It can be seen that in the solution provided by the present invention, the airflow platform can send the data content of the task to be executed to the container cloud platform according to any specified scheduling order. Whenever the container cloud platform receives the data content of at least one task to be executed, it can use the data content of the task to be executed to execute the code content of at least one task to be executed. Therefore, this solution provides a new task scheduling method, which can realize the scheduling of tasks with various scheduling relationships.

[0163] Among them, the designated image file also records the calling relationship of each task to be executed, and the designated orchestration and scheduling order is that the airflow platform parses the calling relationship of each task to be executed from the designated image file, and sends the calling relationship of each task to be executed to the orchestration and scheduling platform. The orchestration and scheduling platform uses the calling relationship to generate and send the designated orchestration and scheduling order to the airflow platform; or, the airflow platform generates the designated orchestration and scheduling order based on the parsed calling relationship of each task to be executed using the designated scheduling component; the designated image file is the file obtained from the designated storage path when the airflow platform scans the designated storage path, and if the designated image file exists in the designated storage path.

[0164] The embodiment of the present invention further provides an electronic device, such as Fig.11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0165] Memory 1103, used for storing computer programs;

[0166] The processor 1101 is used to implement any task scheduling method when executing the program stored in the memory 1103.

[0167] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the above electronic device and other devices.

[0169] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0170] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0171] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned task scheduling methods is implemented.

[0172] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the task scheduling methods in the above embodiments.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0174] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0175] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A task scheduling system, characterized in that: The task scheduling system includes: airflow platform and container cloud platform; The airflow platform is used to load a specified image file containing the data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed, and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; The airflow platform is also used to determine the specified scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified scheduling order; wherein the specified scheduling order is used to characterize the execution order of each task to be executed; The container cloud platform is used to start a target container for processing the task to be executed, using the container start command in the specified data content of the task to be executed, whenever receiving the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, allocate resources to the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the specified data content of the task to be executed.

2. The system according to claim 1, characterized in that The designated image file also records the calling relationship of each task to be executed; The airflow platform is also used to parse the specified image file to obtain the calling relationship of each task to be executed; The system also includes: an orchestration and scheduling platform; The orchestration and scheduling platform is used to obtain the calling relationship of each task to be executed from the airflow platform, and use the calling relationship to generate the specified orchestration and scheduling order, and send it to the airflow platform.

3. The system according to claim 1, characterized in that The airflow platform is also used to generate the specified orchestration scheduling order based on the call relationship of each task to be executed obtained by parsing and using the specified scheduling component.

4. The system according to any one of claims 1 to 3, characterized in that: The system also includes: a devops platform and a code management platform; The code management platform is used to receive code files containing data content of each task to be executed; The devops platform is used to pull the code file from the code management platform, package the code file into the specified image file, and move the specified image file to a specified storage path; The airflow platform is also used to scan the specified storage path, and if the specified image file exists in the specified storage path, obtain the specified image file from the specified storage path.

5. The system according to any one of claims 1 to 3, characterized in that: The airflow platform is also used to display the execution trace of each task to be executed after sending the data content of the task to be executed to the container cloud platform; wherein the execution trace of each task to be executed includes the execution status information of the task to be executed.

6. A task scheduling method, characterized in that: Applied to the airflow platform, the method includes: Loading a specified image file containing the data content of each task to be executed, parsing the specified image file to obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; Determine the designated orchestration and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the designated orchestration and scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, use the container start command in the designated data content of the task to be executed to start the target container for processing the task to be executed, allocate resources for the target container according to the resource parameters recorded in the pod template file in the designated data content of the task to be executed, and use the target container to execute the code content corresponding to the task to be executed based on the environment variables in the designated data content of the task to be executed; wherein the designated orchestration and scheduling order is used to characterize the execution order of the tasks to be executed.

7. A task scheduling method, characterized in that: Applied to a container cloud platform, the method includes: Whenever the data content of at least one task to be executed is received in sequence from the airflow platform according to the specified scheduling order, for each task to be executed currently received, the target container for processing the task to be executed is started using the container start command in the specified data content of the task to be executed; Allocate resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed; Based on the environment variables in the specified data content of the task to be executed, using the target container to execute the code content corresponding to the task to be executed; Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of each task to be executed.

8. A task scheduling device, characterized in that: Applied to the airflow platform, the device comprises: A parsing module, used to load a specified image file containing the data content of each task to be executed, parse the specified image file, and obtain the data content of each task to be executed; wherein the data content of each task to be executed includes the code content of the task to be executed and the specified data content, and the specified data content includes a pod template file for recording resource parameters of the computing resources to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; A sending module is used to determine the specified orchestration and scheduling order corresponding to each task to be executed, and send the data content of each task to be executed to the container cloud platform in sequence according to the specified orchestration and scheduling order, so that whenever the container cloud platform receives the data content of at least one task to be executed sent by the airflow platform, for each task to be executed currently received, the container startup command in the specified data content of the task to be executed is used to start the target container for processing the task to be executed, according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed, resources are allocated to the target container, and based on the environment variables in the specified data content of the task to be executed, the code content corresponding to the task to be executed is executed using the target container; wherein the specified orchestration and scheduling order is used to characterize the execution order of the tasks to be executed.

9. A task scheduling device, characterized in that: Applied to a container cloud platform, the device comprises: The startup module is used to start a target container for processing each task to be executed by using the container startup command in the specified data content of the task to be executed, whenever the data content of at least one task to be executed is sequentially sent by the airflow platform in a specified scheduling order. An allocation module, configured to allocate resources for the target container according to the resource parameters recorded in the pod template file in the specified data content of the task to be executed; An execution module, configured to execute the code content corresponding to the task to be executed using the target container based on the environment variables in the specified data content of the task to be executed; Among them, the data content of each task to be executed includes the code content of the task to be executed, and specified data content, wherein the specified data content includes a pod template file for recording resource parameters of the computing resources required to be allocated when the task is executed, a container startup command for starting the container, and environment variables required when the task is executed; the specified orchestration scheduling order is used to characterize the execution order of each task to be executed.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 6 or 7 when executing a program stored in a memory.

Images