Distributed task deployment method, device, computer equipment and storage medium

Abstract

The present application discloses a distributed task deployment method, deployment device, computer equipment and computer-readable storage medium. The deployment method includes the steps of: receiving a scheduling task and generating a mobilization request according to the scheduling task to start a scheduling program, the scheduling task includes a plurality of sequentially associated subtasks, receiving an execution request of the scheduling program and starting a first execution program according to the execution request to execute a first subtask in the subtask, starting a second execution program according to the execution result of the first execution program and the execution request to execute a second subtask in the subtask, the first subtask is a pre-associated subtask of the second subtask. In this way, the execution of interdependent subtasks is implemented by a scheduling program, avoiding frequent startup of the scheduling program, and making the task management clearer.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a distributed task deployment method, deployment device, computer equipment and computer-readable storage medium. Background Art

[0002] In related technologies, spark's distributed computing tasks can be deployed on Kubernetes (k8s) to solve the problem of resource management and allocation of multiple servers.

[0003] However, since the distributed computing tasks of Spark related to the same business are generally divided into multiple subtasks, and each subtask requires restarting a new Driver Pod to start the executor Pod, the Master node needs to start the Driver Pod frequently. Summary of the invention

[0004] The present application provides a distributed task deployment method, the deployment method comprising:

[0005] Receiving a scheduling task and generating a dispatch request according to the scheduling task to start a scheduling program (driver pod), wherein the scheduling task includes a plurality of sequentially associated subtasks;

[0006] receiving an execution request from the scheduler and starting a first executor pod according to the execution request to execute a first subtask in the subtasks; and

[0007] A second execution program is started according to the execution result of the first execution program and the execution request to execute a second subtask among the subtasks, wherein the first subtask is a predecessor associated subtask of the second subtask.

[0008] In some implementations, starting a second execution program according to the execution result of the first execution program and the execution request to execute the second subtask in the subtasks includes:

[0009] The execution result of the first executor pod is obtained from the HDFS, and the second executor pod is executed according to the execution result of the first executor pod.

[0010] In some embodiments, the deployment method further comprises:

[0011] The running state of the first execution program is monitored to determine whether the first execution program needs to be restarted.

[0012] In some embodiments, the deployment method further comprises:

[0013] Determining whether the execution result is generated;

[0014] If the execution result is not generated, restarting the first execution program to execute the first subtask again;

[0015] When the execution result is generated, it is verified whether the result file is correct.

[0016] In some implementations, the first execution program and the second execution program are executed on multiple child nodes respectively.

[0017] In certain embodiments, the scheduler includes an application program interface, and the deployment method includes:

[0018] generating a log when the first execution program and the second execution program execute the subtask,

[0019] The logs are obtained through the application program interface and the scheduler is enabled to collect and process the logs.

[0020] In some embodiments, the deployment method further comprises:

[0021] After the subtask is executed, the first execution program and the second execution program are deleted.

[0022] The present application also provides a distributed task deployment device, the deployment device comprising a master node and a plurality of sub-nodes, the sub-nodes being connected to the master node;

[0023] The sub-node is used to run a scheduling program, a first execution program and a second execution program.

[0024] The master node is used to:

[0025] Receiving a scheduling task and generating a transfer request according to the scheduling task to start a scheduling program, wherein the scheduling task includes a plurality of sequentially associated subtasks;

[0026] receiving an execution request of the scheduling program and starting a first execution program according to the execution request to execute a first subtask among the subtasks; and

[0027] A second execution program is started according to the execution result of the first execution program and the execution request to execute a second subtask among the subtasks, wherein the first subtask is a predecessor associated subtask of the second subtask.

[0028] In some embodiments, the deployment device further includes a distributed file system, which is connected to the child node; the distributed file system is used to store logs generated by the child node.

[0029] The present application provides a computer device, including one or more processors, a memory; and

[0030] One or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing any one of the distributed task deployment methods described above.

[0031] The present application provides a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the processors execute any one of the above-mentioned distributed task deployment methods.

[0032] In the distributed task deployment method, deployment device, computer equipment and computer-readable storage medium of the implementation mode of the present application, after receiving a scheduling task containing associated subtasks and starting the scheduling program, the scheduling program first starts an execution program, and after the execution program completes the execution of the subtask, the scheduling program starts the next execution program associated with the previous execution program according to the execution result of the execution program. In this way, the execution of interdependent subtasks in the scheduling task is implemented by one scheduling program, which avoids frequent startup of the scheduling program and makes task management clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 It is a flow chart of a deployment method of certain embodiments of the present application.

[0035] Figure 2 It is a module schematic diagram of a deployment device of certain embodiments of the present application.

[0036] Figure 3 It is a module schematic diagram of a computer device of certain embodiments of the present application.

[0037] Figure 4 It is a connection diagram of a processor and a computer-readable storage medium in certain embodiments of the present application.

[0038] Figure 5 It is a scenario diagram of a deployment method of certain embodiments of the present application.

[0039] Figure 6 A flowchart of a deployment method of certain embodiments of the present application.

[0040] Figure 7 It is a flow chart of a deployment method of certain embodiments of the present application.

[0041] Figure 8 It is a flow chart of a deployment method of certain embodiments of the present application.

[0042] Fig. 9 It is a flow chart of a deployment method of certain embodiments of the present application.

[0043] Fig.10 It is a scenario diagram of log collection processing in certain implementation modes of the present application.

[0044] Description of main component symbols:

[0045] Deployment apparatus 10 , master node 12 , child node 14 , storage volume 16 , computer device 100 , processor 20 , memory 30 , program 32 , readable storage medium 40 , computer program 42 . DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0047] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0048] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0049] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0051] In recent years, technologies such as the Internet of Things, big data, cloud computing, and artificial intelligence have been widely used, and big data and cloud computing technologies have also shown a trend of integration. For the computational analysis of big data, the Spark computing framework is currently widely used. The Spark computing framework is a fast and general computing engine designed for big data processing, which can be used to complete a variety of operations.

[0052] In the related art, the computing resources CPU and memory required by the Spark computing framework can be implemented by the Kubernetes (k8s) cluster, that is, the Kubernetes cluster can be used as a resource scheduler for Spark. The Kubernetes cluster is an open source containerized application for managing multiple hosts in a cloud platform, and is used to uniformly manage the resources (CPU and memory) of the host. When the Spark computing framework processes big data computing tasks, it generates scheduling tasks and sends them to the Kubernetes cluster. The Kubernetes cluster can allocate the computing CPU and memory required by the Spark computing framework.

[0053] The Kubernetes cluster includes a master node and several node nodes. Among them, the master node is the main node of the Kubernetes cluster, which is used to record the total number of CPUs and memory of the Kubernetes cluster, and start the pod for execution when Spark submits a scheduling task. The node is a child node of the Kubernetes cluster, which is used to execute the pod assigned by the master node. Pod refers to a running program that occupies a certain number of CPU cores and memory. When Spark is deployed and run on k8s, the pod includes a scheduling pod (driver pod) and an execution pod (executor pod). Usually, after the master node receives the scheduling task, it allocates a node node to start the driver pod, and the driver pod applies to the master node for resource allocation, thereby allocating one or more node nodes again to start the executor pod to execute the task.

[0054] However, a big data computing task may include multiple subtasks, and there is generally a dependency between the subtasks. The output of the calculation result of the previous subtask may be the input of the next subtask, and each subtask needs to restart a new Driver Pod to start the executor Pod, resulting in the Master node needing to frequently start the Driver Pod.

[0055] In view of this, please refer to Figure 1 The embodiment of the present application provides a distributed task deployment method, and the distributed task deployment method includes the steps of:

[0056] S12: receiving a scheduling task and generating a transfer request according to the scheduling task to start a scheduling program, wherein the scheduling task includes a plurality of sequentially associated subtasks;

[0057] S14: receiving an execution request of the scheduling program and starting a first execution program according to the execution request to execute a first subtask in the subtasks; and

[0058] S16: According to the execution result and execution request of the first execution program, the second execution program is started to execute the second subtask in the subtask, and the first subtask is a pre-associated subtask of the second subtask. Figure 2 The embodiment of the present application provides a distributed task deployment device 10. The deployment method of the present application can be applied to the deployment device 10, or in other words, the deployment method can be implemented by the deployment device 10.

[0059] The deployment device 10 includes a main node 12 and a plurality of sub-nodes 14, wherein the sub-nodes 14 are connected to the main node 12; wherein the sub-nodes 14 are used to start a scheduling program, a first execution program, and a second execution program.

[0060] Steps S12, S14 and S16 can be implemented by the master node 12, that is, the master node 12 can be used to receive a scheduling task and generate a mobilization request according to the scheduling task to start a scheduling program, the scheduling task includes a plurality of sequentially associated subtasks, and receive an execution request of the scheduling program and start a first execution program according to the execution request to execute a first subtask in the subtask. The master node 12 is also used to start a second execution program according to the execution result of the first execution program and the execution request to execute a second subtask in the subtask, the first subtask being a pre-associated subtask of the second subtask.

[0061] See also Figure 4 The present application provides a computer device 100, including one or more processors 20, a memory 30; and one or more programs 32, wherein the one or more programs 32 are stored in the memory 30 and executed by one or more processors 20, and the program 32 is executed by the processor 20 to execute instructions of a distributed task deployment method.

[0062] Please combine Figure 5 The present application provides a non-volatile computer-readable storage medium 40 containing a computer program 42. When the computer program 42 is executed by one or more processors 20, the processor 20 performs a distributed task deployment method.

[0063] In the deployment method, deployment apparatus 10, computer device 100 and computer-readable storage medium 40 of these embodiments, after receiving a scheduling task containing associated subtasks and starting the scheduling program, the scheduling program first starts an execution program, and after the execution program completes the execution of the subtask, the scheduling program starts the next execution program associated with the previous execution program according to the execution result of the execution program. In this way, the execution of interdependent subtasks in the scheduling task is implemented by one scheduling program, which avoids frequent startup of the scheduling program and makes task management clearer.

[0064] This embodiment is described by taking the computer device 100 as a computer as an example, that is, the distributed task deployment method and deployment device 10 are applied to but not limited to computers. The deployment device 10 can be hardware or software pre-installed on the computer, and can execute the distributed task deployment method when it is started and run on the computer. For example, the deployment device 10 can be a low-level software code segment of the computer or a part of the operating system.

[0065] In some implementations, the deployment apparatus 10 may be a part of the computer device 100 . In other words, the computer device 100 includes the deployment apparatus 10 .

[0066] In some embodiments, the deployment device 10 may be a discrete component assembled in a certain manner to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a computer software code segment that enables the computer to have the aforementioned functions when running on the computer.

[0067] In some embodiments, as hardware, the deployment device 10 can be independent or installed as an additional peripheral component to a computer or computer system. The deployment device 10 can also be integrated into a computer or computer system. For example, when the deployment device 10 is part of the computer device 100, the deployment device 10 can be integrated into the processor 20.

[0068] In some implementations where the deployment device 10 is a part of the computer device 100, as software, the code segments corresponding to the deployment device 10 can be stored in the memory 30 and executed by the processor 20 to implement the aforementioned functions. In other words, the deployment device 10 includes one or more of the aforementioned programs, or the aforementioned one or more programs include the deployment device 10.

[0069] In some embodiments, the computer-readable storage medium 40 may be a storage medium built into the computer device 100, such as the memory 30, or a storage medium that can be plugged into the computer device 100, such as an SD card.

[0070] It should be noted that the deployment device 10 may be a Kubernetes cluster. The master node 12 is a master node of the Kubernetes cluster, and the child node 14 is a node node of the Kubernetes cluster.

[0071] The scheduler is the driver pod in the node. It should be noted that Pod is the smallest scheduling unit in the Kubernetes cluster. Pod is allocated by the master node. Each Pod is a logical entity with certain computing resources, such as CPU and memory. There are containers in Pod. A container is like a virtual computer with its own independent memory, CPU, and file system. During the running of Pod, Pods are isolated from each other and have their own independent file system, CPU and memory. At the end of Pod, these resources will be released, and the logs generated by the running of the container will also disappear with the end of Pod.

[0072] The first executor and the second executor are executor Pods in the node. It should be noted that the first executor is associated with the second executor, and the first and second only represent the order in which the two associated executors are run. The first executor is started before the second executor.

[0073] For example, the execution programs arranged in order of precedence include execution program No. 1, execution program No. 2, execution program No. 3, and execution program No. 4. For execution program No. 2 and execution program No. 3, since execution program No. 2 is started and run before execution program No. 3, execution program No. 2 is the first execution program, and execution program No. 3 is the second execution program, and execution program No. 3 can be started according to the execution result of execution program No. 2 and the corresponding execution request. For execution program No. 3 and execution program No. 4, since execution program No. 3 is started and run before execution program No. 4, execution program No. 3 is the first execution program, and execution program No. 4 is the second execution program, and execution program No. 4 can be started according to the execution result of execution program No. 3 and the corresponding execution request.

[0074] The scheduling task is a distributed computing task for the spark computing framework. The scheduling task includes multiple associated distributed computing subtasks, and the association relationship of the subtasks in the scheduling task can be defined by the user. In the present application, the subtasks in the scheduling task are executed in order according to the association relationship between each other, and the next subtask can only be executed when the previous scheduling task is executed and generates a calculation result. For example, the subtask includes a first subtask and a second subtask. When the first subtask is executed and generates a calculation result, the second subtask can be executed.

[0075] The scheduling task is executed by the execution program. A subtask can be executed by one or more execution programs at the same time, and there is no limit on the number of execution programs. For example, the number of execution programs can be 1, 2, 3, 5, 8, 10 or even more. The specific number of execution programs can be selected according to the data calculation amount of the subtask. For example, in the present application, a subtask can be executed by two execution programs distributed on different subnodes 14. It can be understood that the larger the data calculation amount of the execution program, the more execution programs can be set, so that the processing time of the subtask can be saved and the processing efficiency of the subtask can be improved.

[0076] A predecessor-associated subtask is a subtask that is associated with the current subtask and has been executed to generate an execution result before the currently executed subtask. For example, the first subtask is a predecessor of the second subtask and is executed before the second subtask. The execution result of the first subtask is used for the second task.

[0077] Please further combine Figure 4Specifically, after receiving the scheduling task, the master node 12 generates a mobilization request, controls the subnode 14 to deploy the scheduling program, and generates an execution request after the scheduling program runs and sends it to the master node 12. After receiving the execution request, the master node 12 allocates the subnode 14 to deploy the first execution program, so that the first subtask can be executed when the first execution program runs.

[0078] Furthermore, after the first execution program completes the execution of the first subtask and generates the execution result, the scheduler generates a scheduling request again and sends it to the master node 12, and then the master node 12 assigns the subnode 14 to deploy the second execution program, and the second execution program executes the second subtask associated with the first subtask. After the second subtask is executed and generates the execution result, the scheduler generates a scheduling request again and sends it to the master node 12, and the master node 12 assigns the subnode 14 to start the subtask associated with the second subtask, and this process is repeated, that is, after the current execution program generates the execution result, the scheduler assigns a new subnode 14 to the master node 12 to deploy the next execution program according to the execution result generated by the previous execution program, until all the subtasks of the scheduled task are executed. In this way, management confusion caused by a large number of subtasks in the scheduled task is avoided.

[0079] Please combine Figure 8 In some embodiments, step S16 further comprises the steps of:

[0080] S162: Obtain the execution result of the first execution program from the distributed file system, and execute the second execution program according to the execution result of the first execution program.

[0081] Please read further Figure 2 In some embodiments, the deployment device 10 further includes a distributed file system (DSF), and the distributed file system is connected to the child node 14. Step S162 can be implemented by a scheduler. In other words, the scheduler can be used to obtain the execution result of the first execution program from the distributed file system, and execute the second execution program according to the execution result of the first execution program.

[0082] In some implementations, the processor 20 may also be configured to obtain an execution result of the first execution program from the distributed file system, and execute the second execution program according to the execution result of the first execution program.

[0083] It can be understood by those skilled in the art that a distributed file system is a way to combine files distributed on different computers into a single namespace, and to make it easier to establish a single, hierarchical multi-file server and server sharing on the network. In this application, the distributed file system may be the Hadoop Distributed File System (HDFS), which refers to a distributed file system designed to run on commodity hardware. HDFS can provide high-throughput data access, and is very suitable for applications on large-scale data sets.

[0084] Specifically, after the execution program generates an execution result after executing the subtask, the execution result can be stored in the distributed file system. After the execution result generated by the first execution program is stored in the distributed file system, the scheduler can obtain the execution result from the distributed file system, and start the execution request again according to the execution result of the first execution program, and send it to the master node 12. After receiving the execution request, the master node 12 allocates the child node 14 to start the second execution program to execute the second subtask.

[0085] Please combine Fig.10 In some embodiments, the deployment method further comprises the steps of:

[0086] S11: monitoring the running status of the first execution program to determine whether the first execution program needs to be restarted.

[0087] In some implementations, step S11 may be implemented by a scheduling program, or in other words, the scheduling program may also be used to monitor the running status of the first execution program to determine whether the first execution program needs to be restarted.

[0088] In some implementations, the processor 20 may be configured to monitor the running status of the first execution program to determine whether the first execution program needs to be restarted.

[0089] It is understandable that an exception may occur during the execution of the execution program. For example, the running memory of the child node 14 is too high, the CPU is abnormal during the execution of the execution program, or the response performance of the child node 14 is reduced, which may cause the execution program to run abnormally, and then may lead to the inability to obtain the execution result. Therefore, it is necessary to monitor the running status of the first execution program to determine whether the first execution program needs to be restarted. If the running status of the first execution program is abnormal, restart the first execution program, so as to ensure that the subsequent second execution program can be executed smoothly and finally complete the scheduling task.

[0090] In some embodiments, the deployment method further comprises:

[0091] S13, determining whether an execution result is generated;

[0092] S15, if the execution result is not generated, restart the first execution program to execute the first subtask again;

[0093] S17, verifying whether the result file is correct when the execution result is generated.

[0094] In some embodiments, steps S13, S15, and S17 may be implemented by a scheduler, or in other words, the scheduler may also be used to determine whether an execution result is generated. If an execution result is not generated, restart the first execution program to execute the first subtask again. The scheduler may also be used to verify whether a result file is correct when the execution result is generated.

[0095] In some embodiments, the processor 20 may be used to determine whether the execution result is generated and restart the first execution program to execute the first subtask again if the execution result is not generated. The processor 20 may also be used to verify whether the result file is correct when the execution result is generated.

[0096] Since the startup and operation of the second execution program is determined by the execution result generated after the first execution program executes the first subtask, it is determined by judging whether the execution result is generated and verifying the result file of the execution result after the execution result is generated to determine whether the result file is correct. If the verification succeeds, the second execution program can be started to execute the second subtask. If the verification fails, the first execution program is started again to execute the first subtask to generate the execution result, and then the result file of the generated execution result is re-verified until the verification succeeds and the second execution program is successfully started to allow the second subtask to be executed. In this way, the second execution program can be smoothly started and run to execute the second subtask.

[0097] See also Fig.10 In some embodiments, the scheduler includes an application programming interface (API), and the first execution program and the second execution program generate logs when executing the subtasks. The deployment method also includes the steps of:

[0098] S18: Obtain logs through the application program interface and enable the scheduler to collect and process the logs.

[0099] Please read further Figure 2 In some implementations, step S18 may be implemented by a scheduler, or in other words, the scheduler may be used to obtain logs through an application program interface and enable the scheduler to aggregate and process the logs.

[0100] In some implementations, the processor 20 may also be configured to obtain logs through an application program interface and enable the scheduler to aggregate and process the logs.

[0101] People in the relevant fields can understand that an application programming interface, also known as an application programming interface, is a collection of definitions, programs and protocols, through which computer software can communicate with each other.

[0102] It should be noted that logs are also generated during the operation of the first execution program and the second execution program. Since the subnode 14 will release the memory and CPU after the execution program ends, the logs generated in the subnode 14 will also disappear when the execution program ends. Therefore, in order to retain some of the generated logs, the scheduler is provided with an application program interface, and the application program may include multiple application programs, one application program interface is connected to one execution program, and then when logs are generated during the execution program, the logs can be obtained by the application program interface, so that the scheduler can collect and process the logs.

[0103] Furthermore, the deployment device also includes a storage volume, and during the operation of the scheduling program, the storage volume is connected to the scheduling program. The scheduling program transmits the collected and processed logs to the storage volume to store the logs, so that the collection and management of the logs is realized, which is conducive to the operation and maintenance personnel to grasp the operation status in time, and provides convenience for the operation and maintenance personnel.

[0104] Furthermore, after the storage volume stores the log, the scheduler deletes the execution program, so that the execution program process ends.

[0105] In addition, it should be noted that the master node 12 also includes an application program interface. During the execution of the execution program, the scheduler can monitor the execution of the execution program for the task through the application program interface of the master node 12. In this way, after the scheduler generates a log, the log can be obtained through the application program interface.

[0106] For example, in some scenarios, data analysts submit corresponding spark tasks to the master node 12 of the Kubernetes cluster. After receiving the spark task, the master node 12 generates a mobilization request after receiving the scheduling task, controls a child node 14 to deploy the scheduling program, and generates an execution request after the scheduling program runs and sends it to the master node 12. After the master node 12 receives the execution request, it allocates a child node 14 to start the execution program, and then the scheduler sends the spark task to the execution program, so that the execution program can execute the spark task when it runs. At the same time, the scheduler also sends an observation request to the master node 12, so that the master node 12 monitors the execution of the spark task by the execution program.

[0107] Further, after the master node 12 detects that the executor has completed the spark task and generated a log, the scheduler obtains the log of the executor through the application program interface, and then exports the log to the storage volume for storage. Finally, the scheduler sends an end execution request to the master node, thereby deleting the executor to end the process of the executor and end the task.

[0108] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed task deployment method, characterized in that: The deployment method includes: Receiving a scheduling task and generating a transfer request according to the scheduling task to start a scheduling program, wherein the scheduling task includes a plurality of sequentially associated subtasks; receiving an execution request of the scheduling program and starting a first execution program according to the execution request to execute a first subtask among the subtasks; and Initiating a second execution program according to the execution result of the first execution program and the execution request to execute a second subtask in the subtasks, wherein the first subtask is a pre-associated subtask of the second subtask; The deployment method further comprises: Determining whether the execution result is generated; If the execution result is not generated, restarting the first execution program to execute the first subtask again; Verifying whether the result file is correct when the execution result is generated; If the verification is successful, the second execution program can be started to execute the second subtask. If the verification fails, the first execution program is started again to execute the first subtask to generate an execution result, and then the result file of the generated execution result is re-verified until the verification is successful and the second execution program is successfully started to allow the second subtask to execute.

2. The deployment method according to claim 1, characterized in that: The starting a second execution program according to the execution result of the first execution program and the execution request to execute the second subtask in the subtasks includes: The execution result of the first execution program is obtained from the distributed file system, and the second execution program is executed according to the execution result of the first execution program.

3. The deployment method according to claim 1, characterized in that: The deployment method includes: The running state of the first execution program is monitored to determine whether the first execution program needs to be restarted.

4. The deployment method according to claim 1, characterized in that: The first execution program and the second execution program are executed on a plurality of child nodes respectively.

5. The deployment method according to claim 1, characterized in that: The scheduling program includes an application program interface, the first execution program and the second execution program generate logs when executing the subtask, and the deployment method includes: The logs are obtained through the application program interface and the scheduler is enabled to collect and process the logs.

6. The deployment method according to claim 5, characterized in that: The deployment method further comprises: After the subtask is executed, the first execution program and the second execution program are deleted.

7. A distributed task deployment device, the device being used to execute the method according to any one of claims 1 to 6, characterized in that: The deployment device includes a master node and a plurality of sub-nodes, wherein the sub-nodes are connected to the master node; Wherein, the subnode is used to run the scheduling program, the first execution program and the second execution program; The master node is used to: Receiving a scheduling task and generating a transfer request according to the scheduling task to start a scheduling program, wherein the scheduling task includes a plurality of sequentially associated subtasks; receiving an execution request of the scheduling program and starting the first execution program according to the execution request to execute a first subtask among the subtasks; and The second execution program is started according to the execution result of the first execution program and the execution request to execute a second subtask among the subtasks, wherein the first subtask is a predecessor associated subtask of the second subtask.

8. The deployment device according to claim 7, characterized in that The deployment device further includes a distributed file system and a storage volume, wherein the distributed file system and the storage volume are connected to the child node; the distributed file system is used to store the execution results of the first execution program and the second execution program; The storage volume is used to store logs generated by the first execution program and the second execution program.

9. A computer device, characterized in that: including one or more processors and memories; and One or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the distributed task deployment method according to any one of claims 1-6.

10. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the processors are enabled to execute the distributed task deployment method according to any one of claims 1 to 6.

Images