Distributed job execution method, master node, system, physical machine and storage medium

Abstract

The embodiment of the present application provides a distributed job execution method, a master node, a system, a physical machine and a storage medium, wherein the method includes: detecting an online service startup instruction and starting a management process; obtaining a startup file of a multi-task management process through the management process; and, through the management process, starting a multi-task management process based on the startup file of the multi-task management process; obtaining a job submitted by a user, performing resource scheduling management of the job through the management process, and managing the DAG corresponding to the job through the multi-task management process. The embodiment of the present application improves the stability and scalability of the distributed system by decoupling the resource scheduling management of the master node from the management of the DAG. Furthermore, the embodiment of the present application can further divide the DAG of the job into subgraphs, so that the vertices in the subgraph are executed in quasi-real-time mode, and the vertices outside the subgraph are executed in offline mode, so as to take into account the low execution latency and high resource utilization of the job execution. The embodiment of the present application can significantly improve the performance of the distributed system.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of distributed technology, and specifically to a distributed job execution method, a master node, a system, a physical machine, and a storage medium. Background Art

[0002] A distributed system is a system composed of multiple physical machines connected through communication lines. It has the characteristics of distribution, autonomy, parallelism, and globality. Using a distributed system to execute jobs submitted by users can improve the efficiency of job execution through the distributed computing capabilities of the distributed system.

[0003] Based on the widespread application of distributed systems, those skilled in the art have been committed to optimizing the execution mode of distributed jobs in distributed systems to improve the performance of distributed systems. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a distributed job execution method, a master node, a system, a physical machine and a storage medium to optimize the execution mode of distributed jobs in a distributed system and improve the performance of the distributed system.

[0005] To achieve the above objectives, the embodiments of the present application provide the following technical solutions.

[0006] In a first aspect, an embodiment of the present application provides a distributed job execution method, which is applied to a master node and includes:

[0007] Detect online service startup instructions and start the management process;

[0008] Obtaining the startup file of the multi-task management process through the management process; and starting the multi-task management process based on the startup file of the multi-task management process through the management process;

[0009] Obtain jobs submitted by users, manage resource scheduling of jobs through management processes, and manage the DAG corresponding to jobs through multi-task management processes.

[0010] In a second aspect, an embodiment of the present application provides a distributed job execution method, including:

[0011] Get the jobs submitted by the user;

[0012] Generate an execution plan for the job, where the execution plan is described by a DAG;

[0013] Configure the physical properties of vertices and connecting edges in the DAG, wherein the physical properties of the connecting edges include sequential edges and parallel edges;

[0014] Traverse each vertex in the DAG and determine whether each vertex meets the preset target conditions for joining the subgraph;

[0015] According to the judgment result of each vertex, adjust the physical properties of the connecting edges in the DAG until the vertex traversal in the DAG is completed;

[0016] After the vertex traversal in the DAG is completed, a subgraph of the DAG is obtained based on the connection edges with adjusted physical properties; the subgraph is formed by vertices connected by parallel edges; wherein the vertices in the subgraph are executed in a quasi-real-time mode, and the vertices outside the subgraph are executed in an offline mode.

[0017] In a third aspect, an embodiment of the present application provides a master node, which is configured to execute the distributed job execution method as described in the first aspect above, or the distributed job execution method as described in the second aspect above.

[0018] In a fourth aspect, an embodiment of the present application provides a distributed system, comprising: a master node and multiple working nodes; the master node is the master node described in the third aspect above, and the online service of the distributed system has one or more service versions.

[0019] In a fifth aspect, an embodiment of the present application provides a physical machine, comprising: at least one memory and at least one processor, the memory storing one or more computer executable instructions, the processor calling the one or more computer executable instructions to execute the distributed job execution method as described in the first aspect above, or the distributed job execution method as described in the second aspect above.

[0020] In a sixth aspect, an embodiment of the present application provides a storage medium, which stores one or more computer-executable instructions. When the one or more computer-executable instructions are executed, the distributed job execution method as described in the first aspect above, or the distributed job execution method as described in the second aspect above is implemented.

[0021] In the embodiment of the present application, when the online service of the distributed system is started, the master node can pull up the management process and the multi-task management process, so that for the job submitted by the user, the management process can be used for the resource scheduling management of the job, and the multi-task management process can manage the DAG corresponding to the job. The embodiment of the present application improves the stability, scalability, and maintainability of the distributed system by decoupling the resource scheduling management of the master node and the management of the DAG, and respectively assigns the management process and the multi-task management process to be responsible, thereby effectively improving the performance of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1A A schematic diagram of the distributed system structure.

[0024] Figure 1B A schematic diagram of the execution plan.

[0025] Figure 1C This is a schematic diagram of DAG.

[0026] Figure 1D A schematic diagram of the mapping between the logical diagram and the physical diagram.

[0027] Figure 2A Another structural diagram of a distributed system.

[0028] Figure 2B A flowchart of a distributed job execution method.

[0029] Figure 2C The following is a diagram showing the relationship between the MJM process and the Labor process under different service versions.

[0030] Figure 2D This is a schematic diagram of the decoupling of resource scheduling and DAG scheduling.

[0031] Figure 3A This is a schematic diagram of DAG in offline mode.

[0032] Figure 3B This is a schematic diagram of DAG in quasi-real-time mode.

[0033] Figure 3C Another flow chart of a distributed job execution method.

[0034] Figure 3D Schematic diagram of data flow of working nodes in quasi-real-time mode.

[0035] Figure 3E This is a diagram of the DAG with physical properties configured in the initial state.

[0036] Figure 3F This is a schematic diagram of DAG after cutting the subgraph.

[0037] Figure 4A The present invention is another flow chart of a distributed job execution method.

[0038] Figure 4B , Figure 4C , Figure 4D and Figure 4E These are example graphs where there is a circular dependency between the current vertex and the current subgraph.

[0039] Figure 5 Request an example graph for the subgraph's resources.

[0040] Figure 6 A block diagram of a physical machine. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] Distributed jobs can be understood as submitting jobs to distributed systems for execution. Figure 1A The schematic diagram of the structure of the distributed system is shown as an example. Figure 1A As shown, the distributed system may include: a master node 110 and multiple working nodes 120. The master node 110 and the working node 120 may be regarded as computing nodes in the distributed system, and the computing nodes may be hosted on a physical machine with data computing capabilities, and one physical machine may host one or more computing nodes.

[0043] In a distributed system, the master node 110 is a computing node used for management and control. For example, the master node 110 can manage the worker nodes 120, coordinate the concurrency and resources corresponding to the execution phase of the execution plan, etc. In some aspects, the master node 110 is the central control node in the distributed system, also known as the execution engine of the distributed system. The worker node 120 is a computing node that specifically executes the job in the distributed system, and can be managed and coordinated by the master node 110 to execute the job.

[0044] When executing a job in a distributed system, the user can submit the job to the cluster resource manager through a terminal, and the cluster resource manager pulls up the master node 110. The master node 110 can then parse the job and generate an execution plan. The execution plan describes the process of how the job data, from the initial source table, undergoes a series of data flows, executions, and changes, and finally generates output. Figure 1B The schematic diagram of the execution plan is shown as an example. Figure 1BAs shown, the execution plan may include: multiple stages (execution stages) with a hierarchical relationship. In some embodiments, the stages may be in a tree-like hierarchical structure. A stage may include one or more tasks (execution tasks). For each stage, the master node 110 may schedule multiple working nodes to execute the tasks of the stage in parallel by configuring the number of working nodes (concurrency), the resources used, etc., thereby realizing the execution of the job in the distributed system.

[0045] In some embodiments, the job is generally submitted by the terminal to the distributed system in the form of a request. In one example, the job submitted by the terminal includes a query statement for querying a database, such as a SQL (Structured Query Language) statement.

[0046] In some further embodiments, the execution plan may be described by a Directed Acyclic Graph (DAG), which includes multiple vertices and connecting edges between vertices. Figure 1C The schematic diagram of DAG is shown as an example. It should be noted that the actual number of vertices, levels, and connecting edges of DAG may be smaller than Figure 1C More complex, Figure 1C This is just a simple DAG example for ease of understanding. Figure 1C As shown, the DAG may include four vertices V1 to V4, and connecting edges 11, 12, 13, and 14. Among them, the connecting edge 11 connects vertices V1 and V2, the connecting edge 12 connects vertices V1 and V3, the connecting edge 13 connects vertices V2 and V4, and the connecting edge 14 connects vertices V3 and V4.

[0047] A vertex in a DAG can represent an independent stage in the execution plan. The edges between vertices can be directed edges, which represent the relationship between vertices. Based on the relationship pointed by the edge, the edge connecting the vertices can be an input edge of the vertex (the input edge points to the vertex) or an output edge of the vertex (the output edge points from the vertex to other vertices). For example Figure 1C In the figure, edge 12 points to V3 and is the input edge of V3; edge 14 is output by V3 and is the output edge of V3; and edge 12 is output by V1, so edge 12 is also the output edge of V1; edge 14 inputs V4, so edge 14 is also the input edge of V4.

[0048] Of the two vertices connected by a connecting edge, the vertex output by the connecting edge is called the direct upstream vertex of the other vertex, and the vertex input by the connecting edge is called the direct downstream vertex of the other vertex. Figure 1C In the example, edge 12 connects V1 and V3, V1 outputs edge 12 and edge 12 inputs V3, then V1 can be called the direct upstream vertex of V3, and V3 can be called the direct downstream vertex of V1. A vertex may have one or more direct upstream vertices and one or more direct downstream vertices. It should be noted that in addition to direct upstream vertices, a vertex may also have indirect upstream vertices. The indirect upstream vertices are not directly connected to the vertex, but are located above the vertex and are connected to the vertex through one or more vertices. For example Figure 1C In the example, V1 is located above V4 and is connected to V4 through V2 or V3, so V1 can be called the indirect upstream vertex of V4. Obviously, in addition to direct downstream vertices, a vertex may also have indirect downstream vertices. The indirect downstream vertices are not directly connected to the vertex, but are located below the vertex and are connected to the vertex through one or more vertices. For example Figure 1C In the example, V4 is at the lower layer of V1 and is connected to V1 through V2 or V3, so V4 can be called the indirect downstream vertex of V1. The upstream vertex of a vertex can include direct upstream vertices and indirect downstream vertices, and the downstream vertex of a vertex can include direct downstream vertices and indirect downstream vertices.

[0049] The execution of a vertex may depend on the directly upstream vertex, that is, there is an execution dependency relationship between the vertex and the directly upstream vertex, and the vertex can only be executed after the directly upstream vertex is executed; the execution of a vertex may not depend on the directly upstream vertex, but can be executed in parallel with the directly upstream vertex.

[0050] In some further embodiments, DAG can have two levels of representation: logical graph and physical graph. The logical graph can be considered as a natural extension of the execution plan, describing the data execution process that the user wants to achieve for the job. The physical graph reflects the physical properties of each stage of the execution plan mapped to the distributed system, describing the physical properties of each stage of the execution plan at the execution level, such as concurrency, resources, and data transmission methods.

[0051] Figure 1D The following is a schematic diagram showing the mapping between the logical diagram and the physical diagram. Figure 1D The execution plan has four stages. Figure 1DAs shown in the figure, the logical graph describes the four vertices of the execution plan (vertices V0, V1, V2, and V3) and the relationship between each vertex (for example, vertex V0 points to vertex V2, and vertex V1 and vertex V2 point to vertex V3). One vertex corresponds to a stage of the execution plan. The logical graph can reflect the data execution process of the execution plan. After mapping the logical graph to a physical graph, the physical graph can describe the physical properties such as the number of working nodes, resources (such as CPU resources, memory resources, etc.), and data transmission methods that need to be configured for each stage. For example, combined with Figure 1D For example, the physical graph describes that vertex V0 needs to be configured with 3 working nodes (concurrency is 3), and vertices V1, V2, and V3 each need to be configured with 2 working nodes (concurrency is 2). In other words, the physical graph can express the physical properties of vertices and connecting edges in the DAG. Through the physical properties of vertices and connecting edges described by the physical graph, the master node can schedule working nodes and resources for each stage, so that tasks in the stage can be executed in parallel by multiple working nodes, realizing the execution of jobs in a distributed system.

[0052] According to the different characteristics of jobs, the distributed system provides different job execution modes: offline mode and quasi-real-time mode.

[0053] Offline mode means that during the job execution process, the master node applies for resources on demand according to the resource requirements of the stage; then, based on the applied resources, it starts a process to execute the tasks in the stage. The process can be regarded as a working node used by the execution stage. In offline mode, each stage executes independently, applies for resources on demand, and uses the data drop mode to perform intermediate data shuffle (shuffle describes the process of data interaction between different stages, such as the process of data from map task output to reduce task input). In offline mode, one master node manages one job, which is aimed at jobs with massive data scale that prioritize throughput and resource utilization.

[0054] Quasi-real-time mode refers to pre-starting a service containing multiple resident processes in a distributed system. The working nodes in the resident processes build a pre-started resource pool (also called a quasi-real-time resource pool). These resident processes are in a started state when the job is submitted; therefore, during the job execution, the master node directly schedules the resident process to execute the stage to reduce the consumption caused by the need to start the process again when executing each stage. In quasi-real-time mode, gangscheduling can be used to apply for and use pre-started resources, and an integrated execution method and memory- or network-based data shuffle can be used for multiple stages. In quasi-real-time mode, one master node can manage multiple jobs, targeting small and medium-sized data-scale jobs with execution latency priority.

[0055] For distributed systems, they run tens of millions of distributed jobs every day, and most of the jobs are executed in quasi-real-time mode. With such a large number of jobs, high requirements are placed on the stability, scalability, and maintainability of distributed systems. In addition, with the update and iteration of service versions of distributed systems, there will be several major version releases and dozens of minor version releases every year. When a new service version needs to be released online, there must be a mechanism to grayscale release the new service version without affecting the released version, so as to support the switching of user jobs to the new service version in batches; therefore, distributed systems need to support the ability to support multiple service versions.

[0056] In an embodiment of the present application, the master node 110 and the working node 120 can form a Master (master) and Slave (slave) architecture of a distributed system. The master node 110 is mainly used for service management as a Master role, and the working node 120 is mainly used for the computing load of the job as a Slave role to realize the specific execution of the job. In some embodiments, the working node can be a resident working node, running in a pre-pull resource pool, and is directly scheduled by the master node 110 in quasi-real-time mode.

[0057] Figure 2A Another structural diagram of the distributed system provided by the embodiment of the present application is exemplarily shown. Figure 1A and Figure 2A As shown, Figure 2A In the distributed system shown, the master node 110 may include: an Admin process 111 and multiple MJM (MultiJobManager) processes 112 . The worker node 120 may include: an Agent process 121 and multiple Labor processes 122 .

[0058] In an embodiment of the present application, the Admin process in the master node can be used for resource scheduling management of jobs, such as managing resource scheduling of resource pools. In some embodiments, the Admin process can manage resource scheduling of pre-pull resource pools, which run resident working nodes and are mainly used for quasi-real-time mode of jobs. In other embodiments, the Admin process can also be used as an internal component of the AM (Application Master, job manager) process to manage resource scheduling of offline resource pools. Resources in offline resource pools need to be applied on the spot, rather than pre-pulled when the job is submitted. Offline resource pools are mainly used for offline mode of jobs.

[0059] The MJM process in the master node can be used to manage DAG, such as life cycle management of DAG, involving: DAG scheduling management, state collection, fault tolerance processing, etc. In some embodiments, an MJM process can manage multiple jobs at the same time. It should be noted that the number of MJM process instances that the master node actually needs to set can be dynamically adjusted (such as horizontal expansion) according to the number of jobs that the master node needs to manage simultaneously and the complexity of the jobs.

[0060] The Agent process and Labor process in the working node can be used to execute the job computing logic and report the execution results to the master node.

[0061] As an optional implementation, Figure 2B The flowchart of the distributed job execution method provided by the embodiment of the present application is shown. The flowchart of the method can be implemented by the master node. Figure 2B The method flow may include the following steps.

[0062] In step S210, an online service start instruction is detected and the Admin process is started.

[0063] When the distributed system starts the online service, the master node can detect the online service startup instruction, thereby starting the Admin process. It can be understood that when the distributed system starts the online service, users can submit jobs through the distributed system so as to use the online service to implement the execution of the job in the distributed system.

[0064] In step S211, the startup file of the MJM process is obtained through the Admin process.

[0065] In step S212, the MJM process is started through the Admin process.

[0066] After the Admin process is started, the Admin process can obtain the startup file of the MJM process and start the MJM process based on the startup file of the MJM process. For example, the Admin process can start the MJM process through the NodeManager (node ​​management) process in the master node.

[0067] In some further embodiments, for the quasi-real-time mode, since the working node needs to be started in advance, the master node can also start the Agent process of the working node through the Admin process during the execution of step S211 and step S212 (the working node here can be a resident working node in the pre-started resource pool). After the Agent process of the working node is started, the Agent process can be used to start the Labor process in the working node; for example, the Agent process can obtain the startup file of the Labor process based on the notification of the Admin process, so that the Agent process can start the Labor process based on the startup file of the Labor process.

[0068] In step S213, the job submitted by the user is obtained, resource scheduling management of the job is performed through the Admin process, and the DAG corresponding to the job is managed through the MJM process.

[0069] After the online service of the distributed system is started, users can submit jobs through the terminal. Jobs can be submitted to the distributed system by the terminal and obtained by the master node. In the master node, the master node can perform resource scheduling management for the job through the Admin process. For example, the Admin process determines the working node used to execute the job and communicates with the Agent process of the working node to determine the Labor process managed by the Agent process for executing the job. The MJM process in the master node can manage the DAG corresponding to the job, such as performing resource scheduling management of the DAG.

[0070] In some embodiments, after the Admin process in the master node obtains the job submitted by the user, the Admin process can transfer the job to the MJM process. The MJM process can generate an execution plan for the job, and the execution plan is described by DAG. Thus, the MJM process can configure the physical properties of the DAG (i.e., the physical graph that maps the DAG); further, after the Labor process in the working node is pulled up, the MJM process can communicate with the Labor process in the working node, so that the Labor process executes the job based on the physical properties of the DAG configured by the MJM process.

[0071] In the embodiment of the present application, when the online service of the distributed system is started, the master node can pull up the Admin process and the MJM process, so that for the job submitted by the user, the Admin process can be used for the resource scheduling management of the job, and the MJM process can manage the DAG corresponding to the job. The embodiment of the present application improves the stability, scalability, and maintainability of the distributed system by decoupling the resource scheduling management of the master node and the management of the DAG, and respectively assigns the Admin process and the MJM process to be responsible, thereby significantly improving the performance of the distributed system.

[0072] In some embodiments, a distributed system may provide online services to support online execution of jobs. The online services of a distributed system may have multiple service versions (e.g., old service versions and new service versions of online services, etc.). In an embodiment of the present application, the MJM process corresponding to each service version may be configured in the master node, and the Labor process corresponding to each service version may be configured in each working node. A service version may be configured with one or more corresponding MJM processes in the master node, and a service version may be configured with one or more corresponding Labor processes in a working node. In some embodiments, the MJM process and the Labor process corresponding to the same service version communicate to execute jobs of the same service version, and the MJM process and the Labor process corresponding to different service versions are isolated from each other. For example, the MJM process corresponding to a service version will only communicate with the Labor process corresponding to the same service version and execute jobs of the same service version, while the MJM process and the Labor process corresponding to different service versions are isolated from each other and do not affect each other. Therefore, for a service version of an online service, the MJM process corresponding to the service version can serve as the management role of the jobs of the service version (the MJM process corresponding to a service version can manage multiple jobs of the service version), and the Labor process corresponding to the service version can serve as the execution role of the jobs of the service version.

[0073] As an example, Figure 2C The figure shows the relationship between the MJM process and the Labor process under different service versions. Figure 2C As shown, the master node 110 can set MJM processes 0 to n to manage the jobs of service versions 0 to n respectively, for example, MJM process 0 manages the jobs of service version 0, MJM process 1 manages the jobs of service version 1, and so on. Each working node 120 can set Labor processes 0 to n to execute the jobs of service versions 0 to n respectively, for example, Labor process 0 in a working node executes the jobs of service version 0, Labor process 1 executes the jobs of service version 1, and so on. The MJM process 0 corresponding to service version 0 communicates with the Labor process 0 corresponding to service version 0, the MJM process 1 corresponding to service version 1 communicates with the Labor process 1 corresponding to service version 1, and so on, so that the MJM process and the Labor process are isolated from each other between different service versions and do not affect each other.

[0074] It should be noted that Figure 2C only uses an example in which one service version corresponds to one MJM process in the master node and one Labor process in a working node. The embodiment of the present application can also support a situation in which one service version corresponds to multiple MJM processes in the master node.

[0075] In some embodiments, the MJM process manages the life cycle of the DAG, for example, the scheduling of the execution plan, the execution process, etc. are described through the DAG. An MJM process can manage the concurrent execution of multiple DAGs and has the ability to scale horizontally. In some embodiments, under multiple service versions of online services, the MJM process corresponding to each service version can be adaptively increased or decreased according to the traffic of the corresponding service version to achieve dynamic load balancing of the MJM process and ensure efficient processing of each service version under multiple service versions. For example, when the master node detects that the traffic of a service version is lower than a preset traffic threshold, it can reduce the number of MJM processes corresponding to the service version.

[0076] The distributed system provided in the embodiment of the present application sets the Admin process and the MJM process in the master node, thereby decomposing the resource scheduling management and DAG scheduling management of the online service into two parts, which are managed by the Admin process and the MJM process respectively, thereby achieving full decoupling of resource scheduling and DAG scheduling. As an example, Figure 2D FIG. 1 shows a schematic diagram of the decoupling of resource scheduling and DAG scheduling. Figure 2D As shown in the figure, the Admin process manages resources (such as pre-pull resource pools) and accepts external job requests as an API Server (interface server); at the same time, the Admin process can manage the MJM process, redirect the submitted job to the specific MJM process corresponding to the service version according to the service version corresponding to the job and the load balancing of each MJM process under the service version, so as to achieve service version control and load balancing of MJM processes. After receiving the job, the MJM process will create a DAG for each job, manage the life cycle of the entire DAG according to the characteristics of the DAG, and communicate with the Labor process of the same service version to complete the execution of the job. Figure 2D In the figure, MJM process 0 can be considered as the MJM process corresponding to service version 0, MJM process 1 is the MJM process corresponding to service version 1, DAG0 is the DAG corresponding to the job of service version 0, and DAG1 is the DAG corresponding to the job of service version 1.

[0077] It should be noted that the distributed system will have a large number of quasi-real-time mode jobs every day, which need to be run on the pre-pull-up resident working nodes. Therefore, managing a large number of quasi-real-time mode jobs poses severe challenges to the stability and scalability of the distributed system. In the embodiment of the present application, the Admin process manages the pre-pull-up resource pool, and the MJM processes of different service versions respectively implement DAG scheduling for jobs of different service versions, which can improve the execution stability of a large number of jobs in the quasi-real-time mode. And by dynamically adjusting (deleting, adding) the MJM processes of different service versions in the master node, the scalability of the distributed system for service versions is improved.

[0078] As an optional implementation, when the online service of the distributed system is started, the master node will start the Admin process. Then, the master node applies for resources from the resource pool through the Admin process, and starts the Agent process in the working node through NodeManger (node ​​management). After that, the master node can determine the service version list of the online service based on the user configuration through the Admin process (the service version list can record multiple service versions of the online service); thus, the Admin process can obtain the startup file of the MJM process corresponding to each service version based on each service version of the online service recorded in the service version list; further, the master node can start the MJM process corresponding to each service version through the Admin process based on the startup file of the MJM process corresponding to each service version. In the process of the Admin process starting the MJM process, the Admin process can transmit the service version list to the Agent process of the working node, so that the Agent process can obtain the startup file of the Labor process corresponding to each service version; further, the working node can start the Labor process corresponding to each service version through the Agent process based on the startup file of the Labor process corresponding to each service version.

[0079] After the MJM process is started, the MJM process can report the process status to the Admin process; similarly, after the Labor process is started, the Labor process will report the process status to the Agent process. The Agent process will summarize the process information of the Labor processes corresponding to the multiple service versions it manages, and report it to the Admin process. In a distributed system, the Agent process in a working node and the Labor process it manages can share a resource group, and the resources such as CPU and memory occupied by the Labor process in a working node cannot exceed the upper limit of the shared resource group. In some further embodiments, in order to avoid multiple Labor processes executing user jobs at the same time, resulting in excessive resource occupation, the embodiments of the present application may stipulate that at the same time, only one Labor process under an Agent process executes a job.

[0080] In some further embodiments, users can dynamically add or delete service versions through RPC (Remote Procedure Call) requests. In some embodiments, if the service version added by the user's request is the first service version, the master node can obtain the new request for the first service version, and the Admin process will send the new information of the first service version to the Agent process; at the same time, the Admin process obtains the startup file of the MJM process that starts the first service version, and based on the startup file of the MJM process of the first service version, adds and pulls up the MJM process of the first service version in the master node. After the Agent process in the working node obtains the new information of the first service version, it can obtain the startup file of the Labor process of the first service version; further, the Agent process can add and pull up the Labor process of the first service version in the working node based on the startup file of the Labor process of the first service version.

[0081] In some embodiments, when a user requests to delete a service version, if the service version requested to be deleted by the user is the second service version, the master node can obtain the deletion request of the second service version, so that the Admin process can send the deletion information of the second service version to the Agent process; at the same time, the Admin process can notify the NodeManager (node ​​management) to stop executing the MJM process corresponding to the second service version; and the Agent process will also notify the NodeManager to stop executing the Labor process corresponding to the second service version.

[0082] In some further embodiments, if a distributed system mounts a large number of service versions, but there is no job submission under a certain service version, a large number of Labor processes will be idle, resulting in a waste of resources in the distributed system. Based on this, if the master node detects that no job is submitted for the third service version within a specified time interval, the Admin process will send a notification to the Agent process, so that the Agent process can stop executing the Labor process corresponding to the third service version based on the notification. After the Labor process corresponding to the third service version stops executing, if the master node subsequently obtains a job request for the third service version, the Admin process can send a notification to the Agent process, so that the Agent process can re-start the Labor process corresponding to the third service version based on the notification. In this way, the embodiment of the present application can effectively save resources when there is no job submission for a certain service version for a long time, and at the same time does not affect the execution of jobs submitted again under the service version.

[0083] In some embodiments, in quasi-real-time mode, user jobs will be submitted to the Admin process of the master node. The user can indicate the service version they want to use in the job. If the job indicates that the service version they want to use is the fourth service version, the master node can transfer the job to the MJM process corresponding to the fourth service version through the Admin process. The MJM process will request resources to execute the job from the Admin process; thus, the master node can respond to the resource request of the MJM process, select an idle target Agent process (all Labor processes under the idle Agent process are in an idle state) through the Admin process, and return the Labor process information of the fourth service version under the target Agent process to the MJM process. Furthermore, the MJM process and the Labor process corresponding to the fourth service version can communicate to implement job execution under the fourth service version.

[0084] After the target Agent process returns the Labor process information of the fourth service version to the MJM process, the target Agent process stops scheduling jobs different from the fourth service version until the job execution of the fourth service version is completed. In some embodiments, when executing a job, the Labor process corresponding to the fourth service version may report the job execution information to the MJM process corresponding to the fourth service version, so that the MJM process corresponding to the fourth service version can obtain the job execution information from the Labor process corresponding to the fourth service version to collect the job status. When the job execution of the fourth service version is completed, the Labor process under the target Agent process will notify the target Agent process, and the target Agent process will send the notification to the Admin process; thus, the Admin process can obtain the notification from the target Agent process, and based on the notification, the master node can re-mark the target Agent process as idle through the Admin process.

[0085] The embodiment of the present application realizes the support capability of the distributed system for multiple service versions by configuring the MJM process corresponding to each service version on the master node, and configuring the Labor process corresponding to each service version on the working node, and the MJM process corresponding to a service version communicates with the Labor process corresponding to the same service version, and the MJM process and the Labor process are isolated from each other and do not affect each other between different service versions. In addition, in terms of adding and deleting service versions, the embodiment of the present application can provide a corresponding MJM process and Labor process adjustment mechanism, which can effectively support the horizontal expansion capability of the distributed system for multiple service versions, realize dynamic load balancing capability, and ensure the efficient scheduling capability of the distributed system for resources and DAG. Furthermore, through the grayscale mechanism of the service version Flighting (functional testing completed by the release team through submitting jobs after the new service version is released, which has no impact on user jobs), it is possible to complete the testing, online, offline and other operations for different service versions, and ensure that all operations are transparent and imperceptible to users. Furthermore, for multiple service versions, the embodiments of the present application can ensure that the resources of the distributed system can be effectively utilized by stopping the Labor process for the idle service version and restoring the Labor process when there is a subsequent job request, while ensuring that the functions of the service version are complete. The embodiments of the present application can significantly improve the performance of the distributed system.

[0086] Based on the offline mode and online mode provided by the distributed system, the offline mode uses the on-demand resource application method to execute jobs, which can ensure a high resource utilization rate but cannot achieve a low execution latency. The quasi-real-time mode uses the resident process pre-launched method to execute jobs, which can ensure a low execution latency for jobs but will also use a high amount of resources. In other words, the offline mode is aimed at pursuing high throughput, and the quasi-real-time mode is aimed at pursuing low latency. Therefore, there are significant differences in the performance indicators of the offline mode and the quasi-real-time mode in various aspects. If the offline mode is used alone to execute jobs, although efficient resource utilization can be achieved, the execution latency of the job cannot be sufficiently reduced; if the quasi-real-time mode is used alone to execute jobs, although the execution latency of the job can be effectively reduced, it will result in higher resource consumption.

[0087] Based on this, in order to seek an effective balance between the efficient resource utilization of the offline mode and the low execution latency of the quasi-real-time mode, an embodiment of the present application provides a hybrid execution mode to execute jobs. The hybrid execution mode can execute jobs in a more fine-grained and general manner than the simple offline mode and the quasi-real-time mode, thereby achieving an effective balance between efficient resource utilization and low execution latency.

[0088] For ease of understanding, the embodiment of the present application introduces the concept of a subgraph (bubble). For the vertices within the bubble, the master node directly schedules the resident process to execute the stage (specifically, execute the task in the stage) based on the quasi-real-time mode. For the vertices outside the bubble, the master node applies for resources according to the resource requirements of the stage, that is, the vertices outside the bubble are executed in offline mode. In terms of the embodiment of the data transmission method, the vertices in the bubble apply for running resources together, and the data of the upstream and downstream working nodes corresponding to the vertices in the bubble are directly transmitted through the network and memory; and the data on the connecting edges outside the bubble are transmitted by dropping data to the disk.

[0089] Based on this, offline mode and quasi-real-time mode can be considered as two extreme scenarios of bubble execution: in offline mode, each vertex in the DAG is treated as a single bubble; in quasi-real-time mode, all vertices of the DAG are treated as a large bubble. Figure 3A The DAG schematic diagram in offline mode is shown as an example. Figure 3A A dotted box in the figure represents a bubble. It can be seen that in offline mode, each vertex in the DAG is regarded as a single bubble. Figure 3B The DAG diagram in quasi-real-time mode is shown as an example. Figure 3B It can be seen that in quasi-real-time mode, all vertices in the DAG are regarded as a large bubble.

[0090] The hybrid execution mode provided in the embodiment of the present application is based on further segmenting the DAG into bubbles. By flexibly and adaptively segmenting the DAG into subgraphs, the embodiment of the present application can provide a more fine-grained, more general, and more reasonable job execution mode (i.e., hybrid execution mode) between the two extremes of offline mode and quasi-real-time mode, achieving an effective balance between high resource utilization and low execution latency for job execution.

[0091] Figure 3C Another flow chart of the distributed job execution method provided by an embodiment of the present application is shown. The method flow can be implemented by the master node. In some embodiments, the method flow can be executed by the MJM process in the master node. In other possible implementations, the method flow can also be executed by the Admin process in the master node. Figure 3C As shown, the method flow may include the following steps.

[0092] In step S310, an execution plan of the job is generated, and the execution plan is described by DAG.

[0093] After the terminal submits a job to the distributed system, the master node in the distributed system can parse the job and generate an execution plan for the job, which is described by a DAG. The DAG may include multiple vertices and connecting edges between vertices.

[0094] In step S311 , physical properties of vertices and connecting edges in the DAG are configured, where the physical properties of the connecting edges include sequential edges and concurrent edges.

[0095] After generating the execution plan, the master node can configure the physical properties of the vertices and connecting edges in the DAG to make the DAG a physical graph. At this time, the physical properties of the DAG are the physical properties in the initial state.

[0096] In the initial state, the physical properties configured by the master node for the connection edge may include sequential edge and concurrent edge. Among them, the sequential edge indicates that there is a sequential execution dependency relationship between the vertices connected by the connection edge, that is, the execution of the vertex needs to depend on the execution of the direct upstream vertex; that is, among the vertices connected by the sequential edge, the direct upstream vertex needs to be executed first, and then the direct downstream vertex is executed. The parallel edge indicates that the vertices connected by the connection edge are executed in parallel at the same time. In some embodiments, the vertices connected by the parallel edge can be executed in quasi-real-time mode, and the vertices connected by the sequential edge can be executed in offline mode. In some further embodiments, based on the concept of bubble, the physical properties of the sequential edge and the parallel edge of the connection edge can coincide with the data transmission mode of the connection edge; for example, the sequential edge transmits data in the form of data disk, which corresponds to the offline mode; the parallel edge transmits data in the form of memory or network connection, which corresponds to the quasi-real-time mode. It should be noted that in possible implementations, the parallel edge can also transmit data by data disk, and the data transmission mode of the memory or network connection of the above parallel edge and the data disk transmission mode of the sequential edge are only an optional implementation.

[0097] As an optional implementation, in the initial state, when the master node configures physical properties for the connection edge, it can determine the barrier vertex among the multiple vertices of the DAG, configure the output connection edge of the barrier vertex as a sequential edge; and configure the other connection edges of the non-sequential edges in the DAG as parallel edges. In some embodiments, the barrier vertex can be considered as a vertex containing a barrier operator. The barrier operator can be considered as an operator in the vertex that contains barrier characteristics and may block the data pipeline.

[0098] It should be noted that for the quasi-real-time mode, whether the data can be effectively pipelined between the working nodes largely determines whether the downstream working nodes will waste resources due to being in an idle state. Figure 3D The following is an exemplary diagram showing the data flow of working nodes in quasi-real-time mode. Figure 3D As shown, Figure 3D A box in the figure represents a worker node. In quasi-real-time mode, all worker nodes are started when the job is submitted. Although this scheduling method allows data to be pipelined when needed, thereby accelerating data processing, not all upstream and downstream worker nodes can have an ideal pipelined dataflow. For example, except for the root worker node (such as Figure 3D In the example, M3, M4, M6 and M7 are root working nodes), the downstream working nodes will be in a certain degree of idle state to some extent, resulting in a waste of resources (for example Figure 3D The downstream working nodes except M3, M4, M6 and M7 will be in a certain degree of idling state).

[0099] The inefficient use of resources caused by the above-mentioned idling is particularly obvious when there are barrier operators in the data processing flow and the pipeline cannot be performed, and when the DAG level is relatively deep. Based on this, in the embodiment of the present application, when the DAG is divided into subgraphs, and the tasks of the vertices in the subgraph are executed in quasi-real-time mode, and the tasks of the vertices of the non-subgraph are executed in offline mode, when the vertex of the DAG contains an operator with barrier characteristics and may block the data pipeline, the embodiment of the present application does not consider cutting the vertex and its downstream vertex into the same subgraph. In one example, operators that rely on global sort operations (such as MergeJoin operators, SorteAggregate operators, etc.) are considered to cause data blocking, while operators based on hash characteristic operations are more friendly to the pipeline. For example, the barrier operator can be a MergeJoin operator, a SorteAggregate operator, and the like.

[0100] Figure 3E The schematic diagram of the DAG with physical attributes configured in the initial state is shown as an example. Figure 3E The content shown is only used as an example. Figure 3E As shown, the example shows 10 vertices (V1 to V10) of the DAG and connecting edges between the vertices, and the 10 vertices form three levels, namely, the 0th level, the 1st level, and the 2nd level. The 0th level can be considered as the top level, and the 2nd level can be considered as the bottom level. Figure 3EThe vertices in the dashed boxes represent barrier vertices, the dashed connecting edges represent sequential edges, and the solid connecting edges represent parallel edges. Figure 3C If vertices V1 and V3 contain barrier operators, V1 and V3 can be marked as barrier vertices, and the output connecting edges of V1 and V3 will be configured as sequential edges in the initial state, while the remaining connecting edges will be configured as parallel edges in the initial state. It should be noted that Figure 3E The value in the vertex indicates the concurrency of the vertex configuration (for example, the number of worker nodes configured for the vertex). For example, 100 in V1 means that vertex V1 is configured with 100 worker nodes. Concurrency is an optional form of physical attribute of vertex configuration.

[0101] In step S312, each vertex in the DAG is traversed to determine whether each traversed vertex meets the preset target condition for joining the subgraph.

[0102] In step S313, the physical properties of the connecting edges in the DAG are adjusted according to the judgment result of each vertex until the vertex traversal in the DAG is completed.

[0103] The physical graph of DAG is realized by materializing the physical properties of vertices and edges in the logical graph (such as data transmission mode, scheduling timing, resource characteristics, etc.). However, the physical properties of the edges configured in the initial state are not necessarily reasonable. Therefore, the master node can traverse each vertex in the DAG and adjust the physical properties of the edges based on the physical properties of each vertex and the relationship between the vertex and other vertices in the DAG, thereby making a clearer and more reasonable description of the offline mode and quasi-real-time mode of the vertices in the DAG.

[0104] Under the above ideas, the embodiment of the present application can pre-set the target conditions for vertices to join the subgraph. Based on the target conditions, after the master node configures the physical properties of the DAG in the initial state, it can traverse each vertex in the DAG and determine whether each traversed vertex meets the target conditions. Then, based on the judgment result of each vertex, the physical properties of the connecting edges in the DAG are adjusted until the vertex traversal in the DAG is completed, so as to achieve a clearer and more reasonable description of the offline mode and quasi-real-time mode of the vertices in the DAG. For example, when the master node traverses to a vertex, based on the judgment result of whether the vertex meets the target conditions, the master node can adjust the physical properties of the connecting edges in the DAG; thus, the above process is executed once each time a vertex is traversed until the vertex traversal in the DAG is completed, so as to achieve continuous iterative adjustment of the physical properties of the connecting edges in the DAG.

[0105] In some embodiments, traversing each vertex in the DAG can be performed in a certain traversal direction. For example, when the DAG has multiple levels, the master node can traverse in a traversal direction from the top layer to the bottom layer (i.e., traversing the vertices in the DAG from top to bottom), or it can traverse in a traversal direction from the bottom layer to the top layer (i.e., traversing the vertices in the DAG from bottom to top).

[0106] In some embodiments, since the traversal direction of the vertices is consistent with the iterative direction of further slicing the DAG into subgraphs, different traversal directions may result in different subgraphs for further slicing of the DAG. Based on this, as an optional implementation, the embodiment of the present application may determine the traversal direction of the vertices according to the data processing direction of the online job. For example, for most online jobs, the processed data is often in the shape of an inverted triangle, and the corresponding DAG is also mostly in the shape of an inverted triangle. Therefore, the traversal direction from the bottom layer to the top layer can be used to traverse the vertices in the DAG, that is, traversing from the vertex farthest from the root vertex to iteratively slice the subgraph. In one example, combined with Figure 3E As shown, the main node can start traversing from the vertices of the second layer, and after traversing the vertices of the second layer, it can traverse the vertices of the first layer, and so on, until the vertices of the zero layer are traversed.

[0107] In the process of the master node traversing the vertices of the DAG, each time the master node traverses a vertex, it can determine whether the vertex meets the target condition, and adjust the physical properties of the connecting edges in the DAG based on the judgment result, so as to iteratively adjust the physical properties of the connecting edges, so that in the process of traversing the vertices of the DAG, the master node can continuously iteratively split the subgraph, and finally complete the splitting of the final subgraph after all vertices are traversed. For example, in the initial state, the current subgraph of the DAG in the initial state can include vertices connected by parallel edges, and in the process of traversing the vertices of the DAG, after the master node adjusts the physical properties of the connecting edges in the DAG once, the subgraph can be adjusted once, for example, after adjusting the physical properties of the connecting edges once, the current subgraph is formed based on the vertices connected by the adjusted parallel edges, so that in the process of traversing the vertices, the current subgraph is continuously iteratively adjusted in this way until all vertices are traversed, and the subgraph finally adjusted is used as the subgraph splitting result of the DAG.

[0108] In some embodiments, the target condition for a vertex to join a subgraph can be set based on the physical properties of the vertex and the relationship (e.g., dependency) between the vertex and other vertices in the DAG. For ease of explanation, the vertex currently traversed by the master node is referred to as the current vertex. In an optional implementation, determining whether the current vertex meets the target condition may include: there is no sequential edge connection between the current vertex and the current subgraph in the DAG; and there is no circular dependency relationship between the current vertex and the current subgraph.

[0109] In some embodiments, after the master node obtains the judgment result of whether the current vertex meets the target condition, if the current vertex meets the target condition, the master node may search the direct upstream vertex and the direct downstream vertex of the current vertex. If the searched vertex and the current vertex cannot be task-connected, the input connection edge of the searched vertex is set to a sequential edge to adjust the physical properties of the connection edge in the DAG when the current vertex meets the target condition; if the current vertex does not meet the target condition, the master node may set the input connection edge of the current vertex to a sequential edge to adjust the physical properties of the connection edge in the DAG when the current vertex does not meet the target condition.

[0110] In step S314, after the vertex traversal in the DAG is completed, a subgraph of the DAG is obtained based on the connecting edges whose physical properties are adjusted; the subgraph is formed by vertices connected by parallel edges; wherein the vertices in the subgraph are executed in a quasi-real-time mode, and the vertices outside the subgraph are executed in an offline mode.

[0111] After completing the vertex traversal in the DAG, the master node completes the physical property adjustment of the connecting edges in the DAG, and completes the final subgraph segmentation through continuous iteration of the current subgraph. For example, in the process of traversing the vertices, based on the continuous adjustment of the connecting edges with physical properties, the master node can continuously and iteratively adjust the vertices in the current subgraph (the vertices in the current subgraph are always formed by vertices connected by parallel edges. When the parallel edges are adjusted to sequential edges, the corresponding vertices will be adjusted and kicked out of the current subgraph); so that when traversing to the last vertex, based on the judgment result of the last vertex, the master node adjusts the physical properties of the connecting edges to obtain the final subgraph. In the final subgraph obtained, the vertices can be configured to execute in quasi-real-time mode, and the vertices outside the subgraph can be configured to execute in offline mode.

[0112] In one example, Figure 3F Schematic diagram of DAG after cutting subgraphs. Figure 3F As shown, in the process of traversing from the second layer to the 0th layer, the main node adjusts the physical properties of the connecting edge once after traversing each vertex based on the method described above. After traversing the nodes of the 0th layer, Figure 3F The final subgraphs shown are: subgraph 0 and subgraph 1. Among them, the tasks corresponding to the vertices V2, V4, V7 and V8 in subgraph 0 can be integrated and parallel executed based on the quasi-real-time mode, the tasks corresponding to the vertices V6 and V10 in subgraph 1 can be integrated and parallel executed based on the quasi-real-time mode, and the tasks corresponding to the vertices V1, V3, V5 and V9 outside the subgraph are executed based on the offline mode.

[0113] As an optional implementation, steps S311 to S314 can be considered as the implementation process of configuring the physical properties of the DAG, which involves configuring the physical properties of the DAG in the initial state, and continuously adjusting the physical properties of the connecting edges in the process of traversing the vertices of the DAG until the final subgraph segmentation is achieved.

[0114] In the distributed job execution method provided in the embodiment of the present application, the master node can generate an execution plan for the job, and describe the execution plan by DAG, and then configure the physical properties of the vertices and connecting edges in the DAG in the initial state, and the physical properties of the connecting edges include sequential edges and parallel edges. Since the physical properties of the connecting edges configured in the initial state are not necessarily reasonable, the master node can traverse each vertex in the DAG, and determine whether each traversed vertex meets the preset target conditions for joining the subgraph, and then adjust the physical properties of the connecting edges in the DAG based on the judgment results of each vertex until the vertex traversal in the DAG is completed. After the vertex traversal in the DAG is completed, the master node can obtain the final subgraph of the DAG, and the final subgraph can be formed by aggregating the vertices connected by the final parallel edges while continuously adjusting the physical properties of the connecting edges during the traversal process; that is, during the vertex traversal process, the master node continuously adjusts the physical properties of the connecting edges, and finally aggregating the vertices connected by the parallel edges to form the final subgraph. When the final subgraph is obtained, the vertices in the subgraph are executed in quasi-real-time mode, and the vertices outside the subgraph are executed in offline mode, thereby achieving a clearer and more accurate description of the offline mode and quasi-real-time mode of the vertices in the DAG. The embodiment of the present application can make a clearer and more accurate description of the offline mode and quasi-real-time mode of the vertex by continuously adjusting the physical properties of the connecting edges during the traversal of the vertex, thereby providing a hybrid execution mode of the hybrid quasi-real-time mode and the offline mode.

[0115] The embodiment of the present application traverses the vertices of the DAG to continuously adjust the physical properties of the connecting edges in the DAG, thereby flexibly and adaptively dividing the DAG into subgraphs based on the connecting edges with continuously adjusted physical properties, so that the execution mode of the vertices in the DAG is clearer and more accurate; therefore, based on the subgraph obtained by the final division, the embodiment of the present application can propose a finer-grained, more general, and more reasonable job execution mode for the DAG in the case of a mixed quasi-real-time mode and an offline mode, taking into account the low execution latency of the quasi-real-time mode and the high resource utilization of the offline mode, achieving an effective balance between high resource utilization and low execution latency during job execution, and significantly improving the performance of the distributed system.

[0116] As an optional implementation, Figure 4A Another flow chart of the distributed job execution method provided by the embodiment of the present application is shown. The method flow can be implemented by the master node. Figure 4A The method flow may include the following steps.

[0117] In step S410, an execution plan of the job is generated, and the execution plan is described by DAG.

[0118] In step S411, physical properties of vertices and connecting edges in the DAG are configured, where the physical properties of the connecting edges include sequential edges and parallel edges.

[0119] In step S412, the vertices of the DAG are traversed in order from the bottom layer to the top layer.

[0120] In step S413, for the current vertex currently traversed, determine whether there is no sequential edge connection between the current vertex and the current subgraph, and whether there is no circular dependency between the current vertex and the current subgraph; if any of the judgment results is no, execute step S414; if all the judgment results are yes, execute step S415.

[0121] In some embodiments, in the initial state of the DAG graph, the current subgraph includes vertices connected by parallel edges configured in the initial state. If the current vertex is the first traversed vertex, the current vertex can be judged as shown in step S413 with the current subgraph in the initial state. After the first traversed vertex, since the physical properties of the connecting edges in the DAG may be adjusted, the current subgraph formed based on the vertices connected by parallel edges may also be adjusted; based on this, if the current vertex is not the first traversed vertex, the current vertex can be judged as shown in step S413 based on the adjusted current subgraph.

[0122] In the judgment of step S413, if there is any "no" in the judgment result, that is, there is a sequential edge connection between the current vertex and the current subgraph, or there is a circular dependency relationship between the current vertex and the current subgraph, it can be determined that the current vertex does not meet the target condition, and step S414 is subsequently performed. In the judgment of step S413, if the judgment results are all "yes", that is, there is no sequential edge connection between the current vertex and the current subgraph, and there is no circular dependency relationship between the current vertex and the current subgraph, it can be determined that the current vertex meets the target condition, and step S415 is subsequently performed.

[0123] In some embodiments, the current vertex and the current subgraph do not have a circular dependency relationship, which may include:

[0124] (1) There is no vertex in the downstream of the current vertex that is upstream of the current subgraph. The downstream vertices of the current vertex can include the downstream direct vertices and downstream indirect vertices of the current vertex. If any downstream vertex of the current vertex is upstream of any vertex in the current subgraph, then there is a vertex in the downstream of the current vertex that is upstream of the current subgraph, and there is a circular dependency relationship between the current vertex and the current subgraph. As an example, Figure 4BAn example of a circular dependency relationship between the current vertex and the current subgraph is shown. Figure 4A As shown, V1 is the current vertex, the subgraph 41 where V2 and V4 are located is the current subgraph, and V3 is the downstream vertex of V1. Since V3 is located upstream of V4 (V4 is located in the subgraph 41), it can be determined that there is a vertex in the downstream vertices of the current vertex that is located upstream of the current subgraph, and there is a circular dependency relationship between the current vertex and the current subgraph.

[0125] (2) There is no vertex in the upstream of the current vertex that is downstream of the current subgraph. The upstream vertices of the current vertex can include the upstream direct vertices and upstream indirect vertices of the current vertex. If any upstream vertex of the current vertex is downstream of any vertex in the current subgraph, then there is a vertex in the upstream of the current vertex that is downstream of the current subgraph, and there is a circular dependency relationship between the current vertex and the current subgraph. As an example, Figure 4C Another example of a circular dependency between the current vertex and the current subgraph is shown. Figure 4C As shown, V3 is the current vertex, the subgraph 42 where V0, V2, V4 and V5 are located is the current subgraph, and V1 is the upstream vertex of V3. Since V1 is located downstream of V0, it can be determined that there is a vertex among the upstream vertices of the current vertex that is located downstream of the current subgraph, and there is a circular dependency relationship between the current vertex and the current subgraph.

[0126] (3) There is no vertex in the downstream subgraph of the current vertex that is upstream of the current subgraph. The DAG can currently have multiple subgraphs. The subgraph currently existing in the DAG is formed by the vertices connected by the current parallel edges, and the subgraph currently existing in the DAG can include the downstream subgraph downstream of the current vertex (for example, the current subgraph where the downstream vertex of the current vertex is located), and the upstream subgraph upstream of the current vertex (for example, the current subgraph where the vertex upstream of the current vertex is located). Take any subgraph currently existing in the DAG as the current subgraph, so that when judging whether there is no circular dependency relationship between the current vertex and the current subgraph, based on the judgment of item (3), if there is any vertex in the downstream subgraph of the current vertex that is upstream of any vertex in the current subgraph, then the current vertex and the current subgraph have a circular dependency relationship.

[0127] As an example, Figure 4D Another example of a circular dependency relationship between the current vertex and the current subgraph is shown. Figure 4DAs shown, V1 is the current vertex, and subgraph 43 where V2 and V4 are located and subgraph 44 where V3 and V5 are located are two subgraphs currently existing in the DAG. Subgraph 43 and subgraph 44 are both downstream subgraphs of V1. When subgraph 43 is used as the current subgraph to determine whether there is no circular dependency with V1, there is a vertex V5 in the downstream subgraph 44 of V1 that is located upstream of vertex V4 (V4 is located in subgraph 43). Therefore, it can be determined that there is a vertex in the downstream subgraph of the current vertex that is located upstream of the current subgraph, and there is a circular dependency between the current vertex and the current subgraph. When subgraph 44 is used as the current subgraph to determine whether there is no circular dependency with V1, the judgment process of item (3) is implemented in the same way as above.

[0128] (4) There is no vertex in the upstream subgraph of the current vertex that is downstream of the current subgraph. When judging whether there is no circular dependency between the current vertex and the current subgraph, based on the judgment of item (4), if there is any vertex in the upstream subgraph of the current vertex that is downstream of any vertex in the current subgraph, then there is a circular dependency between the current vertex and the current subgraph. As an example, Figure 4E Another example of a circular dependency relationship between the current vertex and the current subgraph is shown. Figure 4E As shown, V4 is the current vertex, and subgraph 45 where V0, V3, V5 and V6 are located, and subgraph 46 where V1 and V2 are located are two subgraphs currently existing in the DAG. When subgraph 45 is used as the current subgraph and V4 is used to determine whether there is no circular dependency, there is a vertex V1 in the upstream subgraph 46 of V4 that is located downstream of vertex V0 (V0 is located in subgraph 45). Therefore, it can be determined that there is a vertex in the upstream subgraph of the current vertex that is located downstream of the current subgraph, and there is a circular dependency between the current vertex and the current subgraph. When subgraph 46 is used as the current subgraph and V4 is used to determine whether there is no circular dependency, the judgment process of item (4) is implemented in the same way as above.

[0129] In some embodiments, if the above items (1), (2), (3) and (4) are all satisfied, it can be determined that there is no circular dependency between the current vertex and the current subgraph.

[0130] In step S414, the input connection edge of the current vertex is set as a sequential edge.

[0131] When the current vertex being traversed does not meet the target condition, the master node can set the physical property of the input connection edge of the current vertex to a sequential edge. Figure 3E and Figure 3F As shown, assuming that vertex V9 does not meet the target condition, then Figure 3F The input connection edge of V9 in can be set from parallel edge to sequential edge. For example, the input connection edge of V9 Figure 3E and Figure 3FThe input connection edge from V5 to V9 and the input connection edge from V6 to V9.

[0132] In step S415, the direct upstream vertex and the direct downstream vertex of the current vertex are searched. If the searched vertex cannot be task-connected, the input connection edge of the searched vertex is set as a sequential edge.

[0133] When the current vertex being traversed meets the target condition, the master node can search for the direct upstream vertex and direct downstream vertex of the current vertex (that is, search for vertices directly connected to the current vertex through input connection edges, and vertices directly connected to the current vertex through output connection edges). If the searched vertex cannot be task-connected with the current vertex, the physical property of the input connection edge of the searched vertex is set to a sequential edge.

[0134] In some embodiments, the master node may search the direct upstream vertex of the current vertex first according to the breadth-first traversal algorithm, and then search the direct downstream vertex of the current vertex; thereby judging whether the task connection can be performed for the direct upstream vertex that is searched first, and then searching the direct downstream vertex, and judging whether the task connection can be performed for the direct downstream vertex that is searched. In some embodiments, whether the searched vertex can perform task connection indicates whether the concurrency of the searched vertex is too high; when the concurrency of the vertex is too high, if the vertex is also added to the subgraph, the total concurrency of the subgraph is too high, which will lead to an increase in the probability of failure of the vertices in the subgraph when applying for resources in an integrated manner. Therefore, the embodiment of the present application may consider that the vertex cannot perform task connection when the concurrency of the vertex is too high. As an optional implementation, the embodiment of the present application may set a concurrency threshold to judge whether the concurrency of the searched vertex is higher than the concurrency threshold. If the concurrency of the searched vertex is higher than the concurrency threshold, the searched vertex cannot perform task connection; if the concurrency of the searched vertex is not higher than the concurrency threshold, the searched vertex can perform task connection. When the searched vertex cannot be connected for task connection, the input connection edge of the searched vertex needs to be set as a sequential edge.

[0135] It should be explained that, in the quasi-real-time mode of integrated resource application / operation, when there are too many working nodes configured for the vertex (the concurrency is too high), it may not be possible to apply for resources for the working node, or even if resources can be applied for the working node, the cost of subsequent execution failure may be uncontrollable. Therefore, the embodiment of the present application needs to limit the total size of the concurrency of the subgraph (for example, by limiting the concurrency of the vertices added to the subgraph to not be higher than the concurrency threshold) to avoid the negative impact of too many working nodes for integrated operation.

[0136] Combination Figure 3FAs shown, assuming that the current vertex V2 meets the target condition, according to the breadth-first traversal algorithm, since V2 has no upstream node, the downstream nodes V4, V5 and V6 directly connected to the output connection edge of V2 can be directly searched, among which, the concurrency of V5 is 500 higher than the concurrency threshold, so V5 cannot perform task connection, and the input connection edge of V5 needs to be set to a sequential edge. It should be noted that the specific value of the concurrency threshold can be set according to the actual situation, and the embodiment of the present application is not limited.

[0137] In step S416, after the vertex traversal in the DAG is completed, a subgraph of the DAG is obtained based on the connecting edges whose physical properties are adjusted; the subgraph is formed by vertices connected by parallel edges; wherein the vertices in the subgraph are executed in a quasi-real-time mode, and the vertices outside the subgraph are executed in an offline mode.

[0138] In the process of traversing vertices from the bottom layer to the top layer, each traversed vertex is regarded as the current subgraph, and steps S413 to S416 are iteratively executed. In the process of traversing the vertices of the DAG, the master node can continuously adjust the physical properties of the connecting edges in the DAG, and adjust the physical properties of the connecting edges once for each vertex traversed. Based on the physical properties of the connecting edges adjusted each time, the subgraph currently existing in the DAG will continuously change, thereby making the subgraph iteratively segmented, so that when the last vertex is traversed, the master node can complete the final subgraph segmentation and obtain the subgraph of the DAG.

[0139] Combination Figure 3F As shown in the figure, after traversing all vertices, the vertices V2, V4, V7 and V8 connected by the parallel edges indicated by the solid lines can form subgraph 0, and the vertices V5 and V10 connected by the parallel edges can form subgraph 1. Subgraph 0 and subgraph 1 are the final subgraphs of DAG. After completing the subgraph segmentation of DAG, the vertices in the subgraph can be executed based on the quasi-real-time mode, and the vertices outside the subgraph can be executed based on the offline mode. Figure 3F As shown, vertices V2, V4, V7 and V8 in subgraph 0 can be executed in parallel based on the quasi real-time mode, vertices V6 and V10 in subgraph 0 can be executed in parallel based on the quasi real-time mode, and vertices V1, V3, V5 and V9 outside the subgraph can be executed based on the offline mode.

[0140] In some further embodiments, in data shuffle, the embodiments of the present application may adopt different shuffle methods according to the characteristics of different connection edges inside and outside the subgraph. For the sequential edges outside the subgraph or crossing the bourndary (boundary) of the subgraph (for example, the sequential edges connected to the subgraph), the data shuffle method may be the same as the offline mode, that is, the shuffle data is transmitted by dropping data to the disk. The connection edges within the subgraph (that is, the connection edges connecting the vertices in the subgraph) are parallel edges, which have sufficient data pipeline characteristics and the amount of data transmitted is not large. Therefore, for the data on the parallel edges within the subgraph, the network and memory direct connection method similar to the quasi-real-time mode can be used to implement data shuffle.

[0141] In some further embodiments, in the process of dividing the DAG subgraph, it is also necessary to consider information such as the actual resources and expected running time of the task, such as whether the plan memory of the working node exceeds a certain value, whether the working node contains UDF (user-defined) operators, and whether the estimated execution time of the working node based on historical information during the job execution is too long. In order to further accelerate the computational execution of the job, in some embodiments, the master node can set the working node of the subgraph to come from the pre-pulled working node (for example, the pre-pulled working node in the pre-pulled resource pool). In other possible implementations, in order to achieve flexible pluggability, when necessary, the embodiment of the present application also allows the working node of the subgraph to apply for resources on the spot.

[0142] In some further embodiments, after the subgraph is triggered for scheduling, the master node may apply for resources from the resource management through the Admin process. For example, the default method of applying for resources is an integrated group scheduling method. Figure 5 An example diagram of resource application for a sub-graph is shown. Figure 5 As shown in the figure, when applying for resources in a group scheduling manner, the host node will use the subgraph as a bubble to build the resource request of the subgraph as a whole, and then control the Admin process to allocate resources based on the request. When the Admin process receives the resource request from the subgraph, it will use the corresponding resources in the pre-pull-up resource pool to allocate the working node, thereby realizing the resource scheduling of the subgraph during the job execution process.

[0143] In some further embodiments, in order to achieve dynamic adjustment of resources in a subgraph under tight resource conditions, the subgraph may also support a progressive resource application mode. This mode allows each vertex in the subgraph to apply for resources independently. For such an application, as long as the Admin has incremental resource scheduling, it will schedule the incremental resource scheduling results to the vertices in the subgraph until the resource applications of all vertices in the subgraph are met.

[0144] In some embodiments, after the subgraph is split, the embodiment of the present application can realize a hybrid execution mode based on the split subgraph, making full use of the network, direct memory connection and pre-pulling of working nodes in the subgraph to improve performance, while the vertices outside the subgraph are executed in offline mode, thereby achieving an effective balance between high resource utilization and low execution latency during job execution, significantly improving the performance of the distributed system. As an optional implementation, in the implementation of splitting subgraphs for DAG, the master node can split multiple subgraphs after analyzing the physical properties of the vertex such as the input data volume, operator characteristics, and job scale. For example, the master node can consider the following factors when splitting subgraphs for DAG:

[0145] Operator characteristics of vertices in the DAG. For example, when a vertex contains an operator with barrier characteristics that may block the data pipeline, in the initial state, consider not cutting the vertex and its downstream vertices into the same subgraph, so that the output connection edge of the vertex containing the barrier operator is initially configured as a sequential edge;

[0146] The iteration direction of subgraph segmentation: Considering that the data processed by online jobs is in the shape of an inverted triangle, and most DAGs are also in the shape of an inverted triangle, the vertices in the DAG can be traversed from the bottom layer to the top layer.

[0147] Set the target condition for adding vertices to the subgraph. In the process of traversing the vertices, try to aggregate the vertices of the DAG. When the vertex does not meet the target condition, restore the corresponding parallel edges to sequential edges, so that the subgraph is finally formed by vertices connected by more accurate and reasonable parallel edges.

[0148] In some further embodiments, the subgraphs of the DAG segmentation can be fed back to the user terminal by the master node, so that the user can manually adjust the subgraphs in the DAG, so that the subgraphs of the DAG segmentation are involved in the manual adjustment process, adding human experience and requirements, and further improving the accuracy of the final subgraph segmentation results in the DAG. Based on this, the master node can feed back the DAG containing the subgraphs to the terminal after completing the subgraph segmentation of the DAG, so that after the terminal obtains the adjustment result of the subgraph, the master node can obtain the DAG containing the adjusted subgraph from the terminal; wherein, after the terminal obtains the DAG containing the subgraph fed back by the master node, the terminal user can manually use the terminal to adjust the subgraph, so that the terminal obtains the subgraph adjustment result after the user manually adjusts it.

[0149] In some further embodiments, after the master node has split multiple DAGs into subgraphs, it can form learning experience based on the subgraph splitting results of each DAG, so that based on the learning experience, the master node can train a neural network model that can automatically split the DAG into subgraphs, so as to use the neural network model to realize efficient and accurate subgraph splitting of the DAG. As an optional implementation, after the master node has split a DAG into subgraphs, it can compare the splitting result of the current subgraph with the historical subgraph splitting result of the DAG, calculate the difference between the splitting result of the current subgraph and the historical subgraph splitting result, and optimize the neural network model through the difference (for example, optimize and adjust the parameters of the neural network model), so that the neural network model can be continuously iterated and optimized after the master node completes the subgraph splitting of the DAG each time, so as to improve the accuracy of the neural network model splitting the subgraph. Based on this, the master node can determine the difference between the DAG subgraph and the historical subgraph after completing the subgraph splitting of the DAG, so as to train or optimize the neural network model according to the difference, wherein the neural network model has the ability to split the DAG into subgraphs. In other possible implementations, the neural network model can also learn based on the results of manual adjustments after segmenting the subgraphs, thereby improving the accuracy of the subgraph segmentation effect through manual intervention.

[0150] The hybrid execution mode provided in the embodiment of the present application enables the job execution process to be more flexible and adaptive to match the characteristics of various online jobs, which is of great significance:

[0151] On the one hand, the hybrid execution mode makes it possible to accelerate more jobs. The quasi-real-time mode of integrated parallel scheduling has an entry threshold for the size of the job, and is mainly for jobs with small and medium data sizes. This is for the fair use of limited resources and to control the consumption caused by the failure of the work node. However, for medium and large jobs, although the size of the job exceeds the entry threshold of the quasi-real-time mode, the subgraph in the DAG may be a data size that meets the quasi-real-time mode, and the execution can be accelerated through methods such as data pipeline. Therefore, through the hybrid execution mode, it is possible to execute some or all subgraphs in the DAG of medium and large jobs in the quasi-real-time mode. In addition, some online work nodes cannot be pre-pulled up due to their own characteristics, which makes it impossible for the work nodes to use the quasi-real-time mode to execute jobs. At present, the single use of offline mode or quasi-real-time mode will make a job as long as it contains a work node that cannot be pre-pulled up. It can only be executed in offline mode, and cannot be accelerated in quasi-real-time mode. The hybrid execution mode can better solve this problem.

[0152] On the other hand, the hybrid execution mode connects the online offline resource pool and the quasi-real-time pre-pull-up resource pool. As two online resources with different characteristics, the offline resource pool and the quasi-real-time resource pool are currently completely isolated and managed separately. The separation of the offline resource pool and the quasi-real-time resource pool will lead to irrational use and waste of resources. For example, for large-scale operations, because the quasi-real-time resource pool cannot be used at all, they can only queue up and wait for resources in the offline resource pool, even if the quasi-real-time resource pool is idle at this time. The hybrid execution mode provided in the embodiment of the present application can achieve mixed use of offline resource pools and quasi-real-time resource pools through DAG sub-graph segmentation, so that the two complement each other.

[0153] On the other hand, the hybrid execution mode can improve resource utilization as a whole. From the perspective of resource utilization, for medium-sized jobs that can meet the quasi-real-time mode access requirements, although the use of the quasi-real-time mode can reduce the execution latency of the job and improve the execution speed of the job, it will objectively cause a certain degree of idling and waste of resources (especially when the DAG graph has a deep hierarchy and the calculation logic has barriers). In this case, the embodiment of the present application can divide a DAG in quasi-real-time mode into multiple subgraphs by setting the concurrency of vertices, confirming barrier operators and other conditions, thereby effectively reducing idling consumption, and when the subgraph segmentation conditions are set reasonably, the loss in execution latency can also be kept low.

[0154] On the other hand, the hybrid execution mode can effectively reduce the cost of a single worker node's execution failure. In the integrated quasi-real-time mode, due to the characteristics of the data pipeline, the fault tolerance granularity of the job and the granularity of resource scheduling are closely linked. In other words, as long as one worker node fails to execute, the entire job must be rerun. The larger the job scale, the greater the probability of a worker node failure during the job running process, which undoubtedly limits the scale of jobs that the quasi-real-time mode can support. The hybrid execution mode provides a better balance point, that is, the failure of a single worker node will at most affect the subgraph where the worker node is located, but will not affect all the worker nodes of the DAG.

[0155] In the case of cutting the subgraph and providing a hybrid execution mode in the embodiment of the present application, the hybrid execution mode provided by the embodiment of the present application, the standard offline mode, and the standard quasi-real-time mode are compared in performance, and it can be found that: from the perspective of execution delay, the hybrid execution mode is obviously better than the standard offline mode, and compared with the standard quasi-real-time integrated execution mode, the execution delay of the hybrid execution mode has not increased significantly; from the perspective of resource consumption, the low execution delay of the standard quasi-real-time job is based on the premise that the resource consumption is much greater than the hybrid execution mode and the standard offline mode, while the hybrid execution mode has a lower execution delay than the standard offline mode, while the resource consumption is similar to that of the standard offline mode on the whole. Therefore, in summary, the hybrid execution mode provided by the embodiment of the present application can well combine the advantages of the offline mode and the quasi-real-time mode: at the level of execution delay, the execution delay of the hybrid execution mode is shorter than that of the offline mode, close to the execution delay of the quasi-real-time mode; at the level of resource consumption, the hybrid execution mode is basically equivalent to the offline mode, which is greatly reduced compared to the quasi-real-time mode.

[0156] The embodiments of the present application can face distributed jobs of various scales and various computing characteristics, and make adaptive sub-graph segmentation, realizing the mixed use of multiple resources and execution modes within a job. The hybrid execution mode provided by the embodiments of the present application has high versatility, and when executing jobs, the embodiments of the present application can fully combine the offline offline resource pool and the pre-pulled quasi-real-time resource pool to achieve better resource utilization. The embodiments of the present application achieve an effective balance between high resource utilization and low execution latency, significantly improving the performance of distributed systems.

[0157] The embodiment of the present application also provides a master node, which can be configured to execute the distributed job execution method provided by the embodiment of the present application.

[0158] The embodiment of the present application further provides a distributed system, the structure of which can be combined with the description of the corresponding part above, and the distributed system can include the master node mentioned above. The online service of the distributed system can have one or more service versions.

[0159] The present application embodiment also provides a physical machine, which can be used to set the master node provided by the present application embodiment. As an optional implementation, Figure 6 A block diagram of a physical machine is shown. Figure 6As shown, the physical machine may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4. In an embodiment of the present application, the number of processor 1, communication interface 2, memory 3 and communication bus 4 is at least one, and the processor 1, communication interface 2 and memory 3 complete mutual communication through the communication bus 4. Optionally, the communication interface 2 may be an interface of a communication module for network communication. Optionally, the processor 1 may be a CPU (central processing unit), a GPU (Graphics Processing Unit), an NPU (embedded neural network processor), an FPGA (Field Programmable Gate Array), a TPU (tensor processing unit), an AI chip, a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present application. The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Among them, the memory 3 stores one or more computer executable instructions, and the processor 1 calls the one or more computer executable instructions to execute the distributed job execution method provided in the embodiment of the present application.

[0160] An embodiment of the present application also provides a storage medium, which can store one or more computer-executable instructions. When the one or more computer-executable instructions are executed, the distributed job execution method provided in the embodiment of the present application is implemented.

[0161] The embodiment of the present application also provides a computer program, which can be executed to implement the distributed job execution method provided in the embodiment of the present application.

[0162] The above describes multiple embodiments provided by the embodiments of the present application, and each optional method introduced in each embodiment can be combined and cross-referenced with each other without conflict, thereby extending a variety of possible embodiments, which can be considered as embodiments disclosed and disclosed by the embodiments of the present application. Although the embodiments of the present application are disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application should be based on the scope defined by the claims.

Claims

1. A distributed job execution method, wherein: The method is applied to a master node, and the method comprises: Detect online service startup instructions and start the management process; Obtaining the startup file of the multi-task management process through the management process; and starting the multi-task management process based on the startup file of the multi-task management process through the management process; Obtain jobs submitted by users, manage resource scheduling of jobs through management processes, and manage the directed acyclic graph (DAG) corresponding to jobs through multi-task management processes; The management of the DAG corresponding to the job includes: Generate an execution plan for the job, where the execution plan is described by a DAG; the DAG includes a plurality of vertices and connecting edges between the vertices; Configure the physical properties of vertices and connecting edges in the DAG, wherein the physical properties of the connecting edges include sequential edges and parallel edges; Traverse each vertex in the DAG and determine whether each vertex meets the preset target conditions for joining the subgraph; According to the judgment result of each vertex, adjust the physical properties of the connecting edges in the DAG until the vertex traversal in the DAG is completed; After the vertex traversal in the DAG is completed, a subgraph of the DAG is obtained based on the connection edges with adjusted physical properties; the subgraph is formed by vertices connected by parallel edges; wherein the vertices in the subgraph are executed in a quasi-real-time mode, and the vertices outside the subgraph are executed in an offline mode.

2. The method according to claim 1, wherein: Also includes: Through the management process, pull up the agent process of the working node; The proxy process is used to pull up the working process of the working node. The proxy process and the working process execute the computing logic of the job and transmit the execution result to the master node; The multi-task management process of the master node communicates with the working process of the working node, so that the working process executes the job based on the physical properties of the DAG configured by the multi-task management process.

3. The method according to claim 2, wherein: The online service has multiple service versions, one service version is configured with one or more corresponding multi-task management processes in the master node, and one corresponding working process is configured in a working node; the multi-task management process and the working process of the same service version communicate to execute the jobs of the same service version, and the multi-task management processes and the working processes of different service versions are isolated from each other.

4. The method according to claim 3, wherein: The step of obtaining the startup file of the multi-task management process through the management process includes: obtaining a service version list of the online service through the management process, wherein the service version list records multiple service versions of the online service; and obtaining the startup file of the multi-task management process corresponding to each service version based on each service version recorded in the service version list through the management process; The starting of the multi-task management process based on the startup file of the multi-task management process through the management process includes: starting the multi-task management process corresponding to each service version based on the startup file of the multi-task management process corresponding to each service version through the management process; The method further comprises: The service version list is transmitted to the agent process of the working node through the management process, so that the agent process starts the working process corresponding to each service version on the working node.

5. The method according to claim 3 or 4, wherein: Also includes: Obtain a new addition request for the first service version, obtain a startup file of the multi-task management process of the first service version through the management process, and based on the startup file of the multi-task management process of the first service version, add and start the multi-task management process of the first service version; And, sending the newly added information of the first service version to the agent process through the management process, so that after the agent process obtains the startup file of the working process of the first service version, the agent process adds and starts the working process of the first service version in the working node; and / or, obtaining a deletion request for the second service version, stopping the execution of the multi-task management process corresponding to the second service version through the management process, and sending the deletion information of the second service version to the agent process through the management process, so that the agent process stops executing the work process corresponding to the second service version; and / or, if it is detected that no job is submitted for the third service version within a specified time interval, sending a notification to the agent process through the management process, so that the agent process stops executing the work process corresponding to the third service version; And / or, after the work process corresponding to the third service version stops executing, if a job request for the third service version is obtained, a notification is sent to the agent process through the management process so that the agent process restarts the work process corresponding to the third service version.

6. The method according to claim 3 or 4, wherein: The job indicates that it is desired to use a fourth service version, and the method further includes: Transferring the job to the multi-task management process of the fourth service version through the management process; In response to the resource request of the multi-task management process of the fourth service version, the management process selects an idle target proxy process, and returns the work process information of the fourth service version under the target proxy process to the multi-task management process of the fourth service version; wherein all the work processes under the idle proxy process are in an idle state; Obtaining job execution information from the work process of the fourth service version through the multi-task management process of the fourth service version to collect the job status; When the execution of the job of the fourth service version is completed, the target agent process is re-marked as idle based on the notification obtained by the management process from the target agent process; wherein the target agent process stops scheduling jobs different from the fourth service version until the execution of the job of the fourth service version is completed.

7. The method according to claim 3 or 4, wherein: The agent process in a working node and the working process managed by the agent process share a resource group; the resources occupied by the working process in a working node do not exceed the upper limit of the shared resource group; And at the same time, there is only one worker process executing a job under a proxy process.

8. The method according to claim 1, wherein: The traversal of each vertex in the DAG includes: Traverse each vertex in the DAG from the bottom layer to the top layer, or from the top layer to the bottom layer. The step of adjusting the physical properties of the connecting edges in the DAG according to the judgment result of each vertex includes: If the current vertex traversed does not meet the target condition, the input connection edge of the current vertex is set as a sequential edge; If the current vertex meets the target condition, the direct upstream vertex and the direct downstream vertex of the current vertex are searched. If the searched vertex cannot be task-connected, the input connection edge of the searched vertex is set as a sequential edge.

9. The method according to claim 8, wherein: The searching for the immediate upstream vertex and the immediate downstream vertex of the current vertex includes: According to the breadth-first traversal algorithm, first search the direct upstream vertex of the current vertex, and then search the direct downstream vertex of the current vertex; If the searched vertex cannot be task connected, setting the input connection edge of the searched vertex to a sequential edge includes: Determine whether the concurrency of the searched vertex is higher than the concurrency threshold; if so, the searched vertex cannot be task connected, and the input connection edge of the searched vertex is set to a sequential edge.

10. The method according to claim 8 or 9, wherein: The target condition is set based on the physical properties of the vertex and the relationship of the vertex to other vertices in the DAG.

11. The method according to claim 10, wherein: The step of judging whether each vertex traversed satisfies a preset target condition for joining the subgraph includes: For the current vertex traversed to the current vertex, determine whether there is no sequential edge connection between the current vertex and the current subgraph, and whether there is no circular dependency between the current vertex and the current subgraph; If any of the judgment results is negative, the current vertex does not meet the target condition; if all of the judgment results are positive, the current vertex meets the target condition.

12. The method according to claim 11, wherein: The current vertex and the current subgraph do not have a circular dependency relationship, including: There is no vertex upstream of the current subgraph among the downstream vertices of the current vertex; There is no vertex in the upstream vertex of the current vertex that is downstream of the current subgraph; There is no vertex upstream of the current subgraph in the downstream subgraph of the current vertex; There is no vertex in the upstream subgraph of the current vertex that is downstream of the current subgraph.

13. The method according to claim 1, wherein: Configuring the physical properties of the edges in the DAG includes: In an initial state, a blocking barrier vertex among the plurality of vertices is determined, the output connection edges of the barrier vertex are configured as sequential edges, and other connection edges of non-sequential edges in the DAG are configured as parallel edges; the barrier vertex is a vertex containing a barrier operator; Among them, the sequential edges outside the subgraph or crossing the subgraph boundary use the method of data storage to transmit data; the parallel edges within the subgraph use the network or memory connection method to transmit data.

14. The method according to claim 1, wherein: The method further comprises: Feedback the DAG containing the subgraph to the terminal, so that after obtaining the adjustment result of the subgraph, the terminal obtains the DAG containing the adjusted subgraph; And / or, determining the difference between the subgraph of the DAG and the historical subgraph, and training or optimizing a neural network model based on the difference, wherein the neural network model has the ability to slice the DAG into subgraphs.

15. A distributed job execution method, wherein: include: Get the jobs submitted by the user; Generate an execution plan for the job, where the execution plan is described by a DAG; Configure the physical properties of vertices and connecting edges in the DAG, wherein the physical properties of the connecting edges include sequential edges and parallel edges; Traverse each vertex in the DAG and determine whether each vertex meets the preset target conditions for joining the subgraph; According to the judgment result of each vertex, adjust the physical properties of the connecting edges in the DAG until the vertex traversal in the DAG is completed; After the vertex traversal in the DAG is completed, a subgraph of the DAG is obtained based on the connection edges with adjusted physical properties; the subgraph is formed by vertices connected by parallel edges; wherein the vertices in the subgraph are executed in a quasi-real-time mode, and the vertices outside the subgraph are executed in an offline mode.

16. A master node, wherein: The master node is configured to execute the distributed job execution method according to any one of claims 1 to 14, or the distributed job execution method according to claim 15.

17. A distributed system, wherein: include: A master node and multiple worker nodes; The master node is the master node as claimed in claim 16, and the online service of the distributed system has one or more service versions.

18. A physical machine, wherein: include: At least one memory and at least one processor, the memory storing one or more computer executable instructions, the processor calling the one or more computer executable instructions to execute the distributed job execution method as described in any one of claims 1-14, or the distributed job execution method as described in claim 15.

19. A storage medium, wherein: The storage medium stores one or more computer-executable instructions, and when the one or more computer-executable instructions are executed, the distributed job execution method according to any one of claims 1 to 14 is implemented, or the distributed job execution method according to claim 15 is implemented.

Images