Distributed job adjustment method, master node, system, physical machine and storage medium

Abstract

The embodiments of the present application provide a distributed job adjustment method, a master node, a system, a physical machine and a storage medium, wherein the method includes: obtaining a job submitted by a user; generating an execution plan for the job, wherein the multiple execution stages of the execution plan include an upstream execution stage and a direct downstream execution stage; during the execution of the job, obtaining statistical information of the output data of the upstream execution stage; and configuring the direct downstream execution stage according to the statistical information. The embodiments of the present application can dynamically adjust the configuration of the downstream execution stage based on the output results of the upstream execution stage, such as configuring concurrency, allocating data partitions, selecting subsequent execution paths, etc. for the downstream execution stage. Furthermore, the embodiments of the present application can optimize the description of the execution plan in the field of deep learning and reasonably optimize resource allocation. The embodiments of the present application can improve the performance of distributed systems.

Description

[0001] This application is a divisional application of the application with application number 202110950182.9, application date August 18, 2021, and invention name “Distributed job adjustment method, master node, system, physical machine and storage medium”. Technical Field

[0002] The embodiments of the present application relate to the field of distributed technology, and specifically to a distributed job adjustment method, a master node, a system, a physical machine, and a storage medium. Background Art

[0003] A distributed system is a system composed of multiple physical machines connected through communication lines. It has the characteristics of distribution, autonomy, parallelism, and globality. Using a distributed system to execute jobs submitted by users can improve the efficiency of job execution through the distributed computing capabilities of the distributed system.

[0004] A distributed system mainly includes a master node and a worker node. The master node can generate an execution plan for the job submitted by the user. The master node can configure the generated execution plan. Based on the configuration of the execution plan, the master node can schedule the worker nodes and resources to implement the specific execution of the job during the execution of the job. Based on the wide application of distributed systems, those skilled in the art have been committed to improving the performance of distributed systems. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a distributed job adjustment method, a master node, a system, a physical machine and a storage medium, so as to accurately and reasonably dynamically configure an execution plan during the job execution process, realize dynamic adjustment of the job configuration, and thus improve the performance of the distributed system.

[0006] To achieve the above objectives, the embodiments of the present application provide the following technical solutions.

[0007] In a first aspect, an embodiment of the present application provides a distributed job adjustment method, including:

[0008] Get the jobs submitted by the user;

[0009] Generate an execution plan for the job, the execution plan comprising multiple execution stages, the multiple execution stages comprising an upstream execution stage and a direct downstream execution stage of the upstream execution stage;

[0010] During the execution of the job, obtaining statistical information of output data of the upstream execution stage;

[0011] The direct downstream execution stage is configured according to the statistical information, so that the direct downstream execution stage executes the job based on the configuration result.

[0012] In a second aspect, an embodiment of the present application provides a distributed job adjustment method, including:

[0013] Get deep learning homework;

[0014] Generate an execution plan for the deep learning job, the execution plan comprising multiple execution stages, the multiple execution stages comprising: a worker execution stage and a resource optimization execution stage; the worker execution stage is used to calculate the gradient of the deep learning parameters;

[0015] During the execution of the deep learning job, scheduling the resource optimization node corresponding to the resource optimization execution phase, and determining the historically used resource information matching the current execution state of the deep learning job through the resource optimization node;

[0016] The resource information is configured for the worker execution phase through a resource optimization node.

[0017] In a third aspect, an embodiment of the present application provides a master node, which is configured to execute the distributed job adjustment method as described in the first aspect or the second aspect above.

[0018] In a fourth aspect, an embodiment of the present application provides a distributed system, comprising a master node and multiple working nodes, wherein the master node is the master node described in the third aspect above.

[0019] In a fifth aspect, an embodiment of the present application provides a physical machine, comprising at least one memory and at least one processor, wherein the memory stores one or more computer executable instructions, and the processor calls the one or more computer executable instructions to execute the distributed job adjustment method as described in the first aspect or the second aspect above.

[0020] In a sixth aspect, an embodiment of the present application provides a storage medium, which stores one or more computer-executable instructions, and when the one or more computer-executable instructions are executed, the distributed job adjustment method as described in the first aspect or the second aspect above is implemented.

[0021] In the distributed job adjustment method provided in the embodiment of the present application, after the master node obtains the job submitted by the user, it can generate an execution plan for the job, and the execution plan includes multiple execution stages, and the multiple execution stages include an upstream execution stage and a direct downstream execution stage of the upstream execution stage. During the execution of the job, the master node can obtain statistical information of the output data of the upstream execution stage, and then configure the direct downstream execution stage according to the statistical information, so that the direct downstream execution stage executes the job based on the configuration result. Since the embodiment of the present application can dynamically adjust the configuration of the downstream execution stage based on the data output result of the upstream execution stage during the execution of the job, so that the configuration of the downstream execution stage can adapt to the actual execution result of the upstream execution stage, so that the concurrency, resources and other configurations of the downstream execution stage can fit the specific execution situation of the job, the rationality and accuracy of the execution plan configuration are improved. It can be seen that the distributed job adjustment method provided in the embodiment of the present application can dynamically adjust the configuration of the execution plan during the job execution process, thereby achieving a dynamic adjustment effect of the job and enabling the configuration of the execution plan to fit the specific execution situation of the job; and then realizing the specific execution of the job based on the dynamically configured execution plan, which can enable the distributed system to complete the job execution more reasonably and efficiently, and significantly improve the performance of the distributed system in executing jobs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1A A schematic diagram of the distributed system structure.

[0024] Figure 1B A schematic diagram of the execution plan.

[0025] Figure 1C This is a schematic diagram of DAG.

[0026] Figure 1D A schematic diagram of the mapping between the logical diagram and the physical diagram.

[0027] Figure 2 A flowchart of a distributed job adjustment method provided in an embodiment of the present application.

[0028] Figure 3A Schematic diagram of data distribution for processing in the immediate downstream stage.

[0029] Figure 3BAnother flow chart of the distributed job adjustment method provided in an embodiment of the present application.

[0030] Figure 3C Example graph for assigning a Partition to the immediate downstream stage.

[0031] Figure 4A This is an example diagram of data shuffle.

[0032] Figure 4B Another flow chart of the distributed job adjustment method provided in the embodiment of the present application.

[0033] Figure 4C This is an example diagram of Partition splitting in an embodiment of the present application.

[0034] Figure 5A This is an example diagram of the Join process.

[0035] Figure 5B Another example diagram of the Join process.

[0036] Figure 5C Another flow chart of the distributed job adjustment method provided in an embodiment of the present application.

[0037] Figure 5D This is another example diagram of the Join process.

[0038] Figure 5E Another example diagram of the Join process is shown.

[0039] Fig. 5F For Figure 5D An example diagram of a union operation is further shown based on this.

[0040] Figure 5G For Figure 5E An example diagram of a union operation is further shown based on this.

[0041] Fig. 6A This is an example diagram of Sort Merge Join.

[0042] Figure 6B This is an example diagram of Broadcast Join.

[0043] Figure 6C Yet another flow chart of the distributed job adjustment method provided in an embodiment of the present application.

[0044] Fig.6D A schematic diagram of an execution plan with multiple execution paths.

[0045] Fig. 7AA flowchart for generating an execution plan with multiple execution paths.

[0046] Figure 7B , Figure 7C , Fig.7D and Fig. 7E An example diagram showing the process of converting a physical plan into an execution plan.

[0047] Figure 7F An example diagram of the complete execution plan after the execution path is selected.

[0048] Figure 7G Another example diagram of the complete execution plan after the execution path is selected.

[0049] Fig. 8A This is an example diagram of a PS stage and a Worker stage connected in parallel.

[0050] Figure 8B Yet another flow chart of the distributed job adjustment method provided in an embodiment of the present application.

[0051] Figure 8C An example diagram of adjusting the resources of a Worker node for a Resource Optimization node.

[0052] Fig. 9 This is a structural block diagram of a physical machine. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] Distributed jobs can be understood as submitting jobs to distributed systems for execution. Figure 1A The schematic diagram of the structure of the distributed system is shown as an example. Figure 1A As shown, the distributed system may include: a master node 110 and multiple working nodes 120. The master node 110 and the working node 120 may be regarded as computing nodes in the distributed system, and the computing nodes may be hosted on a physical machine with data computing capabilities, and one physical machine may host one or more computing nodes.

[0055] In a distributed system, the master node 110 is a computing node used for management and control. For example, the master node 110 can manage the worker nodes 120, coordinate the concurrency and resources corresponding to the execution phase of the execution plan, etc. In some aspects, the master node 110 is the central control node in the distributed system, also known as the execution engine of the distributed system. The worker node 120 is a computing node that specifically executes the job in the distributed system, and can be managed and coordinated by the master node 110 to execute the job.

[0056] When executing a job in a distributed system, the user can submit the job to the cluster resource manager through a terminal, and the cluster resource manager pulls up the master node 110. The master node 110 can then parse the job and generate an execution plan. The execution plan describes the process of how the job data, from the initial source table, undergoes a series of data flows, executions, and changes, and finally generates output. Figure 1B The schematic diagram of the execution plan is shown as an example. Figure 1B As shown, the execution plan may include: multiple stages (execution stages) with a hierarchical relationship. In some embodiments, the stages may be in a tree-like hierarchical structure. A stage may include one or more tasks. For each stage, the master node 110 may schedule multiple working nodes to execute the tasks of the stage in parallel by configuring the number of working nodes (concurrency), the resources used, etc., thereby realizing the execution of the job in the distributed system.

[0057] In some embodiments, the job is generally submitted by the terminal to the distributed system in the form of a request. In one example, the job submitted by the terminal includes a query statement for querying a database, such as a SQL (Structured Query Language) statement.

[0058] In some further embodiments, the execution plan may be described by a Directed Acyclic Graph (DAG). The DAG includes a plurality of vertices and edges between the vertices. Figure 1C The schematic diagram of DAG is shown as an example. It should be noted that the actual number of vertices, levels, and connecting edges of DAG may be smaller than Figure 1C More complex, Figure 1C This is just a simple DAG example for ease of understanding. Figure 1C As shown, the DAG may include four vertices V1 to V4, and connecting edges 11, 12, 13, and 14. Among them, the connecting edge 11 connects vertices V1 and V2, the connecting edge 12 connects vertices V1 and V3, the connecting edge 13 connects vertices V2 and V4, and the connecting edge 14 connects vertices V3 and V4.

[0059] A vertex in a DAG can represent an independent stage in the execution plan. The edges between vertices can be directed edges, which represent the relationship between vertices. Based on the relationship pointed by the edge, the edge connecting the vertices can be the input edge of the vertex (the input edge points to the vertex) or the output edge of the vertex (the output edge points from the vertex to other vertices). For example Figure 1C In the figure, edge 12 points to V3 and is the input edge of V3; edge 14 is output by V3 and is the output edge of V3; and edge 12 is output by V1, so edge 12 is also the output edge of V1; edge 14 inputs V4, so edge 14 is also the input edge of V4.

[0060] Of the two vertices connected by a connecting edge, the vertex output by the connecting edge is called the direct upstream vertex of the other vertex, and the vertex input by the connecting edge is called the direct downstream vertex of the other vertex. Figure 1C In the example, edge 12 connects V1 and V3, V1 outputs edge 12 and edge 12 inputs V3, then V1 can be called the direct upstream vertex of V3, and V3 can be called the direct downstream vertex of V1. A vertex may have one or more direct upstream vertices and one or more direct downstream vertices. It should be noted that in addition to direct upstream vertices, a vertex may also have indirect upstream vertices. The indirect upstream vertices are not directly connected to the vertex, but are located above the vertex and are connected to the vertex through one or more vertices. For example Figure 1C In the example, V1 is located above V4 and is connected to V4 through V2 or V3, so V1 can be called the indirect upstream vertex of V4. Obviously, in addition to direct downstream vertices, a vertex may also have indirect downstream vertices. The indirect downstream vertices are not directly connected to the vertex, but are located below the vertex and are connected to the vertex through one or more vertices. For example Figure 1C In the example, V4 is at the lower layer of V1 and is connected to V1 through V2 or V3, so V4 can be called the indirect downstream vertex of V1. The upstream vertex of a vertex can include direct upstream vertices and indirect downstream vertices, and the downstream vertex of a vertex can include direct downstream vertices and indirect downstream vertices.

[0061] The execution of a vertex may depend on the directly upstream vertex, that is, there is an execution dependency relationship between the vertex and the directly upstream vertex, and the vertex can only be executed after the directly upstream vertex is executed; the execution of a vertex may not depend on the directly upstream vertex, but can be executed in parallel with the directly upstream vertex.

[0062] In some further embodiments, DAG can have two levels of representation: logical graph and physical graph. The logical graph can be considered as a natural extension of the execution plan, describing the data execution process that the user wants to achieve for the job. The physical graph reflects the physical properties of each stage of the execution plan mapped to the distributed system, describing the physical properties of each stage of the execution plan at the execution level, such as the concurrency, resources used by the working nodes, and data transmission methods.

[0063] Figure 1D The following is a schematic diagram showing the mapping between the logical diagram and the physical diagram. Figure 1D The execution plan has four stages. Figure 1D As shown in the figure, the logical graph describes the four vertices of the execution plan (vertices V0, V1, V2, and V3) and the relationship between each vertex (for example, vertex V0 points to vertex V2, and vertex V1 and vertex V2 point to vertex V3). One vertex corresponds to a stage of the execution plan. The logical graph can reflect the data execution process of the execution plan. After mapping the logical graph to a physical graph, the physical graph can describe the concurrency that needs to be configured for each stage, the resources used by the working nodes of each stage (such as CPU resources, memory resources, etc.), and the data transmission method and other physical properties. For example, combined with Figure 1D For example, the physical graph describes that vertex V0 needs to be configured with 3 working nodes (concurrency is 3), and vertices V1, V2, and V3 each need to be configured with 2 working nodes (concurrency is 2). In other words, the physical graph can express the physical properties of vertices and connecting edges in the DAG. Through the physical properties of vertices and connecting edges described by the physical graph, the master node can schedule working nodes and resources for each stage, so that tasks in the stage can be executed in parallel by multiple working nodes, realizing the execution of jobs in a distributed system.

[0064] The configuration execution plan referred to in the embodiments of the present application may include configuring the logic of the execution plan and configuring the physical properties of the execution plan. Configuring the logic of the execution plan can be considered as configuring the execution plan at the logical level, for example, configuring the execution process of the execution plan. Configuring the physical properties of the execution plan can be considered as configuring the execution plan at the physical level, for example, configuring the concurrency, resources, data transmission mode and other physical properties of each stage of the execution plan.

[0065] If the execution plan is configured before the specific execution of the job, and the execution plan cannot be adjusted during the specific execution of the job after configuration, this execution plan is called a static execution plan. In other words, the static execution plan is configured before the job is executed and cannot be adjusted during the specific execution of the job. However, before the specific execution of the job, it is impossible to accurately estimate the actual resources required for each stage of the execution plan, as well as the reasonable execution path of the execution plan, which undoubtedly makes it difficult to configure the static execution plan reasonably and accurately, and reduces the performance of distributed system execution of jobs.

[0066] Based on this, an embodiment of the present application provides a new execution plan configuration scheme, which can dynamically adjust the configuration of the downstream stage based on the data output results of the upstream stage during the job execution process, so that the configuration of the downstream stage can adapt to the actual execution results of the upstream stage, thereby making the concurrency, resources and other configurations of the downstream stage fit the specific execution situation of the job, thereby improving the rationality and accuracy of the execution plan configuration.

[0067] As an optional implementation, Figure 2 The flowchart of the distributed job adjustment method provided by the embodiment of the present application is exemplarily shown. The method flow can be implemented by the master node, refer to Figure 2 The method flow may include the following steps.

[0068] In step S210, the job submitted by the user is obtained.

[0069] In step S211, an execution plan of the job is generated, the execution plan includes multiple stages, and the multiple stages include an upstream stage and a direct downstream stage of the upstream stage.

[0070] After the terminal submits the job to the distributed system, the master node in the distributed system can parse the job and generate an execution plan for the job, which can be described by DAG. The execution plan may include multiple stages, which may include an upstream stage and a direct downstream stage of the upstream stage. In some embodiments, any stage in the multiple stages that has a downstream stage can be regarded as the upstream stage, and during the specific execution of the job, the output data of the upstream stage can be input into the direct downstream stage for processing. The output data of an upstream stage can be input into one or more direct downstream stages, and a direct downstream stage may also input the output data of one or more upstream stages; that is, an upstream stage may correspond to one or more direct downstream stages, and a direct downstream stage may also correspond to one or more upstream stages. In some embodiments, the upstream stage may be referred to as the previous stage of the direct downstream stage, and the direct downstream stage may be referred to as the next stage of the upstream stage.

[0071] In one example, combining Figure 1C As shown, V1 is the upstream stage of V2 and V3, and V2 and V3 are the direct downstream stages of V1. During the execution of the job, the output data of V1 can be input into V2 and V3 for processing; and V2 and V3 are the upstream stage of V4, and V4 is the direct downstream stage of V2 and V3. The output data of V2 and V3 can be input into V4 for processing. One aspect of the embodiment of the present application is how much concurrency, resources, etc. the direct downstream stage needs to configure to process the output data of the upstream stage; for example, how much concurrency, resources, etc. V2 and V3 need to configure to process the output data of V1, and how much concurrency, resources, etc. V4 needs to configure to process the output data of V2 and V3.

[0072] In step S212, during the execution of the job, Statistics of the output data of the upstream stage are obtained.

[0073] During the job execution process, after any stage is completed, the master node can collect the statistics of the output data of the working nodes of the stage. Based on this, after a certain upstream stage of the execution plan is completed, the master node can collect the statistics of the output data of the upstream stage. In some embodiments, the Statistics of the output data of the upstream stage may include any of the following: the amount of output data (for example, the amount of output data before and after data compression), the data volume distribution information of the Partition of the output data, the number of Records (serialized data records) of each Partition in the output data, etc.

[0074] In step S213, the direct downstream stage is configured according to the Statistics, so that the direct downstream stage executes the job based on the configuration result.

[0075] Based on the statistical information of the output data of the upstream stage, the master node can configure the direct downstream stage so that the configuration of the direct downstream stage can adapt to the actual execution results of the upstream stage, so that the concurrency, resources and other configurations of the downstream stage can fit the specific execution situation of the job; further, the direct downstream stage can execute the job based on the configuration results, so that the corresponding tasks of the direct downstream stage can be reasonably and efficiently executed.

[0076] During the execution of a job, the direct downstream stage of the upstream stage is dynamically configured in the manner provided by the embodiments of the present application, so that the execution plan can be dynamically adjusted during the execution of the job, and the configuration of the execution plan can be tailored to the specific execution of the job. In other words, during the execution of a job, the embodiments of the present application can dynamically adjust the distributed job based on the output data and statistical information of the upstream stage to achieve the dynamic adjustment effect of the job. Furthermore, the specific execution of the job is achieved based on the dynamically configured execution plan, which can enable the distributed system to complete the execution of the job more reasonably and efficiently, and significantly improve the performance of the distributed system in executing jobs.

[0077] In some embodiments, based on the statistics of the output data of the upstream stage, configuring the immediately downstream stage may include any of the following:

[0078] Based on the statistical information, configure the concurrency of the direct downstream stage;

[0079] Allocate a Partition to the direct downstream stage according to the statistical information; for example, allocate the Partition output by the upstream stage to the working node of the direct downstream stage, or, in a Join scenario, split the Partition output by the upstream stage and allocate it to the direct downstream stage to perform a Join operation on the working node of the direct downstream stage;

[0080] According to the statistical information, a direct downstream stage for subsequent execution is selected; in the execution plan, multiple candidate execution paths can be configured downstream of the upstream stage, so that in the specific execution process of the job, based on the execution result of the upstream stage, the execution path for actual execution is selected from the multiple execution paths, so that the execution logic of the execution plan is more accurate and reasonable; in this case, the stage in an execution path can at least include the direct downstream stage of the upstream stage; by selecting multiple subsequent candidate execution paths of the upstream stage, dynamic adjustment of the execution logic can be achieved, and selection of the direct downstream stage for subsequent execution can be achieved.

[0081] The above-mentioned various situations of configuring the direct downstream stage according to statistical information will be explained later and will not be expanded here.

[0082] In the distributed job adjustment method provided by the embodiment of the present application, after the master node obtains the job submitted by the user, it can generate an execution plan for the job, and the execution plan includes multiple stages, and the multiple stages include an upstream stage and a direct downstream stage of the upstream stage. During the execution of the job, the master node can obtain the statistical information of the output data of the upstream stage, and thus configure the direct downstream stage according to the statistical information. Since the embodiment of the present application can dynamically adjust the configuration of the downstream stage based on the data output result of the upstream stage during the execution of the job, so that the configuration of the downstream stage can adapt to the actual execution result of the upstream stage, so that the concurrency, resources and other configurations of the downstream stage can fit the specific execution of the job, and improve the rationality and accuracy of the execution plan configuration. It can be seen that the distributed job adjustment method provided by the embodiment of the present application can dynamically adjust the configuration of the execution plan during the execution of the job, realize the dynamic adjustment effect of the job configuration, and make the configuration of the execution plan fit the specific execution of the job; and then realize the specific execution of the job based on the dynamically configured execution plan, so that the distributed system can complete the execution of the job more reasonably and efficiently, and significantly improve the performance of the distributed system in executing the job.

[0083] The following describes the implementation scheme for configuring the concurrency of the direct downstream stage based on the statistical information of the output data of the upstream stage.

[0084] For static execution plans, the concurrency of each stage of the execution plan can be determined by the master node through estimation rules. For example, after submitting a job, the master node can use estimation rules to configure the concurrency of each stage of the execution plan based on the total amount of source data of the job (which can be regarded as configuring the concurrency of each vertex in the DAG), or configure the concurrency of each stage of the execution plan based on the concurrency of different types of stages specified by the user. However, the source data processed by distributed jobs is complex and diverse, and it is often difficult for the master node to rely on estimation rules to configure the concurrency that adapts to different jobs, which leads to inaccurate concurrency configuration of each stage of the execution plan. For example, for a stage with a small amount of data processed, if a larger concurrency is statically configured, it will lead to a waste of computing resources of the distributed system; and for a stage with a large amount of data processed, if a smaller concurrency is statically configured, it will lead to a longer execution time of the stage, and even bring various errors such as memory usage exceeding the limit, resulting in job execution failure. Therefore, in the implementation of static configuration of concurrency, in order to avoid the possibility that the stage is configured with a low concurrency and cannot process massive data, the stage often needs to be configured with a higher concurrency, which leads to a lot of waste of computing resources during the actual execution of the job. For example, for a Map-Reduce job, even if the upstream Map stage only generates 1KB of output data during the actual operation, the downstream Reduce stage will still schedule a higher number of worker nodes to process this 1KB of data due to the static configuration of a higher concurrency, which undoubtedly leads to unnecessary waste of computing resources.

[0085] Based on this, during the job execution process, it is particularly necessary to configure the concurrency for the direct downstream stage of the upstream stage based on the output data results of the upstream stage. In some embodiments, for the direct downstream stage of the upstream stage, the master node can distribute the Partitions (data partitions) in the output data of the upstream stage to the work nodes of the direct downstream stage according to the principle of uniform number of Partitions. For example, each work node of the direct downstream stage will process multiple consecutive Partitions output by the upstream stage, and each work node processes the same number of Partitions. This method can be called an Even Reduction strategy based on the number of Partitions, which can achieve a relatively ideal effect when the amount of data in each Partition is uniform; however, in actual environments, the data distribution characteristics are diverse and often have uneven characteristics. For non-uniform data (the amount of data in each Partition is not uniform), the above method may cause a single work node in the direct downstream stage to have a data skew problem, that is, the amount of data processed by a single work node is much larger than that of other work nodes, which further leads to the long tail problem of the work node, causing the job execution time to be unnecessarily prolonged. For example, Figure 3A The following is an exemplary diagram showing the distribution of processing data in the immediate downstream stage. Figure 3A As shown in the figure, the amount of data in each Partition output by the upstream stage is not uniform. The value in the Partition output by the upstream stage can represent the amount of data in the corresponding Partition. If you simply merge multiple Partitions of a specified number to a single worker node in the direct downstream stage, although the number of Partitions processed by each worker node is the same (for example Figure 3A Each worker node in the direct downstream stage processes 2 Partitions), but this may lead to uneven distribution of data processing volume among the worker nodes in the direct downstream stage: on the one hand, some worker nodes process unreasonably small amounts of data (there may even be worker nodes that have no data to process at all); on the other hand, some worker nodes will further aggravate the data skew problem by merging partitions with larger data volumes, resulting in a longer running time of the worker nodes and a long tail.

[0086] In addition, since the number of partitions may be positively correlated with the time required for actual calculations, but for some data types (such as integers), the data compression rate is relatively high, it is possible that the data files are severely compressed after the output data is compressed. In this case, a small number of partitions may correspond to a large number of data records (that is, a large number of data records are recorded in a small number of partitions). Therefore, simply configuring the concurrency of the downstream stage based on the number of partitions in the upstream stage may bring greater uncertainty in actual operations.

[0087] Based on the above situation, the embodiment of the present application provides a solution for dynamically adjusting the concurrency of the direct downstream stage based on the statistical information of the output data of the upstream stage during the job execution process, and ensures that the data processing volume of each working node in the direct downstream stage is close to balanced. As an optional implementation, Figure 3B Another flow chart of the distributed job adjustment method provided by the embodiment of the present application is shown. The method flow can be implemented by the master node, refer to Figure 3B The method flow may include the following steps.

[0088] In step S310, the job submitted by the user is obtained.

[0089] In step S311, an execution plan of the job is generated, the execution plan includes multiple stages, and the multiple stages include an upstream stage and a direct downstream stage of the upstream stage.

[0090] In step S312, the concurrency is initialized and configured for at least the stage that is executed first among the multiple stages.

[0091] The master node can initialize the concurrency of the execution plan. In some embodiments, the master node can initialize the concurrency of the execution plan based on the total amount of source data of the job using an estimation rule; or the master node can initialize the concurrency of the execution plan based on the concurrency of different types of stages specified by the user.

[0092] In an embodiment of the present application, the master node can configure the concurrency of the direct downstream stage based on the amount of data output by the upstream stage; based on this, since the stage executed first in the execution plan has no upstream stage, when the master node initializes the concurrency of the execution plan, it should at least initialize the concurrency of the stage executed first, so that the stage executed first can execute the job with the initialized configured concurrency; and for the stage not executed first in the execution plan, since the stage not executed first generally has an upstream stage, the stage not executed first can be used as the direct downstream stage of one or more upstream stages, so as to configure the concurrency based on the statistical information of the output data of the upstream stage.

[0093] Since the non-first executed stage in the execution plan can dynamically adjust the concurrency during the job execution process, when initializing the concurrency configuration, the embodiment of the present application does not necessarily need to initialize the concurrency configuration for the non-first executed stage. Of course, the embodiment of the present application can also support the initialization of the concurrency configuration for the non-first executed stage, and the embodiment of the present application is not limited to this. In one example, combined with Figure 1C As shown, V1, as the first stage executed, needs to initialize the configuration concurrency, while V2, V3 and V4 all have upstream stages. The embodiment of the present application does not limit the initialization configuration concurrency of V2, V3 and V4.

[0094] In step S313, during the job execution process, the data volume distribution information of the output data of the upstream stage is determined, and the data volume distribution information includes the data volume of multiple data partitions corresponding to the output data.

[0095] During the execution of a job, the master node can schedule the work nodes to process the input data of the upstream stage according to the concurrency of the upstream stage, so that the work nodes scheduled by the upstream stage can generate the output data of the upstream stage after processing the input data. The output data of the upstream stage can be divided into multiple data partitions (Partition). The data volume of the data partition of the output data can be used as the data volume distribution information of the output data, that is, the data volume distribution information of the output data can represent the data volume distributed in each data partition of the output data, and the data volume distribution information can be carried in the statistical information of the output data.

[0096] In an embodiment of the present application, data volume distribution information of the output data of the upstream stage may be collected by the master node.

[0097] In some embodiments, the upstream stage referred to in step S313 may be a stage that is executed first or a stage that is not executed first. It should be noted that if the upstream stage is the stage that is executed first, the upstream stage may schedule the working nodes to process data based on the concurrency configured by the initialization; if the upstream stage is the stage that is not executed first, the upstream stage may dynamically adjust the concurrency based on the solution provided in the embodiment of the present application, and then schedule the working nodes to process data.

[0098] In step S314, the amount of processed data allocated to the working node of the direct downstream stage is determined according to the ideal data size (Ideal DataSize) corresponding to the working node of the direct downstream stage and the data size distribution information.

[0099] After the master node collects the data volume distribution information of the output data of the upstream stage, the master node can allocate the processing data volume to the working nodes of the direct downstream stage based on the data volume distribution information. The embodiment of the present application can automatically configure the concurrency of the direct downstream stage after the working nodes of the direct downstream stage allocate the processing data volume.

[0100] In some embodiments, the embodiments of the present application can pre-set the ideal data volume corresponding to the working node of the direct downstream stage. If the processing data volume of the working node exceeds the ideal data volume, it will exceed the processing load of the working node, resulting in an increased possibility of execution failure of the working node. Therefore, when allocating processing data to the working node of the direct downstream stage, the data volume of the processed data should be close to but not exceed the ideal data volume.

[0101] Based on the data volume distribution information expressing the data volume of the data partition of the upstream stage, when the master node allocates the processing data volume to the working node of the directly downstream stage, it can determine the processing data volume allocated to each working node of the directly downstream stage based on the ideal data volume and the data volume distribution information, and ensure that the processing data volume allocated to the working node does not exceed the ideal data volume.

[0102] In some embodiments, the master node may allocate data partitions that are continuous and whose total data volume does not exceed the ideal data volume output by the upstream stage to a working node of the directly downstream stage, so that the working node of the directly downstream stage can be allocated to data partitions that are continuous and whose total data volume does not exceed the ideal data volume. In some further embodiments, the master node can ensure that the amount of processed data allocated to each directly downstream stage is as balanced as possible on the basis of allocating multiple data partitions that are continuous and whose total data volume does not exceed the ideal data volume to a working node of the directly downstream stage.

[0103] In one example, Figure 3C An example diagram showing data partitions assigned to the immediate downstream stage is shown. Figure 3C As shown, the upstream stage outputs multiple Partitions. Figure 3C The value in the box representing Partition is the data volume of the Partition; it can be seen that when the master node allocates the processing data volume for the direct downstream stage, it allocates the continuous data partitions whose total data volume does not exceed the ideal data volume to a working node of the direct downstream stage, and ensures that the processing data volume of each working node is close to uniform distribution. For example, the processing data volumes allocated to the 4 working nodes of the direct downstream stage are 15, 16, 19 and 10 respectively. It can be seen that after allocating the processing data volume to the working nodes of the direct downstream stage, the master node can complete the configuration of the concurrency of the direct downstream stage, for example, Figure 3C After each working node completes the allocation of the processing data volume, the master node can determine that 4 working nodes (concurrency is 4) are needed to process the output data of the upstream stage. By configuring concurrency in the manner provided in the embodiment of the present application, it is possible to avoid the situation where the processing data volume of a working node in the direct downstream stage does not exceed the ideal data volume, but is too high or too low compared to other working nodes, so that the processing data volume of each working node can be close to balanced and reasonable.

[0104] In step S315, the concurrency of the direct downstream stage is configured according to the number of working nodes in the direct downstream stage that are allocated to process the data volume.

[0105] After the host node determines the amount of processing data allocated to the worker nodes of the direct downstream stage, it can complete the concurrency configuration of the direct downstream stage based on the number of worker nodes allocated to process the data.

[0106] The distributed job adjustment method provided in the embodiment of the present application can, after generating the execution plan of the job, initialize the configuration concurrency for at least the stage that is executed first in the execution plan, so that the job can actually be executed. During the execution of the job, the master node can determine the data volume distribution information of the output data of the upstream stage of the execution plan, and the data volume distribution information can express the data volume of multiple data partitions corresponding to the output data; thereby, based on the ideal data volume corresponding to the working nodes of the direct downstream stage and the data volume distribution information, determine the processing data volume allocated to the working nodes of the direct downstream stage; further, the master node can configure the concurrency of the direct downstream stage according to the number of working nodes allocated to process the data volume in the direct downstream stage, and enable each working node in the direct downstream stage to execute the job with a data volume that does not exceed the ideal data volume, thereby reducing the probability of execution failure caused by excessive processing data volume of a single working node in the direct downstream stage. The embodiment of the present application can allocate the processing data volume to the working nodes of the direct downstream stage based on the data volume distribution information of the output data of the upstream stage and the ideal data volume corresponding to the working nodes of the direct downstream stage during the execution of the job, and automatically configure the concurrency of the direct downstream stage, so as to achieve the effect of dynamically and adaptively adjusting the concurrency of the direct downstream stage based on the output data results of the upstream stage, thereby improving the accuracy and rationality of the concurrency configuration of the direct downstream stage, greatly reducing the waste of computing resources caused by unreasonable concurrency configuration, and can significantly improve the performance of the distributed system.

[0107] The solution provided by the embodiment of the present application can be called a Fair-Parallelism strategy based on the data volume of the Partition. As an optional implementation process, taking the upstream stage to implement data shuffle (the process of calculating the hash value of the data record according to a specific Hash function and sending it to the corresponding working node during the execution of a distributed job) as an example, the master node can obtain the data volume distribution information (the data volume of each Partition) of the Partition output by the working node of the upstream stage after the data shuffle. The master node can allocate the processing data volume for the working node of the direct downstream stage according to the Partition aggregation, for example, by allocating multiple Partitions that are continuous and whose total data volume does not exceed the ideal data volume to a working node of the direct downstream stage. In the above allocation process, if the total amount of data of multiple Partitions allocated to a working node reaches the ideal data volume, the subsequent continuous Partitions whose total data volume does not exceed the ideal data volume are automatically allocated to the next working node, and so on, until all the Partitions output by the upstream stage are allocated. The actual concurrency of the direct downstream stage can be automatically determined after all the Partitions of the upstream stage are allocated, that is, the concurrency of the direct downstream stage is determined based on the number of working nodes that allocate the processing data volume in the direct downstream stage. The embodiment of the present application can distribute the Partition output by the upstream stage to each working node of the direct downstream stage as evenly as possible, thereby realizing more accurate and reasonable configuration of concurrency for the direct downstream stage and avoiding the long tail problem of a single working node.

[0108] It should be noted that after configuring concurrency for the direct downstream stage in step S315, if the direct downstream stage also inputs other downstream stages through connecting edges (that is, the direct downstream stage also has downstream stages), the direct downstream stage that currently configures concurrency in step S315 can become the upstream stage, and based on the solution provided in the embodiment of the present application, continue to adjust the concurrency for subsequent downstream stages until all stages of the execution plan complete the adjustment of concurrency during the job execution process.

[0109] In some further embodiments, the embodiments of the present application can be further improved through technology to reduce the data skew in the working nodes of the direct downstream stage, so as to meet various data feature requirements of job execution, specifically:

[0110] When the data volumes of multiple consecutive Partitions are very small, the sum of the data volumes of multiple Partitions may be less than the ideal data volume. In this extreme case, a large number of Partitions may be assigned to a single work node for processing. If the work node of the direct downstream stage operates a large number of Partitions sequentially, this may lead to a regression in read and write performance. Based on this, the embodiment of the present application can set an upper limit on the number of Partitions allowed to be assigned to a single work node, thereby controlling the upper limit of the number of Partitions added to the work node under the premise of avoiding the data volume processed by the work node being zero.

[0111] When the running time of a work node has little correlation with the size of the processed data, but is related to other characteristics of the data (such as the number of data records, etc.), if the processing data is allocated to the work node according to the size of the data volume of the Partition, the Partition with a small amount of data but a large number of data records may be allocated to a single work node, which will cause the running time of the work node to be prolonged, resulting in a bottleneck in the execution of the job; in addition, it is also necessary to consider the unit data records involved in the work node itself to calculate the complexity, which may be related to factors such as the number of operators of the work node and the characteristics of the operators. This information also needs to be considered as a factor in calculating the concurrency. Based on this, in addition to using the data size of the Partition as a benchmark to allocate the amount of processed data of the work node, the embodiment of the present application can further combine the number of records of the Partition, the number of operators of the work node, and the complexity of the operator, etc., to make a secondary adjustment to the amount of processed data allocated to the work node, and select the result with a larger concurrency obtained in these dimensions, so as to complete the configuration of the final concurrency of the direct downstream stage. Based on this, in the optional implementation of step S315, the embodiment of the present application can further complete the final concurrency configuration of the direct downstream stage according to the number of working nodes allocated to the processing data volume in the direct downstream stage, the number of Records of the Partition allocated to the working node, the number of operators of the working node, and the complexity of the operator.

[0112] When dynamically adjusting the concurrency of job execution, the embodiment of the present application can combine the data characteristics of multiple dimensions, statistically calculate the execution results, and provide more optimization directions for the dynamic execution mode of the distributed system engine; the embodiment of the present application is applied in the actual production environment, which can reduce the overall execution concurrency of the production job by an order of magnitude, significantly improve the computing efficiency of the working nodes in the distributed system, and avoid the waste of computing resources and the consumption of frequent scheduling and pulling up of the working nodes to a great extent. Compared with the simple and direct Even-Reduction strategy, the Fair-Parallelism strategy proposed in the embodiment of the present application can effectively avoid the serious data skew that may be introduced under the Even-Reduction strategy, so that the amount of data processed by all working nodes in the distributed job is distributed as evenly as possible, avoiding the occurrence of prominent long tails or short boards, and avoiding the long tail from becoming a bottleneck of the running time. It can also prevent the working node from frequently processing smaller data and causing the waste of resources caused by repeatedly starting the working node. Further, by limiting the upper limit of the number of continuously merged Partitions when merging Partitions with smaller data volumes, it is prevented that excessive merging of Partitions causes a decline in the performance of reading data in the downstream stage. Furthermore, according to the detailed data information (including data volume and number of Records) generated by each working node during the operation of the statistical operation, the embodiment of the present application can combine the number of Records, the number of operators and the complexity of operators to make a more balanced and perfect concurrency adjustment. It can be seen that the embodiment of the present application can significantly improve the performance of the distributed system.

[0113] Based on the statistical information of the output data of the upstream stage, the embodiment of the present application provides a solution for dynamically merging multiple Partitions. At the same time, for a Partition with a large amount of data, the embodiment of the present application also provides a solution for splitting the Partition. The specific implementation solution is introduced in detail below.

[0114] Effective data orchestration and shuffle are important prerequisites for the horizontal expansion of distributed systems. For the Map-Reduce execution phase, efficient data shuffle has always been one of the important performance indicators of distributed job execution performance. However, full-connection data shuffle (full-shuffle) can only be executed more efficiently in an ideal scenario where data is evenly distributed. In actual production jobs, data distribution is often uneven, and the skewed characteristics of data may be further amplified in full-shuffle mode, resulting in a significant increase in the running time of individual working nodes and causing serious long tails.

[0115] It should be noted that the concept of shuffle comes from the Map-Reduce computing model. Although modern job execution frameworks have evolved to use more general DAG-based descriptions, the Map-Reduce operation mode is still an important description of each subgraph in the DAG. In many more complex DAG topologies, the data flow on each connection edge can also be described by various shuffle models; for example, in the distributed execution framework, an important physical attribute of the DAG connection edge is the transmission of edge data. The transmission of edge data can not only use full-shuffle, but also allow the introduction of more dynamic and intelligent data orchestration methods, and thereby solve the problems faced by full-shuffle in many practical scenarios.

[0116] Taking dynamic partition writing as an example, for a typical distributed data warehouse system, the use of partition tables is extremely extensive. There are usually two ways to write data to a partition table: static partition writing and dynamic partition writing. When the value to be written to the partition can be specified in advance, it is a simpler method to directly use static writing to specify the partition. For partition values ​​that cannot be determined in advance, especially when the data output by a query is distributed in multiple partitions, dynamic partitioning (Dynamic Partition) is used for writing, that is, the value of the data written to the partition will be obtained according to the specific output data during the operation of the job. For example, in the following SQL (Structured Query Language) statement, the data is written as the partition column name specified as country, but the specific partitions to which it will be written and the corresponding partition values ​​are obtained during operation:

[0117] INSERT OVERWRITE TABLE Partitioned_sales Partition(country);

[0118] SELECT country,item,price from sales.

[0119] In distributed systems, due to the uncertainty of the number of partitions, the diversity of data distribution characteristics and the uneven characteristics, it has always been a challenging problem to achieve universal and efficient dynamic partition writing. To this end, the distributed system needs to be able to avoid serious long tails in jobs while not placing a serious burden on the storage system due to the large number of small files.

[0120] In some embodiments, the master node can implement dynamic partitioning through the execution plan of a single Map stage. In this implementation, the execution plan generated by the master node contains only one Map stage, and the Map stage can execute tasks through one or more worker nodes. After each worker node of the Map stage reads the data, it can write files according to the partition value (such as the country in the above example). That is, if a worker node processes data corresponding to different country values, it will generate corresponding files with different paths. The execution plan of a Map stage is simple and intuitive, but it will bring various hidden dangers to actual large-scale distributed systems, the most prominent of which is the problem of fragmentation of small files. In the execution plan of a Map stage, assuming that the concurrency of the Map stage is M, the value of country in the user data may be N. In the case of random data distribution, because each worker node of the Map stage outputs independently, it may eventually cause M files to be written for each partition. Therefore, this execution method may eventually generate M*N data files, and there may be a large number of small files among them. The existence of these large numbers of fragmented small files will have a greater negative impact on a distributed system:

[0121] For the storage of distributed systems, the management of a large number of small files requires a huge amount of Meta information. In extreme cases, it may even crash the master node of the entire distributed system, making the entire distributed system unavailable. In terms of storage efficiency, fragmented small files will also lead to a poor storage compression ratio and occupy more storage space.

[0122] Data always needs to be processed after it is generated. When the upstream stage produces small files, the computational consumption of the downstream stage will increase, and the actual data reading efficiency will also be lower.

[0123] For the Map stage, since M*N small files need to be written, M*N write services (writers) need to be created. Each writer retains a certain Memory Buffer (memory cache) for encoding, compression, etc. If the Buffer is too large, the Map stage will use too much memory, and if the Buffer is too small, the encoding and compression effects will be poor.

[0124] In other embodiments, the master node can implement dynamic partitioning based on a fully connected Reshuffle method. To avoid the above-mentioned small file problem, the master node can generate input data in an execution phase, and then perform a fully connected Reshuffle based on the PartitionKey, that is, aggregate the data of the same Partition to a working node and then write it out. In this way, it is ensured that only one partition file is generated for the partition value of each Partition. However, this mandatory restriction, while reducing the number of files, will also bring another negative effect: data skew.

[0125] It should be noted that for the data generated during the execution phase, the data distribution is varied, and the data distribution of dynamic partitions cannot be obtained before the query is executed. For example, in the above SQL statement example, if the user data of a certain platform is partitioned by country, the data on the home partition will undoubtedly be much larger than that on other partitions, which will cause the work nodes corresponding to the data on the home partition to have a serious long tail, resulting in a significant increase in the running time of the entire job. For severely skewed data, this long tail may cause hundreds or even thousands of times of job delays, and also have a very bad impact on the resource utilization of the entire distributed system.

[0126] In order to alleviate this problem, the implementation of Reshuffle can actually be optimized by introducing an additional Randomshuffle Key. For example, by taking a Reshuffle key with a value in [0,9], the data is randomly scattered into 10 partitions to reduce the degree of data skew. However, such a solution still has certain problems: in the case of severe data skew, even if the data is divided into 10 parts, the data after division may still have skew problems, and may even change from a long tail of data to 10 slightly less severe long tails of data; for partitions with no skew or a small amount of data, if they are also divided into 10 parts, the final number of files will increase (10*N); the distributed system forcibly adds the Random Key shuffle method, which may destroy the idempotence of the data. In a distributed system, if a working node is re-run, there is a risk of data correctness.

[0127] Based on the above situation, the embodiment of the present application performs intelligent dynamic data arrangement according to the data distribution generated in real time by the upstream stage (such as the Map stage), solves the small file problem while ensuring the balance of written data, thereby overcoming the shortcomings of the above two solutions.

[0128] First use Figure 4ALet's briefly describe the shuffle method (for ease of understanding, the details of shuffling according to Reduce concurrent modulus are omitted here). Figure 4A An example diagram of data shuffle is shown. Figure 4A As shown, Figure 4A In the figure, the same linear boxes represent the data with the same Partition Key in the shuffle data generated in the Map phase; for example, the thin dashed box, thin solid box, thick dashed box, and thick solid box represent the data with the same Partition Key. The values ​​in the boxes represent the corresponding data volume. After a shuffle, the data with the same Partition Key will be handed over to the same worker node in the Reduce phase for processing. Figure 4A It can be seen that the amount of data processed by worker node R#0 in the Reduce phase is 8, the amount of data processed by worker node R#1 is 2, the amount of data processed by worker node R#2 is 43, and the amount of data processed by worker node R#3 is 16; there is serious data skew in the data processed by worker node R#2, which results in a very serious long tail when worker node R#2 executes tasks in the Reduce phase.

[0129] For scenarios such as dynamic partition insertion, the same partition data only needs to be written to the same directory, not to the same data file. Therefore, during the shuffle process, it is not necessary to ensure that all data with the same partition key is handed over to the same reduce worker node for processing. Based on this feature, the embodiment of the present application can automatically split the partition with a large amount of data generated in the Map phase, and then hand it over to multiple worker nodes in the Reduce phase for processing.

[0130] As an optional implementation, Figure 4B Another flow chart of the distributed job adjustment method provided in the embodiment of the present application is shown. The method flow can be implemented by the master node, refer to Figure 4B The method flow may include the following steps.

[0131] In step S410, the job submitted by the user is obtained.

[0132] In step S411, an execution plan of the job is generated, the execution plan includes multiple stages, and the multiple stages include an upstream stage and a direct downstream stage of the upstream stage.

[0133] In some embodiments, the upstream stage is, for example, a Map stage, and the immediate downstream stage is, for example, a Reduce stage.

[0134] In step S412, during the job execution, Statistics of the output data of the upstream stage are obtained, and the Statistics include: data volume distribution information of the Partition of the output data.

[0135] In some embodiments, the working node (one or more) of the upstream stage may report the Statistics of the output data to the master node during and after the execution of the task. The Statistics may include: the data volume distribution information of the Partition of the output data, and the data volume distribution information may indicate the data volume of each Partition of the output data. In some further embodiments, the Statistics may also include: the data volume of the output data (for example, the data volume of the output data before and after data compression), the number of Records of each Partition in the output data, etc.

[0136] In step S413, according to the ideal data volume, the Partitions in the output data of the upstream stage whose data volume is larger than the ideal data volume are split.

[0137] The embodiment of the present application can pre-set the ideal data volume for the Partition of the working node of the direct downstream stage. It should be noted that when different stages of the execution phase serve as the direct downstream stage, the set ideal data volume can be different. After obtaining the Statistics of the output data of the upstream stage, the master node can determine the data volume of each Partition in the output data. Therefore, based on the ideal data volume for the Partition by the working node of the direct downstream stage, it is determined whether there is a Partition in the output data with a data volume greater than the ideal data volume; for the Partition in the output data with a data volume greater than the ideal data volume, the embodiment of the present application can split the Partition and make the data volume of the split Partition not greater than the ideal data volume.

[0138] In some embodiments, the embodiments of the present application can split the Partitions in the output data whose data volume is greater than the ideal data volume according to the ideal data volume, so that the data volume of the split Partitions is not greater than the ideal data volume and is close to a uniform distribution.

[0139] In step S414, the split Partition is assigned to the direct downstream stage, and one split Partition is configured to be executed by one working node of the direct downstream stage.

[0140] After splitting the Partition of the output data, the embodiment of the present application can assign the split Partition to the direct downstream stage, and configure a split Partition to be executed by a working node of the direct downstream stage, thereby configuring the data to be processed for the direct downstream stage.

[0141] In some further embodiments, if there is a Partition in the output data whose data volume is less than the ideal data volume, the embodiments of the present application may merge at least two Partitions in the output data whose total data volume is not greater than the ideal data volume, and the data volume of each Partition in the at least two Partitions is less than the ideal data volume, thereby allocating the merged Partition to the direct downstream stage, and a merged Partition is configured to be executed by a working node of the direct downstream stage, so as to achieve efficient utilization of the computing resources of the direct downstream stage.

[0142] The embodiment of the present application obtains the Statistics of the output data of the upstream stage during the execution of the job, and then splits the Partition whose data volume is greater than the ideal data volume based on the data volume of each Partition of the output data indicated in the Statistics, so as to ensure that the data volume of the split Partition is not greater than the ideal data volume. Then the split Partition is assigned to the direct downstream stage, and a split Partition is configured to be executed by a working node of the direct downstream stage, so as to configure the data to be processed for the working node of the direct downstream stage. The embodiment of the present application can dynamically adjust the Partition configured to the direct downstream stage according to the Statistics of the output data of the upstream stage, so as to ensure that the data volume of a Partition processed by each working node in the direct downstream stage will not exceed the ideal data volume, so that the data processed by the working node of the direct downstream stage can be close to uniform distribution, reduce the data skewness of the direct downstream stage, and avoid the long tail problem of the running time of the individual working nodes of the direct downstream stage due to the need to process the Partition with a large data volume. It can be seen that the embodiment of the present application can make the data processed by the working node of the direct downstream stage close to uniform distribution, reduce the data skewness of the direct downstream stage and the long tail problem of the working node, and significantly improve the performance of the distributed system.

[0143] In some embodiments, for the Map-Reduce stage, Map can be considered as the upstream stage, and Reduce can be considered as the direct downstream stage of Map. Each working node in the Map stage can report the Statistics of the output shuffle data to the execution engine (the execution engine can be the master node) during and after the operation; for example, there may be one or more working nodes in the Map stage to execute the task, and each working node can report the Statistics of the output shuffle data to the execution engine during and after the execution of the task. In some embodiments, the Statistics of shuffle data include: the amount of shuffle data before and after compression, the amount of data in each Partition of shuffle data, and the number of Records contained in the Partition, etc.

[0144] In some embodiments, the embodiments of the present application may define an ideal data volume corresponding to a Partition in the Reduce phase. For any Partition output in the Map phase, if the data volume of the Partition is greater than the ideal data volume, the embodiments of the present application may split the Partition according to the ideal data volume, and try to ensure that the data volume processed by each working node in the Reduce phase is uniform; if the data volume of the Partition is less than the ideal data volume, the embodiments of the present application may merge multiple Partitions whose data volume is less than the ideal data volume, and ensure that the data volume of the merged Partition is not greater than the ideal data volume.

[0145] In an implementation example, Figure 4C An example diagram of performing partition splitting in an embodiment of the present application is shown. Figure 4C As shown in the figure, the 4 shuffle data outputted by the Map stage can be divided into 4 Partitions (P#0, P#1, P#2 and P#3). For example, the data with the same linear expression in the 4 shuffle data are aggregated into one Partition, where the data corresponding to the thin dashed line in the shuffle data are aggregated into P#0, the data corresponding to the thin solid line are aggregated into P#1, the data corresponding to the thick dashed line are aggregated into P#2, and the data corresponding to the thick solid line are aggregated into P#3. Figure 4CAs shown in the figure, the data sizes of the four Partitions are 8, 2, 43, and 16, respectively. Assuming that the ideal data size corresponding to a Partition in the Reduce phase is defined as 10, the data sizes of P#0 and P#1 are both less than 10, and the data size after P#0 and P#1 are combined does not exceed the ideal data size of 10. Therefore, P#0 and P#1 can be combined into one Partition and assigned to a working node R#0 in the Reduce phase for processing. Since the data size of P#2, 43, is greater than the ideal data size of 10, P#2 can be split into multiple Partitions as evenly as possible according to the ideal data size, and the data size of each split Partition is not greater than the ideal data size, as shown in the figure. Figure 4C As shown, P#2 can be split into 5 partitions as much as possible according to the ideal data volume, and assigned to the 5 working nodes R#1 to R#5 in the Reduce stage for processing, where R#1 to R#4 are each assigned a partition with a data volume of 9, and R#5 is assigned a partition with a data volume of 7. Similarly, since the data volume of P#3 is larger than the ideal data volume, it is evenly split into 2 partitions and assigned to the 2 working nodes R#6 and R#7 in the Reduce stage for processing.

[0146] The above-mentioned mechanism of splitting the partitions with a data volume larger than the ideal data volume in the Map phase and merging the partitions with a data volume smaller than the ideal data volume can be called the Adaptive shuffle mechanism. In some further embodiments, based on the Adaptive shuffle mechanism, the number of files finally written in the Reduce phase will mainly depend on the size of the input Partition data volume and the size configuration of the ideal data volume. Assuming that the ideal number of Partitions (Ideal Parallelism) input to the Reduce phase is defined as: the total amount of shuffle data divided by the ideal data volume; then after the Adaptive shuffle mechanism of the embodiment of the present application, the maximum number of files finally generated in the Reduce phase is the Ideal Parallelism plus N. Therefore, if it is to be ensured that there is at least one data file on each Partition with output data, only the Partition with a data volume larger than the ideal data volume will increase the number of files after splitting. However, the file sizes generated by this splitting are all of the order of magnitude of the Partition, rather than fragmented small files. Based on this, if the size configuration of the ideal data volume is reasonable, no matter how many files are generated, it is reasonable for the distributed system; that is, as long as the file size is appropriate, massive data can be stored through more files.

[0147] That is to say, based on the Adaptive shuffle mechanism, the amount of data processed by each working node in the Reduce stage will not exceed the given ideal data amount, and each Partition that is smaller than the ideal data amount can output a file by a working node in the Reduce stage. This fundamentally solves the problem of a large number of small files being generated and possible data skew, and can better solve the dilemma faced in dynamic partitioning scenarios. It should also be noted that the use of the Adaptive shuffle mechanism can avoid the need to add additional Random shuffle Keys to reduce data skew. Since the Adaptive shuffle mechanism provided in the embodiment of the present application is deterministic and reentrant throughout the process, it can fundamentally guarantee the correctness of the output data when various retries occur in an unstable distributed system environment.

[0148] The embodiment of the present application realizes adaptive data orchestration by combining the statistical information collection capability during job operation provided by the execution engine (master node) and the dynamic adjustment capability of multi-job execution graph and shuffle mode; it can intelligently distribute and orchestrate the output data based on the data characteristics of the output data of the upstream stage, including automatic splitting of skewed data partitions and merging of multiple small partitions with large data volumes, fundamentally solving the data skew and long-tail problems of working nodes that may be caused in the data shuffle process, while avoiding the data fragmentation problem in other non-shuffle solutions, and can significantly improve the performance of the distributed system.

[0149] In addition to the general optimization of Partition merging and splitting, the embodiments of the present application have been specially optimized for specific scenarios. Taking the Join scenario as an example, the implementation scheme of splitting the Partition output by the upstream stage based on the statistical information of the output data of the upstream stage and assigning it to the direct downstream stage for Join operation is introduced below.

[0150] In the process of executing jobs in distributed systems, the Join operation is the most common and also one of the operations with the highest processing complexity. Since it involves the calculation and processing of multi-channel data, in addition to the challenges that need to be solved in distributed systems in general, the interaction of different channels of data on the Join operator also derives more data processing scenarios. However, uneven data distribution and other situations will lead to data skew and long tail problems in the Join operation, which has become a common problem in distributed systems that has not been systematically solved.

[0151] Figure 5AThe following is an example diagram showing the Join process of distributed SQL. Figure 5A As shown, the upstream stage provides two input data to the downstream stage (such as Figure 5A As shown in the figure, M1 and M2 perform a Join operation in the downstream stage, where the two input data of M1 and M2 can be written out according to the Partition and shuffled to different working nodes of J3 according to the partition number to realize data Join. The intermediate data will be arranged according to the partition and stored on the physical medium.

[0152] Figure 5A The display shows that the Partition data of the upstream stage is evenly distributed. However, in actual query and data processing, the distribution of the Partition data of the upstream stage is likely to be skewed. Figure 5B Another example diagram of the Join process of distributed SQL is shown. Figure 5B As shown in the figure, there is serious data skew in Partition1 (Partition numbered 1) in the input data provided by M1 to J3. At the same time, there is slight data skew in Partition1 in the input data provided by M2 to J3. In this case, when J3 performs a Join operation on Partition1, J3's working node will have a serious long tail, and may even cause the job to fail due to memory overrun.

[0153] To solve the above problems, Figure 5C Another flow chart of the distributed job adjustment method provided by the embodiment of the present application is shown. The method flow can be implemented by the master node, refer to Figure 5C The method flow may include the following steps.

[0154] In step S510, a job submitted by a user is obtained.

[0155] In step S511, an execution plan of the job is generated, the execution plan includes multiple stages, the multiple stages include a Join stage and a direct upstream stage of the Join stage, the direct upstream stage provides multiple input data to the Join stage; wherein one input data includes multiple Partitions.

[0156] In the case of Join operation, the Join stage, as the direct downstream stage of a stage, can input multiple input data provided by the stage, and one input data can include multiple Partitions.

[0157] In step S512, during the job execution, if there is a target partition with data skew in any input data, the target partition is split into multiple partitions, and the multiple sub-partitions are assigned to multiple working nodes of the Join stage.

[0158] During the job execution process, the master node can obtain the Statistics of the output data of any stage. Based on the Statistics, if there is a Partition with data skew in any input data provided to the Join stage by the direct upstream stage (for the sake of convenience, the Partition with data skew can be called the target Partition), then the embodiment of the present application can split the target Partition into multiple Partitions and assign the multiple Partitions to multiple working nodes of the Join stage.

[0159] In some embodiments, the embodiments of the present application can determine whether the data volume of a Partition in any input data is greater than a first data volume threshold. If the data volume of the Partition is greater than the first data volume threshold, the Partition is determined to be a target Partition with data skew.

[0160] In some embodiments, when the target Partition is split, the embodiment of the present application may split the target Partition based on the second data volume threshold, so that the data volume of multiple sub-Partitions after the target Partition is split is evenly distributed and the data volume of each sub-Partition is not greater than the second data volume threshold. In some further embodiments, the second data volume threshold is less than the first data volume threshold, and the specific values ​​of the first data volume threshold and the second data volume threshold can be defined according to actual conditions, and the embodiment of the present application does not limit it. In an implementation example, the second data volume threshold can be, for example, the ideal data volume described above.

[0161] In some embodiments, when multiple sub-Partitions split from a target Partition are allocated to multiple working nodes of a Join stage, one Partition can be allocated to one working node of the Join stage.

[0162] In step S513, the partitions in other input data paths that have the same partition number as the sub-partition are broadcast to the working nodes assigned to the sub-partition.

[0163] In order to achieve correct Join operation in the Join stage, after splitting the target Partition of a certain input data into multiple sub-Partitions and assigning the sub-Partitions to the working nodes of the Join stage, the embodiment of the present application can determine, for each sub-Partition after the target Partition is split, the Partition with the same partition number as the sub-Partition in other input data (input data in multiple input data that is different from the certain input data), and then broadcast the determined Partition to the working nodes assigned to the sub-Partition, so that the sub-Partition and the Partition with the same partition number in other input data can be correctly joined in the Join stage.

[0164] In some embodiments, the multi-channel input data provided by the direct upstream stage may include at least a first channel of input data and a second channel of input data. The first channel of input data and the second channel of input data may be any two channels of input data in the multi-channel input data. The number of channels of the multi-channel input data may be greater than two, and is not limited to only two channels of input data. In this case, there may be the following situations in which the target Partition exists in the multi-channel input data.

[0165] In the first case, there is a target partition in one of the multiple input data. In the first case, taking the case where there is a target partition in the first input data among the multiple input data as an example, the master node can split the target partition in the first input data into multiple sub-partitions and assign each sub-partition to the work node of the Join stage. The partitions in other input data (input data except the first input data among the multiple input data) that have the same partition number as the target partition will be broadcast to the work nodes assigned to each sub-partition.

[0166] As an implementation example, for the first case above, Figure 5D Another example diagram of the Join process is shown. Figure 5D As shown, Figure 5DThe following example shows a scenario where two Join inputs (M1 and M2) have data skew in one of them. Assuming that the worker node that executes Join does not need to maintain operators with features such as Partition and Sort, the master node can determine that Partition 1 output by M1 is the target Partition with data skew after counting the data volume of each Partition output by M1 and M2, and thus split Partition 1 output by M1 into multiple sub-Partitions. Figure 5D In the figure, a small box in the box where Partition1 of M1 is located can represent a split sub-Partition. The multiple sub-Partitions split from Partition1 of M1 can be assigned to multiple working nodes in the Join stage for Join processing. At the same time, in order to perform correct Join operations on the split sub-Partitions in the Join stage, the master node needs to broadcast the Partitions in M2 with the same partition number as the target Partition to the working nodes assigned to each sub-Partition; for example, the master node needs to broadcast Partition1 generated by M2 to the working nodes assigned to each sub-Partition of M1. By broadcasting the Partitions with the same partition number of M1 and M2 to multiple working nodes in the Join stage, it can be ensured that the data with the same partition number can be correctly joined and produce results when the target Partition is split into multiple sub-Partitions.

[0167] In the second case, there is a channel of input data among multiple input data with multiple target partitions; in the second case, taking the case where there are multiple target partitions in the first channel of input data among multiple input data as an example, the processing of each target partition in the first channel of input data is the same as the first case. In some embodiments, for each target partition in the first channel of input data, the master node may split the target partition into multiple sub-partitions, and assign the sub-partitions of the target partition to the working nodes of the Join stage. At the same time, the partitions in other channels of input data that have the same partition number as the target partition will be broadcast to the working nodes assigned to each sub-partition. However, in the case where there are multiple target partitions in the first channel of input data, each target partition needs to be processed in the above manner.

[0168] That is to say, if there is one target Partition or multiple target Partitions in the first input data, then for any target Partition, the embodiment of the present application can split the target Partition into multiple sub-Partitions, and assign each sub-Partition of the target Partition to the working node of the Join stage. At the same time, the Partitions in other input data that have the same partition number as the target Partition are broadcast to the working nodes assigned to each sub-Partition of the target Partition.

[0169] Further integration Figure 5D As shown in the figure, if data skew occurs in multiple partitions in a single input of Join, each partition with data skew can be split and another partition with the same partition number can be broadcast to the working nodes in the Join phase, so that the split sub-partition can be correctly joined with the data of another partition with the same partition number in the Join phase. For example, suppose Figure 5D In the example, if Partition3 output by M1 also has data skew, Partition3 output by M1 can also be split into multiple sub-Partitions and assigned to multiple working nodes in the Join phase. At the same time, Partition3 in M2 will be broadcasted to the working nodes assigned to the sub-Partitions corresponding to Partition3 of M1.

[0170] In the third case, at least two of the multiple input data have a target partition. In the third case, taking the case where both the first and second input data have a target partition, the master node can split a target partition in the first input data into multiple sub-partitions, and assign each of the split sub-partitions to the work nodes of the Join stage. At the same time, the master node can split a target partition in the second input data into multiple sub-partitions, and assign each of the split sub-partitions to the work nodes of the Join stage. For the sub-partitions with the same partition number in the first input data and the second output, the sub-partitions need to be broadcasted to the work nodes assigned to the other sub-partitions with the same partition number. For example, the sub-partitions of the first input data are broadcasted to the work nodes assigned to the sub-partitions with the same partition number in the second input data, and the sub-partitions of the second input data are broadcasted to the work nodes assigned to the sub-partitions with the same partition number in the first input data.

[0171] Figure 5E Another example diagram of the Join process is shown. Figure 5E As shown in the figure, there is data skew in Partition1 of M1 and M2. The Partition1 output by the left-path M1 needs to be split into multiple sub-partitions, and the Partition1 output by the right-path M2 needs to be split into multiple sub-partitions. Then, the multiple sub-partitions split from Partition1 of M1 will be assigned to multiple working nodes in the Join phase, and the multiple sub-partitions split from Partition1 of M2 will also be assigned to multiple working nodes in the Join phase, and one sub-partition is assigned to one working node. Figure 5E A small box in the box where Partition1 of M1 and M2 is located can represent a split sub-Partition. To ensure that the sub-Partitions with the same partition number of M1 and M2 can be correctly joined, the master node can broadcast multiple sub-Partitions split from Partition1 of M1 to the working nodes assigned to the sub-Partitions with the same partition number of M2. Similarly, the master node can broadcast multiple sub-Partitions split from Partition1 of M2 to the working nodes assigned to the sub-Partitions with the same partition number of M1.

[0172] In the fourth case, there are multiple target partitions in at least two of the multiple input data. In the fourth case, taking the case where both the first input data and the second input data have multiple target partitions as an example, the processing process for the same target partition in the first input data and the second input data is the same as the third case. However, when both the first input data and the second input data have multiple target partitions, each of the same target partitions in the first input data and the second input data needs to be processed in the above manner.

[0173] The above example is that the work node in the Join stage mainly implements a simple Join connection, and does not include operators for the data sorted / partitioned attributes. In actual operations, as the most complex and flexible Join operation in SQL syntax, its work node may contain a variety of operators. In this case, when data skew occurs in the Partition input to Join, in addition to splitting the Partition using the method in the above example, a union operation must be added after completing the data Join, so that the split sub-Partitions can be reassembled after Join. When special attributes of the data need to be retained (for example, the data is not directly stored on the disk, but there are downstream operations / stages), a union operation is required to ensure the correctness of subsequent execution.

[0174] Fig. 5F For Figure 5D The following is an example of a union operation. Figure 5D and Fig. 5F As shown in the figure, when data skew occurs in Partition1 of the single input M1 of Join, in addition to using Figure 5D The process shown implements the Join operation on the sub-Parition corresponding to the Parition1 of M1 and the Parition1 of M2, and also combines the Join operation results of each sub-Parition to generate a new Parition1.

[0175] Figure 5G For Figure 5E The following is an example diagram of the union operation based on the above. When data skew exists in both Partition 1 of the Join multi-path input M1 and M2, in addition to using Figure 5EThe process shown implements the Join operation of the sub-Parition of Parition1 of M1 and the sub-Parition of the same partition number of Parition1 of M2, and also combines the Join operation results of each sub-Parition to generate a new Parition1.

[0176] That is to say, the master node also needs to be configured to combine the Join operation results belonging to the same data partition, and the Join operation results of the same data partition include: the Join operation results of the sub-data partition of one input data and the data partition with the same partition number in other input data, or, the Join operation results of the sub-data partition of one input data and the sub-data partition with the same partition number in other input data.

[0177] The embodiment of the present application is aimed at the skewed scenario of the input data of the Join operation. When there is a skewed Partition in the multi-channel or single-channel of the Join input, it can perform adaptive splitting and distribution, so as to achieve uniform distribution of the input data of the Join operation and ensure the correctness of the Join operation in the new shuffle mode. Compared with the method of manually adjusting the job data processing logic and the operations such as data preprocessing that require a large number of terminal users to intervene, the embodiment of the present application can make the skew processing of the input data of the Join operation adaptive and universal. Especially in the case where the master node collects the statistical information of the output data of each stage, the embodiment of the present application enables the adjustment of data skew to be automatically performed without manual adjustment by the user. The adaptability of the solution provided by the embodiment of the present application enables it to make dynamic decisions according to the actual Join input data characteristics during the operation of the job, without the need for terminal users to perceive and participate, and can make the data evenly distributed among multiple distributed working nodes, which can significantly accelerate the operation with data skew and achieve a significant improvement in the performance of the distributed system.

[0178] In addition to the dynamic adjustment of the Partition, according to the statistical information of the upstream stage, the embodiment of the present application can adaptively select the execution path for subsequent execution. The specific implementation scheme is introduced below.

[0179] The embodiment of the present application can dynamically adjust the logic diagram of the execution plan during the execution of the job (i.e., dynamically adjust the logic of the execution plan during the execution of the job). It should be noted that the selection of the logic diagram may be related to the data distribution and characteristics, and these data characteristics can only be accurately obtained during the execution of the job. Different data characteristics may require the configuration of execution plans with different logics to be effectively and accurately implemented. Therefore, for static execution plans, once the execution plan is determined and cannot be dynamically adjusted during the execution of the job, it is undoubtedly impossible to achieve a reasonable and accurate configuration of the logic of the execution plan.

[0180] Taking the Join operation in distributed SQL as an example, to implement the Join of distributed jobs, there are multiple logically equivalent Join algorithms, such as Sort Merge Join and Broadcast Join, etc. Taking the Join of the two source tables Table1 and Table2 as an example, the implementation process of Sort Merge Join and Broadcast Join is described below.

[0181] Fig. 6A An example diagram of Sort Merge Join is shown. Fig. 6A As shown in the figure, the two input source tables Table1 and Table2 can be read and preliminarily processed by M1 and M2. For example, after M1 filters the data of Table1, it partitions the output data according to the shuffle / Join key; the output data of M1 and M2 can be merge-joined according to the same key in the downstream working node. In the above process, the implementation of Merge Join requires full shuffle and sort operations on the output data of M1 and M2, ensuring that data in the same partition can be allocated to the same downstream working node. However, in a distributed system, Sort Merge Join relies on specific implementations such as external row to process any amount of data, but the large number of shuffle and sort operations involved in the process consume more computing and network resources; and when the data is unevenly distributed, the shuffled data may lead to serious long tails, affecting execution efficiency.

[0182] Figure 6B The following figure shows an example of Broadcast Join. A distributed system needs to be able to handle the Join operation between large tables (such as fact tables) and large tables. For the Join operation between a large table and a small table (such as a dimension table), if the size of the small table data source can be loaded into the memory of a single working node, the computing / network consumption of distributed shuffle and sort can be reduced. Figure 6B As shown, Figure 6B In the example, Table1 is a large table and Table2 is a small table. The data of the small table can be broadcasted to all the working nodes of the large table, and a full hash table can be established based on the data of the small table. After the data of the large table is read, it can be joined through hash table lookup. In this way, the data of the large table (Table1) only needs to be read once, and no data shuffle or sort is required. At the same time, in addition to avoiding the computing / network consumption of shuffle and sort, Broadcast Join can also avoid data skew and long tail caused by shuffle. This is because in the Join pattern of Broadcast Join, the Join logic actually occurs in the reading logic of the large table (for example, Map stage). Based on this, Broadcast Join is also called Broadcast Map Join. Of course, while Broadcast Join brings many benefits such as resources / performance, it also has a specific scope of application: the small table data used to establish the hash table must be fully loaded by a single working node. If the optimizer chooses Broadcast Join to execute the job, but finds that the amount of data in the small table exceeds the memory limit during execution, the entire job will fail.

[0183] Based on the description above, Broadcast Join has significant advantages in performance, and Sort Merge Join itself has better versatility. Therefore, if the optimizer of a distributed system needs to choose a suitable Join algorithm between the two, it needs to make more accurate and reasonable judgments: configure efficient execution plan logic as much as possible while ensuring that the job can be successfully completed. However, in actual online scenarios, it is very difficult for the optimizer to make the above judgment before the job is executed, mainly for the following reasons:

[0184] Lack of accurate statistical information: Data stored in distributed systems comes from a variety of sources. Due to various reasons such as the source table data import channel and import time, the source table data's statistical information may be missing or inaccurate (for example, a table has just been imported and the statistical information has not yet been generated, or the table content update has just reached the threshold for discarding statistical data, etc.). In short, the lack of and inaccurate statistical information makes it impossible for the optimizer to accurately estimate the size of the upstream input of the Join, which may be the source table or the output of the source table after a certain logical conversion.

[0185] The data processing logic and data characteristics are complex and changeable. Even if complete and accurate statistical information can be obtained on the source table data, the Join operation may objectively exist in any work node in a data processing flow (within the DAG). The source table data may be interspersed with various user-defined code logics (UDFs) due to complex transformations such as upstream selection / filter / aggregation. All these make it difficult for the optimizer to estimate the amount of Join input data in advance.

[0186] Due to these limitations, if the optimizer statically selects the Join algorithm before the job is executed, it will face a dilemma: on the one hand, when the size of the input data is not accurately estimated, the selection of the Map Join execution plan can only be as conservative as possible, for example, the threshold of the small table is set as low as possible, thus losing a lot of optimization opportunities. Even if the threshold of the small table is configured very low, due to data estimation errors, data expansion and other reasons, the job execution may still fail due to misjudgment of Broadcast Join. Since these extreme cases will result in the failure of job execution, they are fed back to the optimizer's strategy, and the optimizer will further choose a more conservative strategy, thus creating a negative cycle. In addition, the triggering of Broadcast Join is largely based on the manually added Map Join hint, which means that the generation of Broadcast Join plan is left to the user. This outsourcing of optimizer functions brings additional difficulties to user logic maintenance. In fact, users can only perceive the source table data volume relatively accurately, but cannot accurately know the output size of non-source table data after data changes. Therefore, users specifying Map Join hint cannot avoid job execution failures caused by changes in data and upstream processing logic.

[0187] Based on the description above, the optimizer of the master node is required to make an accurate selection of the Join algorithm before the job is executed, and its accuracy is affected by various objective factors. In general, the characteristics of the data (including data size and distribution, etc.) can only be obtained after the upstream working node completes the job execution process. Therefore, to make an accurate judgment on the Join algorithm, it is necessary to make the judgment during the execution of the distributed job, rather than before the job execution. However, the judgment and selection of the Join algorithm during the job execution process poses a challenge to the DAG dynamic capability of the execution engine: when selecting different Join algorithms such as Sort Merge Join and Broadcast Join, the execution plan generated will not only have different physical properties (concurrency, shuffle mode, etc.), but also have significant changes in the topological logical structure of the DAG. Therefore, if dynamic adjustments are to be made during the job execution process, the ability of dynamic logical graphs needs to be provided (that is, the logic of the execution plan can be dynamically adjusted). It should be noted that the adjustment of the logical graph is often accompanied by the adjustment of the physical properties of the execution plan, so this actually puts forward requirements for both the dynamic logical graph and the dynamic physical graph capabilities of the DAG.

[0188] Based on the above description, Figure 6C Another flow chart of the distributed job adjustment method provided in the embodiment of the present application is shown. The method flow can be implemented by the master node. Figure 6C The method flow may include the following steps.

[0189] In step S610, the job submitted by the user is obtained.

[0190] In step S611, an execution plan of the job is generated, the execution plan includes multiple stages and a control node, the multiple stages include an upstream stage, wherein the upstream stage has multiple candidate execution paths downstream, and one execution path includes one or more downstream stages of the upstream stage; the control node is used to select a target execution path that is actually executed by the upstream stage downstream from the multiple execution paths.

[0191] In an embodiment of the present application, when the execution plan of a job is generated, it can carry multiple candidate execution paths downstream of the upstream stage, and one execution path can include one or more downstreams of the upstream stage. For example, an execution path can include the direct downstream stage of the upstream stage, or the direct downstream stage and the indirect downstream stage. In some embodiments, one or more upstream stages in the execution plan can carry multiple candidate execution paths downstream thereof.

[0192] In an embodiment of the present application, in addition to including multiple stages, the execution plan may also set a control node downstream of the upstream stage, and the control node may be connected to multiple execution paths of the upstream stage, and may select an execution path from the multiple execution paths as the target execution path actually executed by the upstream stage downstream. It should be noted that the control node does not belong to the execution stage in the execution plan, and does not actually schedule computing resources such as work nodes, but rather identifies that the execution plan needs to add control logic downstream of the upstream stage in order to select the target execution path actually executed from multiple execution paths.

[0193] In some embodiments, the execution plan referred to in step S611 may be described by a DAG, for example, one or more upstream stages of the DAG may have multiple candidate execution paths downstream. As an example, Fig.6D The schematic diagram of an execution plan carrying multiple execution paths is shown as an example. Fig.6D As shown, there are two candidate execution paths downstream of M1: Path0 and Path1. A control node C8_1 is provided downstream of M1, and the control node C8_1 is connected to Path0 and Path1, and can be used to select a target execution path actually executed downstream of M1 from Path0 and Path1.

[0194] In step S612, during the job execution, statistical information of the output data of the upstream stage is obtained.

[0195] In step S613, a target execution path is selected from the multiple execution paths based on the statistical information through the control node.

[0196] During job execution, the master node can collect statistics on the output data of each stage of the execution plan, including statistics on the output data of the upstream stage with multiple execution paths. Fig.6D As shown in FIG. 1 , after M1 is executed during the job execution process, the statistical information of M1's output data can be collected by the master node. Based on the statistical information of the output data of the upstream stage collected by the master node, the master node can select the target execution path to be actually executed from multiple execution paths of the upstream stage downstream through the control node based on the statistical information. For example, combined with Fig.6D As shown, the control node C8_1 can select a target execution path actually executed downstream of M1 from Path0 and Path1 based on the statistical information of the output data of M1.

[0197] The embodiment of the present application can generate an execution plan so that the upstream stage of the execution plan can have multiple candidate execution paths downstream, and during the execution of the job, based on the statistical information of the output data of the upstream stage, select the target execution path for final execution from the multiple candidate execution paths. In this way, during the execution of the job, the final execution path can be selected according to the actual situation of the output data of the upstream stage, making the selection of the execution path more reasonable and accurate. Executing jobs based on reasonable and accurate execution paths can significantly improve the performance of distributed systems.

[0198] As an optional implementation, the candidate execution paths carried downstream by the upstream stage of the execution plan may include two execution paths, namely, a first execution path and a second execution path. Fig. 7A The flowchart of generating an execution plan with multiple execution paths is shown as an example. The method flow can be implemented by the master node. Fig. 7A The method flow may include the following steps.

[0199] In step S710, a physical plan with multiple candidate information is generated according to the estimated data size of the source data of the job, where one candidate information represents a task that may be used by the source data during the execution of the job, and the multiple candidate information includes first candidate information and second candidate information.

[0200] When a job is submitted, the optimizer may estimate the data size of the source data (e.g., source table) of the job. If the estimated data size of the source data is less than a preset first threshold value, the optimizer may trigger a conditional execution plan. Based on the tasks that the source data may use during the job execution, the optimizer may generate multiple candidates for the source data and a physical plan with the multiple candidates. In some embodiments, a task may include one or more operators, such as implementing the execution of a specific task through the processing of an operator.

[0201] In one example, assuming that the source data is a source table, the source table may use Broadcast Join and Sort Merge Join when performing a Join operation during job execution, so the optimizer may generate two candidates for the source table based on the Broadcast Join and Sort Merge Join that may be used by the source table during the Join operation, where one candidate represents Broadcast Join and the other candidate represents Sort Merge Join. In some embodiments, the physical plan may be considered as a data structure of a node tree.

[0202] In step S711 , the physical plan is converted into an operator tree, where the operator tree includes one or more original tasks of the physical plan, and one original task uses one or more operators.

[0203] In step S712, when the preset operator is traversed for the first time, a task of the first execution path is added to the operator tree according to the first candidate information corresponding to the first execution path, and the operator in the newly added task is connected to the related operator in the original task by a data pipe to convert the first execution path in the operator tree.

[0204] In order to generate an execution plan with multiple execution paths, after converting the physical plan into an operator tree, the optimizer can convert multiple execution paths in the operator tree according to multiple candidate information. In some embodiments, the optimizer can traverse the operator tree multiple times and pre-set a preset operator for execution path conversion. When the optimizer traverses to the preset operator for the first time, it can convert the first execution path corresponding to the first candidate information in the multiple candidate information. In some embodiments, the multiple candidate information includes at least the first candidate information and the second candidate information, wherein the first candidate information corresponds to the first execution path and can record the tasks corresponding to the first execution path, and the second candidate information can correspond to the second execution path and can record the tasks corresponding to the second execution path.

[0205] In some embodiments, when the optimizer traverses to the preset operator for the first time, it can add a task of the first execution path in the operator tree according to the first candidate information. The newly added task can include one or more operators. In order to enable the task of the first execution path to be connected with the original task of the operator tree, the optimizer can connect the operator of the newly added task with the relevant operator in the original task through a pipeline. The relevant operator in the original task can be considered as an operator in the original task that is related to the operator of the newly added task (for example, an operator in the original task that is upstream or downstream of the operator of the newly added task). Through the above processing, the optimizer can convert the first execution path in the algorithm tree.

[0206] In step S713, when the preset operator is traversed for the second time, a task of the second execution path is added to the operator tree according to the second candidate information corresponding to the second execution path, and the operator in the newly added task is connected to the related operator in the existing task by data pipe, so as to convert the second execution path in the operator tree.

[0207] In some embodiments, when the optimizer traverses to the preset operator for the second time, the second execution path can be converted in the operator tree according to the second candidate information. The optimizer can add tasks of the second execution path in the operator tree according to the second candidate information. The newly added tasks can include one or more operators. In order to enable the tasks of the second execution path to be connected with the existing tasks in the operator tree, the optimizer can connect the operators of the newly added tasks with the related operators in the existing tasks through datapipe. The related operators in the existing tasks can be considered as operators in the existing tasks that are related to the operators of the newly added tasks (for example, operators in the original tasks that are upstream or downstream of the operators of the newly added tasks). Through the above processing, the optimizer can convert the second execution path in the algorithm tree.

[0208] As an optional implementation, step S712 can be considered as an optional implementation of transforming the first execution path in the algorithm tree based on the first candidate information, and step S713 can be considered as an optional implementation of transforming the second execution path in the algorithm tree based on the second candidate information.

[0209] In step S714, a control node is set upstream of the first execution path and the second execution path of the operator tree to obtain an execution plan.

[0210] After the conversion of the first execution path and the second execution path is completed in the operator tree, in order to enable the target selection path to be selected from the first execution path and the second execution path after the upstream stage of the first execution path and the second execution path is executed during the job execution process, the optimizer may also set a control node upstream of the first execution path and the second execution path of the operator tree to realize the dynamic selection of the first execution path and the second execution path during the job execution process through the control node. Through the above process, the embodiment of the present application can generate an execution plan carrying multiple execution paths.

[0211] To facilitate the description of the above process of generating an execution plan with multiple execution paths, the embodiment of the present application introduces the concept of conditional execution plan: allowing the optimizer to generate an execution plan with multiple execution paths based on multiple candidates (candidate information) when the job is submitted. The final candidate execution path is selected for use by the master node during the job execution process based on the statistical information of the output data generated by the upstream stage.

[0212] As an optional implementation, the optimizer can generate an execution plan with multiple execution paths through two steps: Cost-based Optimization and Execution plan generation.

[0213] In the Cost-based Optimization phase, taking the Join scenario as an example, the optimizer can estimate the data size of the Join's small table in memory through source table statistics and other information in the Join buildrule. For example, the data size of the small table in memory is estimated based on the RowCount (number of rows) * AverageRowSize (average row size) of the small table. Then, the optimizer can determine whether the data size of the small table in memory is less than the preset first threshold value (the first threshold value can be preset and defined as threshold1). If the data size of the small table in memory is less than the preset first threshold value, the optimizer will trigger the conditional Map Join of the database, for example, generate multiple candidates for Map Join of the small table data, and a candidate can represent a Join operator (such as Broadcast Join or Sort Merge Join) used by the small table. To implement subsequent dynamic decisions based on multiple candidates provided by conditional Map Join, the optimizer can configure the first threshold value threshold1 of the Cost-based Optimization link to be relatively large (the default value is 512M). That is, as long as a job is judged to use Broadcast Join based on a relatively loose probability, a conditional Map Join plan is generated, and the final selection of the execution path is left to the execution engine to make a dynamic selection during the job execution process.

[0214] The above conditional Map Join plan can be regarded as a physical plan, not the final execution plan. The physical plan can be a RelNode tree structure, where RelNode represents a relational expression. In the Cost-based Optimization phase, the physical plan contains a control node such as conditional Map Join, and the control node can contain multiple paths based on multiple candidates to express the possible paths selected by the Join algorithm, such as the execution paths corresponding to Broadcast Join and Sort Merge Join. Regardless of which path is finally selected, the calculation and data of the small table will be shared among the multiple paths. The cost defined in the optimizer's cost model for conditional Map Join can be between the costs of Broadcast Join and Sort Merge Join, and the cost ratio of the two is determined based on the probability of the final selected path.

[0215] In the execution plan generation phase, the optimizer converts the physical plan generated in the previous step into the final execution plan. For example, the physical plan generated by the optimizer is converted into a physical operator tree that can be understood by the runtime and a DAG topology is constructed. Therefore, the final execution plan includes a DAG composed of tasks and edges, as well as an operator tree (i.e., a physical operator tree) inside each work node. In general, the above process is a post-order traversal process of a physical plan, and tasks are dynamically split and edges are added during the traversal process. Different from ordinary queries, conditional Map Join needs to realize dynamic decision-making and path selection at runtime, so control tasks and dependent edges need to be added during the execution plan generation process, so that the scheduling dependency between upstream and downstream work nodes (no data flow) and a simple control node can be accurately described through dependent edges. For example, ordinary work nodes containing normal execution operators are connected to control nodes through dependent edges. With the above settings, during the DAG running process, the control node can select the subsequent execution path by judging the actual size of the small table after running.

[0216] Figure 7B , Figure 7C , Fig.7D and Fig. 7E The process of converting a physical plan into an execution plan is shown as an example. Figure 7B Afterwards you can execute Figure 7C ,Finish Figure 7C Afterwards you can execute Fig.7D ,Finish Fig.7D Afterwards you can execute Fig. 7E. The thick arrows in the diagram represent data pipelines, the thin dashed arrows represent control pipelines, and the thin solid arrows represent runtime operator data flow. In an implementation example, when the master node traverses the physical plan, if it traverses to the conditional Map Join and is ready to transform multiple paths contained in the conditional Map Join, the master node can convert the input relNode tree of the conditional Map Join into an operator tree. The operators in the operator tree can form Figure 7B The two tasks M1 and M2 shown, for example, M1 uses the two operators TableScan1 and Filter1, and M2 uses the three operators TableScan2, Filter2, and StreamlineWrite1.

[0217] Assume that the conditional Map Join contains two candidates, one of which corresponds to Path0 and the other corresponds to Path1. Path0 can correspond to the Join implementation of Broadcast HashJoin, and Path1 can correspond to the Join implementation of Sort Merge Join. Figure 7C As shown in the figure, when the master node is traversing the operator tree, if it traverses filter1 of M1 for the first time, it can transform Path0. Specifically, based on the task recorded by the candidate of Path0, the master node can add a task of Path0 to the operator tree, and connect the operator in the added task to M1 and M2 through a data pipe. Figure 7CAs shown, R3 is a newly added task and has two newly added operators, StreamlineRead1 and StreamlineWrite2. R1 has newly added StreamlineRead2 and ConditionalMapJoin1. Among them, the newly added ConditionalMapJoin1 in M1 is the operator id of Broadcast Hash Join, and its operator type is HashJoin.

[0218] The above process is executed when filter1 in M1 is traversed for the first time, so as to transform Path0 based on the operator tree, and thus express the execution path of Path0 in the execution plan; in this process, the master node can record the original operator of M1 in the operator tree for subsequent copying. From the four newly added operators, it can be inferred that R3 and M1 are currently being processed. At this time, they are recorded as a candidate Path0.

[0219] Then, Path1 is transformed. Path1 and Path0 share StreamlineWrite and Filter, such as StreamlineWrite1 and Filter1. Based on the operator Filter1 shared by Path1 and Path0, when the master node traverses Filter1 for the second time, Path1 can be transformed. Specifically, based on the task recorded by the candidate of Path1, the master node can add a task of Path1 to the operator tree of the transformed Path0, and connect the operator in the newly added task with the relevant operator in the current task through a data pipe. Fig.7D As shown in the figure, M5 and J6 are newly added tasks. When adding the task of M5, the operator of M1 recorded previously (for example Fig.7D TableScan1 and Filter1 in J6 are copied to M5, and a StreamlineWrite3 operator is added to M5. When adding a task to J6, three operators, StreamlineRead3, StreamlineRead4, and ConditionalMapJoin1, can be added to J6, and the data pipe between J6 and M2 and M5 at the operator level can be implemented.

[0220] Combination Fig. 7EAs shown in the figure, after completing the transformation of Path0 and Path1, it is necessary to create a control node C8 in the operator tree. There is only one operator in C8, namely the Conditional operator. Its function is to select the execution path for Join from Path0 and Path1 based on the size of the table during the actual operation of the table.

[0221] In some further embodiments, after generating an execution plan with multiple execution paths, the master node can obtain the amount of output data of the upstream stage during the job execution process, and determine whether the amount of data is less than a preset second threshold value. If so, the first execution path is selected as the target execution path, and if not, the second execution path is selected as the target execution path. Assuming that the data processed by the upstream stage is a small table, during the job execution process, if the task corresponding to the small table is executed, the control node can determine whether the data output result of the small table is less than a preset second threshold value (defined as threshold2) based on the actual execution size of the small table. If the execution output result of the small table is less than the second threshold value, the execution path of Broadcast Join is selected (that is, the first execution path is the execution path for executing Broadcast Join); if the execution output result of the small table is greater than the second threshold value, the execution path of Sort Merge Join is selected (that is, the second execution path is the execution path for executing Sort Merge Join). It should be noted that threshold2 takes into account the memory size occupied by the small table data when loaded into the hash table in the actual operation. Its default value can be the same as threshold1 (for example, 512M). Of course, the embodiment of the present application can also set the values ​​of threshold2 and threshold1 to be different. It should also be noted that the conditional Join plan can only be triggered when the estimated size of the small table meets threshold1. During the operation of the job, when the master node collects the actual size of the small table, the final execution path of the small table Join is determined by threshold2; therefore, threshold1 is used for decision-making in the optimizer stage, and threshold2 is used for decision-making in the DAG operation stage. The entire conditional (condition) generation and the final execution plan selection are completed by adjusting the values ​​of threshold1 and threshold2, and the cooperation between the optimizer and DAG.

[0222] In terms of dynamic selection of execution paths, combined with Fig.6D As shown in Figure 1, when the job is initially submitted, the final execution path has not yet been confirmed, so the submitted DAG contains two possible execution paths (for example, Fig.6D The two execution paths Path0 and Path1 are shown by the dotted lines. At the same time, a new control node (C8_1) is added to the DAG. This control node only has logical control significance and will not pull up any working nodes. The control node chooses to execute Path0 or Path1 based on the output of M1. For example, during the job execution, after M1 reads the candidate output data of the small table and processes it, its actual output data volume will be collected by the master node, and the control node will make a decision on the execution path based on the actual output data volume of the small table. If Path0 is selected, the complete execution plan can be, for example Figure 7F As shown, if you choose to execute Path1, the complete execution plan can be, for example Figure 7G shown.

[0223] The embodiment of the present application can adjust the execution plan before the job is executed, so that the upstream stage of the execution plan can have multiple candidate execution paths in the downstream, and during the execution of the job, based on the statistical information of the output data of the upstream stage, the target execution path for final execution is selected from the multiple candidate execution paths. In this way, during the execution of the job, the final execution path can be selected according to the actual situation of the output data of the upstream stage, so that the selection of the execution path is more reasonable and accurate, so that the job is executed based on a reasonable and accurate execution path, which can significantly improve the performance of the distributed system.

[0224] With the widespread application of deep learning, distributed systems have more and more processing requirements for deep learning jobs, and a variety of execution engines suitable for deep learning jobs have emerged. However, for deep learning jobs, distributed systems still have various defects in scheduling and execution. For example, for the native logic of deep learning systems (such as Tensorflow), the scheduling and execution of native logic completely rely on external systems and are not configured in the execution engine of distributed systems. Taking the Parameter Server (PS) architecture as an example, the working nodes of the distributed system can be divided into two categories: PS nodes and Worker nodes, where PS nodes store deep learning parameters (such as parameters of deep learning models), and Worker nodes are used to calculate the gradients of deep learning parameters. In each iteration of deep learning, the Worker node obtains the deep learning parameters from the PS node, and then returns the calculated gradients to the PS node. The PS node aggregates the gradients returned from the Worker node and updates the deep learning parameters. The PS node broadcasts the updated deep learning parameters to the Worker node, and continuously executes in this way, thereby realizing iterative adjustment of deep learning parameters. In the PS architecture of the distributed system, PS nodes and Worker nodes have the following characteristics during operation:

[0225] The responsibilities of PS nodes and Worker nodes are significantly different, and correspond to different stages of the execution plan. The stage corresponding to the PS node can be called the PS stage (parameter server execution stage), and the stage corresponding to the Worker node can be called the Worker stage (worker execution stage);

[0226] PS node is the serving entity of Parameter and can run independently;

[0227] When the Worker node uses and updates Parameters, it needs the PS node to be running before it can run effectively, and it needs to continuously interact with the PS node during the running process.

[0228] The above characteristics are difficult to describe in many distributed execution frameworks: although there is a scheduling dependency between PS nodes and Worker nodes, because PS nodes and Worker nodes can run at the same time, this dependency cannot be mapped to the logic of scheduling downstream nodes after the upstream node has finished running. Based on this, in many external systems, PS nodes and Worker nodes can only be scheduled and run separately in two isolated and unrelated stages in the execution plan, which may cause the Worker node to be idle before the PS node is scheduled, resulting in a waste of Worker node resources. In addition, due to the lack of description of the relationship between PS nodes and Worker nodes in different stages, many basic functions and dynamic functions cannot be implemented.

[0229] In the field of deep learning, the resources used by deep learning jobs (especially GPU resources) are generally directly open to end users for specification. For example, the end user specifies the concurrency of the working nodes of the deep learning job and the size of the resources used by each working node (such as the number of GPUs used by a working node, etc.). However, it is difficult for users to choose the right resources themselves. In order to ensure that deep learning jobs have enough resources to use, users often apply for excessive resources, resulting in resource waste. For example, in order to ensure that the GPU usage of deep learning jobs is guaranteed, users will apply for multiple GPU cards for each working node when the actual GPU resources used cannot be accurately predicted. However, during the actual execution of the deep learning job, only 25% of them may be used, which causes the remaining GPU resources to be idle and wasted. A cumulative effect of this situation is that a large number of GPU resources on the distributed system are reserved due to user applications, and even the GPU resources applied by users exceed the total GPU resources of the distributed system, resulting in a phenomenon in which the actual GPU resource utilization rate of the distributed system is low, while other jobs need to queue up to use GPU resources. On the other hand, many deep learning jobs are particularly sensitive to the use of resources (especially GPU resources). If the upper limit of resources allowed to users is blindly lowered, the execution performance of deep learning jobs may be reduced.

[0230] Based on the above situation, how to ensure the resource application accuracy of deep learning jobs and improve the resource utilization of distributed systems, so as to save the computing resources of distributed systems while ensuring that the execution performance of deep learning jobs is not affected, has become an urgent problem to be solved.

[0231] To solve the above problems, based on the fact that the vertices and connecting edges in the DAG can correspond to different logical and physical properties, the embodiments of the present application introduce the physical properties of sequential edge and concurrent edge on the physical properties of the connecting edge, and decouple the sequential edge and the parallel edge from the data transmission. The parallel edge describes that the working nodes of the upstream and downstream stages connected by the parallel edge can be in operation at the same time, but there is still a sequence in scheduling, and the timing of scheduling can be customized. For example, the working nodes of the upstream and downstream stages connected by the parallel edge can be scheduled synchronously, or the working nodes of the downstream stage can be scheduled after the working nodes of the upstream stage are running, or the working nodes of the downstream stage can be scheduled by an event after the working nodes of the upstream stage have run to a certain stage, etc. The sequential edge describes the working nodes of the downstream stage connected by the sequential edge, which need to wait until the working nodes of the upstream stage are fully or partially executed before they can be scheduled to run.

[0232] Based on parallel edges, the relationship between PS nodes and Worker nodes of deep learning jobs can be described more accurately. Fig. 8A An example diagram of a PS stage (parameter server execution stage) and a Worker stage (worker execution node) connected by parallel edges is shown. Fig. 8A As shown, the PS stage is the stage corresponding to the PS node in the execution plan, and the Worker stage is the stage corresponding to the Worker node in the execution plan. In the execution plan, the PS stage is connected to the Worker stage through parallel edges, and the parallel edges are input from the PS stage to the Worker stage. Thus, the Worker node and the Worker node can be in operation at the same time, and the scheduling timing can be customized. In some embodiments, the following Table 1 exemplarily shows the type of scheduling timing (referred to as the scheduling type) of the downstream node connected by the parallel edge, which can be used for reference, wherein the upstream node can be considered to be the working node corresponding to the upstream stage connected by the parallel edge (for example Fig. 8A The PS node corresponding to the PS stage shown in the figure), the downstream node can be considered as the working node corresponding to the downstream stage connected by the parallel edge (for example Fig. 8A Worker node corresponding to the Worker stage shown in the figure).

[0233]

[0234] Table 1

[0235] Based on the physical properties of the sequential edges and parallel edges of the connection edges, as well as the scheduling type of the downstream nodes, the embodiments of the present application can fully describe various complex DAG logics, thereby supporting various loads, such as: batch jobs, streaming jobs, near real-time / quasi-real-time jobs, deep learning jobs, etc.

[0236] For deep learning jobs, taking PS jobs as an example, since Worker nodes can use and update Parameters, and Worker nodes can only run after PS nodes are running, in the execution plan of deep learning jobs, the connection edge between PS stage and Worker stage is a parallel edge, and the parallel edge is input from PS stage to Worker stage; that is, in the execution plan, Worker stage is the direct downstream stage of PS stage and is connected by parallel edges. Since Worker can only run after PS starts running, the scheduling type of downstream Worker node is SOURCE_TASK_STARTED, that is, after the instance of upstream PS node is started, the downstream Worker node is scheduled. In terms of data generation, since the data on PS node is only valid when it is in running state, and the upstream data source does not perceive the downstream state, the data source type is EPHEMERAL_STATELESS (transient stateless); in terms of data transmission type, since the data transmission between PS node and Worker node can be not perceived by the execution framework, the data transmission type is NONE (empty). Therefore, the connection edge connecting the PS stage and the Worker stage can be described as {CONCURRENT, EPHEMERAL_STATELESS, NONE, SOURCE_TASK_PROGRESS}.

[0237] Based on the above description, in the execution process of deep learning jobs in the embodiments of the present application, the PS node and the Worker node can run simultaneously, and the Worker node must be running after the PS node is up and running before resources can be scheduled for data processing. This description of the execution plan for deep learning jobs makes dynamic adjustment during the operation of the job possible. Specifically, the execution engine of deep learning requires the end user to provide a large number of configuration parameters for the execution plan, such as the concurrency of the stage, the size and type of required resources, distribution strategy, etc., and these configuration parameters are extremely difficult for the end user to provide. Based on this, the embodiments of the present application introduce a Resource Optimization node in the execution engine of the distributed system. The resource optimization node, as a control node for the execution of a job, can coordinate and dynamically adjust resource-related requests.

[0238] In one example, taking the PS job as an example, the embodiment of the present application adds a new resource optimization node outside the PS node and the Worker node. The resource optimization node is responsible for deciding how to dynamically adjust the resources of the Worker node according to certain rules. On this basis, in addition to setting the parallel edges input from the PS stage to the Worker stage in the execution plan, it is also necessary to add the Resource Optimizationstage (resource optimization execution stage) corresponding to the Resource Optimization node in the execution plan, as well as the parallel edges input from the Resource Optimization stage to the Worker stage. The connection edge connecting the Resource Optimization stage and the Worker stage can be described as {CONCURRENT, EPHEMERAL_STATELESS, NONE, SOURCE_TASK_PROGRESS}. Different from the description of the connection edge connecting the PS stage and the Worker stage, the connection edge connecting the Resource Optimization stage and the Workerstage adopts the scheduling type of the downstream node being SOURCE_TASK_PROGRESS. In other words, the Worker node needs to be scheduled after the execution progress of the upstream Resource Optimization node reaches a certain threshold.

[0239] During the job execution process, the PS node and Resource Optimization node will be started first. When the instance of the PS node is started, it will notify the downstream Worker node for scheduling, and when the instance progress of the Resource Optimization node reaches the threshold, it will notify the downstream Worker node for scheduling. After receiving the above two notifications, the Worker node can update the scheduled resources based on the Resource Optimization node notification and start the corresponding instance (the resources change dynamically based on the control of the ResourceOptimization node). In some embodiments, the Resource Optimization node can dynamically adjust the following resources of the Worker node: node concurrency, node GPU, CPU, MEMORY and other resource usage requirements.

[0240] As an optional implementation, Figure 8B 1 shows another flow chart of the distributed job adjustment method provided in the embodiment of the present application. The method can be implemented by the master node. Figure 8B The method flow may include the following steps.

[0241] In step S810, a deep learning job submitted by a user is obtained.

[0242] In step S811, an execution plan of the deep learning job is generated, and the execution plan includes multiple stages, including: a Worker stage and a Resource Optimization stage.

[0243] In some embodiments, the Resource Optimization stage inputs the Worker stage through a parallel edge.

[0244] In some further embodiments, the multiple stages may further include: a PS stage, wherein the PS stage and the Resource Optimization stage are respectively input into the Worker stage through parallel edges.

[0245] In an embodiment of the present application, a deep learning job (such as a PS job) can be described as a structure of PS, Worker, and Resource Optimization. That is to say, when generating an execution plan for a deep learning job (the execution plan can be described by DAG), the execution plan may include a PS stage, a Worker stage, and a Resource Optimization stage. And in the DAG graph, the PS stage is input into the Worker stage by parallel edges, and the Resource Optimization stage is also input into the Worker stage by parallel edges. In some further embodiments, the parallel edge descriptions of the PS stage and the Worker stage, as well as the parallel edge descriptions of the Resource Optimization stage and the Worker stage, can refer to the corresponding descriptions in the previous text and will not be expanded here.

[0246] In step S812, during the execution of the deep learning job, the Resource Optimization node corresponding to the Resource Optimization stage is scheduled, and the resource information currently applicable to the Worker stage is determined through the Resource Optimization node.

[0247] In an embodiment of the present application, the Resource Optimization node can be scheduled before the Worker node. For example, when the instance progress of the Resource Optimization node reaches a threshold, the Worker node is notified to schedule. In some embodiments, after the Resource Optimization node is scheduled, the Resource Optimization node can determine the resource information currently adapted to the Worker stage. In some embodiments, the Resource Optimization node can determine the historically used resource information that matches the current execution state of the deep learning job, and use the historically used resource information that matches the current execution state as the resource information currently adapted to the Worker stage.

[0248] As an optional implementation, the historically used resource information that matches the current execution state of the deep learning job may include: historically used resource information of historical execution states that are the same as or similar to the current execution state.

[0249] In some embodiments, the Resource Optimization node can determine the resource information suitable for the Worker stage from the historical database based on the current execution state of the deep learning job. The historical database can record the resource information actually used by the historical deep learning job in each historical execution state, that is, the historical database can record the resource information actually used by the deep learning job at the end of the historical execution in each execution state. Thus, the Resource Optimization node can determine the resource information suitable for the current execution state of the deep learning job based on the records in the historical database to configure resources for the Worker node.

[0250] In some embodiments, the Resource Optimization node can search for a historical execution state similar to or identical to the current execution state from a historical database based on the current execution state of the deep learning job, and determine the resource information actually used by the searched historical execution state as the resource information currently adapted to the Worker stage. As an optional implementation, the Resource Optimization node can first search for a historical execution state identical to the current execution state from a historical database, and if found, use the resource information corresponding to the same historical execution state recorded in the historical database as the resource information currently adapted to the Worker stage; if not found, search for a historical execution state similar to the current execution state (e.g., search for a historical execution state with the smallest difference from the current execution state), and use the resource information corresponding to the similar historical execution state recorded in the historical database as the resource information currently adapted to the Worker stage. In some embodiments, the current execution state of the deep learning job can be, for example, the current learning mode of the deep learning job, the characteristics of the current input data (e.g., the amount of data in the current training data), the number of parameter iterations currently remaining, etc.

[0251] In step S813, the resource information is configured for the Worker stage through the Resource Optimization node.

[0252] After determining the resource information currently suitable for the Worker stage (for example, the historically used resource information that matches the current execution state) through the Resource Optimization node, the master node can configure the resource information for the Worker stage in the execution plan through the ResourceOptimization node, so that the resource information configured by the Worker stage is adapted to the current execution state of the deep learning job. Furthermore, when scheduling the Worker node, the Worker node can schedule the resources used based on the resource information, so that the Worker node executes the task with reasonable resources, thereby improving the resource utilization of the Worker node.

[0253] In some embodiments, the Worker node needs to be scheduled after the instance of the PS node is started and when the instance progress of the ResourceOptimization node reaches a threshold.

[0254] In some embodiments, if the user specifies the resource information of the Worker stage in advance, the ResourceOptimization node can adjust the resource information specified by the user based on the determined resource information currently adapted to the Worker stage, to avoid the situation where the resources are idle due to the user over-specifying the resources of the Worker stage. For example, the user specifies that the Worker node uses 2 CPU cores and 1 GPU core resource units. If the Resource Optimization node determines that the Worker node does not actually need to use so many resources based on the current execution status of the deep learning job, the Resource Optimization node can adjust the number of CPU cores and GPU cores specified by the user in advance, for example, to adjust the Worker node to use 1 CPU core and half a GPU core, to avoid idle waste of CPU and GPU resources due to the user specifying too many CPU cores and GPU cores for the Worker node.

[0255] As some implementation examples, Figure 8C The following figure shows an example of how the Resource Optimization node adjusts the resources of the Worker node. Figure 8C As shown in the figure, the resources planned to be configured for the Worker node are 200 CPU cores and 100 GPU cores (specified by the user). During the execution of the deep learning job, the Resource Optimization node can search the historical database for resource information actually used by similar or identical historical execution states based on the current execution state of the job. For example, if the historical execution state similar or identical to the current execution state of the job actually uses 100 CPU cores and 50 GPU cores, the Resource Optimization node can adjust the number of CPU cores configured for the Worker node to 100 and the number of GPU cores to 50, so that the Worker node can ensure the specific execution of the deep learning job by configuring lower resources.

[0256] In some further embodiments, the Resource Optimization node may pre-set multiple resource requirement schemes for the Worker node, and the Resource Optimization node may select a resource requirement scheme that matches the current execution state from multiple resource requirement schemes according to the current execution state of the deep learning job, and configure resource information for the Worker stage with the selected resource requirement scheme. The resource requirement scheme that matches the current execution state can be considered to be a scheme that can meet the job execution of the current execution state and has the least resource requirements.

[0257] The embodiment of the present application can configure more accurate resources for the Worker node based on the current execution state of the deep learning job, and can save distributed system computing resources and improve the resource utilization of the distributed system under the premise of ensuring that the execution performance of the Worker node in executing the deep learning job is not affected. After the large-scale production cluster is put into use, the embodiment of the present application can greatly improve the utilization rate of resources (especially GPU resources) of deep learning jobs without affecting the actual training time of users for deep learning, and at the same time bring about a significant increase in the throughput rate of the job, and the situation of user job queuing can be greatly alleviated.

[0258] The embodiments of the present application can describe and expand the execution plans of Map-Reduce, offline job operation mode, etc., so that the execution plan can accurately describe deep learning jobs including PS. This way of describing the execution plan can avoid the idle waiting of resources of the working node, and can provide a more accurate job execution control process. In addition, the embodiments of the present application can dynamically select and adjust the resources such as GPU actually needed for deep learning jobs during job operation, ensure that the use of resources can be adapted according to the actual operation requirements and algorithm characteristics of the job, and ensure efficient resource use, thereby avoiding the contradiction between excessive resource application and low actual resource utilization in large-scale multi-user distributed systems.

[0259] The embodiment of the present application also provides a master node, which can be configured to execute the distributed job adjustment method provided by the embodiment of the present application.

[0260] An embodiment of the present application also provides a distributed system, the structure of which can be combined with the description of the corresponding part of the foregoing text, and the distributed system can include the master node described above.

[0261] The present application embodiment also provides a physical machine, which can be used to set the master node provided by the present application embodiment. As an optional implementation, Fig. 9 Figure 2 shows the structure diagram of the physical machine. Fig. 9As shown, the physical machine may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4. In an embodiment of the present application, the number of processor 1, communication interface 2, memory 3 and communication bus 4 is at least one, and the processor 1, communication interface 2 and memory 3 complete mutual communication through the communication bus 4. Optionally, the communication interface 2 may be an interface of a communication module for network communication. Optionally, the processor 1 may be a CPU (central processing unit), a GPU (Graphics Processing Unit), an NPU (embedded neural network processor), an FPGA (Field Programmable Gate Array), a TPU (tensor processing unit), an AI chip, a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present application. The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Among them, the memory 3 stores one or more computer executable instructions, and the processor 1 calls the one or more computer executable instructions to execute the distributed job adjustment method provided in the embodiment of the present application.

[0262] An embodiment of the present application also provides a storage medium, which can store one or more computer-executable instructions. When the one or more computer-executable instructions are executed, the distributed job adjustment method provided in the embodiment of the present application is implemented.

[0263] The embodiment of the present application also provides a computer program, which is used to execute the distributed job adjustment method provided in the embodiment of the present application.

[0264] The above describes multiple implementation schemes provided by the embodiments of the present application. The various optional methods introduced in each implementation scheme can be combined and cross-referenced with each other without conflict, thereby extending a variety of possible implementation schemes, which can all be considered as implementation schemes disclosed and open in the embodiments of the present application.

[0265] Although the embodiments of the present application are disclosed above, the present application is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined by the claims.

Claims

1. A distributed job adjustment method, wherein: include: Get deep learning homework; Generate an execution plan for the deep learning job, the execution plan comprising multiple execution stages, the multiple execution stages comprising: a worker execution stage and a resource optimization execution stage; the worker execution stage is used to calculate the gradient of the deep learning parameters; During the execution of the deep learning job, scheduling the resource optimization node corresponding to the resource optimization execution phase, and determining the historically used resource information matching the current execution state of the deep learning job through the resource optimization node; The resource information is configured for the worker execution phase through a resource optimization node.

2. The method according to claim 1, wherein: The execution plan is described by DAG; The multiple execution stages also include: a parameter server execution stage; the parameter server execution stage inputs the worker execution stage through parallel edges, the resource optimization execution stage inputs the worker execution stage through parallel edges, and the worker nodes of the upstream and downstream execution stages connected by the parallel edges can be in operation at the same time.

3. The method according to claim 2, wherein: The downstream worker node is scheduled after the instance of the upstream parameter server node is started and the execution progress of the upstream resource optimization node reaches a certain threshold; the worker node is the worker node corresponding to the worker execution phase, the parameter server node is the worker node corresponding to the parameter server execution phase, and the resource optimization node is the worker node corresponding to the resource optimization execution phase.

4. The method according to claim 1, wherein: The determining, by the resource optimization node, the historically used resource information matching the current execution state of the deep learning job includes: Based on the current execution state of the deep learning job, a historical execution state that is similar or identical to the current execution state is searched from the historical database, and the resource information actually used by the historical execution state is determined as the resource information currently adapted to the worker execution stage; wherein the historical database records the resource information actually used by the historical deep learning job in each historical execution state.

5. The method according to claim 3, wherein: Also includes: When the worker node is scheduled, the worker node schedules the resources to be used based on the resource information.

6. The method according to claim 1, wherein: Also includes: If the user specifies resource information for the worker execution phase in advance, the resource optimization node adjusts the resource information specified by the user based on the determined resource information.

7. A master node, wherein: The master node is configured to execute the distributed job adjustment method as described in any one of claims 1-6.

8. A distributed system, wherein: The distributed system comprises a master node and a plurality of working nodes, wherein the master node is the master node as claimed in claim 7.

9. A physical machine, wherein: The physical machine includes at least one memory and at least one processor, the memory stores one or more computer executable instructions, and the processor calls the one or more computer executable instructions to execute the distributed job adjustment method according to any one of claims 1-6.

10. A storage medium, wherein: The storage medium stores one or more computer executable instructions, and when the one or more computer executable instructions are executed, the distributed job adjustment method according to any one of claims 1 to 6 is implemented.

Images