Parallel decision system for distributed data processing and method thereof

Abstract

The application discloses a parallel decision system and method for distributed data processing. The system comprises: an initial logical node generation component, which generates an initial logical node topology; a direct neighborhood marking component, which traverses the initial logical node topology to obtain a direct neighborhood topology of a selected initial logical node and marks the obtained direct neighborhood topology one by one; a sub-neighborhood cost calculation component, which traverses each sub-neighborhood topology in the selected marked direct neighborhood topology, and calculates the cost sum of the sub-neighborhood topology under each random combination of the sub-neighborhood topology and the node cost sum of each initial logical node; a sub-neighborhood parallel decision component, which calculates the cost sum difference between the node cost sum of each initial logical node under all random combinations, and determines the sub-neighborhood parallel scheme of the sub-neighborhood topology; and a parallel decision termination component, which terminates the repeated traversal process under the condition that the parallel scheme of each direct neighborhood obtained by two consecutive traversals and the node cost sum assigned to each initial logical node remain unchanged.

Description

Technical Field

[0001] This disclosure relates to a data processing technique. More specifically, this disclosure relates to a parallel decision-making system and method for distributed data processing. Background Technology

[0002] With the widespread adoption of deep learning, the increasing number of models and the ever-growing scale of data have made deep learning training impossible on a single computing device. This has led to the development of distributed computing. As distributed computing becomes more prevalent, large jobs or tensors are partitioned, with different parts of the data deployed across various computing devices in different distributed data processing systems. Intermediate parameters need to be exchanged during the computational processes of each part. Thus, during the processing of a specific job, intermediate parameters or results deployed on one computing device become input data for a task on another, incurring data transfer overhead between computing devices. When the job data is very large, this data transfer overhead between different computing devices places a significant computational burden on the distributed data processing system. Therefore, in distributed data processing systems that support data parallelism, model parallelism, hybrid parallelism, and streaming parallelism, reducing this data transfer overhead between different computing devices is a crucial challenge.

[0003] Therefore, people have attempted to find a data processing method that, while satisfying the computational power limitations of distributed computing resources, maximizes parallel data processing capabilities. Clearly, in distributed data processing systems, even after deciding on the number of computing devices for parallel computation, the parallel processing method used is not unique. Typically, each logical node can employ different parallel processing methods for the same computational task. Different parallel processing methods result in different execution times on each logical node. Consequently, the amount of data transmitted between consecutive logical nodes varies due to the different parallel processing methods used, leading to different transmission times. This, in turn, impacts the overall data processing time, resulting in varying computational efficiencies. Obviously, manually adjusting the parallel decisions of thousands or millions of data processing nodes in deep learning inevitably leads to significant human waste. Especially since those capable of such manual adjustments are highly skilled professionals, wasting their time on tedious manual adjustments is a waste of talent. In particular, different tasks require different parallel decisions. Therefore, even when using the same set of distributed data processing resources, different parallel decisions are needed to maximize data processing efficiency, which leads to the need for readjustment for each task. Manual adjustment of parallel decisions firstly increases the workload of technical personnel and consumes their manpower. On the other hand, when applying and implementing different parallel modes on different logical nodes, one also needs to consider memory limitations and runtime consumption, which may and may not bring the ideal parallel effect (compressing the total runtime to the extreme or near the extreme).

[0004] Therefore, a method and system are needed that can automatically determine the parallelism of each logical node based on the amount of distributed data processing resources. Summary of the Invention

[0005] One object of the present invention is to solve at least the above-mentioned problems. Specifically, the present disclosure provides a parallel decision-making system for distributed data processing, comprising: an initial logical node generation component, which receives task configuration data input by a user and generates an initial logical node topology graph for the distributed data processing system, wherein each initial logical node is associated with a set of candidate parallel schemes based on the task configuration data, each candidate parallel scheme specifies the parallel scheme of its associated initial logical node and a candidate computational cost label based on the parallel scheme, and each connection edge between two interconnected initial logical nodes is associated with a label of candidate transmission cost, the candidate transmission cost being determined by the connection edge between the nodes. The parallel schemes for each of the initial logical nodes are determined; a direct neighborhood marking component traverses the initial logical node topology graph to obtain the direct neighborhood topology graph of a selected initial logical node and marks each obtained direct neighborhood topology graph, which includes the selected initial logical node and other initial logical nodes with direct connection edges; a sub-neighborhood cost calculation component, for each marked direct neighborhood topology graph, traverses each sub-neighborhood topology graph in the selected direct neighborhood topology graph and calculates the candidate parallel schemes corresponding to each initial logical node in the traversed sub-neighborhood topology graph. The computational cost and random combinations of candidate transmission costs for each connected edge are used to calculate the sum of costs for the selected sub-neighborhood topology under each random combination, and the sum of node costs for each initial logical node in the selected sub-neighborhood topology under each random combination. Each selected sub-neighborhood topology includes the selected initial logical node in its direct neighbor topology and a predetermined number of directly adjacent initial logical nodes. A sub-neighborhood parallel decision component calculates the cost difference between the sums of node costs for each initial logical node in the selected sub-neighborhood topology under all random combinations. Based on the minimum combined cost of the initial logical nodes with the maximum cost and difference in the selected sub-neighborhood topology, a parallel scheme for the selected sub-neighborhood topology is determined, and the node cost of each initial logical node under the determined parallel scheme is assigned to the corresponding initial logical node. A parallel decision termination component, after multiple traversals of the initial logical node topology, terminates the repeated traversal process if there are two consecutive traversals where the parallel schemes for each direct neighbor and the node cost assigned to each initial logical node remain unchanged, thereby obtaining the resulting logical node topology and thus the parallel scheme of the distributed data processing system.

[0006] According to the parallel decision system for distributed data processing disclosed herein, it further includes: a proxy node insertion component, which traverses the initial logical node topology graph before the direct neighborhood labeling component performs labeling, and when a specific initial logical node has two or more downstream initial logical nodes, inserts a temporary intermediate proxy node between the specific initial logical node and all its downstream initial logical nodes, and assigns the number of the temporary intermediate proxy nodes not exceeding the number of combinations of parallel strategy types of all downstream initial logical nodes as candidate parallel strategies, thereby obtaining an initial logical node topology graph containing the temporary intermediate proxy nodes.

[0007] According to the parallel decision system for distributed data processing disclosed herein, the predetermined number of initial logical nodes directly adjacent to the selected initial logical node contained in each sub-neighborhood topology graph traversed by the sub-neighborhood cost calculation component each time does not exceed 5.

[0008] According to the parallel decision system for distributed data processing disclosed herein, the sub-neighborhood cost calculation component includes: a predetermined configuration acquisition component, which traverses the initial logical node topology graph to acquire predetermined configurations in the initial logical node topology graph, the predetermined configurations including a first predetermined configuration and / or... Or a second predetermined configuration, wherein the first predetermined configuration is an intermediate initial logical node with a first connecting edge and a second connecting edge, and the second predetermined configuration is a pair of initial logical nodes with multiple third connecting edges between them; and a subgraph cost calculation component, for the first predetermined configuration, when the candidate parallel schemes of the first initial logical node of the first connecting edge and the candidate parallel schemes of the second initial logical node of the second connecting edge are determined, obtains each candidate computation cost of the intermediate initial logical node, the candidate transmission cost of the first connecting edge corresponding to the obtained candidate computation cost, and the candidate transmission cost of the second connecting edge, and obtains the first cost sum of the three for each candidate computation cost of each intermediate initial logical node, and selects the minimum first cost sum as the first candidate transmission cost between the first initial logical node and the second initial logical node in the first predetermined configuration when the candidate parallel schemes of each pair of first initial logical nodes and the candidate parallel schemes of the second initial logical node are determined; and for the second predetermined configuration, in the case of the pair of initial logical nodes, the first cost sum of the three is obtained, and ... two Given the candidate parallel schemes for three initial logical nodes and the candidate parallel schemes for a fourth initial logical node in pairs of initial logical nodes, the candidate transmission costs of all connection edges between pairs of initial logical nodes are summed to obtain a second cost sum of the candidate transmission costs between pairs of initial logical nodes, which is used as the second candidate transmission cost. A predetermined configuration transformation component is used to transform the first connection edge, the second connection edge, and the intermediate initial logical node of the first predetermined configuration into a first merged connection edge between the first initial logical node of the first connection edge and the second initial logical node of the second connection edge, and assign all the first candidate transmission costs calculated for the first predetermined configuration to the first merged connection edge as one of the candidate transmission costs of the first merged connection edge, and to transform all the connection edges of the second predetermined configuration into a second merged connection edge in pairs of initial logical nodes, and assign the second candidate transmission costs calculated for the second predetermined configuration to the second merged connection edge in pairs of initial logical nodes as one of the candidate transmission costs of the second merged connection edge.

[0009] According to the parallel decision system for distributed data processing disclosed herein, the predetermined configuration further includes a third predetermined configuration, which is an end initial logical node having only a fourth connection edge. The subgraph cost calculation component, for the third predetermined configuration, when the end initial logical node is determined by a candidate parallel scheme dependent on the initial logical node through the fourth connection edge of the third predetermined configuration, obtains each candidate computation cost of the end initial logical node and the candidate transmission cost of the fourth connection edge corresponding to the candidate computation cost of the end initial logical node, and obtains a third cost sum of the two when the candidate parallel scheme dependent on the initial logical node is determined. The minimum third cost sum is selected as the third additional computation cost when the candidate parallel scheme dependent on the initial logical node is determined. The predetermined configuration transformation component prunes the fourth connection edge and the end initial logical node of the third predetermined configuration and adds the third additional computation cost to the computation cost dependent on the initial logical node.

[0010] According to the parallel decision system for distributed data processing disclosed herein, the predetermined configuration further includes a fourth predetermined configuration, the fourth predetermined configuration including a fifth initial logical node and a sixth initial logical node whose product of the number of candidate parallel decisions located in the same connected component and having no connection edges between them does not exceed a given threshold, and at least one seventh initial logical node connected to the fifth initial logical node and the sixth initial logical node, wherein the subgraph cost calculation component, for the fourth predetermined configuration, obtains the candidate computation costs of the fifth initial logical node and the sixth initial logical node when the candidate parallel schemes of the fifth initial logical node and the sixth initial logical node are determined, and uses the sum of the two candidate computation costs as the fourth cost sum, and in the case of the fifth initial logical node and the sixth initial logical node... Given the candidate parallel schemes for the initial logical node and the seventh initial logical node, the candidate transmission costs of the fifth and sixth connection edges between the fifth and sixth initial logical nodes and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost. The predetermined configuration transformation component merges the fifth and sixth initial logical nodes of the fourth predetermined configuration into a first merged logical node, and merges the fifth and sixth connection edges into a third merged connection edge. The fourth cost calculated for the fourth predetermined configuration and assigned to the first merged logical node are taken as one of its candidate calculation costs, and the third candidate transmission cost is assigned to the third merged connection edge as one of its candidate transmission costs.

[0011] According to the parallel decision-making system for distributed data processing disclosed herein, the predetermined configuration further includes a fifth predetermined configuration. The fifth predetermined configuration includes a fifth initial logical node and a sixth initial logical node, where the product of the number of candidate parallel decisions located within the same connected component and connected to each other by a seventh connection edge does not exceed a given threshold, and at least one seventh initial logical node connected to the fifth and sixth initial logical nodes. The subgraph cost calculation component, for the fifth predetermined configuration, when candidate parallel schemes for the fifth and sixth initial logical nodes are determined, obtains the candidate computation costs of the fifth and sixth initial logical nodes and the transmission costs of the connection edges between them, and uses the sum of the two candidate computation costs and the corresponding transmission costs of the seventh connection edges as the fifth cost. And, given that the candidate parallel schemes for the fifth initial logical node, the sixth initial logical node, and the seventh initial logical node are determined, the candidate transmission costs of the fifth connection edge and the sixth connection edge between the fifth initial logical node and the sixth initial logical node and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost; and the predetermined configuration transformation component merges the fifth initial logical node and the sixth initial logical node of the fifth predetermined configuration into a second merged logical node, and merges the fifth connection edge and the sixth connection edge into a third merged connection edge, and takes the fifth cost calculated for the fifth predetermined configuration and the second merged logical node as one of its candidate calculation costs, and assigns the third candidate transmission cost to the third merged connection edge as one of its candidate transmission costs.

[0012] According to another aspect of this disclosure, a parallel decision-making method for distributed data processing is provided, comprising: an initial logical node generation step, receiving task configuration data input by a user, generating an initial logical node topology graph for the distributed data processing system, wherein each initial logical node is associated with a set of candidate parallel schemes based on the task configuration data, each candidate parallel scheme specifying the parallel scheme of its associated initial logical node and a candidate computational cost label based on the parallel scheme, and each connection edge between two interconnected initial logical nodes is associated with a label of candidate transmission cost, the candidate transmission cost being determined by each interconnected initial logical node. The parallel scheme is determined; the direct neighborhood marking step involves traversing the initial logical node topology graph to obtain the direct neighborhood topology graph of a selected initial logical node and marking each obtained direct neighborhood topology graph. The direct neighborhood topology graph includes the selected initial logical node and other initial logical nodes with direct connection edges to it; the sub-neighborhood cost calculation step involves, for each marked direct neighborhood topology graph, traversing each sub-neighborhood topology graph of the selected marked direct neighborhood topology graph, and calculating the candidate computation cost based on the candidate parallel schemes corresponding to each initial logical node in the traversed and selected sub-neighborhood topology graphs, as well as the candidate computation costs corresponding to each initial logical node in the traversed and selected sub-neighborhood topology graphs. The process involves a random combination of candidate transmission costs for each connecting edge, calculating the sum of costs for each random combination of the selected sub-neighborhood topology, and the sum of node costs for each initial logical node in the selected sub-neighborhood topology under each random combination. Each selected sub-neighborhood topology includes the selected initial logical node in its direct neighbor topology and a predetermined number of directly adjacent initial logical nodes. The sub-neighborhood parallel decision-making step calculates the difference between the sums of node costs for each initial logical node in the selected sub-neighborhood topology under all random combinations, and then... The minimum combined cost of the initial logical nodes with the maximum cost and difference in the selected sub-neighborhood topology is traversed to determine the sub-neighborhood parallel scheme of the selected sub-neighborhood topology, and the node cost of each initial logical node under the determined sub-neighborhood parallel scheme is assigned to the corresponding initial logical node; and the parallel decision termination step, after multiple traversals of the initial logical node topology, if there are two consecutive traversals of the parallel scheme of each direct neighbor and the node cost assigned to each initial logical node remain unchanged, the repeated traversal process is terminated, thereby obtaining the result logical node topology, and thus obtaining the parallel scheme of the distributed data processing system.

[0013] According to the parallel decision-making method for distributed data processing disclosed herein, the method further includes: a proxy node insertion step, wherein before the direct neighborhood labeling component performs labeling, the initial logical node topology graph is traversed, and when a specific initial logical node has two or more downstream initial logical nodes, a temporary intermediate proxy node is inserted between the specific initial logical node and all its downstream initial logical nodes, and a candidate parallel strategy is assigned to the temporary intermediate proxy node, the number of which does not exceed the number of combinations of parallel strategy types of all downstream initial logical nodes, thereby obtaining an initial logical node topology graph containing the temporary intermediate proxy node.

[0014] According to the parallel decision-making method for distributed data processing disclosed herein, the predetermined number of initial logical nodes directly adjacent to the selected initial logical node included in each sub-neighborhood topology graph traversed in each step of the sub-neighborhood cost calculation step does not exceed 5.

[0015] According to the parallel decision-making method for distributed data processing disclosed herein, the method of obtaining a predetermined configuration in the topology graph of the initial logical node further includes obtaining a third predetermined configuration, wherein the third predetermined configuration is an end initial logical node having only a fourth connection edge. The predetermined configuration cost calculation step, for the third predetermined configuration, in the case where the candidate parallel schemes dependent on the initial logical node are determined by the fourth connection edge of the third predetermined configuration, obtains each candidate computation cost of the end initial logical node and the candidate transmission cost of the fourth connection edge corresponding to the candidate computation cost of the end initial logical node, and obtains the third cost sum of the two when the candidate parallel schemes dependent on the initial logical node are determined, and selects the minimum third cost sum as the third additional computation cost when the candidate parallel schemes dependent on the initial logical node are determined. The predetermined configuration transformation step removes the fourth connection edge and the end initial logical node of the third predetermined configuration and adds the third additional computation cost to the computation cost dependent on the initial logical node.

[0016] According to the parallel decision-making method for distributed data processing disclosed herein, obtaining a predetermined configuration in the initial logical node topology graph further includes obtaining a fourth predetermined configuration. The fourth predetermined configuration includes a fifth and a sixth initial logical node located within the same connected component and without any connecting edges between them, and at least one seventh initial logical node connected to the fifth and sixth initial logical nodes. The predetermined configuration cost calculation step, for the fourth predetermined configuration, involves obtaining candidate computational costs for the fifth and sixth initial logical nodes when candidate parallel schemes for the fifth and sixth initial logical nodes are determined, and summing the two candidate computational costs as the fourth cost sum. Additionally, for the fifth and sixth initial logical nodes... Given the candidate parallel schemes for the initial logical node and the seventh initial logical node, the candidate transmission costs of the fifth and sixth connection edges between the fifth and sixth initial logical nodes and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost. The predetermined configuration transformation step merges the fifth and sixth initial logical nodes of the fourth predetermined configuration into a first merged logical node, and merges the fifth and sixth connection edges into a third merged connection edge. The fourth cost calculated for the fourth predetermined configuration and assigned to the first merged logical node are taken as one of its candidate calculation costs, and the third candidate transmission cost is assigned to the third merged connection edge as one of its candidate transmission costs.

[0017] According to the parallel decision-making method for distributed data processing disclosed herein, obtaining a predetermined configuration in the initial logical node topology graph further includes obtaining a fifth predetermined configuration. The fifth predetermined configuration includes a fifth initial logical node and a sixth initial logical node located within the same connected component and connected by a seventh connecting edge, and at least one seventh initial logical node connected to the fifth and sixth initial logical nodes. The predetermined configuration cost calculation step, for the fifth predetermined configuration, involves obtaining the candidate computation costs of the fifth and sixth initial logical nodes and the transmission costs of the connecting edges between them, given that candidate parallel schemes for the fifth and sixth initial logical nodes are determined. The sum of the two candidate computation costs and the corresponding transmission costs of the seventh connecting edges is taken as the fifth cost sum. Given the candidate parallel schemes for the fifth, sixth, and seventh initial logical nodes, the candidate transmission costs of the fifth and sixth connection edges between the fifth and sixth initial logical nodes and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost. The predetermined configuration transformation step merges the fifth initial logical node, the sixth initial logical node, and the seventh connection edge of the fifth predetermined configuration into a second merged logical node, and merges the fifth and sixth connection edges into a third merged connection edge. The fifth cost calculated for the fifth predetermined configuration and assigned to the second merged logical node are taken as one of its candidate calculation costs, and the third candidate transmission cost is assigned to the third merged connection edge as one of its candidate transmission costs.

[0018] By utilizing the parallel decision-making system and method for distributed data processing disclosed herein, the solution space for parallel decision-making in distributed data processing can be minimized from a global perspective. This improves the feasibility and reduces the difficulty of automatic parallel decision-making. Furthermore, the parallel results obtained from parallel decision-making have lower computational and transmission costs, thereby maximizing the computational efficiency of fixed computing resources for the same computational task and accelerating data processing. More importantly, it automates parallel decision-making while minimizing transmission costs, significantly reducing the cost of manual debugging.

[0019] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0020] Figure 1 The diagram shown is a schematic representation of an example of a parallel decision system for distributed data processing according to this disclosure.

[0021] Figure 2The diagram shown is an example of a direct neighborhood topology obtained by traversing the parallel decision system of distributed data processing according to this disclosure.

[0022] Figure 3 The diagram illustrates the insertion process performed by the agent node insertion component in a parallel decision system for distributed data processing according to this disclosure.

[0023] Figure 4 The diagram shown is a schematic representation of another example of a parallel decision system for distributed data processing according to this disclosure.

[0024] Figure 5 The diagram shown is a schematic diagram of a first predetermined configuration according to the present disclosure.

[0025] Figure 6 The diagram shown is a schematic diagram of a second predetermined configuration according to the present disclosure.

[0026] Figure 7 The diagram shown is a schematic diagram of a third predetermined configuration according to the present disclosure.

[0027] Figure 8 The diagram shown is a schematic diagram of the fourth predetermined configuration according to the present disclosure.

[0028] Figure 9 The diagram shown is a schematic diagram of the fifth predetermined configuration according to this disclosure. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings, so that those skilled in the art can implement it based on the description.

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, in the following text, one of two possible objects may be referred to as either a fifth initial logical node or a sixth initial logical node; similarly, the other of two possible devices may be referred to as either a second logical distributed signature or a first logical distributed signature. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] To enable those skilled in the art to better understand this disclosure, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 The diagram shown is a schematic representation of a parallel decision-making system 100 for distributed data processing according to this disclosure. Figure 1 As shown, the distributed signature decision system 100 includes an initial logical node generation component 110, a direct neighborhood marking component 120, a sub-neighborhood cost calculation component 121, a sub-neighborhood parallel decision component 122, and a parallel decision termination component 123.

[0035] The initial logical node generation component 110 receives task configuration data input by the user and generates an initial logical node topology graph for the distributed data processing system. Each initial logical node is associated with a set of candidate parallel schemes based on the task configuration data. Each candidate parallel scheme specifies its associated initial logical node parallel scheme and a candidate computational cost label based on that scheme. Each connection edge between two connected initial logical nodes is also associated with a candidate transmission cost label, determined by the respective parallel schemes of the connected initial logical nodes. The initial logical node generation component 110 receives task configuration data and computational resource data input by the user and generates an initial logical node topology graph 111 for distributed data processing. After job input, the distributed data processing system automatically decomposes the job into numerous smaller job tasks based on the user-input job description. These numerous smaller job tasks consist of various operation components, which are interconnected as logical nodes to form a preliminary tensor processing neural network topology graph. Each layer of these neural networks contains numerous logical nodes, and adjacent layers are interconnected, thus providing guidance for the placement of execution entities (PLACEMENTs) in the distributed data processing system to perform the actual job processing. Figure 1The diagram only schematically illustrates a simple initial logical node topology 111, showing initial logical nodes A, B, C, D, E, F, L, and K. Other nodes not shown are omitted. In actual distributed data processing, the initial logical node topology 111 will be more complex. The initial logical node topology 111 contains the basic computational nodes that implement the computational task described by the user. The initial logical node topology 111 is an undirected graph transformed from a directed acyclic graph. This method of generating the initial logical node topology 111 is a conventional technique in this field and will not be elaborated upon here.

[0036] Each initial logical node in the initial logical node topology diagram 111 contains several candidate parallel strategies. These candidate parallel strategies can be named according to the user's needs. Once the parallel strategy of an initial logical node is selected from its multiple candidate parallel strategies, the parallel mode of the initial logical node is also determined. Candidate parallel strategies can be comprehensively classified into various parallelisms based on the parallel objects and the parallelism location, such as parallelism on multiple devices, parallelism on the same device, and parallelism on multiple workstations. For this purpose, all parallel strategies can be uniformly numbered as parallel strategies P1, P2, P3, P4, P5, P6... In the formed initial logical node topology diagram 111, each initial logical node has multiple candidate parallel strategy P signatures. As a source logical node that has been configured with a P signature by the user or an initial logical node whose unique P signature is determined based on the user's task description, such as initial logical nodes A, E, and C, it only has a unique P signature, such as P-5 for initial logical node A, P-2 for initial logical node C, and P-3 for initial logical node E. In some cases, although some initial logical nodes may have multiple candidate P-signatures, due to specific task requirements, users may specify that one or more initial logical nodes can only use one P-signature. When a unique P-signature is not determined, initial logical nodes typically contain some inherent candidate P-signatures. For example... Figure 1 The initial logical node B in the algorithm has multiple candidate P signatures, for example, three: P-1, P-2, and P-3. Other initial logical nodes also have different candidate P signatures, which are not listed here. Different initial logical nodes will have different fixed candidate P signatures depending on the specific operation they perform.

[0037] To this end, after receiving task configuration data and computing resource data input by the user, the initial logical node generation component 110 generates an initial logical node topology 111 for the distributed data processing system. Each initial logical node in the initial logical node topology 111 is associated with a set of candidate parallel schemes based on the task configuration data. For example, initial logical node B has multiple candidate P signatures, such as three, including P-1, P-2, and P-3. Each candidate parallel scheme specifies the parallel scheme of the initial logical node to which it belongs. Each initial logical node is also associated with a candidate computational cost label C based on the parallel scheme, based on the task configuration data. The computational cost of each initial logical node is based on its data processing type, data block size, and its own parallelism method; its computational cost C is an empirical value. For example, if the parallel decision number for node B is P2, then its computational cost is C2. Similarly, if the parallel decision number for node B is P... k Then its computational cost is C. k The empirical method for obtaining computational cost is an existing technique. For each logical node, the computation time of that logical node under different alternative parallel decision-making scenarios is measured, which is the computational load divided by the computational speed. The running time required per unit of computation can be estimated through numerous experiments. This computation time is called the computational cost of the logical node and is stored in an array to ensure that the computational cost of that logical node under a specific alternative parallel decision-making scenario can be found in O(1) (constant) time. Therefore, it will not be described in detail here.

[0038] Furthermore, each connection edge between any two initially connected logical nodes is labeled with a candidate transmission cost Ct, which is determined by the candidate parallel schemes of the connected initial logical nodes. For example, when the parallel decision numbers of initial logical nodes E and B are P2 and P3, the transmission cost label of the connection edge between them is Ct. The transmission cost label can also be visually identified by including candidate decision numbers, for example, through a two-dimensional array C. t[P2][P3] This expresses the transmission cost of the connecting edge between initial logical nodes E and B when their parallel decision numbers are P2 and P3. Since the computation graph, which is the topology of the logical nodes, is a directed graph, after being converted to an undirected graph, each connecting edge represents data transmission between one node and the other node on that edge. The transmission cost measures the time required for this edge to make different data distribution decisions at its two ends. Specifically, this can be achieved through extensive experiments to measure bandwidth and estimate the time required to transmit a unit of computation between the two nodes. Finally, the transmission cost of this edge under different decisions at its two ends is stored in a two-dimensional array of this edge (e.g., C). t[Pi][Pj] This ensures a constant access time.

[0039] The P-signature disclosed herein is a signature applied in a distributed data processing system. In distributed data processing systems, due to the frequent occurrence of data parallelism, model parallelism, hybrid parallelism, and streaming parallelism, tasks from adjacent logical nodes are often deployed simultaneously to different computing devices. Therefore, during actual data processing, intermediate parameters are exchanged between these computing devices, leading to significant transmission overhead. Since different initial logical nodes choosing different parallel strategies result in varying transmission overhead or costs, and with numerous initial logical nodes, the number of such different parallel strategy combinations becomes astronomical. Therefore, manually selecting a parallel decision combination with lower costs is extremely difficult, and automatic selection, due to the vast number of possible combinations, would instead incur enormous computational overhead.

[0040] To this end, the direct neighbor labeling component 120 of the parallel decision system 100 for distributed data processing of this disclosure traverses the initial logical node topology graph to obtain the direct neighbor topology graph of a selected initial logical node in the initial logical node topology graph and labels the obtained direct neighbor topology graph one by one. The direct neighbor topology graph includes the selected initial logical node and other initial logical nodes that have direct connection edges with it. Figure 2 The diagram shown is an example of a direct neighborhood topology graph obtained by traversing the parallel decision-making system 100 of the distributed data processing disclosed herein. Figure 2 As shown, when traversing the initial logical node topology graph 111, the direct neighbor labeling component 120 determines the other initial logical nodes that are directly connected to each initial logical node. Figure 2 In the shown direct neighborhood topology graph, the initial logical node numbered 00 is the selected initial logical node, and the initial logical nodes numbered 01-10 are other initial logical nodes that have direct connections to the selected initial logical node. Together, they form a traversed direct neighborhood topology graph, which is then assigned a predetermined label, such as DBN00, indicating that this direct neighborhood topology graph is the direct neighborhood topology graph of the selected initial logical node N00. Through this traversal, the direct neighborhood topology graph of each initial logical node can be obtained; for example, node 04 can also have its own direct neighborhood topology graph.

[0041] Subsequently, for each marked direct neighborhood topology, the sub-neighborhood cost calculation component 121 traverses and selects each sub-neighborhood topology in the marked direct neighborhood topology. Based on the random combination of the candidate computation costs corresponding to the candidate parallel schemes of each initial logical node in the traversed and selected sub-neighborhood topology and the candidate transmission costs of each connecting edge, it calculates the combined cost of the sub-neighborhood topology under each random combination, and the node cost of the candidate parallel schemes of each initial logical node in the traversed and selected sub-neighborhood topology under each random combination. Each of the traversed and selected sub-neighborhood topologies contains the selected initial logical node in its direct neighborhood topology and a predetermined number of directly adjacent initial logical nodes. Figure 2 The initial logical nodes in the sub-neighborhood topology graph contained by the dashed line on the left are 00, 01, 02, 03, and 04, while the initial logical nodes in the sub-neighborhood topology graph contained by the dashed line on the right are 00, 02, 03, 04, and 05. However, node 05 has a direct edge adjacent to node 11. Therefore, when traversing the sub-neighborhood topology graph on the right, node 11 is included to fully consider the importance of node 05 in the sub-neighborhood topology graph during subsequent cost calculation by the sub-neighborhood cost calculation component 121. Specifically, for Figure 2In the sub-neighborhood topology graph on the left, the initial logical nodes are 00, 01, 02, 03, and 04. Each of these initial logical nodes has its own candidate parallel schemes. Each candidate parallel scheme corresponds to a node's candidate computational cost. This initial candidate computational cost is not the actual computational cost of that initial logical node in the actual system, but rather an empirical value as described above. Since each of these initial logical nodes has its own candidate parallel scheme, there will be multiple pairs of parallel scheme combinations between two connected nodes (initial logical nodes 00 and 01), and so on, including nodes 00 and 02. Because different pairs of parallel schemes result in different candidate transmission costs between the two nodes represented by the connecting edges formed by different combinations. Since the number of nodes, connecting edges, and candidate parallel schemes in each sub-neighborhood topology graph is finite, the combinations are also finite. The sub-neighborhood cost calculation component 121 sums the computational cost of all nodes and the transmission cost between nodes for each combination in the sub-neighborhood topology graph to obtain the cost sum of the sub-neighborhood topology graph, and calculates the node cost sum of each node for each combination. The node cost sum of each node in each combination includes the computational cost of the node itself and the transmission cost of the connection edges directly associated with it. This includes the transmission cost of some of the directly associated connection edges or the transmission cost of the directly associated connection edges outside the neighborhood. This cost sum is the sum of the computational cost of the node itself and the transmission cost of the connection edges of its directly adjacent external nodes (i.e., nodes not in the neighborhood). This sum does not include the transmission cost within the neighborhood because the decisions within the neighborhood can change and are not fixed; these decisions must be determined, and the node cannot rely on them.

[0042] For each combination of sub-neighborhood cost calculation results in each sub-neighborhood topology, the sub-neighborhood parallel decision component 122 calculates the cost sum and difference between the node costs of each initial logical node in the traversed and selected sub-neighborhood topology under all random combinations. Based on the minimum combination cost sum of the initial logical nodes with the maximum cost sum and difference in the traversed and selected sub-neighborhood topology, it determines the sub-neighborhood parallel scheme for the traversed and selected sub-neighborhood topology and assigns the node cost sum of each initial logical node under the determined sub-neighborhood parallel scheme to the corresponding initial logical node. For example, such as... Figure 2The sub-neighborhood topology on the left, containing initial logical nodes 00, 01, 02, 03, and 04, has 20 possible combinations. Node 02 has a node cost and difference of 10 across these 20 combinations, while the node costs and differences of all other nodes are less than 10. This means that node 02 is the most important node in this sub-neighborhood topology. Therefore, we first determine all parallel schemes for the sub-neighborhood topology based on the parallel schemes for node 02. Then, based on the parallel schemes determined with 02 as the primary factor, we select the parallel scheme with the minimum combined cost among all combinations containing 02 as the sub-neighborhood parallel decision result for this traversal scenario. Specifically, when node 02 has the largest node cost and difference among all nodes, we select the parallel scheme with the smallest combined cost among the combinations where node 02 has the largest node cost and difference. For example, if the node cost and the minimum difference between nodes are 100 and 90 respectively, then the combination with the minimum combination cost among the combinations of node 02 with node cost sums of 100 and 90 is selected as the combination parallel scheme of the sub-neighborhood topology graph.

[0043] After completion Figure 2 After the parallel decision-making process for the sub-neighborhood topology graph in the left dashed line, we move on to the sub-neighborhood topology graph in the right dashed line, which consists of nodes 00, 02, 03, 04, and 05. The same parallel decision-making method as for the left sub-neighborhood topology graph is used. As mentioned above, if node 05 or any other initial node has a node that is not part of the direct neighborhood topology graph but is directly connected to it, it is included in that value's neighborhood topology graph. This process is repeated until all sub-neighborhood topologies in the direct neighborhood topology graph have been traversed, thus forming a parallel scheme for the entire direct neighborhood topology graph.

[0044] By traversing the entire initial logical node topology graph 111 one by one, the parallel scheme for each initial logical node is determined. Subsequently, the initial logical node topology graph 111, which was assigned a parallel scheme in the previous traversal, is traversed again. This may result in adjustments to the parallel schemes or the same parallel schemes may be obtained. Therefore, through multiple traversals, when the parallel decision termination component 123 of the system of this disclosure, after multiple traversals of the initial logical node topology graph, finds that the parallel schemes for each direct neighbor obtained in two consecutive traversals and the node costs assigned to each initial logical node remain unchanged, the repeated traversal process is terminated, thereby obtaining the resulting logical node topology graph, and thus obtaining the parallel scheme of the distributed data processing system.

[0045] Alternatively, according to another aspect of this disclosure, in order to reduce the number of traversals required for the parallel scheme described above, the number of strategies required for parallel decision-making, and to reduce data transmission, the initial logical node topology graph can be preprocessed. For example, by traversing the graph, when a specific initial logical node has two or more downstream initial logical nodes, temporary intermediate proxy nodes can be inserted between the specific initial logical node and all its downstream initial logical nodes. Specifically, see [link to relevant documentation]. Figure 1 The system may include a proxy node insertion component 124, which is used to traverse the initial logical node topology graph before the direct neighborhood marking component marks it. When a specific initial logical node has two or more downstream initial logical nodes, a temporary intermediate proxy node is inserted between the specific initial logical node and all its downstream initial logical nodes. The number of the temporary intermediate proxy nodes is assigned to a candidate parallel strategy that does not exceed the number of combinations of parallel strategy types of all downstream initial logical nodes, thereby obtaining an initial logical node topology graph containing the temporary intermediate proxy nodes.

[0046] Figure 3 The diagram illustrates the insertion process performed by the agent node insertion component in a parallel decision-making system for distributed data processing according to this disclosure. Figure 3 As shown, node A has two downstream nodes B and C. The output tensor y of A is transmitted to B and C. The distribution strategy or parallel strategy SBP of A (see Chinese Patent Publication No. CN110955734A) is S0, while the SBP of nodes B and C is S1. According to the cost model, both edges AB and AC will have a transmission cost Ct, and each will undergo an All2All transmission. To reduce data transmission, we can reduce the amount of data transmitted and the number of transmissions. However, since the data candidate parallel strategy cannot be changed once selected, reducing the number of transmissions can significantly reduce the transmission cost. Therefore, for... Figure 3 As shown, by inserting a temporary intermediate proxy node PN between the upstream and downstream, the number of data transmissions can be reduced from two to one, thereby reducing the overall transmission cost. For example... Figure 3As shown, a temporary intermediate proxy node PN is established between node A and downstream nodes B and C. The output y of A is first processed using All2All and then transmitted to this temporary intermediate proxy node PN. At this time, the SBP of y on this temporary intermediate proxy node PN is S1. From the temporary intermediate proxy node PN to the two downstream nodes B and C, since their SBPs are both S1, no communication occurs. The tensor y is directly used. Therefore, the overall transmission cost is only the data transmission process of transforming the tensor y from the distribution mode S0 to S1. This disclosure abstracts the temporary intermediate proxy node PN (sbp_proxy) in the cost model. After the automatic parallel decision system of this disclosure is completed, it will not form an actual node to participate in the operation during the actual deployment process. It only exists in the automatic parallel decision-making and helps to optimize the automatic parallel decision-making method. When inserting the temporary intermediate proxy node PN, the number of combinations of candidate parallel strategies of the downstream nodes of the temporary intermediate proxy node PN is assigned as a parallel strategy. For example, the candidate parallel strategy of the temporary intermediate proxy node PN is S1.

[0047] Furthermore, in the cost model for automatic parallel decision-making disclosed herein, after the temporary intermediate proxy node PN is established, the parallel strategy of the temporary intermediate proxy node PN is the SBP strategy for the transmitted tensor. When the required SBPs for this tensor differ among downstream nodes, for example, upstream node A outputs tensor y and transmits it to four downstream nodes B, C, D, and E, y is S0 in A, S1 in B and C, and S2 in D and E. In practice, B and C are coordinated, and D and E are also coordinated. In this case, the parallel strategy of the temporary intermediate proxy node PN cannot be S1 or S2, but rather a combination of candidate parallel strategies. The cost from upstream node to downstream node is independent of the number of a certain SBP in the downstream node; it only depends on its existence and the existence of a parallel strategy in the downstream. That is, the cost from upstream A (y: S0) to downstream node B (y: S1) is the same as the cost from upstream node A (y: S0) to 100 downstream nodes (y: S1). Therefore, this disclosure defines the parallel strategy of a temporary intermediate proxy node PN as the set of possible parallel strategies for downstream nodes. For example, if upstream node A transmits tensor y to two downstream nodes B and C, the SBP of tensor y at node B is B, S0, S1, and the SBP of tensor y at node C is B, S0, S2. Then the candidate parallel strategies for the temporary intermediate proxy node PN inserted after upstream node A are {B}, {S0}, {S1}, {S2}, {S0, S1}, {S0, S2}, {S1, S2}. Note that {S0, S1, S2} is not included in the candidate strategies because node A only has two downstream nodes. This disclosure restricts the number of elements in each candidate parallel strategy set of the temporary intermediate proxy node PN to not exceed the number of downstream nodes; otherwise, it would only unnecessarily increase the strategy space. Therefore, the number of candidate parallel strategies assigned to the temporary intermediate proxy node PN cannot exceed the number of downstream nodes. Furthermore, in the candidate parallel strategy of the temporary intermediate proxy node PN, B is not in the same set as other SBPs. This is because the transmission cost from B to other S0, S1, and S2 is 0. When a tensor's SBP at the temporary intermediate proxy node PN is B, it will not communicate with any other downstream SBPs; at most, only some local processing is required. Therefore, as long as there is one B, no SBP is needed in cost estimation. (Of course, in practice, even if the intermediate node has the SBP of B, it will still generate other proxies to store the tensors of other SBPs (such as S0) to avoid duplication of some operations when B is transferred to S, but theoretically, there will be no communication cost here.)

[0048] Based on the rules assigned by the temporary intermediate proxy node PN, the transmission cost (Ct) of the edge from the temporary intermediate proxy node PN to the downstream node can be determined as follows:

[0049] If the downstream node's SBP is included within the SBP of the temporary intermediate proxy node PN, or if the candidate parallel strategy of the temporary intermediate proxy node PN is {B}, then the cost is 0; otherwise, it is GetMaxVal. <float>()=3.4e38.

[0050]

[0051] This can be understood as the communication being completed when it reaches the temporary intermediate proxy node PN; the process from the temporary intermediate proxy node PN to the downstream node is merely a value retrieval process.

[0052] although Figure 2 The displayed sub-neighborhood topology graph has 5 nodes, but this can be adjusted according to actual needs, such as 3 or 4. Generally, 5 is preferred, but more is also possible. If the direct neighborhood topology graph does not exceed the predetermined number of nodes in the sub-neighborhood topology graph, then the direct neighborhood topology graph only needs to perform a sub-neighborhood cost calculation once.

[0053] To reduce the difficulty of parallel decision selection when faced with astronomical numbers of parallel decision combinations, the initial logical node topology graph is reduced to obtain a simpler logical node topology graph, facilitating automatic selection of parallel methods. To this end, the predetermined configuration acquisition component of the topology graph preprocessing component of this disclosure traverses the initial logical node topology graph to obtain multiple predetermined configurations F in the initial logical node topology graph. These predetermined configurations F are transformed into simple subgraph structures using the reduction method of this disclosure. Through repeated iterative reduction, a minimally simplified initial logical node topology graph that facilitates parallel decision selection is finally obtained. Figure 4 The diagram shown is a schematic representation of another example of a parallel decision-making system for distributed data processing according to this disclosure. Figure 1 The difference in the system 100 shown is the use of a dedicated parallel decision signature system, namely the SBP signature system. Logical nodes that have already been configured with SBP signatures by the user, such as initial logical nodes A, E, and C, possess only a unique SBP signature, for example, SBP-5 for initial logical node A, SBP-2 for initial logical node C, and SBP-3 for initial logical node E. In the absence of a unique SBP signature, the initial logical nodes typically contain some inherent candidate SBP signatures. For example... Figure 4 The initial logical node B has multiple candidate SBP signatures, for example, three: SBP-1, SBP-2, and SBP-3. Other initial logical nodes also have different candidate SBP signatures, which are not listed here. Different initial logical nodes will have different fixed candidate SBP signatures depending on the specific operations they perform.

[0054] SBP signatures are signatures applied in a distributed data processing system. In distributed data processing systems, due to the frequent occurrence of data parallelism, model parallelism, hybrid parallelism, and streaming parallelism, tasks of adjacent logical nodes are often deployed simultaneously to different computing devices. Therefore, during actual data processing, intermediate parameters are exchanged between different computing devices, leading to significant transmission overhead. To reduce this data transmission overhead, more logical nodes need to be generated based on the initial logical node topology graph 211 to improve the topology, especially minimizing transmission overhead between upstream and downstream logical nodes by minimizing changes in the data distribution of upstream and downstream logical nodes. Therefore, this disclosure specifies a logical distributed signature for each logical node to obtain better downstream logical nodes. The logical distributed signature uses distributed tensor descriptors to sign the logical nodes. Each distributed tensor descriptor describes the distribution of each tensor throughout the computing system, mainly including Split tensor descriptors, Broadcast tensor descriptors, and Partial Value tensor descriptors.

[0055] Specifically, a split tensor descriptor describes how a tensor is split, such as dividing a data block according to a user's description along specified dimensions and distributing it across different computing devices for specified computational processing. If a data block is two-dimensional, when it is split along its 0th dimension, the distributed descriptor of the data tensor formed by that data block is S(0), and each logical data block obtains a distributed descriptor of this data tensor at its input. Similarly, if a data block is two-dimensional, when it is split along its 1st dimension, the distributed descriptor of the data tensor formed by that data block is S(1), and each logical data block obtains a distributed descriptor of this data tensor at its input. Likewise, if the task data to be processed has more dimensions, there will be more distributed descriptors, such as S(2), S(3), etc. The data mentioned in this way can be the data being processed or the model. If the data itself is split, data parallel processing is achieved on a distributed data processing system; if the model is split, model parallel processing is achieved on a distributed data processing system. If the input of a logical node is such a split tensor descriptor, then in the actual data processing process, if the data size of a tensor is T, and this tensor is distributed across four computing cards for data parallel computation, then the amount of data allocated to each card is one-quarter of the data, and the total amount of data on the four cards is T.

[0056] A broadcast tensor descriptor is used to describe how a tensor is published in a distributed system via broadcast. Typically, for data processing systems that only perform data parallelism, model data is broadcast to various computing devices; therefore, broadcast tensor descriptors are used to describe the broadcast data input to logical nodes. In actual data processing, the broadcast data has the same block size on each actual computing card.

[0057] A partial value tensor descriptor represents that the input or output tensor of a logical node consists of partial values ​​of multiple similar tensors. These partial values ​​include partial sums (Ps), partial products (Pm), partial AND results, partial maximums, and partial minimums. Since data parallel processing is often used, data processing on different devices involves processing partial data. For example, if some tensors are S(0) or S(1), the resulting tensor obtained on some computing devices will be S(0). These partial computing device result tensors are combined to form the partial value tensor. The final output result is obtained by combining similar data from all devices.

[0058] The distributed descriptors of the various tensors described above represent the distribution of these tensors in a distributed computing system. Whether these tensors serve as inputs or outputs of logical nodes, their distribution also describes the distribution of operational data by the logical nodes. For ease of description, this disclosure will simply refer to such distributed descriptors as "SBP descriptors".

[0059] Therefore, with the generation of the initial logical node topology graph 211, the initial logical nodes of this disclosure, that is, some operational nodes, also possess distributed descriptors for each input and output. These input and output distributed descriptors form a signature for the logical node, namely, a signature of the operational logical node using tensor distributed descriptors. For ease of description, this signature is abbreviated as "SBP signature" using the first letters of the English words for these three distributed descriptors.

[0060] Based on the user's description of the computing task and the data parallelism requirements in each distributed computing system, this descriptor will include at least three types: S(0), B, and P. If there are multiple ways to partition the data and model, an additional descriptor is added for each additional partition. For each logical node, its signature contains various combinations of these descriptors. Therefore, in the distributed system according to this disclosure, there are at least three distributed descriptors, and usually four distributed descriptors, such as the following four SBP descriptors: S(0), S(1), P, and B. Depending on the number of tensor dimensions, there can be more distributed descriptors. If there are four SBP descriptors, multiple SBP signatures can be formed according to the permutation and combination of input and output. Some examples of SBP signatures are listed below: (S(0),B)→S(0), (S(1), B)→S(1), P→P, B→B, (S(0),S(1))→P, S(0)→P, S(0)→S(0), S(0)→S(1), P→B, etc. All SBP signatures are combinations of various SBP descriptors. For a matrix multiplication logic node, if its input tensor is cut along the first dimension, its output tensor is also cut along the first dimension. In summary, S, B, and P are descriptors used to describe the distribution of data blocks in a data processing system, while SBP signatures utilize multiple SBP descriptors to describe the task operations of logic nodes. Each data block can have multiple SBP descriptors, and each logic node can represent multiple SBP signature scenarios. For example, Figure 4 The SBP-1 shown can be a signature in the form of (S(0),B)→S(0), while the SBP-2 can be a signature in the form of (S(1),B)→S(1). In practical applications, different signature forms can have different numbers. The numbers given here are only for the convenience of description and do not mean that each signature needs to be assigned a number. There can be no number at all, and different forms of signatures can be distinguished from each other without numbering.

[0061] Each initial logical node can be assigned an SBP signature as described above based on the task description used. Typically, task logical nodes are operation nodes that perform specific operations, and therefore have specific candidate SBP signatures. It should be noted that not every task logical node possesses the same SBP signature. For example, the input tensor of a task logical node performing multiplication typically does not contain partial tensors, therefore its input tensor's SBP descriptor does not contain the distributed descriptor P. For task logical nodes performing addition, the candidate SBP signature can include any combination of various SBP descriptors, either between themselves or with each other. For example, a task logical node performing matrix multiplication, in the case of only data parallelism, typically has candidate SBP signatures such as (S(0),B)→S(0), (S(1), B)→S(1), (S(0),S(1))→P, etc. However, with technological advancements, some signatures previously unsuitable for matrix multiplication can now be applied to matrix multiplication; this is merely an example. Therefore, each initial logical node is associated with a set of candidate logical distributed signatures based on the task configuration data. Each logical distributed signature in the candidate logical distributed signature set specifies the distributed descriptor of each input tensor and the distributed descriptor of each output tensor of its initial logical node.

[0062] While the above describes the general case of determining the final SBP signature from candidate SBP signatures, in certain specific situations, for some logical nodes, with special user configuration or user-specified settings, these logical nodes only have the user-specified SBP signature. Therefore, their downstream logical nodes will determine their SBP signatures based on this specifically specified upstream logical node. Thus, an alternative SBP signature can be used. Figure 1 The parallel policy signature P in the text. Other parts are related to the... Figure 1 The description is the same, so it will not be repeated. (And) Figure 1 Unlike the system 100 shown, the system 200 adds a topology graph preprocessing component 240.

[0063] The topology graph preprocessing component 240 is used to preprocess the initial logical node topology graph before the direct neighbor labeling component performs labeling, to obtain a preprocessed initial logical node topology graph. It includes: a predetermined configuration acquisition component 241, which traverses the initial logical node topology graph to obtain predetermined configurations in the initial logical node topology graph. The predetermined configurations include a first predetermined configuration and / or a second predetermined configuration, wherein the first predetermined configuration is an intermediate initial logical node with a first connecting edge and a second connecting edge, and the second predetermined configuration is a pair of initial logical nodes with multiple third connecting edges between them; subgraph cost calculation; The computation component 242, for a first predetermined configuration, when the candidate parallel schemes for the first initial logical node of the first connection edge and the candidate parallel schemes for the second initial logical node of the second connection edge are determined, obtains the candidate computation cost of each intermediate initial logical node, the candidate transmission cost of the first connection edge corresponding to the obtained candidate computation cost, and the candidate transmission cost of the second connection edge, and obtains the first cost sum of the three for each candidate computation cost of each intermediate initial logical node, and selects the minimum first cost sum as the candidate parallel scheme for each pair of first initial logical nodes and the candidate parallel scheme for the second initial logical node. Under certain conditions, the first candidate transmission cost between the first initial logical node and the second initial logical node in the first predetermined configuration; and for the second predetermined configuration, when the candidate parallel schemes of the third initial logical node and the fourth initial logical node of the paired initial logical nodes are determined, the candidate transmission costs of all connection edges between the paired initial logical nodes are summed to obtain a second cost sum of the candidate transmission costs between the paired initial logical nodes as the second candidate transmission cost; and the predetermined configuration transformation component 243 is used to transform the first connection edge, the second connection edge, and the intermediate initial logical node of the first predetermined configuration into a first merged connection edge between the first initial logical node of the first connection edge and the second initial logical node of the second connection edge, and assign all the first candidate transmission costs calculated for the first predetermined configuration to the first merged connection edge as one of the candidate transmission costs of the first merged connection edge, and transform all the connection edges of the second predetermined configuration into a second merged connection edge of the paired initial logical nodes, and assign the second candidate transmission costs calculated for the second predetermined configuration to the second merged connection edge of the paired initial logical nodes as one of the candidate transmission costs of the second merged connection edge.

[0064] Figure 5 The diagram shown is a schematic representation of a first predetermined configuration according to this disclosure. Figure 5 As shown, the first predetermined configuration F1 is an intermediate initial logic node MN with a first connecting edge Lin and a second connecting edge Lout. The input of the first connecting edge Lin of the first predetermined configuration F1 is connected to the first initial logic node IN, and the output of the second connecting edge Lout is connected to the second initial logic node ON. (See also...) Figure 1 The initial logical node D is an intermediate initial logical node MN, and the initial logical nodes C and G are the first initial logical node IN and the second initial logical node ON, respectively. Therefore, Figure 1 The initial logical nodes C, D, and G in the first predetermined configuration F1 are formed.

[0065] like Figure 5 As shown, in a first predetermined configuration F1, each initial logical node is assigned a candidate parallelism label P and a corresponding computational cost label C. For example, the first initial logical node IN, under candidate parallelism label P2, has a corresponding computational cost label C1; the intermediate initial logical node MN, under candidate parallelism label P1, has a corresponding computational cost label C1; under candidate parallelism label P2, it has a corresponding computational cost label C2; and under candidate parallelism label P4, it has a corresponding computational cost label C3. Similarly, the second initial logical node ON, under candidate parallelism label P2, has a corresponding computational cost label C1; under candidate parallelism label P3, it has a corresponding computational cost label C2; and under candidate parallelism label P4, it has a corresponding computational cost label C3. It should be noted that, for ease of description, although the same computational cost label, such as C1 and C2, is used in different initial logical nodes, the computational cost labels attached to different initial logical nodes do not indicate that the computational costs are the same. They merely serve to mark the different computational costs under different candidate parallelism strategies within the same initial logical node. In practical applications, these cost labels are represented by actual cost values. This labeling method serves only as a marker in subsequent predetermined configurations or explanations, and does not imply that the same cost label format represents the same computational cost.

[0066] like Figure 5 As shown, the first connecting edge Lin has different transmission costs depending on the candidate parallel strategies P of its first initial logical node IN and intermediate initial logical node MN, as well as the determination of computational resources. For example... Figure 2 The first initial logical node IN has one candidate parallel strategy, and the intermediate initial logical node MN has three candidate parallel strategies. These strategies, when combined, form three transmission methods, each corresponding to a transmission cost: Ct1, Ct2, and Ct3. Similarly, Figure 2 There are three candidate parallel strategies for the initial logical node MN in the middle, which, when combined, form nine transmission methods. Therefore, these nine transmission methods correspond to nine transmission costs: Ct1, Ct2, Ct3, Ct4, Ct5, Ct6, Ct7, Ct8, and Ct9. It should be noted that, for ease of description, although the same transmission cost label (e.g., Ct1, Ct2) is used in different connection edges, the transmission cost label attached to different connection edges does not indicate that the transmission costs are the same. It merely serves to mark the different transmission costs under different combinations of candidate parallel strategies within the same connection edge. In practical applications, these transmission cost labels are represented by the actual transmission cost.

[0067] return Figure 4 When the predetermined configuration acquisition component 241 discovers the first predetermined configuration F1 during the process of traversing each initial logical node in the initial logical node topology graph, the subgraph cost calculation component 242, for the first predetermined configuration, under the condition that the candidate parallel scheme of the first initial logical node IN of the first connecting edge and the candidate parallel scheme of the second initial logical node ON of the second connecting edge are determined, acquires each candidate computation cost of the intermediate initial logical node, the candidate transmission cost Ct of the first connecting edge corresponding to the acquired candidate computation cost C, and the candidate transmission cost Ct of the second connecting edge, and acquires the first cost Cs of the three under the condition that each candidate computation cost of the intermediate initial logical node is determined, and selects the minimum first cost Csm as one of the first candidate transmission costs between the first initial logical node and the second initial logical node in the first predetermined configuration under the condition that the candidate parallel scheme of the first initial logical node IN and the candidate parallel scheme of the second initial logical node ON are determined.

[0068] Specifically, combined Figure 5 The first predetermined configuration F1 includes initial logical nodes C, D, and G, corresponding to the first initial logical node IN, the intermediate initial logical node MN, and the second initial logical node ON, respectively. The connection edge CD between initial logical nodes C and D corresponds to the first connection edge Lin, and the connection edge DG between initial logical nodes D and G corresponds to the second connection edge Lout. When initial logical node C selects a candidate parallel decision Pi (where i is the number of the candidate parallel decision for initial logical node C) and initial logical node D selects a candidate parallel decision Pk (where k is the number of the candidate parallel decision for initial logical node D), the transmission cost of connection edge CD is Cost. CD Let [i][k] be the decision group for connecting edge CD. Multiple transmission costs are represented by Cost. CD [i][k] The decision group (i, k) corresponding to the connecting edge CD is sequentially labeled as the candidate transmission costs C1, C2, ..., C1n, C21, C22, ..., C2n, ..., Cm1, Cm2, ..., Cmn. Alternatively, it can be labeled using a two-dimensional array C11, C12, ..., C1n, C21, C22, ..., C2n, ..., Cm1, Cm2, ..., Cmn. Similarly, when the initial logical node D selects a candidate parallel decision Pk (k is the number of the candidate parallel decision of the initial logical node D) and the initial logical node G selects a candidate parallel decision Pj (j is the number of the candidate parallel decision of the initial logical node G), the transmission cost of the connecting edge DG is Cost. DG [k][j], let (k, j) be the decision group of the connection edge DG. Multiple transmission costs are represented by this. DG The decision group (k, j) corresponding to edge DG [k][j] is sequentially labeled as candidate transmission costs C1, C2, ... for edge DG. Therefore, when the candidate parallel decisions for the initial logical nodes C, G, and D are determined as Pi, Pj, and Pk respectively, the subgraph cost calculation component 242 first obtains the transmission cost of edge CD as Cost. CD [i][k], the transmission cost of the connection edge DG is Cost DG The computational cost of [k][j] and the initial logical node D, which serves as the intermediate initial logical node MN, is [cost]. D [k] is summed, and the sum is recorded as Cost. CG [i][j]. Thus, when the candidate parallel decisions for the initial logical nodes C and G are determined, the number of candidate parallel decisions for the initial logical node D, which is the intermediate initial logical node MN, is determined, for example, 3. Thus, the subgraph cost calculation component 242 obtains three costs and a Cost for the initial logical nodes C and G, given a set of determined candidate parallel decisions. CG [i][j], as follows:

[0069] Cost CD [i][1]+Cost D [1]+Cost DG [1][j].

[0070] Cost CD [i][2]+Cost D [2]+Cost DG [2][j].

[0071] Cost CD [i][3]+Cost D [3]+Cost DG [3][j].

[0072] Subgraph cost calculation component 242 will calculate the cost from the three primary costs and Cost mentioned above. CG The minimum sum in [i][j] is selected as the minimum first cost, and Csm is one of the first candidate transmission costs between the first and second initial logical nodes in the first predetermined configuration, given that each pair of candidate parallel schemes for the first initial logical node IN and the second initial logical node ON is determined. Therefore, when (i, j) is the decision group for the connection edge CG, its corresponding minimum first cost and Ctsm are...

[0073] Ctsm = Cost CG [i][j]=MIN{Cost CD [i][k]+Cost D [k]+Cost DG [k][j]}

[0074] 1≤k≤n

[0075] Where n is the number of candidate parallel decisions for the intermediate initial logical node MN. This formula means selecting the minimum value from n sets of first cost sums as one of the first candidate transmission costs for the decision group where (i, j) is the connecting edge CG. If the number of candidate parallel decisions for the initial logical node C and the initial logical node D are X and Y respectively, then X×Y first candidate transmission costs are finally obtained, where each first candidate transmission cost is the minimum first cost sum among the n first cost sums Csm corresponding to the decision group of (i, j) CG.

[0076] Subsequently, the predetermined configuration transformation component 243 transforms the first connecting edge Lin, the second connecting edge Lout, and the intermediate initial logic node MN of the first predetermined configuration F1 into the first merged connecting edge Lm1 between the first initial logic node IN of the first connecting edge and the second initial logic node ON of the second connecting edge. Figure 5 The process of this transformation is shown. Figure 5 The left side represents the original structure of the first predetermined configuration F1, and the right side represents the transformed topology. The transformed first merged connection edge Lm1 has three first candidate transmission costs Ctsm1, Ctsm2, and Ctsm3. As mentioned above, the number of first candidate transmission costs is equal to the product of the number of candidate parallel schemes for each pair of first initial logical nodes IN and the number of candidate parallel schemes for each pair of second initial logical nodes ON.

[0077] like Figure 5 As shown, by performing this transformation on the first predetermined configuration F1 through the predetermined configuration transformation component 122, the number of logical nodes that need to be judged and the number of connection edges that need to be judged are reduced on the one hand, which greatly reduces the difficulty of making overall parallel strategy decisions for the entire logical node topology graph. Moreover, by choosing this locally minimizing cost sum, conditions are provided for selecting the smallest possible parallel cost overall.

[0078] Each time the predetermined configuration transformation component 243 performs such a transformation on the first predetermined configuration F1, the logical nodes in the transformed structure are rearranged at the end of the traversed logical node queue for iterative traversal. That is, the entire logical node topology graph is traversed repeatedly until the first predetermined configuration F1 can no longer be obtained.

[0079] Return to reference Figure 4 The predetermined configuration acquisition component 241 traverses the initial logical node topology graph to obtain a second predetermined configuration F2 among multiple predetermined configurations F in the initial logical node topology graph. The second predetermined configuration F2 consists of pairs of initial logical nodes that are connected to each other by multiple third connecting edges. Figure 3 The diagram shown is a schematic representation of a second predetermined configuration according to this disclosure. Figure 6 As shown, the second predetermined configuration F2 has paired initial logic nodes, a third initial logic node IN and a fourth initial logic node ON, and there are at least two or more third connecting edges L1, L2, etc. between the third initial logic node IN and the fourth initial logic node ON. For ease of description, Figure 6 The image only shows two third connecting edges. In reality, there can be three or more third connecting edges. Figure 4 The initial logical node topology shown is merely an example topology. Figure 1 Part of, in Figure 4 The second predetermined configuration F2 is not shown, but this does not mean that the second predetermined configuration F2 does not exist in all initial logical node topologies.

[0080] like Figure 6 As shown, the third initial logical node IN and the fourth initial logical node ON each have three candidate parallel strategies, and there are two third connecting edges L1 and L2 between them, or more third connecting edges, such as L3 and L4 (not shown). Given that the parallel strategies of the paired initial logical nodes IN and ON are determined, and with a given amount of computing resources, the transmission costs of the third connecting edges L1 and L2 are also determined. Since the third initial logical node IN and the fourth initial logical node ON each have three candidate parallel strategies, there are nine possible combined parallel strategies between them. Therefore, there are also nine possible candidate transmission costs for connecting edges L1 and L2, such as Ct1, Ct2, Ct3, ..., Ct8, Ct9, each corresponding to a set of parallel decision combinations.

[0081] Therefore, when the candidate parallel decisions for the third initial logical node IN and the fourth initial logical node ON are determined as Pi and Pj respectively, the subgraph cost calculation component 242 first obtains the transmission cost of each connection edge L1 and L2 as Cost. L1 [i][j] and Cost L2 [i][j], then calculate the sum of transmission costs for the connection edges under a set of (i,j) candidate decisions for the paired output logical nodes IN and ON. The formula is as follows:

[0082]

[0083] Where k is the edge number, z is the number of edges connecting the paired output logical nodes IN and ON, and Cts is the transmission cost sum, which can also be represented by CostLm[i][j]. For a set of (i,j) candidate parallel decision combinations for the paired output logical nodes IN and ON, a transmission cost sum Cts is formed, for example... Figure 6 The numbers on the left are Cts1, Cts2, Cts3, ..., Cts8, Cts9.

[0084] Subsequently, the predetermined configuration transformation component 243 transforms the connection edge between the paired output logic nodes IN and ON of the second predetermined configuration F2 into a second merged connection edge Lm2 between the paired output logic nodes IN and ON. Figure 6 The process of this transformation is shown. Figure 6 The left side shows the original structure of the second predetermined configuration F2, and the right side shows the transformed topology. The transformed second merged connection edge Lm2 has nine second candidate transmission costs Cts1, Cts2, Cts3, ..., Cts8, Cts9. As mentioned above, the number of second candidate transmission costs is equal to the product of the number of candidate parallel schemes for each pair of output logic nodes IN and ON. Each second candidate transmission cost Cts contains a corresponding combination of the candidate parallel schemes for each pair of output logic nodes IN and ON.

[0085] Each time the predetermined configuration transformation component 122 performs such a transformation for the second predetermined configuration F2, the logical nodes in the transformed structure are rearranged at the end of the traversed logical node queue for iterative traversal. That is, the entire logical node topology graph is traversed repeatedly until the second predetermined configuration F2 can no longer be obtained.

[0086] Return to reference Figure 4 The predetermined configuration acquisition component 241 traverses the initial logical node topology graph to obtain a third predetermined configuration F3 among multiple predetermined configurations F in the initial logical node topology graph. Figure 7 shows a schematic diagram of the third predetermined configuration according to this disclosure. Figure 7 As shown, the third predetermined configuration F3 is an end-initial logic node EN with only a fourth connecting edge L1. This end-initial logic node EN is connected to the dependent initial logic node DN via the fourth connecting edge. The dependent initial logic node DN is also connected to other initial logic nodes, such as N1 and N2, via connecting edges. There may or may not be a connecting edge between N1 and N2. Therefore, in Figure 7 The two are connected by a dashed line. For ease of description, Figure 7 The text only shows two other initial logical nodes that depend on the initial logical node DN. In fact, there can be more other initial logical nodes, or there can be no other initial logical nodes. Figure 4 The initial logical node topology shown is merely an example topology. Figure 1 In some parts, the third predetermined configuration F3 is not shown in Figure 1, but this does not mean that the third predetermined configuration F3 does not exist in all initial logical node topologies. Figure 7 As shown, since both the initial logical node DN and the terminal initial logical node EN have three candidate parallel strategies, there are nine possible combined parallel strategies between them. Therefore, the candidate transmission cost of the fourth connecting edge L1 also has nine possibilities, such as Ct1, Ct2, Ct3, ..., Ct8, Ct9, each corresponding to a set of parallel decision combinations. Therefore, when the candidate parallel decisions Pi and Pj for both the initial logical node DN and the terminal initial logical node EN are determined, the subgraph cost calculation component 121 first obtains the respective computational cost Cost of the initial logical node DN and the terminal initial logical node EN under the determined candidate parallel decisions. DN [i] and Cost EN [j] and the transmission cost of the fourth connection edge L1. L1 [i][j], then calculate Cost EN [j] and the transmission cost of the fourth connection edge L1. L1 [i][j] The sum of costs for both. Given that the candidate parallel decision Pi dependent on the initial logical node DN remains unchanged, the sum of costs for each candidate parallel decision Pj for different final initial logical nodes EN is calculated as follows (when the number of candidate parallel decisions for each final initial logical node EN is three):

[0087] Cost L1 [i][1]|+Cost EN [1]

[0088] Cost L1 [i][2]|+Cost EN [2]

[0089] Cost L1 [i][3]+Cost EN [3]

[0090] Regarding the sum of the costs of the two mentioned above, the minimum sum of costs is obtained when the candidate parallel decision Pi, which depends on the initial logical node DN, remains unchanged, as the minimum third cost sum. Therefore, the above formula can be summarized as follows:

[0091]

[0092] Where n is the number of candidate parallel decisions for each initial logical node EN at the end. Therefore, the subgraph cost calculation component 121 calculates n minimum third costs and the third additional computational cost (Pi) as determined by the candidate parallel schemes depending on the initial logical node DN for the third predetermined configuration. m [i].

[0093] Subsequently, the predetermined configuration transformation component 243 transforms the third predetermined configuration F3 into a new single logic node DN by pruning the fourth connection edge L1 and the terminal initial logic node EN of the third predetermined configuration F3, and adds the third additional computational cost to the computational cost Cost that depends on the initial logic node DN. DN On [i], a new computational cost Cmi is formed that depends on the initial logical node DN, i.e.

[0094]

[0095] like Figure 7 As shown, by performing this transformation on the third predetermined configuration F3 through the predetermined configuration transformation component 243, the number of logical nodes that need to be judged and the number of connection edges that need to be judged are reduced on the one hand, which greatly reduces the difficulty of making overall parallel strategy decisions for the entire logical node topology graph. Moreover, by choosing this locally minimizing cost sum, conditions are provided for selecting the smallest possible parallel cost overall.

[0096] Each time the predetermined configuration transformation component 243 performs this transformation for the third predetermined configuration F3, the logical nodes in the transformed structure are rearranged to the end of the traversed logical node queue for iterative traversal. That is, the entire logical node topology is traversed repeatedly until the third predetermined configuration F3 cannot be obtained through traversal. The new computational cost of the transformed logical node DN, which depends on the initial logical node, will be used as the computational cost for subsequent processing.

[0097] Return to reference Figure 4 When the predetermined configuration acquisition component 241 traverses all queues of logical nodes to be traversed but fails to obtain the first, second, or third predetermined configuration, the predetermined configuration acquisition component 241 begins to traverse the initial logical node topology graph to obtain the fourth predetermined configuration F4 among the multiple predetermined configurations F in the initial logical node topology graph. Figure 8 The diagram shown is a schematic representation of the fourth predetermined configuration F4 according to this disclosure. Figure 8 As shown, the fourth predetermined configuration F4 includes a fifth initial logical node N1 and a sixth initial logical node N2 located within the same connected component and without any connecting edges between them, as well as at least one common seventh initial logical node N3, N4, N5, N6, etc., belonging to the fifth and sixth initial logical nodes. Furthermore, the fourth predetermined configuration F4 may also include an initial logical node N7 connected only to the fifth initial logical node N1. The fourth predetermined configuration F4 may also include an initial logical node N8 connected only to the sixth initial logical node N2. The connecting edge between any two initial logical nodes is L; for example, the connecting edge between N1 and N4 is L14, and similarly, the connecting edge between N2 and N5 is L25, and so on.

[0098] During the process of obtaining the fourth predetermined configuration F4 by the predetermined configuration acquisition component 120 in traversing the initial logical nodes of the topology graph, the product of the number of candidate parallel decisions of the fifth initial logical node N1 and the sixth initial logical node N2 constituting the fourth predetermined configuration F4 cannot exceed a predetermined threshold, such as 128 or 64. Without this threshold limitation, the traversal of the fourth predetermined configuration F4 would be repeated repeatedly in a complete topology graph, eventually resulting in the formation of a single logical node in the complete topology graph, leading to excessively long traversal times and a lengthy determination of the final parallel decision. Furthermore, during the merging process of the fifth initial logical node N1 and the sixth initial logical node N2 constituting the fourth predetermined configuration F4, a large amount of label data is generated, such as computational cost, transmission cost, and cost sum. Therefore, if the number of candidate parallel decisions of the fifth initial logical node N1 and the sixth initial logical node N2 constituting the fourth predetermined configuration F4 is too large, it will also require more memory space to store this label data, resulting in excessive data space occupied by nodes and connecting edges in the merged topology graph. For example, the space occupied by a merged node is O(threshold), and the space occupied by the connecting edges after merging is O(threshold squared). Therefore, thresholding the candidate parallel decision tree for the predetermined configuration F4 is also necessary to reduce data space usage. Restrictions can be placed on the initial logical nodes and initial connecting edges; nodes with more than the threshold of input candidate decisions are acceptable. However, it should be noted that the merged node cannot be merged with other nodes. Therefore, without thresholding, the merging of nodes in the fourth predetermined configuration F4 will continue indefinitely, eventually generating a large node with a candidate decision count equal to the product of the candidate decisions of all original nodes, which is undesirable.

[0099] Optionally, during the process of traversing the initial logical nodes in the topology graph to obtain the fourth predetermined configuration F4, the predetermined configuration acquisition component 241 should select the fourth predetermined configuration F4 with the highest possible direct neighborhood overlap. A direct neighborhood refers to a logical node and the set of all logical nodes connected to it. For the fifth initial logical node N1 and the sixth initial logical node N2 constituting the fourth predetermined configuration F4, if the product of their candidate decisions is less than or equal to the threshold O, the overlap of their direct neighborhoods can be found using a set comparison algorithm based on bitwise operations, and the pair of nodes with the highest overlap is recorded for node merging. The higher the overlap, the more connecting edges are merged. Therefore, the predetermined configuration acquisition component 241 prioritizes traversing and obtaining the fourth predetermined configuration F4 with the highest overlap that satisfies the threshold constraint.

[0100] Given the candidate parallel schemes for the fifth initial logical node N1 and the sixth initial logical node N2, the subgraph cost calculation component 242 obtains the candidate computation costs for the fifth initial logical node and the sixth initial logical node, and uses the sum of the two candidate computation costs as the fourth cost sum.

[0101] If the fifth initial logical node N1 has m alternative decisions {0,1,2,…,m-1} and the sixth initial logical node N2 has n alternative decisions {0,1,2,…,n-1}, then, given that the candidate parallel schemes for the fifth and sixth initial logical nodes N1 and N2 are determined, for example, if their candidate parallel decision combination is (i,j), and its corresponding combination number is k, where k = i*n + j, then the candidate computation cost for the fifth initial logical node N1 is Cost. N1 [i], the candidate computation cost of the sixth initial logical node N2 is Cost N2 [j], the cost of the parallel decision combination of the two is denoted as

[0102] Where k / n represents the quotient of k divided by n, and k%n represents the remainder of k divided by n. The subgraph cost calculation component 242 uses the sum of the two candidate transmission costs of the fifth initial logical node N1 and the sixth initial logical node N2 as the fourth cost sum.

[0103] like Figure 8 As shown, each of the seventh initial logical nodes N3, N4, N5, and N6 is simultaneously connected to both the fifth initial logical node N1 and the sixth initial logical node N2. Taking the seventh initial logical node N4 as an example, its connection edges with the fifth initial logical node N1 and the sixth initial logical node N2 are L14 and L24. Given that the parallel decision combination of the fifth initial logical node N1 and the sixth initial logical node N2 is determined (e.g., Pi, Pj), the subgraph cost calculation component 242 can also determine the transmission cost of connection edges L14 and L24 for any candidate parallel decision numbered r of the seventh initial logical node N4, provided that the computational resources are determined; that is, the Cost. L14 and Cost L24 For the candidate parallel decision combination (i, j) with corresponding combination number k, the sum of the transmission costs of its connecting edges L14 and L24 can be generally expressed as follows:

[0104] Cost Lmr [k][r] = Cost L14 [k / n][r]+Cost L24 [k%n][r]

[0105] Where k / n represents the quotient of k divided by n, k%n represents the remainder of k divided by n, and r is the candidate parallel decision number of the seventh initial logical node.

[0106] In addition, such as Figure 8 As shown, the fourth predetermined configuration F4 can also include an initial logical node N7 connected only to the fifth initial logical node N1. The fourth predetermined configuration F4 can also include an initial logical node N8 connected only to the sixth initial logical node N2. As an eighth initial logical node, the transmission cost of the connection edge L17 between the initial logical node N7 and the fifth initial logical node N1 remains unchanged. However, for the candidate parallel decision combination (i, j) of the fifth initial logical node N1 and the sixth initial logical node N2, the candidate transmission cost of the connection edge L17 with candidate parallel decision number s of the eighth initial logical node N7 can be expressed as:

[0107] Cost Lms [k][s] = Cost L17 [k / n][s]

[0108] Where k is the number of the candidate parallel decision combination (i, j) of the fifth initial logical node N1 and the sixth initial logical node N2, s is the number of the candidate parallel decision of the eighth initial logical node N7, n is the number of the candidate parallel decisions of the sixth initial logical node N2, and k / n represents the quotient of k divided by n.

[0109] The transmission cost of the connection edge L28 between the initial logical node N8 and the sixth initial logical node N2 remains unchanged, as the eighth initial logical node. However, for the candidate parallel decision combination (i, j) of the fifth initial logical node N1 and the sixth initial logical node N2, the candidate transmission cost of the connection edge L28 with candidate parallel decision number s of the eighth initial logical node N8, as the fourth candidate transmission cost, can be expressed as:

[0110] Cost Lms [k][s] = Cost L28 [k%n][s]

[0111] Where k is the number of the candidate parallel decision combination (i, j) of the fifth initial logical node N1 and the sixth initial logical node N2, s is the number of the candidate parallel decision of the eighth initial logical node N8, n is the number of the candidate parallel decisions of the sixth initial logical node N2, and k%n represents the remainder when k is divided by n.

[0112] Subsequently, the predetermined configuration transformation component 243 merges the fifth initial logic node N1 and the sixth initial logic node N2 of the fourth predetermined configuration F4 into a first merged logic node N1. 2 And merge the fifth connecting edge (e.g., L14) and the sixth connecting edge (e.g., L24) into a third merged connecting edge (e.g., Lm4), and assign the fourth cost calculated for the fourth predetermined configuration to the first merged logical node N1. 2 As one of its candidate computational costs, the third candidate transmission cost is assigned to the third merged connection edge. In addition, the fourth candidate transmission cost can be assigned to connection edge L17 (or Lm7) or connection edge L28 (or Lm8) that is connected only to one of the fifth initial logical node N1 and the sixth initial logical node N2.

[0113] Figure 8 The right side of the graph does not explicitly list the candidate computational or transmission costs of each transformed logical node or connecting edge within curly braces, but these costs are actually present. The general order of traversing the predetermined configurations is: first predetermined configuration F1, second predetermined configuration F2, and third predetermined configuration F3. Optionally, the first three predetermined configurations can be traversed in any order. The fourth predetermined configuration F4 is only traversed when the first three predetermined configurations cannot be found. In short, if the first three predetermined configurations cannot be found and a connected component of the transformed topology still contains more than two nodes (i.e., each logical node in that connected component has a degree greater than or equal to 3), then the fourth predetermined configuration F4 is obtained for further simplification of the topology.

[0114] Figure 9 The diagram shown is a schematic representation of the fifth predetermined configuration F5 according to this disclosure. Figure 9 As shown, the only difference between the fifth predetermined configuration F5 and the fourth predetermined configuration F4 is that the fifth initial logical node N1 and the sixth initial logical node N2 have a seventh connecting edge L12. Given the candidate parallel schemes for the fifth initial logical node N1 and the sixth initial logical node N2, the subgraph cost calculation component 121 obtains the candidate computation costs of the fifth initial logical node N1 and the sixth initial logical node N2, as well as the transmission costs of the corresponding seventh connecting edges, and uses the sum of the two candidate computation costs and the transmission costs of the corresponding seventh connecting edges as the fifth cost sum.

[0115] Specifically, if the fifth initial logical node N1 has m alternative decisions {0,1,2,…,m-1} and the sixth initial logical node N2 has n alternative decisions {0,1,2,…,n-1}, then, given any combination of candidate parallel schemes for the fifth initial logical node N1 and the sixth initial logical node N2, for example, a candidate parallel decision combination of (i,j) with a corresponding combination number k, where k = i*n + j, then the candidate computation cost of the fifth initial logical node N1 is Cost. N1 [i], the candidate computation cost of the sixth initial logical node N2 is Cost N2 [j], the transmission cost of the seventh connecting edge L12 is Cost L12 [i][j]. The sum of the computational cost of the parallel decision combination of the two and the transmission cost of the seventh connection edge is denoted as...

[0116]

[0117] Where k / n represents the quotient of k divided by n, and k%n represents the remainder of k divided by n. The subgraph cost calculation component 242 uses the sum of the two candidate transmission costs of the fifth initial logical node N1 and the sixth initial logical node N2, as well as the transmission cost of the seventh connecting edge, as the fifth cost sum. Other processing methods are exactly the same as those for the fourth predetermined configuration, and will not be described again here.

[0118] As described above, for a given computational graph or a subgraph thereof, candidate parallel decisions for each logical node are given, and the computational cost of each logical node under different parallel decisions and the transmission cost of each connection edge under different parallel decisions at its endpoints are calculated, along with given computational resources (e.g., the number of computing cards or devices). Assuming that each program segment within the computational subgraph runs simultaneously on each card or device, the time required to run the entire computational subgraph is approximately equal to the time it takes for each card to run all its programs. By amortizing the cost, when all logical nodes have decided on a parallel decision, the computational cost of each card and the transmission cost between cards are the same. Therefore, a final parallel decision is selected, which determines which parallel decision each logical node should take, such that the total cost on each card is minimized or reduced.

[0119] As described above, after the predetermined configuration transformation component 243 performs any transformation, it will arrange the transformed logical nodes as initial logical nodes after the traversed initial logical node queue. When the predetermined configuration acquisition component 241 has traversed all initial logical nodes twice and has not found any predetermined configuration, it will end the traversal and transformation operation and output the transformed logical node topology graph.

[0120] By transforming the four predetermined configurations as described above, the candidate space for parallel decision-making can be reduced to a great extent, and the topology of the computation graph will definitely be reduced. Therefore, the number of times parallel decision-making can be performed is finite. If M is the maximum value between the maximum number of candidate decisions for the initial logical node (the number of candidate parallel decisions for each initial logical node may be different) and the threshold O, then the results obtained after transforming the four predetermined configurations are shown in the table below:

[0121] Predetermined configuration Reduce solution space Reduce the number of nodes Reduce the number of connected edges Decision complexity F1 yes 1 1 item <![CDATA[O(M 3 )]]> F2 yes 1 1 item <![CDATA[O(M 2 )]]> F3 no 0 k items, k≥1 <![CDATA[O(kM 2 )]]> F4 no 1 k items, k≥1 <![CDATA[O((k+l)M 2 ) ]]>

[0122] In the table above, l represents the sum of the degrees of the two initial logical nodes N1 and N2 of the fourth predetermined structure F4.

[0123] By utilizing the preprocessing components of the parallel decision-making system for distributed data processing according to this disclosure, the solution space faced by parallel decision-making in distributed data processing can be minimized from a global perspective. This improves the feasibility and reduces the difficulty of automatic parallel decision-making. Furthermore, it enables the parallel results obtained from parallel decision-making to have lower computational and transmission costs, thereby maximizing the computational efficiency of fixed computing resources for the same computational task and accelerating data processing speed. More importantly, it achieves automation of parallel decision-making while minimizing transmission costs, significantly reducing the cost of manual debugging.

[0124] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that those skilled in the art will understand that all or any step or component of the methods and apparatus of this disclosure can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of this disclosure.

[0125] Therefore, the object of this disclosure can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of this disclosure can also be achieved simply by providing a program product containing program code implementing the method or apparatus. That is, such a program product also constitutes this disclosure, and the storage medium storing such a program product also constitutes this disclosure. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future.

[0126] It should also be noted that, in the apparatus and method of this disclosure, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of this disclosure. Furthermore, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order. Some steps can be performed in parallel or independently of each other.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / float>

Claims

1. A parallel decision-making system for distributed data processing, comprising: An initial logical node generation component receives task configuration data input by the user and generates an initial logical node topology graph for a distributed data processing system. Each initial logical node is associated with a set of candidate parallel schemes based on the task configuration data. Each candidate parallel scheme specifies the parallel scheme of the initial logical node to which it belongs and a candidate computation cost label based on the parallel scheme. Each connection edge between two interconnected initial logical nodes is associated with a candidate transmission cost label, which is determined by the respective parallel scheme of the interconnected initial logical nodes. The direct neighbor labeling component traverses the initial logical node topology graph to obtain the direct neighbor topology graph of a selected initial logical node in the initial logical node topology graph and labels the obtained direct neighbor topology graph one by one. The direct neighbor topology graph includes the selected initial logical node and other initial logical nodes that have direct connection edges with it. The sub-neighborhood cost calculation component, for each marked direct neighborhood topology, traverses and selects each sub-neighborhood topology in the marked direct neighborhood topology, and calculates the cost sum of the sub-neighborhood topology under each random combination of the candidate computation cost corresponding to the candidate parallel scheme of each initial logical node in the traversed and selected sub-neighborhood topology and the node cost sum of the candidate parallel scheme of each initial logical node in the traversed and selected sub-neighborhood topology under each random combination, and each of the traversed and selected sub-neighborhood topology contains the selected initial logical node in its direct neighborhood topology and a predetermined number of directly adjacent initial logical nodes; The sub-neighborhood parallel decision component calculates the cost and difference between the node costs of each initial logical node in the traversed and selected sub-neighborhood topology graph under all random combinations, and determines the sub-neighborhood parallel scheme of the traversed and selected sub-neighborhood topology graph based on the minimum combination cost of the initial logical node with the maximum cost and difference in the traversed and selected sub-neighborhood topology graph, and assigns the node cost of each initial logical node under the determined sub-neighborhood parallel scheme to the corresponding initial logical node. as well as The parallel decision termination component, after traversing the initial logical node topology multiple times, terminates the repeated traversal process if there are parallel schemes for each direct neighbor obtained in two consecutive traversals and the node cost assigned to each initial logical node remains unchanged, thereby obtaining the result logical node topology and thus obtaining the parallel scheme of the distributed data processing system.

2. The parallel decision-making system for distributed data processing according to claim 1, further comprising: The proxy node insertion component is used to traverse the initial logical node topology graph before the direct neighbor labeling component performs labeling. When a specific initial logical node has two or more downstream initial logical nodes, a temporary intermediate proxy node is inserted between the specific initial logical node and all its downstream initial logical nodes. The number of the temporary intermediate proxy nodes is assigned to a candidate parallel strategy that does not exceed the number of combinations of parallel strategy types of all downstream initial logical nodes, thereby obtaining an initial logical node topology graph containing the temporary intermediate proxy nodes.

3. The parallel decision system for distributed data processing according to claim 1 or 2, wherein the predetermined number of initial logical nodes directly adjacent to the selected initial logical node included in each sub-neighborhood topology graph traversed by the sub-neighborhood cost calculation component each time does not exceed 5.

4. The parallel decision-making system for distributed data processing according to claim 1 or 2 further includes a topology graph preprocessing component, used to preprocess the initial logical node topology graph before the direct neighborhood labeling component performs labeling, to obtain a preprocessed initial logical node topology graph, comprising: A predetermined configuration acquisition component traverses the initial logical node topology graph to obtain a predetermined configuration in the initial logical node topology graph. The predetermined configuration includes a first predetermined configuration and / or a second predetermined configuration, wherein the first predetermined configuration is an intermediate initial logical node with a first connecting edge and a second connecting edge, and the second predetermined configuration is a pair of initial logical nodes with multiple third connecting edges between them. as well as The subgraph cost calculation component, for a first predetermined configuration, when the candidate parallel schemes of the first initial logical node of the first connecting edge and the candidate parallel schemes of the second initial logical node of the second connecting edge are determined, obtains the candidate computation cost of each intermediate initial logical node, the candidate transmission cost of the first connecting edge corresponding to the obtained candidate computation cost, and the candidate transmission cost of the second connecting edge, and obtains the first cost sum of the three for each candidate computation cost of each intermediate initial logical node, and selects the minimum first cost sum as the first candidate transmission cost between the first initial logical node and the second initial logical node in the first predetermined configuration when the candidate parallel schemes of the first initial logical node and the candidate parallel schemes of the second initial logical node are determined for each pair of first initial logical nodes; And for the second predetermined configuration, when the candidate parallel schemes of the third initial logical node and the fourth initial logical node of the pair of initial logical nodes are determined, the candidate transmission costs of all connection edges between the pair of initial logical nodes are summed to obtain the second cost sum of the candidate transmission costs between the pair of initial logical nodes as the second candidate transmission cost. A predetermined configuration transformation component is used to transform a first connecting edge, a second connecting edge, and an intermediate initial logical node of a first predetermined configuration into a first merged connecting edge between the first initial logical node of the first connecting edge and the second initial logical node of the second connecting edge, and to assign all first candidate transmission costs calculated for the first predetermined configuration to the first merged connecting edge as one of the candidate transmission costs of the first merged connecting edge; and to transform all connecting edges of a second predetermined configuration into second merged connecting edges with paired initial logical nodes, and to assign the second candidate transmission costs calculated for the second predetermined configuration to the second merged connecting edges with paired initial logical nodes as one of the candidate transmission costs of the second merged connecting edge.

5. The parallel decision system for distributed data processing according to claim 4, wherein the predetermined configuration further includes a third predetermined configuration, the third predetermined configuration being an end initial logical node having only a fourth connection edge, wherein the subgraph cost calculation component, for the third predetermined configuration, when the end initial logical node is determined by the candidate parallel scheme dependent on the initial logical node through the fourth connection edge of the third predetermined configuration, obtains each candidate computation cost of the end initial logical node and the candidate transmission cost of the fourth connection edge corresponding to the candidate computation cost of the end initial logical node, and obtains the third cost sum of the two when the candidate parallel scheme dependent on the initial logical node is determined, and selects the minimum third cost sum as the third additional computation cost when the candidate parallel scheme dependent on the initial logical node is determined, and the predetermined configuration transformation component prunes the fourth connection edge and the end initial logical node of the third predetermined configuration, and adds the third additional computation cost to the computation cost dependent on the initial logical node.

6. The parallel decision system for distributed data processing according to claim 4, wherein the predetermined configuration further includes a fourth predetermined configuration, the fourth predetermined configuration including a fifth initial logical node and a sixth initial logical node whose product of the number of candidate parallel decisions located in the same connected component and having no connection edges between them does not exceed a given threshold, and at least one seventh initial logical node connected to the fifth initial logical node and the sixth initial logical node, wherein the subgraph cost calculation component, for the fourth predetermined configuration, when candidate parallel schemes for the fifth initial logical node and the sixth initial logical node are determined, obtains candidate computation costs for the fifth initial logical node and the sixth initial logical node, and uses the sum of the two candidate computation costs as the fourth cost sum, and in the case of the fifth initial logical node... Given the candidate parallel schemes for the sixth and seventh initial logical nodes, the candidate transmission costs of the fifth and sixth connection edges between the fifth and sixth initial logical nodes and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost. The predetermined configuration transformation component merges the fifth and sixth initial logical nodes of the fourth predetermined configuration into a first merged logical node, and merges the fifth and sixth connection edges into a third merged connection edge. The fourth cost calculated for the fourth predetermined configuration and assigned to the first merged logical node are taken as one of its candidate calculation costs, and the third candidate transmission cost is assigned to the third merged connection edge as one of its candidate transmission costs.

7. The parallel decision system for distributed data processing according to claim 4, wherein the predetermined configuration further includes a fifth predetermined configuration, the fifth predetermined configuration including a fifth initial logical node and a sixth initial logical node whose product of the number of candidate parallel decisions located in the same connected component and connected to each other by a seventh connection edge does not exceed a given threshold, and at least one seventh initial logical node connected to the fifth initial logical node and the sixth initial logical node, wherein the subgraph cost calculation component, for the fifth predetermined configuration, when the candidate parallel schemes of the fifth initial logical node and the sixth initial logical node are determined, obtains the candidate computation costs of the fifth initial logical node and the sixth initial logical node and the transmission cost of the connection edge between them, and takes the sum of the two candidate computation costs and the corresponding transmission cost of the seventh connection edge as the fifth initial logical node. The cost is calculated as follows: Given the candidate parallel schemes for the fifth, sixth, and seventh initial logical nodes, the candidate transmission costs of the fifth and sixth connection edges between the fifth and sixth initial logical nodes and the seventh initial logical node are obtained, and the sum of the two candidate transmission costs is taken as the third candidate transmission cost. The predetermined configuration transformation component merges the fifth and sixth initial logical nodes of the fifth predetermined configuration into a second merged logical node, and merges the fifth and sixth connection edges into a third merged connection edge. The fifth cost calculated for the fifth predetermined configuration is assigned to the second merged logical node as one of its candidate computation costs, and the third candidate transmission cost is assigned to the third merged connection edge as one of its candidate transmission costs.

8. A parallel decision-making method for distributed data processing, comprising: The initial logical node generation step involves receiving task configuration data input by the user and generating an initial logical node topology graph for the distributed data processing system. Each initial logical node is associated with a set of candidate parallel schemes based on the task configuration data. Each candidate parallel scheme specifies the parallel scheme of the initial logical node to which it belongs and a candidate computation cost label based on the parallel scheme. Each connection edge between two interconnected initial logical nodes is associated with a label of candidate transmission cost, which is determined by the respective parallel scheme of the interconnected initial logical nodes. The direct neighbor labeling step involves traversing the initial logical node topology graph to obtain the direct neighbor topology graph of a selected initial logical node in the initial logical node topology graph and labeling the obtained direct neighbor topology graph one by one. The direct neighbor topology graph includes the selected initial logical node and other initial logical nodes that have direct connection edges with it. The sub-neighborhood cost calculation step involves, for each marked direct neighbor topology, traversing and selecting each sub-neighbor topology in the marked direct neighbor topology, and calculating the cost sum of the sub-neighbor topology under each random combination of the candidate computation cost corresponding to the candidate parallel scheme of each initial logical node in the traversed and selected sub-neighbor topology and the node cost sum of the candidate parallel scheme of each initial logical node in the traversed and selected sub-neighbor topology under each random combination. Each of the traversed and selected sub-neighbor topology includes the selected initial logical node in its direct neighbor topology and a predetermined number of directly adjacent initial logical nodes. The sub-neighborhood parallel decision-making step calculates the cost and difference between the node costs of each initial logical node in the traversed and selected sub-neighborhood topology under all random combinations, and determines the sub-neighborhood parallel scheme of the traversed and selected sub-neighborhood topology based on the minimum combination cost of the initial logical node with the maximum cost and difference in the traversed and selected sub-neighborhood topology, and assigns the node cost of each initial logical node under the determined sub-neighborhood parallel scheme to the corresponding initial logical node. as well as The parallel decision termination step, after multiple traversals of the initial logical node topology graph, terminates the repeated traversal process if there are parallel schemes for each direct neighbor obtained by two consecutive traversals and the node cost assigned to each initial logical node remains unchanged, thereby obtaining the result logical node topology graph, and thus obtaining the parallel scheme of the distributed data processing system.

9. The parallel decision-making method for distributed data processing according to claim 8, further comprising: In the proxy node insertion step, before the direct neighbor labeling component performs labeling, the initial logical node topology graph is traversed. When a specific initial logical node has two or more downstream initial logical nodes, a temporary intermediate proxy node is inserted between the specific initial logical node and all its downstream initial logical nodes. The number of temporary intermediate proxy nodes is assigned to a candidate parallel strategy that does not exceed the number of combinations of parallel strategy types of all downstream initial logical nodes, thereby obtaining an initial logical node topology graph containing the temporary intermediate proxy nodes.

10. The parallel decision-making method for distributed data processing according to claim 8 or 9, wherein the predetermined number of initial logical nodes directly adjacent to the selected initial logical node is included in each sub-neighborhood topology graph traversed in each step of the sub-neighborhood cost calculation step does not exceed 5.

Images