A data processing method, apparatus and device

Abstract

The application provides a data processing method, device and equipment, wherein the data processing method comprises the following steps: determining a target tree height and a target width corresponding to each layer of the target tree height according to a target tree topology; constructing a tree with each processing node as a root according to the target tree height and the target width; equally dividing a total data block according to a total number of the processing nodes to obtain at least two sub-blocks; and performing full reduction processing on each of the sub-blocks by using each of the constructed trees. The scheme can support reducing the communication step number and the length of the dependent chain, thereby reducing the communication delay; can avoid the communication congestion caused by the multiple one by regulating the width, thereby improving the bandwidth utilization; and can well solve the problems of the large communication delay and the multiple one communication congestion in the full reduction data processing scheme in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, device and equipment. BACKGROUND

[0002] Allreduce is an operation of full synchronization among multiple processes and multiple machines, and is widely used in distributed systems and is an important link affecting the performance of distributed systems.

[0003] At present, commonly used multi-machine communication libraries include MPI, NCCL and the like, which contain the implementation of several allreduce methods. In actual application, they select a specific method for allreduce operation according to the number of nodes participating in synchronization and the size of data. However, the basis for selection is an empirical parameter, and the environmental differences of different clusters are ignored, which performs poorly in the current environment of high-speed iteration of hardware.

[0004] In addition, from each specific allreduce method, these methods have the following shortcomings:

[0005] 1) too many communication steps, and communication delay has too great influence on overall performance;

[0006] 2) only a small amount of data blocks are merged each time, and the memory read-write overhead is too large;

[0007] 3) there is network traffic of 'all-to-all', and bandwidth competition reduces network utilization.

[0008] Therefore, the data processing scheme for allreduce in the prior art has problems of large communication delay and one-to-many communication blocking. SUMMARY

[0009] The purpose of the present application is to provide a data processing method, device and equipment to solve the problems of large communication delay and one-to-many communication blocking in the prior art data processing scheme for allreduce.

[0010] In order to solve the above technical problems, the embodiment of the present application provides a data processing method, comprising:

[0011] determining a target tree height and a target width corresponding to each layer under the target tree height according to a target tree topology;

[0012] constructing a tree with each processing node as the root according to the target tree height and the target width;

[0013] equally dividing total data blocks according to the total number of processing nodes to obtain at least two sub-blocks;

[0014] Each of the constructed trees is used to perform a full reduction process on one of the sub-blocks.

[0015] Optionally, the step of performing full reduction processing on each of the constructed trees for one of the sub-blocks includes:

[0016] Based on the target width corresponding to the first layer, determine the grouping of processing nodes and perform the first merging process of sub-blocks;

[0017] Based on the first result of the first merging process, perform at least one first operation until each node completes the global merging of a sub-block;

[0018] Based on the second result corresponding to the global merge, reverse node broadcasting is performed;

[0019] The first operation includes: redetermining the grouping of processing nodes based on the target width corresponding to the current layer and the previous merging result, and performing sub-block merging processing again;

[0020] The previous merge result is the result of the merge process corresponding to the previous layer.

[0021] The tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is the adjacent layer of the previous layer;

[0022] The number of sub-blocks to be merged in the previous layer is greater than the number of sub-blocks to be merged in the current layer.

[0023] Optionally, before determining the target tree height and the target width corresponding to each layer under the target tree height based on the target tree topology, the method further includes:

[0024] Based on the total number, obtain each candidate tree topology;

[0025] Using the cost model, obtain the cost value corresponding to each candidate tree topology;

[0026] Based on the cost value, a target tree topology is determined from the candidate tree topologies;

[0027] The overhead model includes at least one of the following: memory read / write overhead factor and bandwidth contention factor.

[0028] Optionally, before obtaining each candidate tree topology based on the total number, the method further includes:

[0029] If the number of factors in the total quantity is less than a threshold value, the total quantity is split into a first quantity plus a first value; the first value is less than the first quantity.

[0030] The acquiring each candidate tree topology according to the total quantity comprises:

[0031] The acquiring each candidate tree topology according to the first quantity and the first value.

[0032] Optionally, the grouping of the processing nodes comprises:

[0033] Grouping the nodes corresponding to the first quantity, and dividing the nodes corresponding to the first value into the last group.

[0034] The last group is the last group in the groups of the nodes corresponding to the first quantity.

[0035] The embodiment of the application further provides a data processing device, comprising:

[0036] The first determining module is configured to determine a target tree height and target widths corresponding to each layer at the target tree height according to a target tree topology.

[0037] The first processing module is configured to construct trees with each processing node as a root according to the target tree height and the target widths.

[0038] The first dividing module is configured to divide total data blocks equally according to a total quantity of the processing nodes to obtain at least two sub-blocks.

[0039] The second processing module is configured to perform global reduction processing on each of the sub-blocks by using each of the constructed trees.

[0040] Optionally, the performing global reduction processing on each of the sub-blocks by using each of the constructed trees comprises:

[0041] Grouping the processing nodes according to the target width corresponding to the first layer, and performing first merging processing of the sub-blocks.

[0042] Performing at least one first operation according to a first result of the first merging processing until each node completes global merging of one sub-block.

[0043] Performing reverse node broadcasting according to a second result corresponding to the global merging.

[0044] The first operation comprises: re-determining the grouping of the processing nodes according to the target width corresponding to the current layer and a last merging result, and performing merging processing of the sub-blocks again.

[0045] The last merging result is a result of merging processing corresponding to the last layer.

[0046] The tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is a neighboring layer of the previous layer.

[0047] The number of sub-blocks of the merging processing corresponding to the previous layer is greater than the number of sub-blocks of the merging processing corresponding to the current layer.

[0048] Optionally, the method further comprises:

[0049] The first obtaining module is configured to, before determining the target tree height and the target width corresponding to each layer under the target tree height according to the target tree topology, obtain each candidate tree topology according to the total number.

[0050] The second obtaining module is configured to obtain the overhead value corresponding to each candidate tree topology by using an overhead model.

[0051] The second determining module is configured to determine the target tree topology from the candidate tree topologies according to the overhead value.

[0052] The overhead model comprises at least one of a memory read-write overhead factor and a bandwidth competition factor.

[0053] Optionally, the method further comprises:

[0054] The third processing module is configured to, before obtaining each candidate tree topology according to the total number, split the total number into a first number plus a first value in a case where the factor number of the total number is less than a threshold value; the first value is less than the first number.

[0055] The obtaining each candidate tree topology according to the total number comprises:

[0056] Obtaining each candidate tree topology according to the first number and the first value.

[0057] Optionally, the determining the grouping of processing nodes comprises:

[0058] Grouping the nodes corresponding to the first number, and dividing the nodes corresponding to the first value into the last group.

[0059] The last group is the last group in the grouping of the nodes corresponding to the first number.

[0060] Embodiments of the present application also provide a data processing device, comprising a processor and a transceiver.

[0061] The processor is configured to determine a target tree height and a target width corresponding to each layer under the target tree height according to a target tree topology.

[0062] constructing a tree with each processing node as a root according to the target tree height and the target width;

[0063] equally dividing the total data block according to the total number of the processing nodes to obtain at least two sub-blocks;

[0064] performing full reduction processing on each of the sub-blocks by using the constructed trees respectively.

[0065] Optionally, the full reduction processing on each of the sub-blocks by using the constructed trees respectively comprises:

[0066] determining the grouping of the processing nodes according to the target width corresponding to the first layer, and performing first merging processing of the sub-blocks;

[0067] performing at least one first operation according to a first result of the first merging processing until each node completes global merging of one sub-block;

[0068] performing reverse node broadcasting according to a second result corresponding to the global merging;

[0069] The first operation comprises: re-determining the grouping of the processing nodes according to the target width corresponding to the current layer and a last merging result, and performing merging processing of the sub-blocks again.

[0070] The last merging result is a result of merging processing corresponding to a last layer;

[0071] The tree height corresponding to the current layer is greater than the tree height corresponding to the last layer, and the current layer is a neighboring layer of the last layer.

[0072] The number of the sub-blocks of the merging processing corresponding to the last layer is greater than the number of the sub-blocks of the merging processing corresponding to the current layer.

[0073] Optionally, the processor is further configured to:

[0074] before determining the target tree height and the target width corresponding to each layer under the target tree height according to the target tree topology, obtaining each candidate tree topology according to the total number;

[0075] obtaining an overhead value corresponding to each candidate tree topology by using an overhead model;

[0076] determining the target tree topology from the candidate tree topologies according to the overhead value;

[0077] The overhead model comprises at least one of a memory read-write overhead factor and a bandwidth competition factor.

[0078] Optionally, the processor is further configured to:

[0079] Before obtaining each candidate tree topology according to the total number, in a case that a factor number of the total number is less than a threshold value, the total number is split into a first number plus a first value; the first value is less than the first number;

[0080] The obtaining each candidate tree topology according to the total number comprises:

[0081] Obtaining each candidate tree topology according to the first number and the first value.

[0082] Optionally, the determining the grouping of the processing nodes comprises:

[0083] Grouping the nodes corresponding to the first number, and dividing the nodes corresponding to the first value into the last group;

[0084] The last group is the last group in the grouping of the nodes corresponding to the first number.

[0085] Embodiments of the present application also provide a data processing device, comprising a memory, a processor and a program stored in the memory and executable on the processor; the processor implements the data processing method when executing the program.

[0086] Embodiments of the present application also provide a readable storage medium, which stores a program, and the program is executable on a processor to implement the steps of the data processing method.

[0087] The above technical solutions of the present application have the following advantages:

[0088] In the above scheme, the data processing method determines a target tree height according to a target tree topology, and a target width corresponding to each layer under the target tree height; constructs a tree with each processing node as a root according to the target tree height and the target width; divides total data blocks equally according to the total number of the processing nodes to obtain at least two sub-blocks; and performs full reduction processing on one of the sub-blocks by using each of the constructed trees. The method can support reducing the number of communication steps and the length of the dependent chain, thereby reducing communication delay; can also avoid communication congestion caused by multiple hits by regulating the width, thereby improving bandwidth utilization; and solves the problems of large communication delay and multiple hits in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 The figure is a flowchart of the data processing method of the embodiments of the present application;

[0090] Figure 2 The figure is a schematic diagram of constructing a tree with six nodes according to the embodiments of the present application;

[0091] Figure 3 A schematic diagram of constructing a tree for six nodes of an embodiment of the present application and completing full reduction;

[0092] Figure 4 A schematic diagram of constructing a tree for seven nodes of an embodiment of the present application;

[0093] Figure 5 A schematic diagram of constructing a tree for seven nodes of an embodiment of the present application and completing full reduction;

[0094] Figure 6 A schematic diagram of the structure of a data processing device of an embodiment of the present application;

[0095] Figure 7 A schematic diagram of the structure of a data processing device of an embodiment of the present application. DETAILED DESCRIPTION

[0096] To make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the accompanying drawings.

[0097] The present application aims at the problems of large communication delay and one-to-many communication blocking in the prior art data processing scheme for full reduction, and provides a data processing method, as shown in Figure 1 The method comprises the following steps:

[0098] Step 11: determining a target tree height according to a target tree topology, and a target width corresponding to each layer under the target tree height;

[0099] Step 12: constructing a tree with each processing node as a root according to the target tree height and the target width;

[0100] Step 13: equally dividing total data blocks according to the total number of the processing nodes to obtain at least two sub-blocks;

[0101] Step 14: performing full reduction processing on one of the sub-blocks by using each constructed tree.

[0102] The data processing method provided by the embodiment of the application can support reducing the communication step number and the length of the dependent chain, thereby reducing the communication delay, and can also avoid the communication congestion caused by the multiple transmission by regulating the width, thereby improving the bandwidth utilization, and can well solve the problems of large communication delay and multiple transmission communication congestion in the prior art.

[0103] The full reduction processing of each tree on one sub-block comprises: determining the grouping of the processing nodes according to the target width corresponding to the first layer, and performing first merging processing of the sub-block; performing at least one first operation according to the first result of the first merging processing until each node completes the global merging of one sub-block; and performing reverse node broadcasting according to the second result corresponding to the global merging; wherein the first operation comprises: re-determining the grouping of the processing nodes according to the target width corresponding to the current layer and the previous merging result, and performing merging processing of the sub-block again; the previous merging result is the result of the merging processing corresponding to the previous layer; the tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is the adjacent layer of the previous layer; and the number of sub-blocks of the merging processing corresponding to the previous layer is greater than the number of sub-blocks of the merging processing corresponding to the current layer.

[0104] The number of groupings of each layer can be equal to the total number of nodes divided by the width of the layer; the reverse node broadcasting can be understood as: feeding back the global merging result of the sub-block obtained by the node to the node (which can be specifically the node that has sent the sub-block data before) that performs communication; in addition, the number of sub-blocks merged by each layer of nodes can be equal to the total number of nodes divided by the product of the target width corresponding to the current layer and the target width corresponding to each layer before the current layer; specifically, when the number of layers is greater than 3, the number of sub-blocks can be equal to the total number of nodes divided by (the target width corresponding to the current layer multiplied by the target width corresponding to the previous layer multiplied by the target width of the first layer); when the number of layers is equal to 2, the number of sub-blocks can be equal to the total number of nodes divided by (the target width corresponding to the current layer multiplied by the target width of the first layer); and when the number of layers is equal to 3, the number of sub-blocks can be equal to the total number of nodes divided by (the target width corresponding to the current layer multiplied by the target width corresponding to the previous layer multiplied by the target width of the first layer), but the number of sub-blocks is not limited to this.

[0105] Further, before determining the target tree height according to the target tree topology, and the target width corresponding to each layer under the target tree height, the method further comprises: obtaining each candidate tree topology according to the total number; obtaining an overhead value corresponding to each candidate tree topology by using an overhead model; and determining the target tree topology from the candidate tree topologies according to the overhead value; wherein the overhead model comprises at least one of a memory read-write overhead factor and a bandwidth competition factor.

[0106] In this way, the optimal logical topology can be ensured. In the embodiment of the present application, the overhead model can further comprise at least one of a delay overhead factor, a bandwidth overhead factor and a calculation overhead factor, which are not limited herein.

[0107] Further, before obtaining each candidate tree topology according to the total number, the method further comprises: splitting the total number into a first number and a first value in a case that the number of factors of the total number is less than a threshold value; the first value is less than the first number; and the obtaining each candidate tree topology according to the total number comprises: obtaining each candidate tree topology according to the first number and the first value.

[0108] In this way, the stability of the overall performance can be maintained at the cost of a small amount of time consumption, and the universality of the method can be ensured.

[0109] The determining the grouping of processing nodes comprises: grouping the nodes corresponding to the first number, and dividing the nodes corresponding to the first value into the last group; wherein the last group is the last group in the groups of the nodes corresponding to the first number.

[0110] In this way, an incomplete symmetric tree can be constructed to ensure the stability of the overall performance and the universality of the method.

[0111] The "redetermining the grouping of processing nodes" can adopt similar specific operations of the "determining the grouping of processing nodes" herein, but is not limited herein.

[0112] The data processing method provided by the embodiment of the present application will be described below. The following processing nodes are referred to as nodes.

[0113] To solve the above technical problems, the embodiment of the present application provides a data processing method, which can be specifically implemented as a universal full-reduction logical topology generation and optimization method, and relates to a full-reduction and automatic logical topology generation optimization based on a tree topology. The method is universal for most cluster environments, can select the optimal logical topology according to the specific environment, and can improve the full-reduction performance. The scheme provided by the embodiment of the present application mainly involves the following three steps:

[0114] 1. For a new real deployment environment, a one-time automated test is performed to collect environment information (corresponding to the following test script by automation, parameters in the Cost Model can be determined) to form a Cost Model (overhead model) for evaluating the performance of logical topology.

[0115] 2. According to the total number of nodes (corresponding to the above total number), all possible logical topologies (corresponding to the above candidate tree topologies) are generated, and the best one is selected according to the Cost Model (corresponding to the target tree topology). According to the total number of nodes, the candidate tree topologies are obtained; the overhead values corresponding to the candidate tree topologies are obtained by using the overhead model; and the target tree topology is determined from the candidate tree topologies according to the overhead values.

[0116] 3. The full reduction process is completed using the optimal performance topology.

[0117] The following is a specific example of logical topology selection and full reduction process.

[0118] I. Regarding the selection of logical topology.

[0119] In addition to the communication delay (i.e. delay overhead factor), communication overhead (i.e. bandwidth overhead factor) and computation overhead (factor) in the Cost Model in this scheme, memory read-write overhead factor and bandwidth competition factor are also added to depict the differences between different logical topologies of the same node. The following formulas can be used:

[0120]

[0121] where 2hα represents the delay overhead factor, represents the bandwidth overhead factor, represents the computation overhead factor, represents the memory read-write overhead factor, represents the bandwidth competition factor.

[0122] h represents the height of the constructed tree (i.e. total tree height, total number of layers), N represents the total number of nodes, S represents the total data amount (i.e. total data block), i represents the i-th layer, j represents the j-th layer, Π represents the continuous product symbol, w j represents the width of the j-th layer, w i represents the width of the i-th layer, w t represents a fixed threshold. α, β, γ, δ and ε represent calculation coefficients, which can correspond to the above environment information. In the above formulas, h can be greater than or equal to 1, N can be greater than or equal to 2, w i can be greater than or equal to 2, w tmay be greater than or equal to 2, and the other parameters can be numbers greater than 0 respectively.

[0123] The memory read-write overhead factor is used to quantify all memory read-write time consumption involved in the full-reduction process, and the bandwidth competition factor is used to reflect the factor of bandwidth utilization rate reduction caused by multi-shot communication. In an actual application scenario, the parameters in the Cost Model (corresponding to environmental information, which can include hardware environmental information, software environmental information, etc.) can be determined through an automated test script; thereafter, only two independent variables, the total number of nodes N and the total data amount S, need to be substituted, so as to select the optimal full-reduction topology (specifically, the overhead cost can be obtained by using the overhead model, and the one with the minimum overhead cost is selected).

[0124] II. Regarding the full-reduction process.

[0125] The full-reduction method in the present solution is based on tree topology. Assuming that there are N nodes in total, the height of the constructed tree is h, and the width from bottom to top is w1, w2, w3, …, w h (w1 to w h The product of which is equal to N). N trees are constructed with N nodes as roots respectively (corresponding to the above-mentioned construction of trees with each processing node as the root); meanwhile, the total data block is equally divided into N blocks (corresponding to the above-mentioned at least two sub-blocks), and each tree is responsible for the full-reduction of one block (corresponding to the above-mentioned full-reduction processing of each constructed tree on one of the sub-blocks).

[0126] Taking the construction of a 3x2 tree with 6 nodes as an example, Figure 2 the structure of the six trees generated is shown Figure 2 Six 3x2 trees constructed with six nodes, and their equivalent Full-Mesh (complete graph, full mesh) grouping, are shown, wherein:

[0127] Each layer of the tree represents a communication stage, and the data block is transmitted along the edge (i.e., the connection line between nodes). In fact, the communication stage corresponding to this layer with a width of w i will divide the N nodes into N / w i groups, each with w i nodes, and Full-Mesh communication is performed within the group. Similarly, taking 6 nodes as an example, Figure 3 the nodes are numbered 1-6, and the data block is equally divided into 6 parts, numbered a-f. Corresponding to the above-mentioned determination of the target tree height according to the target tree topology, and the target width corresponding to each layer under the target tree height; construction of a tree with each processing node as the root according to the target tree height and the target width; equal division of the total data block to obtain at least two sub-blocks according to the total number of processing nodes; subsequent full-reduction processing of each constructed tree on one of the sub-blocks can be performed, specifically as follows:Figure 3 As shown, 6 nodes construct a 3x2 tree, and complete the Reduce step, including:

[0128] 1. In the first stage (i.e. the first layer), w1=3, the nodes are divided into N / w1=2 groups, and without loss of generality, let 1~3 be a group (the grouping can also be out of order), and 4~6 be a group. (1~3)(a~b) will be collected and merged on node 1, (1~3)(c~d) will be collected and merged on node 2, and (1~3)(e~f) will be collected and merged on node 3; (4~6)(a~b) will be collected and merged on node 4, (4~6)(c~d) will be collected and merged on node 5, and (4~6)(e~f) will be collected and merged on node 6.

[0129] This step corresponds to the above determination of the grouping of the processing nodes according to the target width corresponding to the first layer, and the first merging processing of the sub-blocks is performed.

[0130] 2. In the second stage, w2=2, the nodes are divided into N / w2=3 groups. The grouping at this time will be generated based on the first stage, and those with the same sub-block number will be divided into a group, i.e. 1, 4 a group, 2, 5 a group, and 3, 6 a group. (1~6)a will be collected and merged on node 1, (1~6)c will be collected and merged on node 2, (1~6)e will be collected and merged on node 3, (1~6)b will be collected and merged on node 4, (1~6)d will be collected and merged on node 5, and (1~6)f will be collected and merged on node 6. It can be seen that at this time, each node has completed the global merging of a sub-block.

[0131] This step corresponds to the above execution of at least one first operation until each node respectively completes the global merging of a sub-block according to the first result of the first merging processing.

[0132] 3. In the third stage and the fourth stage, according to the step opposite to the previous collection, broadcasting can be performed (for example, node 1 returns the global data a after reduction to node 4; also to node 2, etc.).

[0133] This step corresponds to the above reverse node broadcasting according to the second result corresponding to the global merging.

[0134] It is explained that the above numbers 1~6 represent nodes 1~6, and the like; the letters a~b represent sub-blocks a~b obtained by dividing data blocks, and the like; (1~3)(a~b) represents sub-blocks a~b of nodes 1~3 (i.e. sub-blocks a~b of node 1, sub-blocks a~b of node 2, and sub-blocks a~b of node 3), and the like.

[0135] As to the process of the above global reduction, it can be understood that the nodes are grouped and merged according to the width corresponding to the layer; then grouped and merged; until each sub-block is grouped on a node, and global merging is realized.

[0136] In addition, as to the merging of the above first layer, there is no specific limitation, and it can be preferred to be merged in order, i.e. the above (a~b) is merged into 1, or it can not be merged in order, such as a and c are merged into 1, which is not limited here; each subsequent layer is processed based on the result of the previous layer; wherein a node, the number of merged blocks in the i-th layer is equal to N / X, X is equal to the product of the width from the first layer to the width of the i-th layer, such as i is equal to 2, then X = w1 x w2, and such as i is equal to 3, then X = w1 x w2 x w3.

[0137] In addition, the number of groups divided by each layer of nodes can be determined by the width of the layer.

[0138] Further, in general, the tree width w i N can be taken as a factor. But if the factor of N is too small (corresponding to the case that the number of factors of the total number is less than the threshold value), the method of +n can also be used to split N into N = N' + n, where n is a small number (corresponding to the above first value), and N' is a number that is convenient for factor splitting (corresponding to the above first number). Based on the splitting of N', a tree that is not completely symmetrical (included in the above according to the first number and the first value, obtaining each candidate tree topology) can be constructed, which can improve the stability of the overall performance at the cost of a small amount of time consumption, and ensure the universality of the method. Taking 7 nodes as an example (i.e. the total number of nodes is taken as 7), it is split into 7 = 3 x 2 + 1, as shown in Figure 4 , (such as Figure 4 only two trees are given as examples, and other similar ones) ; at this time, the data block is equally divided into 7 parts, denoted as a~g. Corresponding to the above according to the target tree topology, the target tree height is determined, and the target width corresponding to each layer under the target tree height is determined; according to the target tree height and the target width, a tree is constructed with each processing node as the root; according to the total number of processing nodes, the total data block is equally divided to obtain at least two sub-blocks; subsequently, each constructed tree is used to process a sub-block respectively, and the specific method can be as shown in Figure 5 , the method of completing global reduction of 7 nodes is divided into four steps (i.e. stages) in total:

[0139] 1. In the first stage (i.e. the first layer), the nodes are divided into N / w1=2 groups, without loss of generality, let 1~3 be a group, 4~6 be a group, and node 7 is attached to the last group for communication (i.e. node 7 belongs to the last group, which corresponds to the above grouping of the nodes corresponding to the first number, and the nodes corresponding to the first number are divided into the last group). (1~3)(a~b) will be collected and merged into node 1, (1~3)(c~d) will be collected and merged into node 2, (1~3)(e~g) will be collected and merged into node 3; (4~7)(a~b) will be collected and merged into node 4, (4~7)(c~d) will be collected and merged into node 5, (4~7)(e~f) will be collected and merged into node 6, (4~7)(g) will be collected and merged into node 7.

[0140] This step corresponds to the above determination of the grouping of the processing nodes according to the target width corresponding to the first layer, and the first merging processing of the sub-blocks is performed.

[0141] 2. In the second stage, the nodes are divided into N / w2=3 groups. At this time, the grouping is referenced to the example of 6 nodes, which is the same as 6=3x2, i.e. 1, 4 a group, 2, 5 a group, 3, 6 a group, and node 7 will be attached to the last group (which corresponds to the above grouping of the nodes corresponding to the first number, and the nodes corresponding to the first number are divided into the last group). (1~7)a will be collected and merged into node 1, (1~7)c will be collected and merged into node 2, (1~7)e will be collected and merged into node 3, (1~7)b will be collected and merged into node 4, (1~7)d will be collected and merged into node 5, (1~7)f will be collected and merged into node 6, (1~7)g will be collected and merged into node 7. It can be seen that at this time each node has completed a global merging of a sub-block.

[0142] This step corresponds to the above execution of at least one first operation until each node respectively completes a global merging of a sub-block according to the first result of the first merging processing.

[0143] 3. In the third stage and the fourth stage, the steps are opposite to the previous collection, and broadcasting is sufficient (for example, node 1 returns the reduced global data a to node 4; also to node 2, etc.).

[0144] The above numbers 1~7 represent nodes 1~7, and the like; the letters a~b represent sub-blocks a~b obtained by dividing the data block, and the like; (1~3)(a~b) represents sub-blocks a~b of nodes 1~3 (i.e. sub-blocks a~b of node 1, sub-blocks a~b of node 2, and sub-blocks a~b of node 3), and the like.

[0145] As to the process of the above global reduction, it can be understood that the nodes are grouped and merged according to the width corresponding to the layer; then grouping and merging are performed; and this is continued until each sub-block is grouped on a node to achieve global merging.

[0146] In addition, as to the merging of the above first layer, there is no specific limitation, and it can be preferred to merge in order, that is, the above (a~b) is merged into 1, or it can not be merged in order, such as a and c are merged into 1, which is not limited here; each subsequent layer is processed based on the result of the previous layer; wherein, usually, the number of merging blocks of a node at the i-th layer is equal to N / X, X is equal to the product of the width from the first layer to the width of the i-th layer, such as i is equal to 2, X = w1 x w2, and such as i is equal to 3, X = w1 x w2 x w3. However, in the example of 7 nodes, in the non-final stage (such as the first stage above), the number of merging blocks of the last node in the non-final group needs to be increased by 1 on this basis, and the number of merging blocks of the last node in the last group does not apply to the formula, but is directly equal to 1; in the last stage (such as the second stage above), the number of merging blocks of all nodes is equal to 1.

[0147] In addition, the number of groups divided by each layer of nodes can be determined by the width of the layer.

[0148] It is explained here that the example of 7 nodes can refer to the related content of the example of 6 nodes, and similar content will not be repeated here. In addition, Figure 3 and Figure 5 In "number Y (letters z1~zn)", the sub-blocks z1~zn of node Y are represented, such as 2(a~b) represents the sub-blocks a~b of node 2; and in "number Y (letter z)", the sub-block z of node Y is represented, such as 5(g) represents the sub-block g of node 5.

[0149] The above content, through the universal definition of the tree-shaped global reduction logical topology, the irregular expansion design of the tree-shaped logical topology, the automatic detection of the actual physical environment, and the accurate modeling description of the physical environment, enables efficient completion of global reduction operation on most clusters, without being limited by the number of nodes, bandwidth, delay, and other physical parameters of the cluster, and has good universality.

[0150] As to the application scenarios of the present scheme:

[0151] At present, an important application of distributed systems is distributed machine learning. Data parallelism is an important distributed method, and global reduction is a main scheme to support data parallelism, which has a crucial influence on the performance of distributed machine learning. Taking a common OpenMPI application as an example:

[0152] 1. Run the real machine test to automatically parameterize the Cost Model.

[0153] 2. Modify the existing Cost Model used in OpenMPI when choosing specific all-reduce method (i.e. use the above overhead model provided in the present scheme), and add the tree all-reduce method proposed in the present scheme, and recompile OpenMPI.

[0154] 3. Train the model using OpenMPI.

[0155] In this way, the performance optimization can be completed without modifying the training code, and user transparency is achieved.

[0156] From the above, the scheme provided by the embodiments of the present application mainly relates to: a tree topology-based all-reduce method and a new Cost Model-based automatic logical topology generation optimization; can support any number of nodes, reduce memory read-write overhead by adjusting tree width (i.e. the above width), and avoid bandwidth competition caused by multiple hits (i.e. avoid multiple hits blocking). The present scheme can be specifically implemented as a tree topology-based all-reduce method that can form an efficient all-reduce for any given number of nodes, and an automatic logical topology optimization method.

[0157] In summary, the present scheme proposes a tree topology-based all-reduce method and an automatic logical topology generation optimization method (also referred to as a general all-reduce logical topology generation method and a new Cost Model-based automatic logical topology optimization method). Compared with the existing scheme, the present scheme has the following advantages:

[0158] 1. Reduces the number of communication steps and the length of the dependency chain, thereby reducing the impact of communication delay.

[0159] 2. Reduces the amount of memory read and write, thereby reducing the memory read and write overhead.

[0160] 3. Relieves the network bandwidth contention problem caused by multiple hits.

[0161] 4. For any number of nodes, an efficient all-reduce logical topology can be formed.

[0162] 5. In any environment, the bandwidth and memory impact are considered comprehensively, and the logical topology is automatically optimized.

[0163] In addition, it is explained herein that the "one-to-many communication" involved in the embodiments of the present application refers to that in a network, multiple nodes simultaneously send data to one node. In a unit of time, the data simultaneously transmitted by the multiple sending nodes will be much larger than the data that the receiving node can receive. This can result in (1) packet loss and retransmission caused by packet loss; (2) in order to avoid packet loss, some networks enable a flow control or congestion control mechanism, but still can cause the link to be suspended at some time. In general, this will result in a decrease in the utilization of bandwidth, thereby resulting in a decrease in the performance of the all-reduce.

[0164] The embodiments of the present application also provide a data processing apparatus, as shown in the accompanying drawings, comprising: Figure 6

[0165] The first determining module 61 is used for determining a target tree height according to a target tree topology, and a target width corresponding to each layer under the target tree height;

[0166] The first processing module 62 is used for constructing a tree with each processing node as a root according to the target tree height and the target width;

[0167] The first dividing module 63 is used for equally dividing total data blocks according to the total number of the processing nodes to obtain at least two sub-blocks;

[0168] The second processing module 64 is used for performing all-reduce processing on one of the sub-blocks by using each of the constructed trees.

[0169] The data processing apparatus provided by the embodiments of the present application can support reducing the number of communication steps and the length of a dependent chain, thereby reducing the communication delay, and can also avoid the communication blockage caused by one-to-many communication by regulating the width, thereby improving the bandwidth utilization, and thus solves the problems of large communication delay and one-to-many communication blockage in the prior art.

[0170] ​The step of performing full reduction processing on a sub-block using each constructed tree includes: determining the grouping of processing nodes based on the target width corresponding to the first layer, and performing a first merging process on the sub-blocks; performing at least one first operation based on the first result of the first merging process, until each node completes the global merging of a sub-block; and performing reverse node broadcasting based on the second result corresponding to the global merging. The first operation includes: re-determining the grouping of processing nodes based on the target width corresponding to the current layer and the previous merging result, and performing the sub-block merging process again; the previous merging result is the result of the merging process corresponding to the previous layer; the tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is an adjacent layer to the previous layer; the number of sub-blocks merged by the previous layer is greater than the number of sub-blocks merged by the current layer.

[0171] Furthermore, the data processing device further includes: a first acquisition module, configured to acquire each candidate tree topology based on the total number before determining the target tree height and the target width corresponding to each layer under the target tree height according to the target tree topology; a second acquisition module, configured to acquire the overhead value corresponding to each candidate tree topology using an overhead model; and a second determination module, configured to determine the target tree topology from the candidate tree topologies based on the overhead value; wherein the overhead model includes at least one of a memory read / write overhead factor and a bandwidth contention factor.

[0172] Furthermore, the data processing device further includes: a third processing module, configured to, before obtaining each candidate tree topology based on the total quantity, split the total quantity into a first quantity plus a first value if the number of factors in the total quantity is less than a threshold value; the first value is less than the first quantity; obtaining each candidate tree topology based on the total quantity includes: obtaining each candidate tree topology based on the first quantity and the first value.

[0173] In this embodiment of the invention, determining the grouping of processing nodes includes: grouping the nodes corresponding to the first quantity, and assigning the nodes corresponding to the first value to the last group; wherein, the last group is the last group among the groups of nodes corresponding to the first quantity.

[0174] The implementation embodiments of the above data processing method are all applicable to the embodiments of the data processing device and can achieve the same technical effect.

[0175] This invention also provides a data processing device, such as... Figure 7 As shown, it includes: a processor 71 and a transceiver 72;

[0176] The processor 71 is configured to determine a target tree height according to a target tree topology, and a target width corresponding to each layer of the target tree height;

[0177] According to the target tree height and the target width, a tree is constructed with each processing node as a root;

[0178] According to a total number of the processing nodes, a total data block is equally divided to obtain at least two sub-blocks;

[0179] Each of the constructed trees is used to perform a full reduction processing on one of the sub-blocks.

[0180] In the embodiment of the application, the transceiver 72 can communicate with the processor 71, which is not limited herein.

[0181] The data processing device provided by the embodiment of the application can determine a target tree height according to a target tree topology, and a target width corresponding to each layer of the target tree height; construct a tree with each processing node as a root according to the target tree height and the target width; equally divide a total data block according to a total number of the processing nodes to obtain at least two sub-blocks; and use each of the constructed trees to perform a full reduction processing on one of the sub-blocks. The data processing device can support reducing the number of communication steps and the length of a dependent chain, thereby reducing communication delay. The data processing device can also avoid communication congestion caused by multiple punches by regulating the width, thereby improving bandwidth utilization. The data processing device can solve the problems of large communication delay and multiple punch communication congestion in the prior art.

[0182] The full reduction processing on each of the constructed trees includes: determining the grouping of the processing nodes according to the target width corresponding to the first layer, and performing a first merging processing of the sub-blocks; performing at least one first operation according to a first result of the first merging processing until each node completes a global merging of one sub-block; and performing a reverse node broadcast according to a second result corresponding to the global merging; the first operation includes: re-determining the grouping of the processing nodes according to the target width corresponding to the current layer and a previous merging result, and performing a merging processing of the sub-blocks again; the previous merging result is a result of a merging processing corresponding to a previous layer; the tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is an adjacent layer of the previous layer; the number of sub-blocks of the merging processing corresponding to the previous layer is greater than the number of sub-blocks of the merging processing corresponding to the current layer.

[0183] Further, the processor is further configured to: before determining the target tree height according to the target tree topology and determining the target width corresponding to each layer of the target tree height, acquire each candidate tree topology according to the total number; acquire an overhead value corresponding to each candidate tree topology by using an overhead model; and determine the target tree topology from the candidate tree topologies according to the overhead value; wherein the overhead model comprises at least one of a memory read-write overhead factor and a bandwidth competition factor.

[0184] Further, the processor is further configured to: before acquiring each candidate tree topology according to the total number, split the total number into a first number and a first value in a case where a factor number of the total number is less than a threshold value; the first value is less than the first number; and the acquiring each candidate tree topology according to the total number comprises acquiring each candidate tree topology according to the first number and the first value.

[0185] In the embodiment of the application, the grouping of the processing nodes comprises: grouping the nodes corresponding to the first number, and dividing the nodes corresponding to the first value into the last group; wherein the last group is the last group in the groups of the nodes corresponding to the first number.

[0186] The implementation embodiments of the data processing method described above are applicable to the implementation embodiments of the data processing device, and the same technical effects can be achieved.

[0187] The embodiment of the application further provides a data processing device, which comprises a memory, a processor and a program stored in the memory and executable on the processor; and the processor implements the data processing method described above when executing the program.

[0188] The implementation embodiments of the data processing method described above are applicable to the implementation embodiments of the data processing device, and the same technical effects can be achieved.

[0189] The embodiment of the application further provides a readable storage medium, which stores a program; and the program is executed by a processor to implement the steps of the data processing method described above.

[0190] The implementation embodiments of the data processing method described above are applicable to the implementation embodiments of the readable storage medium, and the same technical effects can be achieved.

[0191] It should be noted that many functional components described in the specification are referred to as modules, so as to more particularly emphasize the independence of their implementation.

[0192] In the embodiments of the present application, the modules can be implemented in software, and executed by various types of processors. For example, an identified module of executable code can consist of one or more physical or logical blocks of computer instructions. For example, a module can be implemented in object-oriented programming, such as a Java® class, or in procedural programming, such as a C function. However, the executable code of an identified module need not be physically located together, but can be distributed in various places of a computer memory, and across multiple storage devices. Similarly, operational data can be identified within modules and can be

[0193] Indeed, an executable code module can be a single instruction, or many instructions, and can even be distributed over several different code segments, among different programs, and across several memory devices. Also, operational data can be identified within modules and can be

[0194] When a module is implemented in software, the module can be stored in any desired manner, and in any desired computer readable medium, including a storage device, such as a hard disk, floppy disk, or compact disk, a memory, such as a random access memory (RAM), a program cartridge, a cassette, or a cartridge, or any other medium from which a computer can read. Similarly, the software implementing a module can be distributed over the internet, computer

[0195] The preferred embodiments of the present application have been described above with the aid of numerous reference to drawings of various embodiments of the application. These embodiments are illustrative only, and changes and modifications can be suggested to one skilled in the art, without departing from the spirit and scope of the application.

Claims

1. A data processing method, characterized in that, include: Based on the target tree topology, determine the target tree height and the target width corresponding to each layer under the target tree height; Based on the target tree height and target width, construct a tree with each processing node as the root; Based on the total number of processing nodes, the total data block is divided into equal parts to obtain at least two sub-blocks; Each of the constructed trees is used to perform a full reduction process on one of the sub-blocks; Before determining the target tree height and the target width corresponding to each layer under the target tree height based on the target tree topology, the process further includes: Based on the total number, obtain each candidate tree topology; Using the cost model, obtain the cost value corresponding to each candidate tree topology; Based on the cost value, a target tree topology is determined from the candidate tree topologies; The overhead model includes at least one of the following: memory read / write overhead factor, latency overhead factor, bandwidth contention factor, and computation overhead factor; the memory read / write overhead factor is used to quantify the memory read / write time involved in the full reduction process, and the bandwidth contention factor is used to reflect the factor of bandwidth utilization reduction caused by many-to-one communication. The process of performing full reduction on a sub-block using each constructed tree includes: Based on the target width corresponding to the first layer, determine the grouping of processing nodes and perform the first merging process of sub-blocks; Based on the first result of the first merging process, perform at least one first operation until each node completes the global merging of a sub-block; Based on the second result corresponding to the global merge, reverse node broadcasting is performed; The first operation includes: redetermining the grouping of processing nodes based on the target width corresponding to the current layer and the previous merging result, and performing sub-block merging processing again; The previous merge result is the result of the merge process corresponding to the previous layer. The tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is the adjacent layer of the previous layer; The number of sub-blocks to be merged in the previous layer is greater than the number of sub-blocks to be merged in the current layer.

2. The data processing method according to claim 1, characterized in that, Before obtaining each candidate tree topology based on the total number, the process also includes: If the number of factors in the total quantity is less than a threshold value, the total quantity is split into a first quantity plus a first value; the first value is less than the first quantity. The step of obtaining each candidate tree topology based on the total number includes: Based on the first quantity and the first value, obtain each candidate tree topology.

3. The data processing method according to claim 2, characterized in that, Determine the grouping of processing nodes, including: Group the nodes corresponding to the first quantity, and assign the nodes corresponding to the first value to the last group; The last group is the last group among the groups of nodes corresponding to the first quantity.

4. A data processing apparatus, characterized in that, include: The first determining module is used to determine the target tree height and the target width corresponding to each layer under the target tree height based on the target tree topology. The first processing module is used to construct a tree with each processing node as the root based on the target tree height and target width. The first partitioning module is used to divide the total data block equally according to the total number of processing nodes, so as to obtain at least two sub-blocks; The second processing module is used to perform full reduction processing on one of the sub-blocks using each of the constructed trees; The first acquisition module is used to acquire each candidate tree topology based on the total number before determining the target tree height and the target width corresponding to each layer under the target tree height according to the target tree topology. The second acquisition module is used to obtain the cost value corresponding to each candidate tree topology using the cost model; The second determining module is used to determine the target tree topology from the candidate tree topologies based on the overhead value; The overhead model includes at least one of the following: memory read / write overhead factor, latency overhead factor, bandwidth contention factor, and computation overhead factor; the memory read / write overhead factor is used to quantify the memory read / write time involved in the full reduction process, and the bandwidth contention factor is used to reflect the factor of bandwidth utilization reduction caused by many-to-one communication. The process of performing full reduction on a sub-block using each constructed tree includes: Based on the target width corresponding to the first layer, determine the grouping of processing nodes and perform the first merging process of sub-blocks; Based on the first result of the first merging process, perform at least one first operation until each node completes the global merging of a sub-block; Based on the second result corresponding to the global merge, reverse node broadcasting is performed; The first operation includes: redetermining the grouping of processing nodes based on the target width corresponding to the current layer and the previous merging result, and performing sub-block merging processing again; The previous merge result is the result of the merge process corresponding to the previous layer. The tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is the adjacent layer of the previous layer; The number of sub-blocks to be merged in the previous layer is greater than the number of sub-blocks to be merged in the current layer.

5. The data processing apparatus according to claim 4, characterized in that, Also includes: The third processing module is used to, before obtaining each candidate tree topology based on the total quantity, split the total quantity into a first quantity plus a first value if the number of factors in the total quantity is less than a threshold value. The first value is less than the first quantity; The step of obtaining each candidate tree topology based on the total number includes: Based on the first quantity and the first value, obtain each candidate tree topology.

6. The data processing apparatus according to claim 5, characterized in that, Determine the grouping of processing nodes, including: Group the nodes corresponding to the first quantity, and assign the nodes corresponding to the first value to the last group; The last group is the last group among the groups of nodes corresponding to the first quantity.

7. A data processing device, characterized in that, include: Processor and transceiver; The processor is configured to determine the target tree height and the target width corresponding to each layer under the target tree height based on the target tree topology. Based on the target tree height and target width, construct a tree with each processing node as the root; Based on the total number of processing nodes, the total data block is divided into equal parts to obtain at least two sub-blocks; Each of the constructed trees is used to perform a full reduction process on one of the sub-blocks; The processor is further configured to: before determining the target tree height and the target width corresponding to each layer under the target tree height based on the target tree topology, obtain each candidate tree topology based on the total number; and obtain the cost value corresponding to each candidate tree topology using the cost model. Based on the cost value, a target tree topology is determined from the candidate tree topologies; The overhead model includes at least one of the following: memory read / write overhead factor, latency overhead factor, bandwidth contention factor, and computation overhead factor; the memory read / write overhead factor is used to quantify the memory read / write time involved in the full reduction process, and the bandwidth contention factor is used to reflect the factor of bandwidth utilization reduction caused by many-to-one communication. The process of performing full reduction on a sub-block using each constructed tree includes: Based on the target width corresponding to the first layer, determine the grouping of processing nodes and perform the first merging process of sub-blocks; Based on the first result of the first merging process, perform at least one first operation until each node completes the global merging of a sub-block; Based on the second result corresponding to the global merge, reverse node broadcasting is performed; The first operation includes: redetermining the grouping of processing nodes based on the target width corresponding to the current layer and the previous merging result, and performing sub-block merging processing again; The previous merge result is the result of the merge process corresponding to the previous layer. The tree height corresponding to the current layer is greater than the tree height corresponding to the previous layer, and the current layer is the adjacent layer of the previous layer; The number of sub-blocks to be merged in the previous layer is greater than the number of sub-blocks to be merged in the current layer.

8. A data processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the data processing method as described in any one of claims 1 to 3.

9. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the data processing method as described in any one of claims 1 to 3.

Images