Data distribution method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure disclose a data distribution method and device, electronic equipment and storage medium. The method comprises: obtaining grouping matrix construction parameters, and constructing a grouping matrix based on the grouping matrix construction parameters, each column element in the grouping matrix corresponding to a heterogeneous node group, and one row element in adjacent two rows being sorted in ascending order of expected latency, and the other row element being sorted in descending order of expected latency; obtaining a pre-generated data distribution matrix, updating the data distribution matrix based on the heterogeneous node group in the grouping matrix; and distributing a subset of data to be processed to the heterogeneous nodes based on the updated data distribution matrix. By placing the heterogeneous nodes with good performance and the heterogeneous nodes with poor performance in the same heterogeneous node group, the average error of the master node to recover the complete processing result is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a data allocation method, apparatus, electronic device, and storage medium. Background Technology

[0002] To improve the processing efficiency of large-scale datasets, distributed systems are widely adopted. The mainstream distributed system organization is a master-slave architecture, consisting of a master node and multiple compute nodes. The master node is responsible for allocating and summarizing tasks, while the compute nodes process subtasks. Typically, the master node divides the dataset into multiple subsets and distributes these subsets to multiple compute nodes, each processing a different subset. After processing its assigned subset, each compute node sends a result back to the master node. Upon receiving results from all compute nodes, the master node summarizes these results to reconstruct the complete result corresponding to the entire dataset. However, due to performance differences or variations in network connectivity among compute nodes, some nodes become lagging nodes. Lagging nodes process their assigned subsets slowly and may even fail to send back results, preventing the master node from reconstructing the complete result.

[0003] To ensure the master node can recover complete result information, gradient coding, an encoding scheme for distributed gradient computation, has been proposed in recent years. The core idea of ​​gradient coding is to simultaneously allocate each subset of data to be processed to multiple computing nodes, repeating the computation on these nodes. Even if some computing nodes become lagging, other nodes can still successfully feed back result information, allowing the master node to recover the complete result information from the partially received feedback. The key to gradient coding lies in designing the allocation scheme for the subsets of data to be processed—that is, which subsets of data to be processed are assigned to which computing nodes. In existing technologies, random grouping is often used, randomly dividing multiple computing nodes into groups and assigning the same subset of data to be processed to computing nodes within the same group. This way, even if a lagging node occurs within the same group, the master node can still receive the result information fed back from other computing nodes within that group, thus ensuring the effective recovery of the complete result information.

[0004] However, in a heterogeneous node environment, due to the large performance differences between computing nodes, using random grouping can easily lead to computing nodes that are more likely to become lagging nodes being assigned to the same group. The probability that all computing nodes in the group will become lagging nodes will increase accordingly. When every computing node in the group is a lagging node, the master node will not be able to receive any result information of the subset of data to be processed assigned to the group, resulting in a large average error between the complete result information recovered by the master node and the accurate complete result information.

[0005] Therefore, in heterogeneous node environments, reducing the average error of the master node in recovering complete result information has become a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the problems in the related technologies, embodiments of this disclosure provide a data allocation method, apparatus, electronic device, and storage medium.

[0007] Firstly, this disclosure provides a data allocation method.

[0008] Specifically, the data allocation method includes:

[0009] Obtain the group matrix construction parameters, which include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes. The heterogeneous node group includes at least one heterogeneous node.

[0010] A grouping matrix is ​​constructed based on the grouping matrix construction parameters. The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column of the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes.

[0011] Obtain a pre-generated data allocation matrix, and update the data allocation matrix based on the heterogeneous node groups in the grouping matrix;

[0012] The updated data allocation matrix is ​​used to allocate a subset of data to be processed to heterogeneous nodes.

[0013] In one implementation of this disclosure, constructing the grouping matrix based on the grouping matrix construction parameters includes:

[0014] The heterogeneous nodes are sorted in ascending order of expected latency;

[0015] The grouping matrix is ​​constructed based on the sorted heterogeneous nodes and the grouping matrix construction parameters.

[0016] One implementation of this disclosure also includes:

[0017] Get the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes;

[0018] The number of heterogeneous node groups is determined based on the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes.

[0019] Based on the total number of heterogeneous nodes and the number of heterogeneous node groups, the number of heterogeneous nodes included in each heterogeneous node group is determined.

[0020] One implementation of this disclosure also includes:

[0021] Based on the total number of heterogeneous nodes, the number of heterogeneous node groups, the total number of data subsets to be processed, and the number of data subsets to be processed allocated to heterogeneous nodes, a data allocation matrix is ​​constructed. The data allocation matrix has a dimension of n rows × k columns, where n is the total number of heterogeneous nodes and n = d × q, k is the total number of data subsets to be processed and k = p × q, and p is the number of data subsets to be processed allocated to heterogeneous nodes. Each row of the data allocation matrix corresponds to a heterogeneous node, and each column corresponds to a data subset to be processed. The data allocation matrix includes q × q block sub-matrices, each block sub-matrice having a dimension of d rows × p columns. The elements of the block sub-matrices located on the diagonal have a value of 1, and the elements of the block sub-matrices located off-diagonally have a value of 0. The heterogeneous nodes corresponding to the block sub-matrices located on the diagonal constitute a heterogeneous node group.

[0022] In one implementation of this disclosure, updating the data allocation matrix based on the heterogeneous node groups in the grouping matrix includes:

[0023] Determine the block submatrices on the diagonal of the data allocation matrix;

[0024] Determine the heterogeneous node groups corresponding to the block submatrices on the diagonal and the heterogeneous node groups corresponding to a column of elements in the grouping matrix;

[0025] Based on the heterogeneous node group corresponding to a column of elements in the grouping matrix, update the heterogeneous node group corresponding to the block submatrix on the diagonal to obtain the updated data allocation matrix.

[0026] In one implementation of this disclosure, the step of allocating a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix includes:

[0027] Determine the heterogeneous nodes and data subsets to be processed corresponding to the diagonal submatrices in the updated data allocation matrix;

[0028] Assign the data subset to be processed corresponding to the diagonal submatrix to the heterogeneous nodes corresponding to the diagonal submatrix.

[0029] One implementation of this disclosure also includes:

[0030] In response to receiving a first aggregation result from at least one of the heterogeneous nodes, the current node's failure status is determined based on the heterogeneous node that sent the first aggregation result. The first aggregation result is determined based on the first result information obtained by the heterogeneous node from processing the allocated subset of data to be processed and the heterogeneous node aggregation coefficient vector of the heterogeneous node. The heterogeneous node aggregation coefficient vector has a dimension of 1 row × p columns, and the value of p is the number of subsets of data to be processed allocated by the master node to the heterogeneous node.

[0031] Based on the current node dropout situation, a set of master node linear combination coefficients corresponding to the current node dropout situation is determined from the pre-generated master node aggregation coefficient matrix. The master node aggregation coefficient matrix has a dimension of m rows × n columns, where m is the number of node dropout situations and n is the total number of heterogeneous nodes. Each row of the master node aggregation coefficient matrix corresponds to a node dropout situation, and each column corresponds to a heterogeneous node. Each element corresponds to the linear coefficient used by the master node to return the first aggregation result to the heterogeneous node under different combinations of heterogeneous nodes and different node dropout situations.

[0032] The second aggregation result is determined based on the linear combination coefficients of the master node and at least one of the first aggregation results.

[0033] One implementation of this disclosure also includes:

[0034] Obtain various node dropout scenarios under the total number of heterogeneous nodes;

[0035] The master node aggregation coefficient matrix is ​​constructed based on the total number of heterogeneous nodes and the various node dropout scenarios.

[0036] Secondly, this disclosure provides a data distribution device.

[0037] Specifically, the data distribution device includes:

[0038] The acquisition module is configured to acquire group matrix construction parameters, which include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes, wherein the heterogeneous node group includes at least one heterogeneous node.

[0039] The grouping matrix construction module is configured to construct a grouping matrix based on the grouping matrix construction parameters. The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column of the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes.

[0040] The update module is configured to obtain a pre-generated data allocation matrix and update the data allocation matrix based on the heterogeneous node groups in the grouping matrix;

[0041] The allocation module is configured to allocate a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix.

[0042] Thirdly, embodiments of this disclosure provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in the first aspect and any possible implementation thereof.

[0043] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described in the first aspect and any possible implementation thereof.

[0044] According to the technical solution provided in this disclosure, the heterogeneous node groups defined in the pre-acquired data allocation matrix are updated using the heterogeneous node groups defined in the constructed grouping matrix, and the updated data allocation matrix is ​​used to allocate a subset of data to be processed to each heterogeneous node in each heterogeneous node group. Since in the constructed grouping matrix, one row of elements in the odd-numbered and even-numbered rows is sorted in ascending order of expected latency, and the other row is sorted in descending order of expected latency, and each column corresponds to a heterogeneous node group, high-performance heterogeneous nodes and low-performance heterogeneous nodes can be placed in the same heterogeneous node group, and medium-performance heterogeneous nodes can be placed in the same heterogeneous node group. This makes the performance of each heterogeneous node group more even, reducing the probability that all heterogeneous nodes in the same heterogeneous node group become lagging nodes. This ensures that each heterogeneous node group can successfully send back the result information to the master node, and the master node can receive the processing results of all the data subsets to be processed, thereby recovering a relatively accurate complete processing result. Without increasing additional computational redundancy, the average error of the master node in recovering the complete processing result is reduced.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0047] Figure 1 A flowchart illustrating a data allocation method according to an embodiment of the present disclosure is shown.

[0048] Figure 2 A schematic diagram of a grouping matrix according to an embodiment of the present disclosure is shown.

[0049] Figure 3 A schematic diagram of a data allocation matrix according to an embodiment of the present disclosure is shown.

[0050] Figure 4 A schematic diagram of an updated data allocation matrix according to an embodiment of the present disclosure is shown.

[0051] Figure 5 A structural block diagram of a data distribution apparatus according to an embodiment of the present disclosure is shown.

[0052] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing a data allocation method according to an embodiment of the present disclosure. Detailed Implementation

[0053] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings.

[0054] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0055] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] As mentioned above, current gradient coding typically employs random grouping, randomly dividing multiple computing nodes into groups and assigning the same subset of data to be processed to nodes within the same group. This ensures that even if a node falls behind within a group, the master node can still receive the results from other nodes in that group, guaranteeing the effective recovery of complete results. However, in heterogeneous node environments, due to significant performance differences between computing nodes, random grouping can easily lead to computing nodes more likely to fall behind being assigned to the same group. This increases the probability that all nodes in the group will fall behind. When every node in the group is a fallback node, the master node will be unable to receive any results from the subset of data assigned to that group, resulting in a larger average error between the recovered complete results and the accurate complete results.

[0057] To address the aforementioned shortcomings, this disclosure provides a data allocation method. This method updates the heterogeneous node groups defined in a pre-acquired data allocation matrix using heterogeneous node groups defined in a constructed grouping matrix, and then uses the updated data allocation matrix to allocate a subset of data to be processed to each heterogeneous node within each heterogeneous node group. Since in the constructed grouping matrix, one row of odd-numbered and even-numbered rows is sorted in ascending order of expected latency, and the other row is sorted in descending order of expected latency, and each column corresponds to a heterogeneous node group, high-performance and low-performance heterogeneous nodes can be placed in the same heterogeneous node group, while medium-performance heterogeneous nodes can be placed in the same heterogeneous node group. This ensures that the performance of each heterogeneous node group is relatively even, reducing the probability that all heterogeneous nodes within the same heterogeneous node group will become lagging nodes. This ensures that each heterogeneous node group can successfully send back the result information to the master node, and the master node can receive the processing results of all the data subsets to be processed, thereby recovering a relatively accurate complete processing result. Without increasing additional computational redundancy, the average error of the master node in recovering the complete processing result is reduced.

[0058] The details of the embodiments of this disclosure are described in detail below through specific examples.

[0059] Figure 1 A flowchart illustrating a data allocation method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the data allocation method includes the following steps S101-S104.

[0060] In step S101, the grouping matrix construction parameters are obtained.

[0061] The grouping matrix construction parameters include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes. The heterogeneous node group includes at least one heterogeneous node.

[0062] In step S102, a grouping matrix is ​​constructed based on the grouping matrix construction parameters.

[0063] The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column in the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes.

[0064] In step S103, a pre-generated data allocation matrix is ​​obtained, and the data allocation matrix is ​​updated based on the heterogeneous node groups in the grouping matrix.

[0065] In step S104, a subset of data to be processed is allocated to heterogeneous nodes based on the updated data allocation matrix.

[0066] The aforementioned data allocation method is applied in distributed machine learning scenarios. This data allocation method can be executed on the master node, by a physical device capable of providing master node functionality, by a component (such as a chip) configured in the physical device, or by a module capable of implementing some or all of the master node's functions, etc. This disclosure does not limit this. For ease of understanding, this disclosure uses the master node as an example to describe the method provided by this disclosure in the following description.

[0067] In this embodiment, since the computing speed and information transmission speed of different computing nodes may vary significantly, i.e., the latency between computing nodes is independent and follows different distributions, these computing nodes can be referred to as heterogeneous nodes, and the environment composed of heterogeneous nodes can be referred to as a heterogeneous node environment. For ease of understanding, this disclosure will use the example of all computing nodes under the master node being heterogeneous nodes to describe the method of this disclosure.

[0068] To enable data allocation to heterogeneous nodes, the master node can first obtain the following parameters: total number of heterogeneous nodes, number of heterogeneous node groups, and expected latency of heterogeneous nodes.

[0069] The total number of heterogeneous nodes can be the number of latency-independent computing nodes under the master node that follow different distributions. In some embodiments, when all computing nodes under the master node are heterogeneous nodes, the total number of heterogeneous nodes is the total number of computing nodes under the master node.

[0070] The number of heterogeneous node groups can be determined by the master node. The master node can divide multiple heterogeneous nodes into multiple heterogeneous node groups, each heterogeneous node group including at least one heterogeneous node, and different heterogeneous node groups contain the same number of heterogeneous nodes. That is, the number of heterogeneous node groups is equal to the number of heterogeneous node groups created.

[0071] The expected latency of a heterogeneous node is the expected time consumed for the heterogeneous node to complete the subset of data to be processed assigned by the master node and send the corresponding result information back to the master node. The master node can obtain the expected latency of each heterogeneous node. In some embodiments, the expected latencies of multiple heterogeneous nodes are different.

[0072] It's important to note that the performance of heterogeneous nodes can be reflected in their expected latency. The shorter the expected latency of a heterogeneous node, the faster it can process the subset of data to be processed and the faster it can send results back to the master node. This indicates better performance and a lower probability of the node becoming a lagging node. Conversely, the longer the expected latency of a heterogeneous node, the slower it can process the subset of data to be processed and the slower it can send results back to the master node. This indicates poorer performance and a higher probability of the node becoming a lagging node.

[0073] After obtaining the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of each heterogeneous node, the master node can construct a grouping matrix based on these parameters. The grouping matrix defines the grouping of multiple heterogeneous nodes under the master node.

[0074] In this disclosure, the constructed grouping matrix has a dimension of d rows × q columns, and the grouping matrix satisfies the following conditions 1) to 4).

[0075] 1) The number of rows d in the grouping matrix is ​​determined by the number of heterogeneous nodes contained in the heterogeneous node group, the number of columns q is determined by the number of heterogeneous node groups, and the value of d×q is the total number of heterogeneous nodes.

[0076] 2) Each element in the grouping matrix represents a heterogeneous node.

[0077] 3) Each column element in the grouping matrix represents a heterogeneous node group.

[0078] 4) In the grouping matrix, the elements in one row of the odd-numbered row and the elements in the even-numbered row are sorted in ascending order of the expected delay of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected delay of the heterogeneous nodes.

[0079] After constructing the grouping matrix, the master node can further obtain a pre-generated data allocation matrix. This data allocation matrix is ​​a block matrix that randomly defines the grouping of heterogeneous nodes. Specifically, it randomly divides the multiple heterogeneous nodes under the master node into heterogeneous node groups, and defines the number of data subsets to be processed allocated to each heterogeneous node group, as well as the specific subsets of data to be processed allocated to each heterogeneous node group. In short, the heterogeneous node groups defined by the data allocation matrix are randomly divided, while the data subsets to be processed allocated to each heterogeneous node group are pre-defined.

[0080] After obtaining the pre-generated data allocation matrix and constructing the grouping matrix, the master node can update the heterogeneous node groups defined in the data allocation matrix based on the heterogeneous node groups defined in the grouping matrix, ensuring that the updated data allocation matrix defines the same heterogeneous node groups as the grouping matrix. Therefore, the master node can allocate a subset of data to be processed to each heterogeneous node within each heterogeneous node group based on the grouping of heterogeneous nodes defined in the updated data allocation matrix. When allocating the subset of data to be processed, the master node can allocate data to heterogeneous nodes within each heterogeneous node group based on the specific subset of data to be processed allocated to each heterogeneous node group as defined in the data allocation matrix. The master node can partition large-scale datasets to obtain multiple subsets of data to be processed.

[0081] In some embodiments, heterogeneous nodes within the same heterogeneous node group are assigned the same subset of data to be processed, while different heterogeneous node groups are assigned the same number of subsets of data to be processed. In other words, heterogeneous nodes within the same heterogeneous node group are assigned the same number of subsets of data to be processed and the same data content, while different heterogeneous node groups are assigned the same number of subsets of data to be processed, but the data content is different.

[0082] This disclosure proposes a data allocation method. This method updates the heterogeneous node groups defined in a pre-acquired data allocation matrix using heterogeneous node groups defined in a constructed grouping matrix, and then uses the updated data allocation matrix to allocate a subset of data to be processed to each heterogeneous node within each heterogeneous node group. Since in the constructed grouping matrix, one row of elements in the odd-numbered and even-numbered rows is sorted in ascending order of expected latency, and the other row is sorted in descending order of expected latency, and each column corresponds to a heterogeneous node group, high-performance and low-performance heterogeneous nodes can be placed in the same heterogeneous node group, while medium-performance heterogeneous nodes can be placed in the same heterogeneous node group. This ensures that the performance of each heterogeneous node group is relatively even, reducing the probability that all heterogeneous nodes within the same heterogeneous node group will become lagging nodes. This ensures that each heterogeneous node group can successfully send back the result information to the master node, and the master node can receive the processing results of all the data subsets to be processed, thereby recovering a relatively accurate complete processing result. Without increasing additional computational redundancy, the average error of the master node in recovering the complete processing result is reduced.

[0083] In an optional implementation of this embodiment, step S102, namely constructing the grouping matrix based on the grouping matrix construction parameters, may further include the following steps:

[0084] The heterogeneous nodes are sorted in ascending order of expected latency;

[0085] The grouping matrix is ​​constructed based on the sorted heterogeneous nodes and the grouping matrix construction parameters.

[0086] In this optional implementation, when constructing the grouping matrix, the master node can first sort the heterogeneous nodes according to their expected latency. For example, suppose there are 6 heterogeneous nodes under the master node, with identity documents (ID): T1, T2, T3, T4, T5, and T6. The expected latencies of these 6 heterogeneous nodes are t1, t2, t3, t4, t5, and t6, respectively, and t1 < t3 < t2 < t4 < t5 < t6. Then, the performance of these 6 heterogeneous nodes can be sorted in ascending order of expected latency, resulting in the order: T1 < T3 < T2 < T4 < T5 < T6.

[0087] In some embodiments, after sorting the heterogeneous nodes according to their expected latency in ascending order, and before constructing the grouping matrix, the sorted heterogeneous nodes can be numbered sequentially from heterogeneous node 1 to heterogeneous node n, where n is the number of heterogeneous nodes. For example, assuming the sorting according to expected latency in ascending order is: T1 < T3 < T2 < T4 < T5 < T6, the sequential numbering order is: heterogeneous node 1 < heterogeneous node 2 < heterogeneous node 3 < heterogeneous node 4 < heterogeneous node 5 < heterogeneous node 6. Here, heterogeneous node 2 essentially corresponds to the heterogeneous node with ID T3, and heterogeneous node 3 essentially corresponds to the heterogeneous node with ID T2.

[0088] Then, the master node can construct the grouping matrix based on the sorted heterogeneous nodes and the parameters of the grouping matrix, according to the conditions 1) to 4) mentioned above that need to be satisfied to construct the grouping matrix. Specifically, the first row of the grouping matrix with dimensions of d rows × q columns corresponds to heterogeneous nodes 1 to q, and these q heterogeneous nodes are sorted in ascending order of expected latency; the second row of the grouping matrix corresponds to heterogeneous nodes q+1 to 2q, and these q heterogeneous nodes are sorted in descending order of expected latency; the third row of the grouping matrix corresponds to heterogeneous nodes 2q+1 to 3q, and these q heterogeneous nodes are sorted in ascending order of expected latency; and so on up to the dth row, where the elements in odd-numbered rows are sorted in ascending order of expected latency of heterogeneous nodes, and the elements in even-numbered rows are sorted in descending order of expected latency of heterogeneous nodes.

[0089] For example, Figure 2 A schematic diagram of a grouping matrix according to an embodiment of the present disclosure is shown.

[0090] Assume the performance order of the 6 heterogeneous nodes after sorting is: T1 < T3 < T2 < T4 < T5 < T6. The master node divides the nodes into 3 heterogeneous node groups, and each heterogeneous node group contains 2 heterogeneous nodes. Figure 2 As shown, a 2×3 grouping matrix can be constructed. The elements in the first row and first column of this matrix represent heterogeneous nodes T1, T3, T2, T4, T5, and T6, respectively. It can be seen that each column of this grouping matrix forms a heterogeneous node group. Specifically, T1 and T6 form heterogeneous node group 1, T3 and T5 form heterogeneous node group 2, and T2 and T4 form heterogeneous node group 3. It can also be seen that the elements in the first row (odd-numbered rows) of this grouping matrix are sorted from left to right in ascending order of expected latency, and the elements in the second row (even-numbered rows) are sorted from left to right in descending order of expected latency.

[0091] It can be observed that the grouping matrix constructed in the above manner places the worst-performing heterogeneous node T1 and the best-performing heterogeneous node T6 in the same heterogeneous node group, the relatively poor-performing heterogeneous node T3 and the relatively good-performing heterogeneous node T5 in the same heterogeneous node group, and the moderately performing heterogeneous nodes T2 and T4 in the same heterogeneous node group. In this way, by placing high-performing and low-performing heterogeneous nodes in the same heterogeneous node group, the performance of each heterogeneous node group is relatively even, reducing the probability that all heterogeneous nodes in the same heterogeneous node group will become lagging nodes.

[0092] In one optional implementation of this embodiment, the following steps may also be included:

[0093] Get the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes;

[0094] The number of heterogeneous node groups is determined based on the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes.

[0095] Based on the total number of heterogeneous nodes and the number of heterogeneous node groups, the number of heterogeneous nodes included in each heterogeneous node group is determined.

[0096] In this optional implementation, the number of heterogeneous node groups can be determined as follows: the number of heterogeneous node groups is determined by the quotient between the total number of data subsets to be processed and the number of data subsets to be processed allocated to each heterogeneous node. For example, if the total number of data subsets to be processed is 6, and each heterogeneous node is allocated 2 data subsets, then the number of heterogeneous node groups is 6 ÷ 2 = 3 groups.

[0097] In some embodiments, the number of data subsets to be processed allocated to each heterogeneous node is the same. The number of data subsets to be processed allocated to heterogeneous nodes can be determined as follows: the total number of data subsets to be processed is determined based on the data subsets to be processed obtained by partitioning the dataset to be processed; the number of data subsets to be processed allocated to each heterogeneous node is determined based on the quotient between the total number of data subsets to be processed and the total number of heterogeneous nodes.

[0098] In some embodiments, the number of data subsets to be processed allocated to each heterogeneous node can be divided by the total number of data subsets to be processed.

[0099] In some embodiments, to facilitate the construction of the grouping matrix, the total number of subsets of data to be processed can be the same as the total number of heterogeneous nodes.

[0100] The number of heterogeneous nodes in a heterogeneous node group can be determined as follows: the number of heterogeneous nodes in a heterogeneous node group is determined by the quotient between the total number of heterogeneous nodes and the number of heterogeneous node groups. For example, if the total number of heterogeneous nodes is 6 and the number of heterogeneous node groups is 3, then each heterogeneous node group contains 6 ÷ 3 = 2 heterogeneous nodes.

[0101] In one optional implementation of this embodiment, the following steps may also be included:

[0102] Based on the total number of heterogeneous nodes, the number of heterogeneous node groups, the total number of data subsets to be processed, and the number of data subsets to be processed allocated to heterogeneous nodes, a data allocation matrix is ​​constructed. The data allocation matrix has a dimension of n rows × k columns, where n is the total number of heterogeneous nodes and n = d × q, k is the total number of data subsets to be processed and k = p × q, and p is the number of data subsets to be processed allocated to heterogeneous nodes. Each row of the data allocation matrix corresponds to a heterogeneous node, and each column corresponds to a data subset to be processed. The data allocation matrix includes q × q block sub-matrices, each block sub-matrice having a dimension of d rows × p columns. The elements of the block sub-matrices located on the diagonal have a value of 1, and the elements of the block sub-matrices located off-diagonally have a value of 0. The heterogeneous nodes corresponding to the block sub-matrices located on the diagonal constitute a heterogeneous node group.

[0103] In this optional implementation, the pre-generated data allocation matrix obtained in step S103 can be constructed based on the total number of heterogeneous nodes, the number of heterogeneous node groups, the total number of data subsets to be processed, and the number of data subsets to be processed allocated to heterogeneous nodes.

[0104] The constructed data allocation matrix has an dimension of n rows × k columns. This data allocation matrix satisfies the following conditions 1) to 5).

[0105] 1) The number of rows n in the data allocation matrix is ​​determined by the total number of heterogeneous nodes, which is equal to the product of the number of heterogeneous nodes contained in a heterogeneous node group and the number of heterogeneous node groups. The number of columns k is determined by the total number of subsets of data to be processed, which is equal to the product of the number of subsets of data to be processed allocated to heterogeneous nodes and the number of heterogeneous node groups.

[0106] 2) Each row corresponds to a random heterogeneous node, and each column corresponds to a subset of data to be processed.

[0107] 3) The data allocation matrix is ​​a block matrix with a number of blocks of q×q, where q is the number of heterogeneous node groups. All elements in the diagonal block submatrices are 1, and all elements in the off-diagonal block submatrices are 0.

[0108] 4) The dimension of the block submatrix is ​​d rows × p columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, and p is the number of data subsets to be processed allocated to the heterogeneous nodes.

[0109] In this system, any submatrix located on the diagonal can form a heterogeneous node group by representing its corresponding heterogeneous nodes. The number of heterogeneous node groups is the same as the number of submatrixes located on the diagonal.

[0110] 5) Elements with a value of 1 in each row correspond to the subset of data to be processed assigned to the heterogeneous node, and elements with a value of 1 in each column correspond to the heterogeneous node of the subset of data to be processed.

[0111] For example, Figure 3 A schematic diagram of a data allocation matrix according to an embodiment of the present disclosure is shown.

[0112] Assuming there are a total of 6 heterogeneous nodes, each heterogeneous node group contains 2 heterogeneous nodes, and there are 3 heterogeneous node groups, then the data allocation matrix will have 2 × 3 = 6 rows. And assuming that each heterogeneous node receives 2 subsets of data to be processed, then the data allocation matrix will have 2 × 3 = 6 columns.

[0113] like Figure 3As shown, the heterogeneous nodes corresponding to the elements in rows 1 to 6 of this data allocation matrix can be randomly assigned as T1, T2, T3, T4, T5, and T6. The subsets of data to be processed corresponding to the elements in columns 1 to 6 of this data allocation matrix can be assigned as D1, D2, D3, D4, D5, and D6. This data allocation matrix consists of 3×3=9 sub-matrices. The sub-matrices located on the diagonal are A1, A2, and A3, where all elements are 1. The elements in the other six off-diagonal sub-matrices are 0. Each sub-matrice has a dimension of 2 rows × 2 columns.

[0114] It is known that the heterogeneous node group corresponding to the block submatrix A1 includes two heterogeneous nodes T1 and T2, the heterogeneous node group corresponding to the block submatrix A2 includes two heterogeneous nodes T3 and T4, and the heterogeneous node group corresponding to the block submatrix A3 includes two heterogeneous nodes T5 and T6. Furthermore, the data subsets to be processed allocated to the heterogeneous node group corresponding to the block submatrix A1 are D1 and D2, the data subsets to be processed allocated to the heterogeneous node group corresponding to the block submatrix A2 are D3 and D4, and the data subsets to be processed allocated to the heterogeneous node group corresponding to the block submatrix A3 are D5 and D6.

[0115] It should be understood that Figure 3 The data allocation matrix shown is for illustrative purposes only. In practical applications, the specific heterogeneous node corresponding to each element in a row is randomly generated, as long as each element in a row corresponds to one heterogeneous node.

[0116] Because the data allocation matrix constructed in this optional embodiment assigns identical subsets of data to be processed to heterogeneous nodes within the same heterogeneous node group, meaning that heterogeneous nodes within the same heterogeneous node group are assigned identical data processing tasks, if one or more heterogeneous nodes within the same heterogeneous node group become lagging nodes, the master node can still effectively recover the complete result information corresponding to the large-scale dataset to be processed based on the result information returned by other heterogeneous nodes within the same heterogeneous node group. This also reduces the average error of the master node when recovering the complete result information to a certain extent.

[0117] In an optional implementation of this embodiment, step S103, updating the data allocation matrix based on the heterogeneous node groups in the grouping matrix, may further include the following steps:

[0118] Determine the block submatrices on the diagonal of the data allocation matrix;

[0119] Determine the heterogeneous node groups corresponding to the block submatrices on the diagonal and the heterogeneous node groups corresponding to a column of elements in the grouping matrix;

[0120] Based on the heterogeneous node group corresponding to a column of elements in the grouping matrix, update the heterogeneous node group corresponding to the block submatrix on the diagonal to obtain the updated data allocation matrix.

[0121] In this optional implementation, after obtaining the data allocation matrix, the heterogeneous node groups defined in the grouping matrix can be further updated based on the heterogeneous node groups defined in the data allocation matrix, thereby updating the data allocation matrix.

[0122] When determining the heterogeneous node group defined by the data allocation matrix, the heterogeneous node group can be determined based on the heterogeneous nodes corresponding to the allocation submatrices located on the diagonal of the data allocation matrix. Figure 3 For example, if the diagonal submatrix A1 corresponds to two heterogeneous nodes T1 and T2, then T1 and T2 form a heterogeneous node group; if the diagonal submatrix A2 corresponds to two heterogeneous nodes T3 and T4, then T3 and T4 form another heterogeneous node group; if the diagonal submatrix corresponds to two heterogeneous nodes T5 and T6, then T5 and T6 form yet another heterogeneous node group.

[0123] Accordingly, when determining the heterogeneous node groups defined by the grouping matrix, the heterogeneous node groups can be determined based on the heterogeneous nodes corresponding to the elements in each column of the grouping matrix. Figure 2 For example, if the elements in the first column correspond to two heterogeneous nodes T1 and T6, then T1 and T6 can form heterogeneous node group 1; if the elements in the second column correspond to two heterogeneous nodes T3 and T5, then T3 and T5 can form heterogeneous node group 2; if the elements in the third column correspond to two heterogeneous nodes T2 and T4, then T2 and T4 can form heterogeneous node group 3.

[0124] Furthermore, the master node can update the heterogeneous node group defined by the data allocation matrix based on the heterogeneous node group defined by the grouping matrix. For example, suppose the master node determines... Figure 2 The heterogeneous node groups defined by the grouping matrix shown are Figure 3 The heterogeneous node group defined by the data allocation matrix shown can be updated by changing the heterogeneous node corresponding to the block submatrix A1 on the diagonal of the data allocation matrix from T2 to T6, the heterogeneous node corresponding to the block submatrix A2 on the diagonal from T4 to T5, and the heterogeneous nodes corresponding to the block submatrix A3 on the diagonal from T5 and T6 to T2 and T4. In other words, by updating the heterogeneous node corresponding to the element in the second row of the data allocation matrix from T2 to T6, the heterogeneous node corresponding to the element in the fourth row from T4 to T5, the heterogeneous node corresponding to the element in the fifth row from T5 to T2, and the heterogeneous node corresponding to the element in the sixth row from T6 to T4, the following can be obtained: Figure 4 The updated data allocation matrix is ​​shown.

[0125] It should be understood that Figure 4 The updated data allocation matrix shown is for illustrative purposes only. In practical applications, any heterogeneous node group defined in the grouping matrix can update any heterogeneous node group defined in the data allocation matrix, all within the scope of this disclosure.

[0126] In some embodiments, since the heterogeneous node groups of the data allocation matrix are randomly defined, if one or more heterogeneous node groups defined by the grouping matrix contain the same heterogeneous nodes as one or more heterogeneous node groups defined by the data allocation matrix, then there is no need to update the one or more heterogeneous node groups defined by the data allocation submatrix.

[0127] In an optional implementation of this embodiment, step S104, which involves allocating a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix, may further include the following steps:

[0128] Determine the heterogeneous nodes and data subsets to be processed corresponding to the diagonal submatrices in the updated data allocation matrix;

[0129] Assign the data subset to be processed corresponding to the diagonal submatrix to the heterogeneous nodes corresponding to the diagonal submatrix.

[0130] In this optional implementation, after obtaining the updated data allocation matrix, a subset of data to be processed can be allocated to each heterogeneous node based on the updated data allocation matrix. The master node can determine the heterogeneous node group and its corresponding subset of data to be processed for each block submatrix located on the diagonal of the updated data allocation matrix, and send the subset of data to be processed corresponding to that heterogeneous node group to each heterogeneous node in each heterogeneous node group, thereby completing the allocation of the subset of data to be processed to all heterogeneous nodes.

[0131] by Figure 4 Taking the updated data allocation matrix as an example, based on the heterogeneous nodes and data subsets to be processed defined by the diagonal submatrix A1, data subsets D1 and D2 to be processed can be allocated to heterogeneous nodes T1 and T6 respectively; based on the heterogeneous nodes and data subsets to be processed defined by the diagonal submatrix A2, data subsets D3 and D4 to be processed can be allocated to heterogeneous nodes T3 and T5 respectively; based on the heterogeneous nodes and data subsets to be processed defined by the diagonal submatrix A3, data subsets D5 and D6 to be processed can be allocated to heterogeneous nodes T2 and T4 respectively.

[0132] In an optional implementation of this embodiment, the method may further include:

[0133] In response to receiving a first aggregation result from at least one of the heterogeneous nodes, the current node's failure status is determined based on the heterogeneous node that sent the first aggregation result. The first aggregation result is determined based on the first result information obtained by the heterogeneous node from processing the allocated subset of data to be processed and the heterogeneous node aggregation coefficient vector of the heterogeneous node. The heterogeneous node aggregation coefficient vector has a dimension of 1 row × p columns, and the value of p is the number of subsets of data to be processed allocated by the master node to the heterogeneous node.

[0134] Based on the current node dropout situation, a set of master node linear combination coefficients corresponding to the current node dropout situation is determined from the pre-generated master node aggregation coefficient matrix. The master node aggregation coefficient matrix has a dimension of m rows × n columns, where m is the number of node dropout situations and n is the total number of heterogeneous nodes. Each row of the master node aggregation coefficient matrix corresponds to a node dropout situation, and each column corresponds to a heterogeneous node. Each element corresponds to the linear coefficient used by the master node to return the first aggregation result to the heterogeneous node under different combinations of heterogeneous nodes and different node dropout situations.

[0135] The second aggregation result is determined based on the linear combination coefficients of the master node and at least one of the first aggregation results.

[0136] In this optional implementation, after the master node allocates a certain number of data subsets to be processed to each heterogeneous node, the heterogeneous node can process the allocated data subsets to obtain the corresponding first result information. The heterogeneous node needs to process the corresponding number of data subsets allocated by the master node, thus obtaining the corresponding number of first result information. Each first result information corresponds to one data subset. Simultaneously, the heterogeneous node can also obtain the corresponding heterogeneous node aggregation coefficient vector based on the number of data subsets allocated to it by the master node. This aggregation coefficient vector has a dimension of 1 row × p columns, where p is the number of data subsets allocated by the master node, and p ≥ 1. In other words, the value of p is the same as the number of first result information. Furthermore, after determining the same amount of first result information as the allocated subset of data to be processed and the corresponding heterogeneous node aggregation coefficient vector, the heterogeneous node can determine the first aggregation result. After determining the first aggregation result, the heterogeneous node feeds it back to the master node. It should be noted that the result information returned by the heterogeneous node to the master node mentioned above is this first aggregation result.

[0137] In some embodiments, the first aggregation result can be determined as follows:

[0138] g = a1·b1 + ... + ap·bp

[0139] Where g is the first aggregation result, a1 to ap are aggregation coefficients in the heterogeneous node aggregation coefficient vector, and b1 to bp are first result information in multiple first result information.

[0140] Within a preset time period, if the master node receives the first aggregation result returned by at least one heterogeneous node, the current node out-of-line status can be determined based on the heterogeneous node that sent the first aggregation result. For example, assuming there are four heterogeneous nodes under the master node, namely T1, T2, T3, and T4, if the master node receives the first aggregation results returned by heterogeneous nodes T1 and T2 within the preset time period, but does not receive the first aggregation results returned by heterogeneous nodes T3 and T4, then the current node out-of-line status is: heterogeneous nodes T3 and T4 are out of line.

[0141] It should be understood that those skilled in the art can set preset time periods according to actual needs, and this disclosure does not limit this, as all such settings are within the scope of protection.

[0142] After determining the current node's failure status, a set of linear combination coefficients of the master control nodes can be further determined to recover the complete result information under the current node failure situation. This set of linear combination coefficients of the master control nodes can be determined based on the master control node aggregation coefficient matrix. The master control node aggregation coefficient matrix defines the linear combination coefficients of the master control nodes required to recover the complete result information under different node failure situations.

[0143] In this disclosure, the aggregation coefficient matrix of the master node has a dimension of m rows × n columns, and it satisfies the following characteristics:

[0144] 1) m is the total number of nodes that have fallen out of the queue, and n is the total number of heterogeneous nodes.

[0145] 2) Each row of elements corresponds to one type of node dropout, and each column of elements corresponds to a heterogeneous node.

[0146] 3) Each element takes the value of a linear coefficient, which corresponds to the case where the node in the row where the element is located falls out of the queue and the case where the node in the column is located is located under the case of heterogeneous node combination.

[0147] Since each row of the master node aggregation coefficient matrix corresponds to a node dropout scenario, when the current node dropout scenario is determined, the row corresponding to that scenario can be identified from the master node aggregation coefficient matrix. The values ​​of these elements constitute a set of master node linear combination coefficients. Then, the second aggregation result can be recovered based on the determined set of master node linear combination coefficients and the received first aggregation result. This second aggregation result is the complete result information described above, used to characterize the result obtained after processing a large-scale dataset.

[0148] For example, suppose there are four heterogeneous nodes under the master node. Within a preset time period, only the first aggregation result g1 returned by heterogeneous node T1, the first aggregation result g3 returned by heterogeneous node T3, and the first aggregation result g3 returned by heterogeneous node T4 are received. Using the aggregation coefficient matrix of the master node, the values ​​of the row corresponding to the time when heterogeneous node T2 falls behind are determined to be 0.667, 0, 0.667, and 1.333, and each column corresponds to T1, T2, T3, and T4 respectively. Then the second aggregation result... It can be determined in the following way:

[0149]

[0150] In an optional implementation of this embodiment, the method may further include:

[0151] Obtain various node dropout scenarios under the total number of heterogeneous nodes;

[0152] The master node aggregation coefficient matrix is ​​constructed based on the total number of heterogeneous nodes and the various node dropout scenarios.

[0153] In this optional implementation, when constructing the master node aggregation coefficient matrix, the total number of heterogeneous nodes and the identity identifier of each heterogeneous node can be obtained. The number of columns in the master node aggregation coefficient matrix is ​​determined based on the total number of heterogeneous nodes, with each column corresponding to one heterogeneous node. Furthermore, based on the total number of heterogeneous nodes and their identities, multiple node dropout scenarios can be preset, thus obtaining the total number of node dropout scenarios. The number of rows in the master node aggregation coefficient matrix is ​​then determined based on the total number of node dropout scenarios, with each row corresponding to one node dropout scenario. Finally, the linear coefficient corresponding to each heterogeneous node under each dropout scenario is used as an element of the master node aggregation coefficient matrix, thereby obtaining the master node aggregation coefficient matrix.

[0154] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0155] Figure 5A structural block diagram of a data distribution apparatus according to an embodiment of the present disclosure is shown. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5 As shown, the data distribution device includes:

[0156] The acquisition module 501 is configured to acquire group matrix construction parameters, which include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes, wherein the heterogeneous node group includes at least one heterogeneous node.

[0157] The grouping matrix construction module 502 is configured to construct a grouping matrix based on the grouping matrix construction parameters. The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column of the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes.

[0158] Update module 503 is configured to obtain a pre-generated data allocation matrix and update the data allocation matrix based on the heterogeneous node groups in the grouping matrix;

[0159] The allocation module 504 is configured to allocate a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix.

[0160] In an optional implementation of this embodiment, the grouping matrix construction module 502 includes:

[0161] The sorting submodule is configured to sort heterogeneous nodes in ascending order of desired latency;

[0162] The grouping matrix construction submodule is configured to construct the grouping matrix based on the sorted heterogeneous nodes and the grouping matrix construction parameters.

[0163] In an optional implementation of this embodiment, the acquisition module 501 is further configured to acquire the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes;

[0164] The device further includes:

[0165] The determination module is configured to determine the number of heterogeneous node groups based on the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes; and to determine the number of heterogeneous nodes contained in each heterogeneous node group based on the total number of heterogeneous nodes and the number of heterogeneous node groups.

[0166] In an optional implementation of this embodiment, the apparatus further includes:

[0167] The data allocation matrix construction module is configured to construct the data allocation matrix based on the total number of heterogeneous nodes, the number of heterogeneous node groups, the total number of data subsets to be processed, and the number of data subsets to be processed allocated to heterogeneous nodes. The data allocation matrix has a dimension of n rows × k columns, where n is the total number of heterogeneous nodes and n = d × q, k is the total number of data subsets to be processed and k = p × q, and p is the number of data subsets to be processed allocated to heterogeneous nodes. Each row of the data allocation matrix corresponds to a heterogeneous node, and each column corresponds to a data subset to be processed. The data allocation matrix includes q × q block sub-matrices, each block sub-matrice having a dimension of d rows × p columns. The elements of the block sub-matrices located on the diagonal have a value of 1, and the elements of the block sub-matrices located off-diagonally have a value of 0. The heterogeneous nodes corresponding to the block sub-matrices located on the diagonal constitute a heterogeneous node group.

[0168] In an optional implementation of this embodiment, the update module 503 includes:

[0169] The first determining submodule is configured to determine the block submatrix on the diagonal of the data allocation matrix;

[0170] The second determining submodule is configured to determine the heterogeneous node group corresponding to the block submatrix on the diagonal and the heterogeneous node group corresponding to a column of elements in the grouping matrix.

[0171] The update submodule is configured to update the heterogeneous node group corresponding to the block submatrix on the diagonal based on the heterogeneous node group corresponding to a column of elements in the grouping matrix, so as to obtain the updated data allocation matrix.

[0172] In an optional implementation of this embodiment, the allocation module 504 includes:

[0173] The third determining submodule is configured to determine the heterogeneous nodes and data subsets to be processed corresponding to the diagonal block submatrices in the updated data allocation matrix.

[0174] The allocation submodule is configured to allocate the data subset to be processed corresponding to the block submatrix on the diagonal to the heterogeneous nodes corresponding to the block submatrix on the diagonal.

[0175] In an optional implementation of this embodiment, the determining module is further configured to, in response to receiving a first aggregation result from at least one of the heterogeneous nodes, determine the current node's failure status based on the heterogeneous node that sent the first aggregation result. The first aggregation result is determined based on first result information obtained by the heterogeneous node from processing the allocated subset of data to be processed and the heterogeneous node aggregation coefficient vector of the heterogeneous node. The heterogeneous node aggregation coefficient vector has a dimension of 1 row × p columns, and the value of p is the number of subsets of data to be processed allocated by the master node to the heterogeneous node.

[0176] Based on the current node dropout situation, a set of master node linear combination coefficients corresponding to the current node dropout situation is determined from the pre-generated master node aggregation coefficient matrix. The master node aggregation coefficient matrix has a dimension of m rows × n columns, where m is the number of node dropout situations and n is the total number of heterogeneous nodes. Each row of the master node aggregation coefficient matrix corresponds to a node dropout situation, and each column corresponds to a heterogeneous node. Each element corresponds to the linear coefficient used by the master node to return the first aggregation result to the heterogeneous node under different combinations of heterogeneous nodes and different node dropout situations.

[0177] The second aggregation result is determined based on the linear combination coefficients of the master node and at least one of the first aggregation results.

[0178] In an optional implementation of this embodiment, the acquisition module 501 is further configured to acquire various node dropout situations under the total number of heterogeneous nodes;

[0179] The device further includes:

[0180] The master node aggregation coefficient matrix construction module is configured to construct the master node aggregation coefficient matrix based on the total number of heterogeneous nodes and the various node dropout scenarios.

[0181] The data allocation device in this embodiment corresponds to the data allocation method described above. For details, please refer to the description of the data allocation method above, which will not be repeated here.

[0182] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing a data allocation method according to an embodiment of the present disclosure.

[0183] like Figure 6As shown, the electronic device 600 includes a processing unit 601, which can be implemented as a CPU, GPU, FPGA, NPU, or other processing unit. The processing unit 601 can execute various processes according to any of the methods described above in this disclosure, based on a program stored in the read-only memory (ROM) 602 or a program loaded from the storage portion 608 into the random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0184] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0185] In particular, according to embodiments of this disclosure, any of the methods described above in the embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing any of the methods in the embodiments of this disclosure. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0188] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0189] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A data allocation method, characterized in that, include: Obtain the grouping matrix construction parameters, which include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes. The heterogeneous node group includes at least one heterogeneous node. The expected latency of the heterogeneous node is the expected time consumed by the heterogeneous node to complete the subset of data to be processed assigned by the master node and send back the corresponding result information to the master node. A grouping matrix is ​​constructed based on the grouping matrix construction parameters. The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column of the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes. Obtain a pre-generated data allocation matrix, and update the data allocation matrix based on the heterogeneous node groups in the grouping matrix; The updated data allocation matrix is ​​used to allocate a subset of data to be processed to heterogeneous nodes.

2. The method according to claim 1, characterized in that, The construction of the grouping matrix based on the grouping matrix construction parameters includes: The heterogeneous nodes are sorted in ascending order of expected latency; The grouping matrix is ​​constructed based on the sorted heterogeneous nodes and the grouping matrix construction parameters.

3. The method according to claim 2, characterized in that, Also includes: Get the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes; The number of heterogeneous node groups is determined based on the total number of data subsets to be processed and the number of data subsets to be processed allocated to heterogeneous nodes. Based on the total number of heterogeneous nodes and the number of heterogeneous node groups, the number of heterogeneous nodes included in each heterogeneous node group is determined.

4. The method according to claim 3, characterized in that, Also includes: Based on the total number of heterogeneous nodes, the number of heterogeneous node groups, the total number of data subsets to be processed, and the number of data subsets to be processed allocated to heterogeneous nodes, a data allocation matrix is ​​constructed. The data allocation matrix has a dimension of n rows × k columns, where n is the total number of heterogeneous nodes and n = d × q, k is the total number of data subsets to be processed and k = p × q, and p is the number of data subsets to be processed allocated to heterogeneous nodes. Each row of the data allocation matrix corresponds to a heterogeneous node, and each column corresponds to a data subset to be processed. The data allocation matrix includes q × q block sub-matrices, each block sub-matrice having a dimension of d rows × p columns. The elements of the block sub-matrices located on the diagonal have a value of 1, and the elements of the block sub-matrices located off-diagonally have a value of 0. The heterogeneous nodes corresponding to the block sub-matrices located on the diagonal constitute a heterogeneous node group.

5. The method according to claim 4, characterized in that, The step of updating the data allocation matrix based on the heterogeneous node groups in the grouping matrix includes: Determine the block submatrices on the diagonal of the data allocation matrix; Determine the heterogeneous node groups corresponding to the block submatrices on the diagonal and the heterogeneous node groups corresponding to a column of elements in the grouping matrix; Based on the heterogeneous node group corresponding to a column of elements in the grouping matrix, update the heterogeneous node group corresponding to the block submatrix on the diagonal to obtain the updated data allocation matrix.

6. The method according to claim 5, characterized in that, The process of allocating a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix includes: Determine the heterogeneous nodes and data subsets to be processed corresponding to the diagonal submatrices in the updated data allocation matrix; Assign the data subset to be processed corresponding to the diagonal submatrix to the heterogeneous nodes corresponding to the diagonal submatrix.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: In response to receiving a first aggregation result from at least one of the heterogeneous nodes, the current node's failure status is determined based on the heterogeneous node that sent the first aggregation result. The first aggregation result is determined based on the first result information obtained by the heterogeneous node from processing the allocated subset of data to be processed and the heterogeneous node aggregation coefficient vector of the heterogeneous node. The heterogeneous node aggregation coefficient vector has a dimension of 1 row × p columns, and the value of p is the number of subsets of data to be processed allocated by the master node to the heterogeneous node. Based on the current node dropout situation, a set of master node linear combination coefficients corresponding to the current node dropout situation is determined from the pre-generated master node aggregation coefficient matrix. The master node aggregation coefficient matrix has a dimension of m rows × n columns, where m is the number of node dropout situations and n is the total number of heterogeneous nodes. Each row of the master node aggregation coefficient matrix corresponds to a node dropout situation, and each column corresponds to a heterogeneous node. Each element corresponds to the linear coefficient used by the master node to return the first aggregation result to the heterogeneous node under different combinations of heterogeneous nodes and different node dropout situations. The second aggregation result is determined based on the linear combination coefficients of the master node and at least one of the first aggregation results.

8. The method according to claim 7, characterized in that, Also includes: Obtain various node dropout scenarios under the total number of heterogeneous nodes; The master node aggregation coefficient matrix is ​​constructed based on the total number of heterogeneous nodes and the various node dropout scenarios.

9. A data distribution device, characterized in that, include: The acquisition module is configured to acquire group matrix construction parameters, which include: the total number of heterogeneous nodes, the number of heterogeneous node groups, and the expected latency of heterogeneous nodes. The heterogeneous node group includes at least one heterogeneous node. The expected latency of the heterogeneous node is the expected time consumed by the heterogeneous node to complete the subset of data to be processed assigned by the master node and send the corresponding result information back to the master node. The grouping matrix construction module is configured to construct a grouping matrix based on the grouping matrix construction parameters. The grouping matrix has a dimension of d rows × q columns, where d is the number of heterogeneous nodes contained in the heterogeneous node group, q is the number of heterogeneous node groups, and d × q is the total number of heterogeneous nodes. Each element in the grouping matrix corresponds to a heterogeneous node, and each column of the grouping matrix corresponds to a heterogeneous node group. In the adjacent rows of the grouping matrix, the elements in one row of the odd-numbered row and the elements in the other row are sorted in ascending order of the expected latency of the heterogeneous nodes, and the elements in the other row are sorted in descending order of the expected latency of the heterogeneous nodes. The update module is configured to obtain a pre-generated data allocation matrix and update the data allocation matrix based on the heterogeneous node groups in the grouping matrix; The allocation module is configured to allocate a subset of data to be processed to heterogeneous nodes based on the updated data allocation matrix.

10. An electronic device, wherein, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-8.

11. A computer-readable storage medium having stored thereon computer instructions, wherein, When executed by a processor, the computer instructions implement the method described in any one of claims 1-8.

Images