A data processing method and apparatus applied to a heterogeneous cluster

Abstract

This specification discloses a data processing method and apparatus applied to a heterogeneous cluster. The method includes: performing reduction calculation on task data on each computing node in a first computing cluster to obtain first result data corresponding to the first computing cluster; detecting whether a completion signal sent by a second computing cluster before the current moment is received, and if so, sending the completion signal to the second computing cluster. If not, sending the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, performing reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and performing reduction calculation based on the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until a termination condition is satisfied to obtain target result data; broadcasting the target result data to each second computing cluster.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular, to a data processing method and apparatus applied to heterogeneous clusters. Background Art

[0002] Collective communication, as an indispensable collaboration mechanism in the field of distributed computing, enables multi-node computing clusters to efficiently interact and jointly tackle complex computing problems. This model has demonstrated strong vitality in many fields such as distributed model training, big data analysis, and high-performance scientific computing.

[0003] In a heterogeneous computing environment, different computing clusters are often constructed by diverse hardware architectures, such as those equipped with Graphics Processing Units (GPUs), Central Processing Units (CPUs), Tensor Processing Units (TPUs), and other artificial intelligence acceleration chips. These heterogeneous computing nodes not only possess unique architectural advantages but also exhibit distinct performance characteristics, forming distinct multi-dimensional heterogeneous features in terms of hardware configuration, which poses new challenges for system-level optimization.

[0004] However, due to the heterogeneity between computing clusters, the time taken for different computing clusters to complete intra-cluster communication varies. Currently, during the process of cross-cluster communication, it is necessary to wait for all computing clusters to complete the reduction calculation within the cluster before further performing inter-cluster communication. This results in the inability to synchronize the communication times of each computing cluster, wasting communication time and seriously affecting the data processing efficiency between heterogeneous clusters. Summary of the Invention

[0005] This specification provides a data processing method and apparatus applied to heterogeneous clusters to partially solve the above problems existing in the prior art.

[0006] This specification adopts the following technical solutions:

[0007] This specification provides a data processing method applied to a heterogeneous cluster. The method is applied to a first computing cluster, where the first computing cluster is any one of multiple computing clusters jointly executing a target task, and the multiple computing clusters are heterogeneous clusters. The method includes:

[0008] Performing a reduction calculation on the task data on each computing node in the first computing cluster to obtain first result data corresponding to the first computing cluster, and generating a completion signal for the reduction calculation;

[0009] Detect whether a completion signal sent by the second computing cluster before the current moment is received. If so, send the completion signal to the second computing cluster; wherein, the second computing cluster is any one of the multiple computing clusters other than the first computing cluster;

[0010] If not, send the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, perform reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until the termination condition is met to obtain the target result data;

[0011] Broadcast the target result data to each second computing cluster.

[0012] Optionally, detecting whether a completion signal sent by the second computing cluster before the current moment is received specifically includes:

[0013] Query through the first target node in each computing node of the first computing cluster whether a completion signal sent by the second computing cluster before the current moment is received;

[0014] If not, sending the completion signal to each second computing cluster specifically includes:

[0015] Send the completion signal to the second target node in each second computing cluster;

[0016] Performing reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data specifically includes:

[0017] Perform reduction calculation on the first result data on the first target node and the first result data on the second target node to obtain the second result data.

[0018] Optionally, the method further includes:

[0019] Construct a first communication domain according to the computing processes running on each computing node in the first computing cluster;

[0020] Broadcast the target result data in the first communication domain through the first target node to synchronize the target result data to each computing node in the first computing cluster.

[0021] Optionally, before sending the completion signal to each second computing cluster, the method further includes:

[0022] Construct a second communication domain according to the computing processes running on the first target node and the computing processes running on the second target nodes in each second computing cluster;

[0023] Send the completion signal to the second target nodes in each second computing cluster, specifically including:

[0024] Send the completion signal to the second target nodes in each second computing cluster through the second communication domain.

[0025] Optionally, before performing reduction calculation based on the second result data and the first result data corresponding to other second computing clusters for which the completion signal is subsequently sent, the method further includes:

[0026] Construct a signal queue on the first target node, and add the received completion signals to the signal queue in the order of sending of the completion signals, where the completion signal carries the device identifier corresponding to the second target node that sends the completion signal.

[0027] Optionally, perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters for which the completion signal is subsequently sent, specifically including:

[0028] Poll the completion signals in the signal queue in sequence, and perform reduction calculation based on the second result data and the first result data of the second computing cluster corresponding to the device identifier carried by the polled completion signal.

[0029] Optionally, the termination condition includes: the first computing cluster completes the reduction calculation with each second computing cluster;

[0030] The target task includes: a model training task.

[0031] This specification provides a data processing device applied to a heterogeneous cluster, which is applied to a first computing cluster, and the first computing cluster is any one of multiple computing clusters that jointly execute a target task, and the multiple computing clusters are heterogeneous clusters, including:

[0032] A first computing module, configured to perform reduction calculation on task data on each computing node in the first computing cluster to obtain first result data corresponding to the first computing cluster, and generate a completion signal for the reduction calculation;

[0033] A detection module, configured to detect whether a completion signal sent by a second computing cluster before the current moment is received, and if so, send the completion signal to the second computing cluster; where the second computing cluster is any one of the multiple computing clusters other than the first computing cluster;

[0034] A second computing module, configured to, if not, send the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, perform reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until the termination condition is met to obtain target result data;

[0035] A broadcast module, configured to broadcast the target result data to the second computing clusters.

[0036] This specification provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the data processing method for heterogeneous clusters described above is implemented.

[0037] This specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the data processing method for heterogeneous clusters described above is implemented.

[0038] At least one of the technical solutions adopted in this specification can achieve the following beneficial effects:

[0039] In the cross-cluster data processing method provided in this specification, reduction calculation is performed on the task data on each computing node in the first computing cluster to obtain the first result data corresponding to the first computing cluster, and a completion signal for the reduction calculation is generated; it is detected whether a completion signal sent by the second computing cluster before the current moment is received; if not, the completion signal is sent to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, reduction calculation is performed based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data; reduction calculation is performed based on the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until the termination condition is met to obtain target result data, and the target result data is broadcast to each second computing cluster.

[0040] As can be seen from the above method, in this solution, when each computing cluster completes the reduction calculation within the cluster, it will query whether it has received the completion signal sent by other computing clusters before. If it has received the completion signal sent by other computing clusters before, it will only send the completion signal to that computing cluster. If it has not received the completion signals of other clusters, it will send the completion signal to all other computing clusters. For any computing cluster, when it detects that other computing clusters have not sent the completion signal before, it means that this computing cluster is the first computing cluster to complete the reduction calculation, and then this computing cluster sends the reduction completion signal to all other computing clusters; when it detects that other computing clusters have sent the completion signal before, it means that this cluster is not the first computing cluster to complete the reduction calculation, and the received completion signal is the reduction completion signal of the first computing cluster. Then, at this time, it sends the reduction completion signal to the first computing cluster that has completed the reduction calculation. After the first computing cluster that has completed the reduction calculation receives the completion signal of any computing cluster, it can immediately execute the reduction calculation with that computing cluster without waiting for the computing clusters that have not completed the intra-cluster calculation, thereby shortening the time of the collective communication process and improving the data processing efficiency between heterogeneous clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present specification and form a part of the present specification. The schematic embodiments of the present specification and their descriptions are used to explain the present specification and do not constitute an improper limitation to the present specification. In the drawings:

[0042] Figure 1 It is a schematic flow chart of a data processing method applied to heterogeneous clusters provided in the present specification;

[0043] Figure 2 It is a schematic hardware architecture diagram of cross-cluster communication provided in the present specification;

[0044] Figure 3 It is a flow chart of cross-heterogeneous cluster communication provided in the present specification;

[0045] Figure 4 It is a schematic diagram of a data processing device applied to heterogeneous clusters provided in the present specification;

[0046] Figure 5 It is provided in the present specification corresponding to Figure 1 a schematic diagram of an electronic device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] To make the objectives, technical solutions, and advantages of this specification clearer, the following will clearly and completely describe the technical solutions of this specification in combination with specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this specification without creative efforts fall within the scope of protection of this specification.

[0048] In recent years, large models such as GPT-3 and GPT-4 have gradually emerged. These large models can perform general and somewhat complex tasks such as code generation, and have an increasingly significant impact on the field of artificial intelligence. However, training these large models requires a large number of intelligent hardware devices. If only using the intelligent hardware devices of a single manufacturer, there will be a dependence on the intelligent hardware devices of that single manufacturer. Due to the delivery time issue of the manufacturer's intelligent hardware devices, it may affect the construction and training of the cluster. In this context, purchasing intelligent hardware devices from multiple manufacturers and building a heterogeneous cluster by combining the intelligent hardware devices of multiple manufacturers becomes a feasible solution.

[0049] During the process of collective communication between heterogeneous clusters, the heterogeneous collective communication requests can be transformed into collective communication within homogeneous clusters and collective communication operations between heterogeneous clusters to achieve efficient heterogeneous collective communication. For example, the communication between heterogeneous clusters can be realized by using in-cluster reduction calculation (homogeneous reduce) + cross-cluster reduction calculation (heterogeneous Allreduce) + homogeneous broadcast.

[0050] However, due to the inconsistent computing capabilities and bandwidths of clusters with different architectures, the time for in-cluster reduction calculation within clusters of different architectures is also different. As the number of computing nodes within the homogeneous cluster increases, the time for in-cluster reduction calculation also increases, and the gap in the completion time of in-cluster reduction calculation between clusters of different architectures will increase. If the traditional method of in-cluster reduction calculation + cross-cluster reduction calculation + homogeneous broadcast is used for cross-cluster communication, after the in-cluster reduction calculation of all clusters is completed, the cross-cluster reduction calculation is simultaneously performed among the target nodes of the in-cluster reduction calculation. There will be a situation where the target nodes that have completed the in-cluster reduction calculation within the homogeneous cluster earlier wait for other homogeneous clusters to complete the in-cluster reduction calculation. The larger the scale of the homogeneous cluster, the greater the gap in the completion time of in-cluster reduction calculation, and the longer the waiting time. Moreover, the more the number of clusters with different architectures, the more target nodes of the in-cluster reduction calculation within the homogeneous clusters that need to perform cross-cluster reduction calculation, and the longer the time for cross-cluster reduction calculation among the target nodes of the in-cluster reduction calculation within the homogeneous clusters.

[0051] The following will detail the technical solutions provided by each embodiment of this specification in combination with the drawings.

[0052] Figure 1 A schematic flow diagram of a data processing method applied to a heterogeneous cluster provided in this specification, including the following steps:

[0053] S101: Perform reduction calculation on the task data on each computing node in the first computing cluster to obtain the first result data corresponding to the first computing cluster, and generate a completion signal for the reduction calculation.

[0054] In practical applications, a collective communication system may include several computing clusters that jointly execute a target task. These computing clusters can be heterogeneous clusters, that is, the computing nodes in different computing clusters show heterogeneous configurations, and one or more of the types (such as GPUs and NPUs), device models, and manufacturers of the computing nodes in different computing clusters are different.

[0055] Among them, the above computing nodes may include: GPUs, CPUs, NPUs, TPUs, neural network processors (Neural network Processing Unit, NPU), and other AI chips, which are not specifically limited in this specification.

[0056] The above target task may be a model training task. Correspondingly, its task data may be parameter data of the model training task (such as gradient data, feature vectors, loss values). Of course, the target task may also be other distributed computing tasks. Correspondingly, the task data may be transaction data, risk control data for distributed data processing tasks in the financial field, medical data for distributed data processing tasks in the medical field, etc.

[0057] In the process of cross-cluster data processing of the task data of the target task, for any computing cluster (hereinafter referred to as the first computing cluster) that executes the target task, the computing cluster can first perform reduction calculation on the task data on each computing node in the cluster to obtain the first result data within the cluster, and can generate a completion signal for the reduction calculation. And synchronize the first result data to each computing node in the computing cluster. Among them, the above completion signal is used to indicate that the current computing cluster has completed the reduction calculation among the computing nodes within the cluster.

[0058] In this process, for each computing cluster, the computing cluster can construct the first communication domain corresponding to the cluster according to the computing processes running on each computing node therein.

[0059] In addition, there is at least one target node among the computing nodes in each computing cluster, and each target node is marked with a unique identifier. The target node can be selected and set in advance among the computing nodes. Of course, it can also be automatically selected by the computing cluster. For example, the target node can be the master node among these nodes (that is, the node with the serial number 0 among each node).

[0060] For example, for computing clusters A, B, and C that execute a target task, the first communication domains corresponding to the respective computing clusters are Ca, Cb, and Cc, and the target nodes corresponding to the respective computing clusters are A1 (the first target node), B1 (the second target node), and C1 (the third target node).

[0061] S102: Detect whether a completion signal sent by a second computing cluster before the current moment is received. If so, send the completion signal to the second computing cluster; where the second computing cluster is any computing cluster other than the first computing cluster among the multiple computing clusters;

[0062] S103: If not, send the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, perform reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until a termination condition is met to obtain target result data.

[0063] The first computing cluster can send a completion signal to other computing clusters (hereinafter simply referred to as second computing clusters) and simultaneously receive the completion signal sent by the second computing cluster.

[0064] During this process, a second communication domain can be constructed according to the computing processes running on the first target node and the computing processes running on the second target nodes in each second computing cluster, so as to construct a second communication domain according to the computing processes running on the target nodes in each computing cluster. For the sake of understanding, this specification provides a schematic diagram of a communication architecture between heterogeneous clusters, as Figure 2 shown.

[0065] Figure 2 This is a schematic diagram of a communication architecture between heterogeneous clusters provided in this specification.

[0066] Among them, the first communication domain corresponding to each computing cluster is composed of the respective computing nodes included in the computing cluster, and the second communication domain between heterogeneous clusters is composed of the target nodes in each computing cluster.

[0067] The first computing cluster can send the completion signal to the second target node through the first target node, and, through the first target node, receive the completion signal sent by the second target node in the second communication domain.

[0068] Among them, the first computing cluster can query through the first target node whether it has received the completion signal sent by the second computing cluster before the current moment. If it has received the completion signal sent by the second computing cluster before the current moment, it sends the completion signal to the second target node in the second cluster through the second communication domain. If it has not received the completion signal sent by the second computing cluster before the current moment, it sends the completion signal to the second target node in each second computing cluster through the second communication domain.

[0069] When the first computing cluster has not received the completion signal sent by the second computing cluster before the current moment, it indicates that each second computing cluster has not completed the in-cluster reduction calculation locally, that is, the first computing cluster is the first computing cluster to complete the local reduction calculation.

[0070] At this time, the first target node can be used as the root computing node, and a signal queue can be constructed on the first target node. When the first target node receives the completion signals sent by the computing nodes in other computing clusters, it can add these completion signals to the signal queue in the order of their respective sending times.

[0071] Among them, the completion signal sent by each target node can carry the device identifier of the computing cluster to which the second target node belongs, so as to determine which computing cluster the target node indicates has completed the in-cluster reduction calculation.

[0072] When the first target node receives the completion signal sent by the second computing cluster, it indicates that the second computing cluster has completed the in-cluster reduction calculation. Therefore, the first target node can perform the reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster, so as to obtain the second result data.

[0073] It should be added that for each computing cluster, if the computing cluster has received the completion signal sent by other computing clusters before its in-cluster calculation is completed, the computing node corresponding to the computing cluster that first completes the signal is used as the root computing node, and the above operations are performed in the root computing node.

[0074] For example, assume that at time tA, the isomorphic reduce in the communication domain Ca of computing cluster A is completed first. The target node A1 queries whether it has previously received the completion signal of the isomorphic reduce in the communication domain Cb and the completion signal of the isomorphic reduce in the communication domain Cc.

[0075] If the target node A1 does not receive any isomorphic reduce completion signals within the corresponding communication domain of any other computing clusters, the target node A1 can place its device identifier into the isomorphic reduce completion signal of Ca. In the communication domain Cd, the target node A1 sends the isomorphic reduce completion signal of Ca to other target nodes in Cd.

[0076] Suppose at time tB (tB > tA), the isomorphic reduce of the first communication domain Cb corresponding to the computing cluster B is completed. The target node B1 queries whether it has received any isomorphic reduce completion signals within other first communication domains. If it queries that it has received the isomorphic reduce completion signal of the communication domain Ca. The target node B1 can place its device identifier into the isomorphic reduce completion signal of the communication domain Cb and send the isomorphic reduce completion signal of the communication domain Cb to the target node A1.

[0077] The target node A1 can poll the signal queue in real time to determine whether the signal queue is empty. If not, it takes out the first element from it and pauses polling.

[0078] After the target node A1 receives the isomorphic reduce completion signal of the communication domain Cb, it adds the isomorphic reduce completion signal of the communication domain Cb to the signal queue constructed locally. When the target node A1 polls the isomorphic reduce completion signal of the communication domain Cb from it, it can form a new communication domain Cu (the third communication domain) with the target node B1 based on this completion signal, with the target node A1 as the root node. Perform heterogeneous reduce within Cu and stop polling at the same time. Thus, the second result data is obtained.

[0079] The first target node in the first computing cluster can sequentially poll the completion signals in the signal queue, and perform reduction calculations based on the second result data and the first result data of the second computing cluster corresponding to the device identifier carried by the polled completion signal until a preset condition is met to obtain the target result data.

[0080] Among them, the above preset condition can include that the first computing cluster completes the reduction calculation with each second computing cluster, or the number of second computing clusters performing the reduction calculation with the first computing cluster exceeds a set number.

[0081] S104: Broadcast the target result data to each second computing cluster.

[0082] After determining the target result data, the first computing cluster can, through its first target node, broadcast the target result data to each second computing cluster so that the target nodes in each second cluster can receive the target result data.

[0083] For any computing cluster, if the target result data is stored in the target node therein, the target result data can be broadcast in the first communication domain to synchronize the target result data to each computing node in the computing cluster, thereby completing the entire process of cross-cluster data processing.

[0084] Here, continuing from the above example, assume that at time tC (tC > tB), the isomorphic reduce in the first communication domain Cc corresponding to the computing cluster C is completed. The computing node C1 queries whether it has received the completion signal of the isomorphic reduce in other first communication domains.

[0085] If it is queried that the completion signal of the isomorphic reduce in the communication domain Ca has been received. The computing node C1 can place the device identifier of the computing node C1 into the completion signal of the isomorphic reduce in the communication domain Cc and send the completion signal of the isomorphic reduce in the communication domain Cc to the target node A1 in the communication domain Ca.

[0086] After the target node A1 receives the completion signal of the isomorphic reduce in the communication domain Cc, it can add the completion signal of the isomorphic reduce in the communication domain Cc to the signal queue, and after completing the reduction calculation with the target node B1, continue to poll, take out the completion signal of the isomorphic reduce in the communication domain Cc in the signal queue, form a new communication domain Cp (the third communication domain) with the target computing node A1 and the target node C1, with the target node A1 as the root node, and perform the reduction calculation between the target computing node A1 and the target node C1 within Cp. At this time, the data used by the target node A1 for the reduction calculation is the second result data obtained after the reduction calculation between the target node A1 and the target node B1.

[0087] Within the second communication domain Cd, with the target node A1 as the root computing node, the target result data can be broadcast to other target nodes (B1 and C1) in the communication domain Cd, thereby synchronizing the target result data to the target nodes in each computing cluster.

[0088] In the communication domain Ca, with the target node A1 as the root node, the target result data is broadcast in the first communication domain Ca to other computing nodes in the computing cluster A.

[0089] The target node B1 and the target node C1 will respectively receive the target result data and broadcast the target result data to each computing node in the cluster B and each computing node in the cluster C in the communication domains Cb and Cc respectively.

[0090] Based on the above steps, each heterogeneous computing cluster participating in the target task and each computing node in each computing cluster can obtain the target result data. For ease of understanding, this specification also provides a flowchart of cross-heterogeneous cluster communication, asFigure 3 as shown

[0091] Figure 3 It is a flowchart of cross - heterogeneous cluster communication provided in this specification.

[0092] Among them, each computing cluster performs homogeneous reduce within the cluster: all computing nodes within each computing cluster form a communication domain Ci, and homogeneous reduce operations are performed within the homogeneous cluster communication domain Ci. The target node in the first computing cluster that completes homogeneous reduce is denoted as RankF;

[0093] Since the time to complete homogeneous reduce in different computing clusters is different, before all computing clusters complete homogeneous reduce, heterogeneous reduce between the computing clusters that have completed homogeneous reduce is performed in advance. During this process, a communication domain Cv is constructed based on the target nodes in each computing cluster.

[0094] RankF performs heterogeneous reduce with other target nodes in sequence according to the time when different computing clusters complete homogeneous reduce, and finally obtains the target result data. Then RankF broadcasts the target result data to the target nodes of other computing clusters in the communication domain Cv, so that each target node obtains the target result data. After that, for each target node Ranki storing the target result data, the target result data can be further broadcast in its communication domain Ci, so as to synchronize the target result data to each computing node.

[0095] As can be seen from the above method, in the process of data processing across heterogeneous clusters in this solution, homogeneous reduce within the homogeneous cluster is first performed. Before all homogeneous reduces within the homogeneous clusters are completed, heterogeneous reduce between the clusters that have completed homogeneous reduce is performed in advance, avoiding the waiting time for the calculation results of other clusters that have not completed homogeneous reduce within the cluster, and improving the efficiency of data processing across heterogeneous clusters.

[0096] The above is a method for executing a computing task applied to collective communication in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding device for executing a computing task applied to collective communication, such as Figure 4 as shown

[0097] Figure 4 It is a schematic diagram of a data processing device applied to a heterogeneous cluster provided in this specification, including:

[0098] The first computing module 401 is configured to perform reduction computing on the task data on each computing node in the first computing cluster, obtain the first result data corresponding to the first computing cluster, and generate a completion signal for the reduction computing;

[0099] The detection module 402 is configured to detect whether a completion signal sent by the second computing cluster before the current moment is received. If so, the completion signal is sent to the second computing cluster; wherein, the second computing cluster is any computing cluster other than the first computing cluster among the multiple computing clusters;

[0100] The second computing module 403 is configured to, if not, send the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, perform reduction computing according to the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and perform reduction computing according to the second result data and the first result data corresponding to other second computing clusters that send the completion signal subsequently until a termination condition is met to obtain target result data;

[0101] The broadcast module 404 is configured to broadcast the target result data to each second computing cluster.

[0102] Optionally, the detection module 402 is specifically configured to query, through the first target node in each computing node of the first computing cluster, whether a completion signal sent by the second computing cluster before the current moment is received;

[0103] The second computing module 403 is specifically configured to send the completion signal to the second target node in each second computing cluster;

[0104] Performing reduction computing according to the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data specifically includes:

[0105] Performing reduction computing on the first result data on the first target node and the first result data on the second target node to obtain the second result data.

[0106] Optionally, the broadcast module 404 is further configured to construct a first communication domain according to the computing processes running on each computing node in the first computing cluster; and through the first target node, broadcast the target result data in the first communication domain to synchronize the target result data to each computing node in the first computing cluster.

[0107] Optionally, before sending the completion signal to each second computing cluster, the detection module 402 is further configured to construct a second communication domain according to the computing processes running on the first target node and the computing processes running on the second target nodes in each second computing cluster.

[0108] The second computing module 403 is specifically configured to send the completion signal to the second target nodes in each second computing cluster through the second communication domain.

[0109] Optionally, before performing reduction calculation according to the second result data and the first result data corresponding to other second computing clusters to which the subsequent sent completion signals belong, the detection module 402 is further configured to construct a signal queue on the first target node, and add the received completion signals to the signal queue in the order in which the completion signals are sent, where the completion signal carries the device identifier corresponding to the second target node that sends the completion signal.

[0110] Optionally, the second computing module 403 is specifically configured to sequentially poll the completion signals in the signal queue, and perform reduction calculation according to the second result data and the first result data of the second computing cluster corresponding to the device identifier carried by the polled completion signal.

[0111] Optionally, the termination condition includes: the first computing cluster completes the reduction calculation with each second computing cluster; the target task includes: a model training task.

[0112] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 provided data processing method for heterogeneous clusters.

[0113] This specification also provides Figure 5 a schematic structural diagram of an electronic device corresponding to Figure 1 as shown. As Figure 5 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 described data processing method for heterogeneous clusters. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or logic devices.

[0114] For an improvement in a technology, it can be clearly distinguished whether it is a hardware improvement (e.g., improvement in circuit structures such as diodes, transistors, switches, etc.) or a software improvement (improvement in method processes). However, with the development of technology, many improvements in method processes today can be regarded as direct improvements in hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method process into the hardware circuit. Therefore, it cannot be said that an improvement in a method process cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user programming the device. The designer programs by himself to "integrate" a digital system on a piece of PLD, without the need to ask a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog.

[0115] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to make the controller implement the same function in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or the structures within the hardware component.

[0116] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0117] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0119] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0122] In a typical configuration, a computing node includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0123] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing nodes. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0128] The various embodiments in this specification are described in a progressive manner. For the parts that are the same or similar among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0129] The above description is only for the embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A data processing method applied to a heterogeneous cluster, characterized in that The method is applied to a first computing cluster, which is any one of multiple computing clusters that jointly execute a target task. The multiple computing clusters are heterogeneous clusters. The method includes: Performing reduction calculation on task data on each computing node in the first computing cluster to obtain first result data corresponding to the first computing cluster, and generating a completion signal for the reduction calculation; Detecting whether a completion signal sent by a second computing cluster before the current moment is received. If so, sending the completion signal to the first second computing cluster that sent the completion signal, and when the second computing cluster performs reduction calculation, sending the first result data to the second computing cluster, so that the second computing cluster performs reduction calculation in the order of the received completion signals based on its own corresponding first result data and the received first result data until a termination condition is met to obtain target result data; where the second computing cluster is a computing cluster other than the first computing cluster among the multiple computing clusters; If not, sending the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, performing reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and performing reduction calculation based on the second result data and the first result data corresponding to other second computing clusters that send subsequent completion signals until the termination condition is met to obtain target result data; Broadcasting the target result data to each computing cluster.

2. The method according to claim 1, wherein Detecting whether a completion signal sent by a second computing cluster before the current moment is received specifically includes: Querying through a first target node in each computing node of the first computing cluster whether a completion signal sent by a second computing cluster before the current moment is received; If not, sending the completion signal to each second computing cluster specifically includes: Sending the completion signal to a second target node in each second computing cluster; Performing reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data specifically includes: Performing reduction calculation on the first result data on the first target node and the first result data on the second target node to obtain the second result data.

3. The method according to claim 2, wherein The method further includes: Constructing a first communication domain according to the computing processes running on each computing node in the first computing cluster; Broadcasting the target result data in the first communication domain through the first target node to synchronize the target result data to each computing node in the first computing cluster.

4. The method according to claim 2, wherein Before sending the completion signal to each second computing cluster, the method further includes: Constructing a second communication domain according to the computing process running on the first target node and the computing processes running on the second target nodes in each second computing cluster; Sending the completion signal to a second target node in each second computing cluster specifically includes: Send the completion signal to the second target nodes in each second computing cluster via the second communication domain.

5. The method according to claim 2, wherein Before performing reduction calculation based on the second result data and the first result data corresponding to other second computing clusters for which the completion signal is subsequently sent, the method further includes: Construct a signal queue on the first target node, and add the received completion signals to the signal queue in the order in which the completion signals are sent, where the completion signal carries the device identifier corresponding to the second target node that sent the completion signal.

6. The method according to claim 5, characterized in that, Perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters for which the completion signal is subsequently sent, specifically including: Poll the completion signals in the signal queue in sequence, and perform reduction calculation based on the second result data and the first result data of the second computing cluster corresponding to the device identifier carried by the polled completion signal.

7. The method according to claim 1, wherein The termination condition includes: the first computing cluster completes the reduction calculation with each second computing cluster; The target task includes: a model training task.

8. A data processing device applied to a heterogeneous cluster, characterized in that, Applied to a first computing cluster, the first computing cluster is any one of multiple computing clusters that jointly execute a target task, and the multiple computing clusters are heterogeneous clusters, including: A first computing module, configured to perform reduction calculation on the task data on each computing node in the first computing cluster to obtain the first result data corresponding to the first computing cluster, and generate a completion signal for the reduction calculation; A detection module, configured to detect whether a completion signal sent by a second computing cluster before the current moment is received. If so, send the completion signal to the first second computing cluster that sent the completion signal, and when the second computing cluster performs reduction calculation, send the first result data to the second computing cluster, so that the second computing cluster performs reduction calculation in the order of the received completion signals according to its own corresponding first result data and the received first result data until the termination condition is met to obtain the target result data; where the second computing cluster is a computing cluster other than the first computing cluster among the multiple computing clusters; A second computing module, configured to, if not, send the completion signal to each second computing cluster, and after receiving the completion signal sent by the second computing cluster, perform reduction calculation based on the first result data corresponding to the first computing cluster and the first result data corresponding to the second computing cluster to obtain second result data, and perform reduction calculation based on the second result data and the first result data corresponding to other second computing clusters for which the completion signal is subsequently sent until the termination condition is met to obtain the target result data; A broadcast module, configured to broadcast the target result data to each computing cluster.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 above is implemented.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 7 above is implemented.

Images