Heterogeneous node communication methods, computing nodes, computing clusters and storage media

Abstract

This application provides a heterogeneous node communication method, a computing node, a computing cluster, and a storage medium. The heterogeneous node communication method includes: obtaining a communication group required for the communication task; obtaining the device identifier of the second processor bound to each process in the communication group; determining whether the communication group includes heterogeneous nodes based on the device identifier; and, if the communication group includes heterogeneous nodes, controlling each process to interact with other processes in the communication group by calling a preset heterogeneous communication library when executing the communication task. This method enables communication between processors other than the CPU in heterogeneous nodes without using CPU intermediaries, reducing CPU resource consumption and limiting the performance limitations of computing nodes.

Description

Technical Field

[0001] This application relates to the field of integrated circuits, and more specifically, provides a heterogeneous node communication method, a computing node, a computing cluster, and a storage medium. Background Technology

[0002] A computing cluster consists of multiple computing nodes, and each computing node includes multiple processor devices. The processors can be such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), AI (Artificial Intelligence) accelerator cards, etc. For example, a computing node may include 1 to 2 CPUs, 8 GPUs or AI accelerator cards.

[0003] While the processors used by computing nodes may come from the same manufacturer, the processors used by different computing nodes may come from different manufacturers. Different manufacturers will configure corresponding communication libraries for their respective processor devices. Since the communication libraries of different manufacturers are different, the processor devices of different manufacturers cannot communicate directly to exchange data, resulting in a communication isolation problem. Computing nodes using different communication libraries can also be called heterogeneous nodes.

[0004] When multiple computing nodes work together in a communication group, heterogeneous nodes may be used. Currently, to achieve cross-device communication, a CPU-intermediate communication mode is used, which relays data through the CPU's memory. For example, if the communication group includes computing node A and computing node B, when computing node A and computing node B communicate, the GPU in computing node A must first send instructions to the CPU in computing node A via the PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard). Then, the CPU in computing node A forwards the instructions to the CPU in computing node B, and the CPU in computing node B executes the corresponding operations on the GPU in computing node B, thus realizing the data interaction between the GPUs of computing node A and computing node B.

[0005] Using the CPU as a relay not only consumes CPU resources but also limits the size of the computing cluster and communication group due to CPU performance limitations. For example, when processors other than GPUs need to perform data interaction, they consume CPU computing resources and bandwidth, preventing the CPU from focusing on computational tasks. Furthermore, when CPU performance is limited, it's difficult for the CPU to support a large number of GPU interactions, resulting in some GPUs being idle, which in turn hinders the expansion of the computing cluster and communication group. Summary of the Invention

[0006] In view of this, this application aims to provide a heterogeneous node communication method, computing node, computing cluster, and storage medium, so that processors other than CPUs can also communicate directly between heterogeneous nodes, such as enabling direct communication between GPUs of heterogeneous nodes, reducing or avoiding the need for data transfer through the CPU during data interaction between heterogeneous nodes, thereby reducing the CPU resource consumption of data interaction between heterogeneous nodes and reducing the limitation of CPU performance on the scale of the computing cluster.

[0007] First, this application provides a heterogeneous node communication method applied to a first processor in a current computing node, the current computing node further including multiple second processors; the heterogeneous node communication method includes: obtaining a communication group required for a communication task; the communication task includes multiple processes executing the communication task, each process being bound to a different second processor, and each process in the communication task being allowed to be bound to a second processor in different computing nodes; obtaining the device identifier of the second processor bound to each process in the communication group; the device identifier is used to characterize the type of communication library used by the second processor; determining whether the communication group includes heterogeneous nodes based on the device identifier, wherein the heterogeneous nodes characterize that the second processor bound to each process comes from computing nodes using different types of communication libraries; if it is determined that the communication group includes the heterogeneous nodes, controlling each process to call a preset heterogeneous communication library to interact with other processes in the communication group when executing the communication task.

[0008] If the second processors used in the communication group include processors from different manufacturers and using different types of communication libraries, i.e., the second processors in the communication group come from heterogeneous nodes, this application provides a heterogeneous communication library to address this issue. This heterogeneous communication library supports use by heterogeneous nodes. Based on a unified heterogeneous communication library, there is no need for communication library conversion between heterogeneous nodes, thereby enabling communication between the second processors. Therefore, all computing nodes involved in the communication group are first linked to this heterogeneous communication library. Then, before the communication group starts working, it is determined whether there are heterogeneous nodes based on the device identifier of each second device. If heterogeneous nodes exist, the heterogeneous communication library is selected for communication. This allows the second processors in the communication group to achieve direct communication and establish direct end-to-end communication. During the communication process, the CPU does not need to perform calculations related to communication library conversion for data interaction, and can directly perform data interaction, reducing CPU relay and bandwidth consumption. Therefore, by adopting the above method, the computational resource and bandwidth consumption of the CPU can be effectively reduced, enabling data interaction between the heterogeneous nodes involved in the communication group to effectively reduce or even avoid CPU dependence, thereby reducing the limitation of CPU performance on the scale of the computing cluster. Furthermore, this method involves modifications to the communication library, which eliminates the need to modify the communication logic and code of the processors within the computing nodes themselves, helping to reduce implementation difficulty.

[0009] In one embodiment, each computing node includes a compatible interface layer; before controlling each process to execute the communication task and calling a preset heterogeneous communication library to interact with other processes in the communication group, the method further includes: when each process calls the initialization interface, hijacking the initialization interface based on the compatible interface layer, so as to control the calling of the heterogeneous communication library through the compatible interface layer; the initialization interface is used to call the communication library.

[0010] In this embodiment, after the upper-layer application of the computing node is compiled, it links to the underlying communication library. Typically, the linking instruction is sent to the dynamic link library (DLL), which then connects to the corresponding communication library for each device on the computing node. If this linking instruction is not intercepted, the computing node will normally link to its respective native communication library, thus preventing the selection of heterogeneous communication libraries. Therefore, in this embodiment, a compatibility interface layer can be used to replace the DLL, preventing the DLL from receiving the linking instruction and thus directly linking to its corresponding native communication library. Instead, the DLL links to both heterogeneous and native communication libraries through the compatibility interface layer, selecting the appropriate communication library based on the needs of the communication group.

[0011] In one embodiment, after determining whether the communication group includes heterogeneous nodes based on the device identifier, the method further includes: when it is determined that the communication group consists entirely of homogeneous nodes, controlling each process to call the native communication library corresponding to the current computing node to interact with other processes in the communication group when executing the communication task; wherein, the homogeneous node indicates that the second processor bound to each process comes from a computing node that uses the same type of communication library.

[0012] Compared to heterogeneous communication libraries, the native communication library corresponding to the computing node is more compatible with the devices within the computing node. For example, compared to using a heterogeneous communication library for communication, using a native communication library has lower requirements for communication latency and computational demands. Therefore, in the embodiments of this application, if the computing nodes of each processor in the communication group are homogeneous nodes from the same manufacturer and using the same type of communication library, then each computing node in the communication group can call the corresponding native communication library to maintain high-performance communication between the computing nodes in the communication group.

[0013] In one embodiment, each processor in each computing node uses the same communication library; the device identifier of the second processor to which each process is bound includes a first device identifier and a second device identifier; the first device identifier is the device identifier of the current computing node; the second device identifier is the second device identifier of each second processor in other computing nodes to which the process is bound; obtaining the device identifier of the second processor to which each process in the communication group is bound includes: obtaining the first device identifier of the current computing node; obtaining the second device identifier broadcast by other computing nodes based on a preset general communication protocol; the preset general communication protocol is the communication protocol supported by the second processors to which each process in the communication group is bound when no end-to-end connection is established.

[0014] Before linking the communication library, the processors of the computing nodes within the communication group cannot establish end-to-end links. Consequently, the processors of the computing nodes within the communication group cannot directly exchange data. Therefore, before linking the communication library, the local computing node cannot scan or know the communication library types of other computing nodes involved in the communication group. To address this issue, in this embodiment, each computing node can broadcast its device identifier through a preset general communication protocol. The general communication protocol is a communication protocol typically configured by each computing node, such as a communication protocol for communication with devices like CPUs or other types of devices supported within the node. By broadcasting, the device identifier can be sent and received. The broadcasting method does not require direct communication between computing nodes. Thus, even if an end-to-end connection has not yet been established, the computing nodes can still exchange information to obtain the device identifiers of other computing nodes.

[0015] In one embodiment, after obtaining the first device identifier, the method further includes: broadcasting the first device identifier based on the preset general communication protocol so that the first device identifier can be obtained by other computing nodes.

[0016] In this embodiment of the application, the processors of other computing nodes in the communication group also need to establish end-to-end communication with the processor of the local computing node. Similarly, they need to construct complete computing node topology information within the communication group. Therefore, the local computing node also needs to broadcast its own first device identifier in order to achieve information exchange.

[0017] In one embodiment, determining whether the communication group includes heterogeneous nodes based on the device identifier includes: determining whether there are different device identifiers among the first device identifier and all the second device identifiers; if different device identifiers are determined, determining that the communication group includes the heterogeneous nodes; if all device identifiers are determined to be the same, determining that each computing node in the communication group is a homogeneous node.

[0018] In this embodiment, it is not necessary to identify the communication library to determine whether each computing node in the communication group uses the same type of communication library. Instead, it is determined whether the communication library types used by each computing node are the same by whether the device identifiers are the same. This method can simplify the efficiency of determining whether the devices of each computing node are from the same manufacturer and use the same type of communication library.

[0019] In one embodiment, before invoking a preset heterogeneous communication library to interact with other processes in the communication group when controlling each of the processes to execute the communication task, the method further includes: obtaining first communication connection information broadcast by at least one other computing node based on a preset general communication protocol; the preset general communication protocol is a communication protocol supported by the second processor bound to each process in the communication group when no end-to-end connection is established; and controlling the second processor in the current computing node to establish a communication connection with the second processor in the computing node that broadcast the first communication connection information based on the first communication connection information.

[0020] Establishing an end-to-end connection between computing nodes requires communication connection information such as ports from both parties. Only after obtaining the communication connection information can an end-to-end connection be established. However, in this embodiment, after linking the heterogeneous communication library, the processors in different computing nodes within the communication group have not yet established an end-to-end connection between devices and cannot directly exchange communication connection information. Therefore, after linking the heterogeneous communication library, a common communication protocol can be used to broadcast the communication connection information so that the second processors in different computing nodes that need to establish communication can obtain the communication connection information of the other end and then establish an end-to-end connection.

[0021] In one embodiment, the computing nodes of different types of communication libraries record communication connection information using different structures. After obtaining the first communication connection information broadcast by at least one other computing node based on a preset general communication protocol, and before controlling the second processor in the current computing node to establish a communication connection with the second processor in the computing node that broadcast the first communication connection information based on the first communication connection information, the method further includes: reconstructing the first communication connection information based on a preset structure to configure the first communication connection information as a structure supported by the current computing node.

[0022] The devices used by heterogeneous nodes come from different manufacturers. Different manufacturers may have used different structures to define and record data during the development process, and the definitions of each piece of information may not match the heterogeneous communication library. This means that after the local computing node receives the first communication connection information sent by other computing nodes, the local computing node may not be able to directly use the first communication connection information. Therefore, in the embodiments of this application, after receiving the first communication connection information, a preset structure can be used to reconstruct the first communication connection information so that the local computing node can use the reconstructed first communication connection information and establish an end-to-end communication connection with the heterogeneous nodes in the communication group based on the reconstructed first communication connection information, thereby enabling accurate communication through the heterogeneous communication library.

[0023] In one embodiment, the first communication connection information includes the communication protocol supported by the second processor in the first target computing node that sent the first communication connection information. The step of obtaining the first communication connection information broadcast by at least one other computing node based on a preset general communication protocol includes: obtaining the second communication connection information of the current computing node; the second communication connection information includes the communication protocol supported by the second processor in the current computing node; determining whether the second processor in the current computing node supports IB (InfiniBand) communication based on the second communication connection information; if it is determined that the second processor in the current computing node supports IB communication, determining whether the second processor in the first target computing node supports IB communication based on the first communication connection information; if it is determined that the second processor in the first target computing node supports IB communication, controlling the second processor in the current computing node to establish a communication connection based on IB communication with the second processor in the first target computing node.

[0024] IB communication can bypass the operating system kernel protocol stack to achieve remote direct memory access in user space. This makes the communication method have lower latency and higher bandwidth. Based on this, in the embodiments of this application, it can be determined whether the two heterogeneous nodes supporting IB communication support it. If they do support it, data can be transmitted based on IB communication to obtain better communication results.

[0025] In one embodiment, if it is determined that the second processor in the current computing node does not support the IB communication or the second processor in the first target computing node does not support the IB communication, the method further includes: controlling the second processor in the current computing node to establish a communication connection with the second processor in the first target computing node based on the TCP / SOCKET protocol.

[0026] In this embodiment of the application, if either of the heterogeneous nodes in the communication does not support IB communication, the communication will revert to using the TCP (Transmission Control Protocol) / SOCKET protocol. The TCP / SOCKET protocol is a general communication protocol that is configured on various types of devices. This method has a wider range of applications and can meet the needs of more devices to establish end-to-end communication.

[0027] In one embodiment, each computing node further includes a network interface card (NIC); the step of controlling each process to execute the communication task and calling a preset heterogeneous communication library to interact with other processes in the communication group includes: for any process, in response to the execution of the process, determining the target second processor bound to the target process that interacts with the process; controlling the second processor bound to the process to call the NIC and communicate with the target second processor based on the heterogeneous communication library.

[0028] Communication between different computing nodes is achieved through network interface cards (NICs). Therefore, after establishing an end-to-end connection between the second processors, the second processors can be controlled to call the NICs to achieve end-to-end communication based on the execution of the processes.

[0029] In one embodiment, the step of controlling each process to execute the communication task and calling a preset heterogeneous communication library to interact with other processes in the communication group includes: for any process, in response to the execution of the process, controlling the second processor bound to the process to divide the interaction data into multiple data blocks based on a preset size; the interaction data is the interaction data sent by the second processor bound to the process to a target second processor; the target second processor is the second processor bound to any other process in the communication group; and sequentially transmitting the data blocks to the target second processor based on the heterogeneous communication library, wherein the target second processor supports performing calculations on previously received data blocks while receiving the data blocks.

[0030] In this embodiment, a data is divided into data blocks of fixed tube size. Thus, during communication, the sending, transmission, receiving, and local calculation of different data blocks overlap in time, forming a pipeline. This method can hide the potentially high network latency between heterogeneous nodes, make full use of network bandwidth, shorten the time required for data transmission, and improve the overall efficiency of heterogeneous communication.

[0031] In a second aspect, embodiments of this application provide a computing node, the computing node including a first processor, at least two second processors, and a network interface card (NIC), wherein the first processor is connected to each of the second processors and the NICs respectively, and the NICs are also connected to each of the second processors; the first processor is used to implement the heterogeneous node communication method as described in any of the first aspects.

[0032] Thirdly, embodiments of this application provide a computing cluster, including at least two computing nodes as described in the second aspect, wherein any number of the first processors and the second processors of any computing node in the computing cluster are used to form a communication group for communication.

[0033] Fourthly, embodiments of this application provide a readable storage medium storing a program that, when run on a processor, causes the processor to implement the heterogeneous node communication method as described in any of the first aspects. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram illustrating the communication between existing heterogeneous nodes. Figure 2 A flowchart illustrating a heterogeneous node communication method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a heterogeneous node communication process provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computing node provided in an embodiment of this application.

[0036] Icons: Compute node 400; First processor 410; Second processor 420; Network interface card 430; Storage unit 440. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] First, this application provides a heterogeneous node communication method, which can be applied to any computing node on a computing cluster, or configured on every computing node in the same computing cluster using the heterogeneous node communication method provided in this application.

[0039] To facilitate understanding of the heterogeneous node communication method provided in this application, the relationship and communication method between the computing cluster, computing nodes, and devices within the computing nodes will be explained first.

[0040] In the embodiments of this application, the computing cluster includes multiple computing nodes, each of which is a fully functional device. For example, a computing node may include processor devices such as CPUs, GPUs, and AI accelerator cards, as well as communication devices such as network cards. Exemplarily, some computing nodes may include 1 to 2 CPUs, 8 GPUs or AI accelerator cards, and the computing node may also include a network card. The processors and network cards within the same computing node are generally from the same manufacturer.

[0041] For ease of understanding, the following explanation will use an example of a computing node comprising a first processor, at least two second processors, and a network interface card (NIC). The first processor can be a CPU, and the second processors can be GPUs. The heterogeneous node communication method provided in this application can be executed by the first processor (CPU). After controlling the communication of each second processor (GPU), the second processors of different computing nodes can call the network of their respective nodes for end-to-end communication.

[0042] The processors and other devices and components in the same compute node are usually from the same manufacturer. However, due to differences in equipment cost and performance, different compute nodes in the same compute cluster may need to use equipment from different manufacturers. For example, compute node 1 and compute node 2 use equipment from different manufacturers, while compute node 1 and compute node 3 use equipment from the same manufacturer. In this case, compute node 1 and compute node 3 constitute homogeneous nodes, while compute node 1 and compute node 2 constitute heterogeneous nodes.

[0043] In the same computing cluster, any number of processors on any computing node can form a communication group. For example, computing node 1 and computing node 2 are heterogeneous nodes, and both computing node 1 and 2 include 4 secondary processors. In a communication group, there may be 3 secondary processors of computing node 1 and 2 secondary processors of computing node 2.

[0044] Different manufacturers develop their products relatively independently, resulting in differences between the products they create. One such difference lies in the different communication libraries used by each product. For example, they may provide their own native communication libraries for their respective products, which means that computing nodes composed of devices from different manufacturers may use different communication libraries.

[0045] After forming a communication group, the processors in the same communication group need to communicate with each other, which requires communication between the computing nodes. However, the communication libraries used by different manufacturers' devices are different, which causes serious communication isolation problems when heterogeneous nodes work together in a cluster environment. The communication libraries provided by each manufacturer can only support data exchange between their own devices, forming closed communication islands.

[0046] Therefore, current methods typically require CPU as an intermediary; please refer to [link / reference]. Figure 1 , Figure 1 This diagram illustrates communication between existing heterogeneous nodes. Compute Node 1 and Compute Node 2 are heterogeneous nodes. During communication, the CPU needs to receive operation instructions from the initiating device (such as any GPU in Compute Node 1). The CPU uses the Gloo library to translate the operation instructions and then performs tasks such as memory allocation, data serialization, asynchronous copying, and synchronous coordination for devices like GPUs based on the operation instructions. The CPU of the peer compute node receives the operation instructions, performs the corresponding translation calculations, and sends them to the receiving device (such as one of the GPUs in Compute Node 2). This consumes CPU computing resources and bandwidth, limiting the size of compute nodes and the compute cluster to CPU performance. It also makes communication between compute nodes more complex, increasing the difficulty of development and programming.

[0047] With the development of artificial intelligence and neural networks, the requirements for computing power of computing nodes and computing clusters have increased. The heterogeneous node communication method based on CPU relay has limited the improvement of computing power of computing nodes and computing clusters and cannot meet the needs.

[0048] Unlike heterogeneous node communication methods that rely on CPU relay, the heterogeneous node communication method provided in this application offers a heterogeneous communication library. This library allows heterogeneous nodes to use the same heterogeneous communication library, enabling other processors besides the CPU to achieve end-to-end communication. This reduces or avoids the use of the CPU as a relay, thereby mitigating the impact of CPU performance on communication between heterogeneous nodes and the improvement of computing performance of computing nodes and computing clusters.

[0049] Therefore, in the embodiments of this application, before enabling end-to-end communication between processors in each computing node through the heterogeneous node communication method provided in the embodiments of this application, it is also necessary to enable each heterogeneous node to call the heterogeneous communication library. Therefore, the heterogeneous node communication method in the embodiments of this application may include: S210 links to heterogeneous communication libraries and native communication libraries based on a compatible interface layer.

[0050] In the embodiments of this application, each computing node includes a compatible interface layer, which can be used to link to a preset heterogeneous communication library and the native communication library corresponding to each computing node, so that when a communication group is formed to perform tasks, the heterogeneous communication library can be selected to be called according to the actual computing nodes included in the communication group.

[0051] In the embodiments of this application, the communication library is a binary file that implements communication. The communication library is stored in the memory of each computing node device in the computing cluster, and the required communication library can be selected according to various situations. The method of linking the communication library can refer to existing technologies, for example, using a Loader (dynamic loader) to call the dlopen function, and then using the dlopen function to call the API of the communication library to complete the linking of the communication library.

[0052] In the embodiments of this application, the heterogeneous communication library uses an environment variable system independent of vendor-specific communication libraries to enforce the unification of key communication parameters and computational logic across computing nodes, including heterogeneous nodes. The heterogeneous communication library is derived by engineers through summarizing different types of communication libraries. By unifying core parameters such as data block size, communication algorithms, and protocols across different vendors' devices, this library ensures consistency in understanding and execution across devices, resolving collaboration failures or performance degradation caused by internal differences in native communication libraries from different vendors. This, in turn, improves the stability and efficiency of communication between heterogeneous nodes.

[0053] Native communication libraries can be communication libraries provided by various processor manufacturers, which will not be elaborated on here. Compared with heterogeneous communication libraries, native communication libraries have higher adaptability and can achieve better communication results for each computing node. Therefore, in the embodiments of this application, in addition to heterogeneous communication libraries, native communication libraries can also be selected to be called according to the actual task to obtain better communication results.

[0054] Next, the heterogeneous node communication method provided in this application will be elaborated. Please refer to [link / reference]. Figure 2 , Figure 2 This is a flowchart illustrating a heterogeneous node communication method provided in an embodiment of this application. The heterogeneous node communication method includes: S220, Obtain the communication group required for the communication task.

[0055] A communication task is a task in a computing cluster that requires communication between computing nodes to execute. This communication task may involve different computing nodes and may call the first and second processors of different computing nodes to execute the task.

[0056] In embodiments of this application, the communication task includes multiple processes executing the communication task, each process being bound to a different second processor. Accordingly, all processors used in executing the communication task together form a communication group.

[0057] In the embodiments of this application, a communication task can select any second processor from any computing node in the computing cluster to form a communication group. That is, each process of the communication task is allowed to be bound to a second processor in different computing nodes, which makes it possible for the second processor bound to each process in the communication group to come from heterogeneous nodes. Among them, the second processors of heterogeneous nodes use different types of communication libraries. Therefore, heterogeneous nodes can also represent that the second processor bound to the process comes from computing nodes using different types of communication libraries.

[0058] S230: Obtain the device identifier of the second processor to which each process in the communication group is bound.

[0059] In the embodiments of this application, it can be determined whether the second processor bound to each process in the communication group comes from a heterogeneous node. If it does not include a second processor from a heterogeneous node, the heterogeneous communication library can be used instead of the native communication library. Correspondingly, if it needs to be determined that the second processor bound to each process in the communication group includes a heterogeneous node, the heterogeneous communication library can be called.

[0060] In the embodiments of this application, because different manufacturers configure different communication libraries for their respective devices, device identifiers can be used to characterize the type of communication library used by each computing node and its internal first processor, second processor, and network interface card, etc. These device identifiers include, but are not limited to, PCIe bus ID, device ID, and vendor ID. Therefore, the device identifiers of the second processors to which each process is bound can be used to characterize the device identifiers, and thus, the existence of heterogeneous nodes can be determined based on these device identifiers to determine whether to call heterogeneous communication libraries.

[0061] In application development, once the upper-layer application on a compute node is compiled, it is linked to a communication library. This allows the upper-layer application to call hardware devices based on the API interfaces provided by the communication library. The communication library is a software component used to implement data exchange between devices or systems, typically providing interface and protocol support to simplify the communication process. Therefore, if an upper-layer application needs to access hardware resources of devices on different compute nodes to perform communication tasks, it must first link to the communication library. The upper-layer application can be something like PyTorch (a deep learning framework) or TensorFlow (an end-to-end open-source machine learning framework).

[0062] After the upper-layer application is developed, the lower-layer files dynamically compile and call the corresponding native communication libraries of each computing node. If it is necessary for each heterogeneous node to call a heterogeneous communication library, the call instruction needs to be intercepted so that the upper-layer application can call the heterogeneous communication library provided in this embodiment. If the link instruction is not intercepted, the device will normally link to its corresponding native communication library. In some embodiments of this application, the aforementioned compatible interface layer can be used to implement interception.

[0063] For example, in some embodiments of this application, before obtaining the device identifier of the second processor bound to each process in the communication group, the method may further include: hijacking the initialization interface based on the compatibility interface layer when each process calls the initialization interface, so as to determine whether the call to the heterogeneous node and the control heterogeneous communication library is included through the compatibility interface layer.

[0064] The initialization interface is used to call the communication library. That is, in the embodiments of this application, the compatibility interface layer can first link the heterogeneous communication library and each native communication library. When actually executing the communication task, the compatibility interface layer can intercept the initialization interface, so that it does not directly access the native communication library, but determines whether the communication group executing the communication task involves heterogeneous nodes, so as to select to call the heterogeneous communication library or the native communication library.

[0065] In the embodiments of this application, there are various ways to use the compatibility interface layer for interception. For example, in some embodiments of this application, the compatibility interface layer can be configured to replace the dynamic link library. For example, libnccl.so.2 can be replaced with libxccl.so, where libnccl.so.2 represents the dynamic link library and libxccl.so represents the compatibility interface layer.

[0066] In this embodiment, a compatible interface layer is used to replace the initialization interface so that the dynamic link library cannot receive the initialization interface and thus cannot call its corresponding communication library. Accordingly, the compatible interface layer receives the initialization interface and performs the subsequent process of determining the device type and selecting whether to call the heterogeneous communication library.

[0067] The initialization command uses an environment variable to point to the dynamic link library, sending the initialization command to the dynamic link library and invoking the communication library. Therefore, in some other embodiments of this application, the environment variable can also be adjusted to point to the compatible interface layer. For example, the environment variable can be adjusted to call the compatible interface layer, where the environment variable can be LD_PRELOAD. By modifying the environment variable, the link command can also be made to call the compatible interface layer, realizing the interception of link commands and the invocation of heterogeneous communication libraries, avoiding the device from directly calling its corresponding native communication library.

[0068] In the above embodiments of this application, by replacing the original dynamic link library with a compatible interface layer, the changes to the original code of the device are minimal, regardless of whether environment variables are modified. This approach can effectively reduce the threshold for use and migration costs, and expand the scope of application of this application.

[0069] In the embodiments of this application, after receiving the initialization interface, the compute node enters the network initialization phase, which may be the bootstrap phase. Device identifier exchange can be performed during the network initialization phase.

[0070] In the embodiments of this application, the device identifier of the second processor to which each process is bound includes a first device identifier and a second device identifier. The first device identifier is the device identifier of the current computing node, and the second device identifier is the second device identifier of the second processor in each of the other computing nodes to which the process is bound. The first and second device identifiers are used only for ease of distinction. In the embodiments of this application, any computing node in the communication group can be the current computing node, with the first device identifier being the current computing node, and the second device identifier being the device identifier of the other computing nodes involved in the communication group.

[0071] At this stage, communication connections have not yet been established between the computing nodes, and direct data exchange is impossible. Consequently, local computing nodes cannot obtain the device identifiers of other computing nodes. Therefore, the methods for obtaining device identifiers will differ depending on the specific computing node.

[0072] In the embodiments of this application, obtaining the device identifier of each device in the computing cluster may include: obtaining the first device identifier of the current computing node; and obtaining the second device identifier broadcast by other computing nodes based on a preset general communication protocol.

[0073] Local compute nodes can obtain their unique device identifier information, i.e., obtain the first device identifier, by calling the runtime API corresponding to each vendor's hardware. The methods by which local compute nodes obtain their own device identifier will not be elaborated upon here.

[0074] In this embodiment, the preset general communication protocol is the communication protocol supported by the second processor to which each process in the communication group is bound when no end-to-end connection is established. For example, the preset general communication protocol may be the TCP / SOCKETS protocol, or other implementations and communication protocols between the processor and other types of devices within the CPU or computing node, which are not limited here.

[0075] Since no end-to-end connection has been established between computing nodes, direct communication is impossible using only a preset general communication protocol. Therefore, in the embodiments of this application, each computing node can be further configured to send and receive device identifiers via broadcast. Broadcasting eliminates the need for direct communication between computing nodes. Thus, even without an end-to-end connection, devices on each computing node can exchange information to obtain the second device identifiers of other computing nodes / second processors.

[0076] Accordingly, after obtaining the first device identifier, the local computing node can broadcast the first device identifier based on a preset general communication protocol so that it can be obtained by other processors. Thus, other processors may also establish end-to-end communication with devices within the computing cluster, requiring the construction of complete device topology information within the computing cluster. Therefore, the local computing node also needs to broadcast its own first device identifier to improve information exchange.

[0077] In the embodiments of this application, each computing node in the communication group may use the allgather set operation to obtain the device identifiers of other computing nodes and complete the exchange of global information.

[0078] In some embodiments of this application, after obtaining the device identifiers of all computing nodes within the communication group, each device can construct device topology information based on the device identifiers of each computing node, so as to record the device identifiers of each computing node through the device topology information. In some embodiments, the device topology information may include the hardware device signal, manufacturer, etc., corresponding to each computing node in the communication group, and the specific configuration content is not limited.

[0079] In this way, even if the end-to-end communication connection between the second processors has not yet been established, the local computing node and other computing nodes can complete the collection and discovery of global device information, realize the dynamic perception of the computing nodes and processors included in the communication group, and then construct device topology information and determine whether heterogeneous nodes are included.

[0080] It should be noted that before establishing an end-to-end connection between computing nodes, such as between GPUs, the first processor (CPU) controls the work related to linking communication libraries between the computing nodes. After establishing end-to-end communication, GPUs can communicate directly during the execution of communication tasks. Therefore, the achievement in this embodiment of the application of avoiding CPU resource consumption without CPU involvement refers to the effect achievable after establishing end-to-end communication.

[0081] Accordingly, in the embodiments of this application, after obtaining the first device identifier, the current computing node can also broadcast the first device identifier based on a preset general communication protocol so that the first device identifier can be obtained by other computing nodes. That is, the current computing node also needs to broadcast its own first device identifier so that other computing nodes can obtain the first device identifier.

[0082] S240 determines whether the communication group includes heterogeneous nodes based on the device identifier.

[0083] In the embodiments of this application, the device identifier of each computing node can be used to characterize whether the communication libraries used by each computing node are the same. If they are different, it indicates that there are heterogeneous nodes in the communication group. For communication groups that use the same type of communication library, it can be determined that the communication group includes homogeneous nodes.

[0084] Therefore, in one embodiment of this application, determining whether a communication group includes heterogeneous nodes based on device identifiers may include: determining whether there are different device identifiers among the first device identifier and all second device identifiers; if different device identifiers are determined, determining that the communication group includes heterogeneous nodes; if all device identifiers are determined to be the same, determining that each computing node in the communication group is a homogeneous node.

[0085] In this embodiment, when determining whether a node is heterogeneous, it is not necessary to identify the communication library. Instead, the type of communication library used is determined by whether the device identifiers are the same. This method simplifies the determination and improves the efficiency of determining whether each device is from the same manufacturer and uses the same type of communication library.

[0086] S250, when it is determined that the communication group includes heterogeneous nodes, controls each process to call the preset heterogeneous communication library to interact with other processes in the communication group when performing communication tasks.

[0087] In the embodiments of this application, the second processors from heterogeneous nodes bound to each process in the communication group can communicate based on the same heterogeneous communication library, thereby establishing end-to-end communication without the need for CPU relay, effectively reducing CPU performance usage and the limitation of CPU performance on the performance of the computing cluster.

[0088] As mentioned earlier, in the absence of an end-to-end communication connection, each computing node broadcasts its own device identifier. Similarly, establishing an end-to-end communication connection between computing nodes also requires communication connection-related information. Therefore, communication connection-related information can also be broadcast to enable communication connections between computing nodes.

[0089] For example, in one embodiment of this application, before calling a preset heterogeneous communication library to interact with other processes in the communication group when controlling each process to perform a communication task, the method may further include: obtaining first communication connection information broadcast by at least one other computing node based on a preset general communication protocol; and controlling a second processor in the current computing node to establish a communication connection with the second processor in the computing node that broadcast the first communication connection information based on the first communication connection information.

[0090] The first communication connection information is used for each computing node in the communication group to establish a connection with each other. Each computing node needs to know the connection information of the peer device to be connected, including but not limited to port information, address information, etc. Therefore, the two parties establishing the communication connection need to exchange their respective first communication connection information.

[0091] In this embodiment, the preset general communication protocol is the communication protocol supported by the second processor to which each process in the communication group mentioned in the previous embodiment is bound when no end-to-end connection is established. As before, the preset general communication protocol is the communication protocol supported by each computing node in the computing cluster when no end-to-end connection is established, and it can be the same protocol as described above, for example, it can be the TCP protocol, which will not be repeated here.

[0092] Furthermore, in embodiments of this application, the current computing node may also broadcast its own communication connection information.

[0093] In some embodiments of this application, the first communication connection information includes the communication protocol supported by the second processor in the first target computing node that sends the first communication connection information. Obtaining the first communication connection information broadcast by at least one other computing node based on a preset general communication protocol may include: obtaining the second communication connection information of the current computing node; determining whether the second processor in the current computing node supports Infinite Bandwidth (IB) communication through the second communication connection information; if it is determined that the second processor in the current computing node supports IB communication, determining whether the second processor in the first target computing node supports IB communication through the first communication connection information; if it is determined that the second processor in the first target computing node supports IB communication, controlling the second processor in the current computing node to establish a communication connection based on IB communication with the second processor in the first target computing node.

[0094] Here, to facilitate the differentiation of communication connection information, the first communication connection information represents the communication connection information of other processors, and the second communication connection information represents the communication connection information supported by the local computing node. The second communication connection information includes the communication protocols supported by the local computing node, and the local computing node can directly obtain the second communication connection information. The first communication connection information can be obtained in accordance with the methods provided in the foregoing embodiments.

[0095] In the embodiments of this application, the first communication connection information and the second communication connection information include whether each device of the computing node supports IB communication. IB communication is a communication method that bypasses the operating system kernel protocol stack, enabling user-mode remote direct memory access with lower latency and higher bandwidth. Therefore, when establishing a communication connection, it can be determined whether the local computing node and the peer computing node (i.e., the first target computing node) support IB communication. If both support it, IB communication is established to obtain better communication performance.

[0096] Conversely, if it is determined that the local computing node does not support IB communication or the first target computing node does not support IB communication, the second processor in the current computing node is controlled to establish a communication connection with the second processor in the first target computing node based on the TCP / SOCKET protocol. The TCP / SOCKET protocol is a protocol that is configured on various devices. Although its performance is lower than that of IB, the TCP / SOCKET protocol has higher versatility, enabling the method of this application to be used in more scenarios.

[0097] In the embodiments of this application, both IB communication and TCP / SOCKET protocol can be encapsulated. When configuration is required, the corresponding interface can be directly called for configuration, thereby realizing pluggable and unified management of communication protocols and improving the efficiency of configuring the heterogeneous communication method on the device.

[0098] Furthermore, although second processors on different computing nodes can establish end-to-end communication, the actual communication between them is still achieved by calling the network interface card (NIC). That is, each computing node also includes a NIC; controlling each process to execute communication tasks and interact with other processes within the communication group by calling a pre-defined heterogeneous communication library can include: for any given process, in response to the execution of that process, determining the target second processor bound to the target process interacting with that process; controlling the second processor bound to that process to call the NIC and communicate with the target second processor based on the heterogeneous communication library.

[0099] It should be noted that in existing methods where data is forwarded via CPU relay, different nodes also send data through network interface cards (NICs). That is, the path can be represented as: GPU (Compute Node 1) - CPU (Compute Node 1) - NIC (Compute Node 1) - NIC (Compute Node 2) - CPU (Compute Node 2) - GPU (Compute Node 2). However, using the method provided in this application, the path can be simplified to: GPU (Compute Node 1) - NIC (Compute Node 1) - NIC (Compute Node 2) - GPU (Compute Node 2), reducing the required relay path and thus lessening the limitation of CPU performance on the communication group and computing cluster size.

[0100] In compute nodes, data is recorded based on structures, which include data width, format, etc. Different manufacturers may define structures differently; that is, devices with different types of communication libraries record communication connection information using different structures. However, the TCP / SOCKET protocol stipulates that the transmitted and received structures must be consistent. Therefore, the first communication connection data broadcast by each compute node may differ in format. If a local compute node receives the first communication connection data sent by a heterogeneous node, it may be unable to use it directly or it may encounter anomalies after use.

[0101] Therefore, in some embodiments of this application, the structures used by computing nodes of different types of communication libraries to record communication connection information are different. After obtaining the first communication connection information broadcast by at least one other computing node based on a preset general communication protocol, and before controlling the second processor in the current computing node to establish a communication connection with the second processor in the computing node that broadcast the first communication connection information based on the first communication connection information, the method may further include: reconstructing the first communication connection information based on a preset structure to configure the first communication connection information as a structure supported by the current computing node. Thus, reconstructing the first communication connection information based on a preset structure to configure the first communication connection information as a structure supported by the local computing node enables the local computing node to accurately use the first communication connection information based on the preset structure.

[0102] In one embodiment of this application, after determining whether the communication group includes heterogeneous nodes based on the device identifier, the method may further include: S260, when it is determined that all nodes in the communication group are homogeneous, controls each process to execute communication tasks and calls the native communication library corresponding to the current computing node to interact with other processes in the communication group.

[0103] In the embodiments of this application, homogeneous nodes represent that the second processors bound to each process come from computing nodes that use the same type of communication library, that is, the computing nodes of each processor in the communication group use the same type of communication library.

[0104] In the embodiments of this application, the native communication library types corresponding to each homogeneous node are the same. If the device topology information shows that the second processors bound to each process in the communication group belong to the same manufacturer and use the same type of communication library, then the processors involved in the communication group are determined to be homogeneous nodes. The communication method between homogeneous nodes can also be called homogeneous mode.

[0105] In the embodiments of this application, the compatibility interface layer can use the dlopen function to load the deeply optimized native communication library provided by the vendor. All subsequent calls to the second processor of each computing node within the communication group will be directly forwarded to the loaded native communication library for execution via function pointers, thereby achieving optimal performance.

[0106] The native communication library corresponding to the computing node is more compatible with that computing node and can provide better performance. Compared with using heterogeneous communication libraries, using the native communication library has lower requirements for communication latency and computational demands. Therefore, in this embodiment, if the computing nodes to which the second processors bound to each process in the communication group belong are homogeneous nodes from the same vendor and using the same type of communication library, then the corresponding native communication library can be configured for all computing nodes and second processors in the communication group to maintain high-performance communication between devices in the computing cluster. The communication methods between homogeneous nodes can refer to the prior art and will not be elaborated here.

[0107] In the embodiments described above, the system includes three stages: perception, decision-making, and routing. During the perception stage, in the network initialization bootstrap phase, each device can automatically collect global device topology information through broadcasting and the AllGather operation. In the decision-making stage, the communication mode (heterogeneous or homogeneous) can be dynamically determined based on this device topology information. The routing stage includes calling the corresponding underlying communication library (heterogeneous or homogeneous). This process is completely transparent to upper-layer applications, enabling automated operation and maintenance and performance optimization.

[0108] In some embodiments of this application, controlling each process to execute a communication task by calling a preset heterogeneous communication library to interact with other processes in the communication group may further include: for any process, in response to the execution of the process, controlling the second processor bound to the process to divide the interactive data into multiple data blocks based on a preset size; and sequentially transmitting the data blocks to a target second processor based on the heterogeneous communication library, wherein the target second processor is the second processor bound to any other process in the communication group, and the target second processor supports performing calculations on previously received data blocks while receiving data blocks.

[0109] In this embodiment, the interactive data refers to the data sent from the second processor bound to the process to the target second processor. Guided by unified communication parameters, large-scale data is fragmented into preset sizes, and during communication, the sending, transmission, receiving, and local computation of different data blocks overlap in time, forming a pipeline. This approach can fully utilize network bandwidth, significantly shorten the overall data transmission time, and effectively address the potentially high network latency between heterogeneous nodes, thereby greatly improving the overall efficiency of heterogeneous communication.

[0110] To facilitate understanding, an example is provided here to illustrate the method provided in this application. Please refer to [link / reference]. Figure 3 , Figure 3 This is an example diagram illustrating a heterogeneous node communication process provided in an embodiment of this application. The heterogeneous node communication process includes: First, linking to the native communication library and the heterogeneous communication library through a compatible interface layer. During the usage phase, the initialization interface is intercepted through the compatible interface layer. After intercepting the initialization interface, the perception-decision-routing phase begins. Perception is used to identify all computing nodes and their corresponding device identifiers within the communication group. During this period, each computing node can broadcast its own device identifier. After the local computing node collects the device identifiers of each computing node, the decision phase determines whether heterogeneous nodes are included, i.e., whether devices from different manufacturers are included, based on the device identifiers. If devices from different manufacturers are included, heterogeneous nodes are determined to be included; otherwise, homogeneous nodes are included. Then, the routing phase begins, routing the communication libraries that need to be called. If heterogeneous nodes are included, the heterogeneous communication library is called; if homogeneous nodes are included, the corresponding native communication library is called.

[0111] Finally, an end-to-end connection is established between the processors to determine if IB communication is supported. If supported, IB communication is initiated between the processors; otherwise, TCP / SOCKET protocol communication is established. Furthermore, data can be transmitted in chunks to improve data transmission and processing efficiency.

[0112] In the embodiments provided in this application, a heterogeneous communication library is first provided. Based on a unified heterogeneous node library, heterogeneous nodes can communicate with each other. Through this library, the CPU does not need to perform data interaction calculations between GPUs, thus reducing the CPU's computational resource consumption. Next, to further avoid using the CPU as an intermediary, during the device's computing framework / upper-layer application linking to the communication library stage, each device directly links to the heterogeneous communication library without first linking to its corresponding communication library. Since the communication libraries of all heterogeneous nodes are identical, they can establish end-to-end connections based on the same library, enabling direct communication. Furthermore, by eliminating the need for CPU data interaction calculations, direct end-to-end communication can be established between heterogeneous nodes, further reducing CPU intermediaries and bandwidth consumption. Therefore, by using the above methods, the computational resources and bandwidth usage of the CPU can be effectively reduced, and the data interaction between heterogeneous nodes can effectively reduce or even avoid the dependence on the CPU. This reduces the limitation of CPU performance on the scale of computing nodes in the computing cluster, fundamentally eliminates the performance bottleneck of multiple PCIe bus and memory copying, and achieves ultra-low latency communication close to the theoretical bandwidth of the hardware.

[0113] Furthermore, this approach involves modifying the communication library, which eliminates the need to modify the communication logic and code between devices, effectively reducing implementation difficulty. The method provided in this application can be implemented as a plugin-like program, which can be configured according to needs without modifying the device's native code. Instead, it only requires adding the content corresponding to the above method, such as adding a compatible interface layer and adding device awareness and heterogeneous communication libraries. The method of linking the communication libraries and the native communication libraries do not need to be adjusted. This method provides a flexible and independently upgradeable solution for heterogeneous scenarios, while avoiding the complexity and dependency conflicts caused by integrating all communication implementation code into a single library.

[0114] Furthermore, the communication group may be expanded or reduced in size as needed, and some computing nodes may be added or removed, which may cause the original homogeneous to become heterogeneous, or vice versa. Based on this, in the embodiments of this application, the initialization interface of each communication task can be intercepted by the compatibility interface layer to execute the above-mentioned heterogeneous node communication method, so as to dynamically adjust the communication mode according to the actual access computing node type.

[0115] In the embodiments of this application, the above method can be implemented using a neural network model, and the training data used when training the neural network model can include homogeneous nodes and heterogeneous nodes.

[0116] Based on the same inventive concept, this application also provides a computing node, which can be included in a computing cluster, that is, the computing node can be one of the computing nodes in the computing cluster.

[0117] Please see Figure 4 , Figure 4 This is a schematic diagram of a computing node provided in an embodiment of this application. In this embodiment, the computing node 400 includes a first processor 410, at least two second processors 420, and a network interface card (NIC) 430. The first processor 410 is connected to each of the second processors 420 and the NIC 430, and the NIC 430 is also connected to each of the second processors 420. The first processor 410 is used to obtain linking instructions, which are used by the upper-layer application to instruct the linking of the communication library during the compilation phase; the first processor is used to execute the heterogeneous node communication method executed by the first processor as provided in the foregoing embodiments. The specific process can be found in the foregoing content and will not be elaborated here.

[0118] In this embodiment of the application, the first processor 410 may be a CPU, and the second processor 420 may be an NPU, GPU, AI accelerator card, etc.

[0119] In this embodiment, the first processor 410 can implement the heterogeneous node communication method provided in any of the foregoing embodiments. In this embodiment, any second processor can also divide the interactive data into multiple data blocks based on a preset size, and sequentially transmit the data blocks to other second processors based on the heterogeneous communication library. The other second processors 420 can simultaneously perform calculations on previously received data blocks while receiving the data blocks.

[0120] Based on the same inventive concept, this application also provides a computing cluster, which may include at least two computing nodes 400, wherein each computing node may be a computing node provided in the foregoing embodiments. Any number of first processors 410 and second processors 420 of any computing node are used to form a communication group for communication, and the first processor of any computing node 400 in the communication group is used to execute the heterogeneous node communication method as provided in the foregoing embodiments.

[0121] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when run on a computer, causes the computer to perform the methods provided in the above embodiments.

[0122] The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs (digital video discs)), or semiconductor media (e.g., SSDs (solid state disks)).

[0123] If the method is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0124] In the embodiments provided in this application, it should be understood that the disclosed methods and apparatus can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. The functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] The above embodiments can be freely combined without conflict, and the resulting embodiments are covered within the protection scope of this application.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A heterogeneous node communication method, characterized in that, A first processor is applied in the current computing node, and the current computing node further includes multiple second processors; The heterogeneous node communication method includes: Obtain the communication group required for the communication task; the communication task includes multiple processes that execute the communication task, each process is bound to a different second processor, and each process in the communication task is allowed to be bound to the second processor in different computing nodes; Obtain the device identifier of the second processor to which each process in the communication group is bound; the device identifier is used to characterize the type of communication library used by the second processor. Based on the device identifier, it is determined whether the communication group includes heterogeneous nodes, wherein the heterogeneous nodes represent that the second processor bound to each process comes from a computing node using a different type of communication library; When it is determined that the communication group includes the heterogeneous nodes, each process is controlled to execute the communication task and call a preset heterogeneous communication library to interact with other processes in the communication group.

2. The heterogeneous node communication method according to claim 1, characterized in that, Each of the aforementioned computing nodes includes a compatibility interface layer; Before obtaining the device identifier of the second processor to which each of the processes in the communication group is bound, the method further includes: When each of the processes calls the initialization interface, the initialization interface is hijacked based on the compatibility interface layer, so as to determine whether the call to the heterogeneous node and the control of the heterogeneous communication library are included through the compatibility interface layer; the initialization interface is used to call the communication library.

3. The heterogeneous node communication method according to claim 2, characterized in that, After determining whether the communication group includes heterogeneous nodes based on the device identifier, the method further includes: When it is determined that all communication groups are homogeneous nodes, each process is controlled to execute the communication task and call the native communication library corresponding to the current computing node to interact with other processes in the communication group; wherein, the homogeneous node indicates that the second processor bound to each process comes from a computing node that uses the same type of communication library.

4. The heterogeneous node communication method according to claim 1, characterized in that, Each processor in each computing node uses the same communication library; the device identifier of the second processor to which each process is bound includes a first device identifier and a second device identifier; the first device identifier is the device identifier of the current computing node; the second device identifier is the second device identifier of the second processor in each of the other computing nodes to which the process is bound; The step of obtaining the device identifier of the second processor to which each of the processes in the communication group is bound includes: Obtain the first device identifier of the current computing node; The second device identifier is obtained by acquiring broadcasts from other computing nodes based on a preset general communication protocol; the preset general communication protocol is the communication protocol supported by the second processor to which each process in the communication group is bound when no end-to-end connection is established.

5. The heterogeneous node communication method according to claim 4, characterized in that, After obtaining the first device identifier, the method further includes: The first device identifier is broadcast based on the preset general communication protocol so that the first device identifier can be obtained by the other computing nodes.

6. The heterogeneous node communication method according to claim 4, characterized in that, The step of determining whether the communication group includes heterogeneous nodes based on the device identifier includes: Determine whether there are any different device identifiers among the first device identifier and all the second device identifiers; If different device identifiers are determined to exist, the communication group is determined to include the heterogeneous nodes; If all device identifiers are identical, then each computing node in the communication group is determined to be a homogeneous node.

7. The heterogeneous node communication method according to any one of claims 1-6, characterized in that, Before invoking a preset heterogeneous communication library to interact with other processes within the communication group when controlling each of the aforementioned processes to execute the communication task, the method further includes: The first communication connection information broadcast by at least one other computing node is obtained based on a preset general communication protocol; the preset general communication protocol is the communication protocol supported by the second processor to which each process in the communication group is bound when no end-to-end connection is established. Based on the first communication connection information, the second processor in the current computing node is controlled to establish a communication connection with the second processor in the computing node that broadcasts the first communication connection information.

8. The heterogeneous node communication method according to claim 7, characterized in that, The structures for recording communication connection information differ among computing nodes of different types of communication libraries. After obtaining first communication connection information broadcast by at least one other computing node based on a preset general communication protocol, and before controlling the second processor within the current computing node to establish a communication connection with the second processor within the computing node that broadcast the first communication connection information based on the first communication connection information, the method further includes: The first communication connection information is reconstructed based on a preset structure to configure the first communication connection information as a structure supported by the current computing node.

9. The heterogeneous node communication method according to claim 7, characterized in that, The first communication connection information includes the communication protocol supported by the second processor in the first target computing node that sent the first communication connection information. The step of obtaining the first communication connection information broadcast by at least one other computing node based on a preset general communication protocol includes: Obtain the second communication connection information of the current computing node; the second communication connection information includes the communication protocol supported by the second processor in the current computing node; The second communication connection information is used to determine whether the second processor in the current computing node supports unlimited bandwidth IB communication. If it is determined that the second processor in the current computing node supports IB communication, the first communication connection information is used to determine whether the second processor in the first target computing node supports IB communication. If it is determined that the second processor in the first target computing node supports the IB communication, the second processor in the current computing node is controlled to establish a communication connection based on the IB communication with the second processor in the first target computing node.

10. The heterogeneous node communication method according to claim 9, characterized in that, If it is determined that the second processor in the current computing node does not support the IB communication or the second processor in the first target computing node does not support the IB communication, the method further includes: The system controls the second processor in the current computing node to establish a communication connection with the second processor in the first target computing node based on the TCP / SOCKET protocol.

11. The heterogeneous node communication method according to any one of claims 1-6, characterized in that, Each of the aforementioned computing nodes also includes a network interface card (NIC); The process of controlling each of the aforementioned processes to execute the communication task calls a preset heterogeneous communication library to interact with other processes within the communication group, including: For any given process, in response to the execution of that process, determine the target second processor to which the target process interacting with that process is bound; The second processor bound to the control process calls the network card and communicates with the target second processor based on the heterogeneous communication library.

12. The heterogeneous node communication method according to any one of claims 1-6, characterized in that, The process of controlling each of the aforementioned processes to execute the communication task calls a preset heterogeneous communication library to interact with other processes within the communication group, including: For any given process, in response to the execution of that process, the control of the second processor bound to that process divides the interactive data into multiple data blocks based on a preset size; the interactive data is the interactive data sent by the second processor bound to that process to a target second processor; the target second processor is the second processor bound to any other process in the communication group. The data blocks are sequentially transmitted to the target second processor based on the heterogeneous communication library, wherein the target second processor supports performing calculations on previously received data blocks while receiving the data blocks.

13. A computing node, characterized in that, The computing node includes a first processor, at least two second processors, and a network interface card (NIC). The first processor is connected to each of the second processors and the NICs respectively, and the NICs are also connected to each of the second processors. The first processor is used to implement the heterogeneous node communication method as described in any one of claims 1-12.

14. A computing cluster, characterized in that, The computing cluster includes at least two computing nodes as described in claim 13, wherein any number of the first processors and the second processors of any computing node in the computing cluster are used to form a communication group for communication.

15. A readable storage medium, characterized in that, The readable storage medium stores a program that, when run on a processor, causes the processor to implement the heterogeneous node communication method as described in any one of claims 1-12.

Images