Heterogeneous dynamic fault-tolerant distributed set communication method and system and storage medium

Abstract

The invention relates to a heterogeneous dynamic fault-tolerant distributed set communication method and system and a storage medium. The method comprises the following steps: a process or a node participating in set communication sends an HTTP request to a proxy server, and applies for registering a set communication service; the proxy server establishes a set communication group according to the unique identifier, and maintains participation node information; the proxy server dynamically manages the set communication group according to the set communication ID, and joins and quits the processing node; the proxy servers receive a set communication request of a client and route and forward a message to a corresponding node according to the information of a set communication group, and the proxy servers communicate with each other through an HTTP (Hyper Text Transport Protocol); and executing set communication, and realizing broadcasting, specification, collection and dispersion operations. According to the method, broadcasting, reduction, collection and dispersion of primitives compatible with set communication are realized through an HTTP protocol, the configuration of a hardware environment is transparent, and an inherent adaptive dynamic node is added into and withdrawn from a fault-tolerant node through stateless communication.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a heterogeneous dynamic fault-tolerant distributed collective communication method, system and storage medium. Background Art

[0002] Collective communication refers to the communication operation of data exchange and synchronization between multiple processes or nodes in a distributed computing environment. Collective communication primitives include broadcast, reduce, gather, scatter, etc. These operations are very important in parallel and distributed computing, and are used to coordinate and synchronize data among multiple computing nodes. With the rapid development of large-scale AI model training and reasoning applications, collective communication plays a more important role in fast data synchronization and data exchange between distributed processes or nodes.

[0003] At present, the implementation of collective communication primitives usually relies on underlying communication protocols and libraries, such as Gloo, MPI (Message Passing Interface), and NCCL (NVIDIA Collective Communication Library). Gloo, MPI, NCCL and dedicated hardware are forcibly bound and difficult to adapt to heterogeneous computing hardware. They can usually only run in a homogeneous intelligent center environment. In addition, collective communication assumes a static topological environment and is difficult to adapt to dynamic node joining. At the same time, the failure of collective communication operation due to node failure will cause the fault tolerance and adaptability of dynamic joining to fail. With the growth of intelligent computing capabilities of end-side devices, there is an urgent need to support collective communication in heterogeneous hardware environments. Summary of the invention

[0004] Based on this, it is necessary to provide a dynamic fault-tolerant distributed collective communication method, system and storage medium to address the above technical problems.

[0005] A heterogeneous dynamic fault-tolerant distributed collective communication method, the method comprising: The process or node participating in the collective communication sends an HTTP request to the proxy server to apply for registration of the collective communication group record; the collective communication group record is encoded in the form of a dictionary, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1, ..., node n}; The proxy server establishes a collective communication group based on a unique identifier, and stores the globally consistent collective communication group state through a distributed hash table (DHT), and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations.

[0006] The proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and leaving of nodes.

[0007] The proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group. The proxy servers communicate with each other through the HTTP protocol.

[0008] The server performs collective communication and implements broadcast, protocol, collection, and distribution operations.

[0009] In one of the embodiments, the proxy service includes a sending proxy server, a relay proxy server, and a receiving proxy server; each proxy server joins a global distributed hash table DHT to maintain a global state required for collective communication.

[0010] The sending proxy server is used to receive client requests, process collective communication operations, and forward data.

[0011] Relay proxy server is used to receive data, process collective communication operations, and forward it to the next proxy server or server.

[0012] The receiving proxy server is used to receive data and forward it to the server.

[0013] In one embodiment, the specific process of the broadcast operation includes: The sending node prepares the data to be broadcast and sends the data to be broadcast to the proxy server through an HTTP POST request.

[0014] The proxy server receives the HTTP POST request, queries the DHT to obtain the target set information according to the communication group, searches the set communication group in the DHT through the unique identifier, and obtains the address and status information of the target node.

[0015] The proxy server forwards the data to be broadcast to the proxy servers of all target nodes or recursively delivers it to the relay proxy server; In one embodiment, the specific process of the protocol operation includes: All nodes prepare the data to be reduced and send the data to be reduced to the sending proxy server through HTTP POST request; The sending proxy server receives the HTTP POST request, queries and obtains the target address in the DHT (distributed hash table) according to the unique identifier of the communication group, and forwards the data to the proxy server of the target node or recursively delivers it to the relay proxy server to achieve multi-level forwarding.

[0016] The target node receives all data to be reduced, performs reduction operations, and then returns the reduction results to the client.

[0017] In one embodiment, the specific process of the collection operation includes: All nodes prepare the data to be collected and send the data to be collected to the sending proxy server through HTTP POST request.

[0018] The sending proxy server receives the request, queries and obtains the address and status of the target node in the DHT according to the unique identifier of the communication group, and forwards the data to be collected to the proxy server of the target receiving node, or recursively delivers it to the relay proxy server to achieve multi-level forwarding.

[0019] The target node receives all the data to be collected, performs the collection operation, and then returns the collection result to the client.

[0020] In one embodiment, the specific process of the dispersion operation includes: The sending node prepares the data to be dispersed and divides the data to be dispersed into multiple data blocks, each data block corresponding to a target node.

[0021] The proxy server receives the HTTP POST request, queries and obtains the target set information in the DHT according to the unique identifier of the communication group, including the address and status of the target node, and forwards the data to be dispersed to the proxy servers of all target nodes, or recursively delivers it to the relay proxy server; The proxy server of the target node receives the data to be dispersed from the proxy server of the sending node and forwards it to the target node; The destination node receives the data block and processes and saves it.

[0022] In one embodiment, a process or node participating in collective communication sends an HTTP request to a proxy server to apply for registration of a collective communication group record, including: The processes or nodes participating in collective communication register collective communication group records with the proxy server; Each node registers a proxy server for receiving or sending, and the corresponding proxy server provides transparent delivery of data.

[0023] Each proxy server registers a relay proxy server, and the corresponding proxy server provides data delivery to the destination node. At the same time, each relay proxy server can register subsequent proxy servers to achieve recursive data delivery.

[0024] Each new process or node dynamically queries the proxy server's communication group record and joins the existing collective communication group.

[0025] In one embodiment, the proxy server dynamically manages the collective communication group according to the collective communication ID, and the processing of the joining and leaving of the node includes: The proxy server establishes and maintains the collective communication group according to the collective communication ID and records the information of the participating nodes.

[0026] Update the collective communication group based on the joining and leaving of nodes.

[0027] Through the self-healing characteristics of DHT, the proxy server automatically redistributes and recovers the tasks of failed nodes, ensuring the continuity of collective communication operations.

[0028] A heterogeneous dynamic fault-tolerant distributed collective communication system comprises: a client, a proxy server and a server.

[0029] The client is a process or node that is the sending side of collective communication and is used to send HTTP requests to the proxy server.

[0030] The proxy server includes: the proxy service includes a sending proxy server, a relay proxy server and a receiving proxy server; each proxy server joins the global distributed hash table DHT to maintain the global state required for collective communication; the sending proxy server is used to receive client requests, process collective communication operations, and forward data; the relay proxy server is used to receive data, process collective communication operations, and forward to the next proxy server or server; the receiving proxy server is used to receive data and forward it to the server.

[0031] A server is a process or node that is the response side of collective communication, processing requests and returning responses to clients.

[0032] The client participating in collective communication sends an HTTP request to the proxy server to apply for registration of collective communication group records; the collective communication group records are encoded in dictionary form, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1,..., node n}; the proxy server establishes a collective communication group based on the unique identifier, and implements the storage of globally consistent collective communication group status through the distributed hash table DHT, and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations; the proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and exit of nodes; the proxy server receives the client's collective communication request, and routes and forwards the message to the corresponding node according to the information of the collective communication group, and the proxy servers communicate with each other through the HTTP protocol; the server executes collective communication to implement broadcast, protocol, collection, and dispersion operations.

[0033] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0034] The above-mentioned heterogeneous dynamic fault-tolerant distributed collective communication method, system and storage medium, the method includes: the process or node participating in the collective communication sends an HTTP request to the proxy server to apply for registration of the collective communication service; the proxy server establishes a collective communication group according to the unique identifier and maintains the information of the participating nodes; the proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and leaving of the nodes; the proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group, and the proxy servers communicate with each other through the HTTP protocol; executes collective communication to implement broadcast, reduction, collection and dispersion operations. The method implements broadcast, reduction, collection and dispersion primitives compatible with collective communication through the HTTP protocol, is transparent to the configuration of the hardware environment, and inherently adapts to dynamic node joining and leaving or fault-tolerant nodes through stateless communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a heterogeneous dynamic fault-tolerant distributed collective communication method in one embodiment; Figure 2 A flowchart of a broadcast operation in another embodiment; Figure 3 A flowchart of the protocol operation in another embodiment; Figure 4 A schematic diagram of gather communication topology in another embodiment; Figure 5 Collecting operation flow chart for another embodiment; Figure 6 A flowchart of a distributed operation in another embodiment; Figure 7 The figure is a message communication flow chart in another embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] In order to support heterogeneous, dynamic, and fault-tolerant collective communication, the present application proposes a heterogeneous, dynamic, and fault-tolerant distributed collective communication method, which is a distributed collective communication method based on the HTTP (HyperText Transfer Protocol) protocol, and implements broadcast, reduce, gather, and scatter primitives compatible with collective communication through the HTTP protocol.

[0038] The HTTP protocol runs at the application layer above the TCP protocol layer and adopts request-response stateless communication. Each request is independent and the server does not retain the client status. Therefore, it is transparent to the configuration of the hardware environment. Moreover, through stateless communication, it can inherently adapt to dynamic node joining and exiting or fault-tolerant nodes.

[0039] At the same time, the HTTP protocol defines a variety of methods, such as GET, POST, PUT, DELETE, etc. Mainstream web servers and client libraries, such as Apache HTTP Server, Nginx, Requests (Python library, etc.) implement the HTTP protocol.

[0040] In one embodiment, Figure 1 As shown, a heterogeneous dynamic fault-tolerant distributed collective communication method is provided, and the method includes the following steps: Step 100: The process or node participating in the collective communication sends an HTTP request to the proxy server to apply for registration of the collective communication group record; the collective communication group record is encoded in a dictionary form, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1,..., node n}.

[0041] Specifically, the communication object ID is represented by a globally unique integer identifier.

[0042] Node ID: Represented by a globally unique identifier, which can be an integer sequence, a UUID identifier, or an IP communication address, etc. Only a unique identifier is required.

[0043] Step 102: The proxy server establishes a collective communication group according to the unique identifier, and implements storage of globally consistent collective communication group states through a distributed hash table DHT, and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations.

[0044] Specifically, the collective communication group is maintained by a proxy server, and the processes or nodes participating in the collective communication.

[0045] Step 104: The proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and exiting of nodes.

[0046] Step 106: The proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group. The proxy servers communicate with each other through the HTTP protocol.

[0047] Step 108: The server performs collective communication to implement broadcast, protocol, collection, and dispersion operations.

[0048] In the above heterogeneous dynamic fault-tolerant distributed collective communication method, the method includes: the process or node participating in the collective communication sends an HTTP request to the proxy server to apply for registration of the collective communication service; the proxy server establishes a collective communication group according to the unique identifier and maintains the information of the participating nodes; the proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and exiting of nodes; the proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group, and the proxy servers communicate with each other through the HTTP protocol; executes collective communication to implement broadcast, reduction, collection, and dispersion operations. The method implements broadcast, reduction, collection, and dispersion primitives compatible with collective communication through the HTTP protocol, is transparent to the configuration of the hardware environment, and inherently adapts to dynamic node joining and exiting or fault-tolerant nodes through stateless communication.

[0049] In one embodiment, the proxy service includes a sending proxy server, a relay proxy server, and a receiving proxy server; each proxy server joins the global distributed hash table DHT to maintain the global state required for collective communication; the sending proxy server is used to receive client requests, process collective communication operations, and forward data; the relay proxy server is used to receive data, process collective communication operations, and forward to the next proxy server or server; the receiving proxy server is used to receive data and forward it to the server.

[0050] Among them, data object: generated by the client or server, a data object can be a file, a NUMPY array, a dictionary, a byte sequence, or multiple files, arrays, dictionaries, and byte sequences.

[0051] In one embodiment, the specific process of the broadcast operation includes: Step 200: The sending node prepares the data to be broadcast, and sends the data to be broadcast to the proxy server via an HTTP POST request; Step 202: The proxy server receives the HTTP POST request, queries the DHT according to the communication group to obtain the target set information, searches the set communication group in the DHT by the unique identifier, and obtains the address and status information of the target node; Step 204: The proxy server forwards the data to be broadcast to the proxy servers of all target nodes or recursively delivers it to the relay proxy server; Step 206: The proxy server of the target node receives the data to be broadcast, and sends the data to be broadcast to the target node; Step 208: After receiving and processing the data to be broadcast, the target node may directly respond to the sending node or relay the response through a proxy server.

[0052] Specifically, a broadcast message is generated by the client and responded to by the server, including the broadcast data object, the broadcast sender, and the broadcast receiver sequence.

[0053] The broadcast operation process is as follows Figure 2 shown.

[0054] In one embodiment, the specific process of the protocol operation includes: Step 300: All nodes prepare the data to be reduced, and send the data to be reduced to the sending proxy server via HTTP POST request; Step 302: The sending proxy server receives the HTTP POST request, queries and obtains the target address in the DHT (distributed hash table) according to the unique identifier of the communication group, and forwards the data to the proxy server of the target node or recursively delivers it to the relay proxy server to achieve multi-level forwarding; Step 304: The proxy server of the target node receives the data and forwards the data to be reduced to the target node; Step 306: The target node receives all the data to be reduced, performs reduction operations (such as summing, averaging, etc.), and then returns the reduction results to the client.

[0055] Specifically, the protocol message is generated by the client and responded to by the server, including the protocol data object, protocol participants, and protocol receivers.

[0056] The operating procedures are as follows: Figure 3 shown.

[0057] In one embodiment, the specific process of the gather operation includes: Step 400: All nodes prepare the data to be collected, and send the data to be collected to the sending proxy server via HTTP POST request; Step 402: The sending proxy server receives the request, queries and obtains the address and status of the target node in the DHT (distributed hash table) according to the unique identifier of the communication group, and forwards the data to be collected to the proxy server of the target node, or recursively delivers it to the relay proxy server to achieve multi-level forwarding; Step 404: the proxy server of the target node receives the data to be collected, and forwards the data to be collected to the target node; Step 406: The target node receives all the data to be collected, performs collection operations (such as merging lists, building dictionaries, etc.), and then returns the collection results to the client.

[0058] Specifically, the gather message is generated by the client and responded by the server, including the collected data object, the collected participants, and the collected receivers. The gather communication topology is as follows: Figure 4 As shown, the collection operation process is as follows Figure 5 shown.

[0059] In one embodiment, the specific process of the dispersion operation includes: Step 500: The sending node prepares the data to be distributed and divides the data to be distributed into multiple data blocks, each data block corresponds to a target node; Step 502: The sending node sends each data block to the proxy server of the corresponding target node through an HTTP POST request; Step 504: The proxy server receives the HTTP POST request, queries and obtains the target set information in the DHT (distributed hash table) according to the unique identifier of the communication group, including the address and status of the target node, and forwards the data to be dispersed to the proxy servers of all target nodes, or recursively delivers it to the relay proxy server to achieve multi-level forwarding; Step 506: the proxy server of the target node receives the data to be distributed from the proxy server of the sending node, and forwards it to the target node; Step 508: The target node receives the data block and processes and stores it.

[0060] Specifically, decentralized messages are generated by the client and responded to by the server, including decentralized data objects, decentralized participants, and decentralized receivers. The decentralized operation process is as follows: Figure 6 shown.

[0061] In one embodiment, step 100 includes: the processes or nodes participating in the collective communication register the collective communication group record with the proxy server; each node registers a proxy server that provides receiving or sending, and the corresponding proxy server provides transparent delivery of data; each proxy server registers a relay proxy server, and the corresponding proxy server provides data delivery to the destination node. At the same time, each relay proxy server can register subsequent proxy servers to achieve recursive data delivery; each new process or node dynamically queries the communication group record of the proxy server and joins the existing collective communication group.

[0062] In one of the embodiments, step 104 includes: the proxy server establishes and maintains a collective communication group based on the collective communication ID and records the information of participating nodes; updates the collective communication group based on the joining and leaving of nodes; through the self-healing characteristics of DHT, the proxy server automatically reallocates and restores the tasks of failed nodes to ensure the continuity of collective communication operations.

[0063] In one embodiment, a collective communication method based on the HTTP protocol is provided, comprising the following steps: (1) Registration of collective communication services: a) The process or node participating in collective communication registers the collective communication group record with the proxy server: collective communication ID: {node 1, ..., node n}; b) Each node registers a proxy server for receiving or sending, and the corresponding proxy server provides transparent delivery of data; c) Each proxy server registers a relay proxy server, and the corresponding proxy server provides data delivery to the destination node. At the same time, each relay proxy server can register subsequent proxy servers to achieve recursive data delivery; d) Each new process or node dynamically queries the proxy server's communication group record and joins the existing collective communication group.

[0064] (2) Establishing a collective communication group: The proxy server establishes a collective communication group based on the unique identifier and maintains the information of participating nodes.

[0065] a) State storage management: The proxy server uses the distributed hash table DHT to store the globally consistent state of the collective communication group; b) Add, delete, query and modify operations: The proxy server can dynamically add, delete, modify and query the information of collective communication groups; (3) Manage collective communication groups: The proxy server dynamically manages collective communication groups and handles the joining and exit of nodes.

[0066] a) Establish and maintain a collective communication group based on the collective communication ID and record the information of participating nodes.

[0067] b) Update the collective communication group, handle the joining and leaving of nodes, and ensure that the member information of the collective communication group is always up to date.

[0068] c) Through the self-healing characteristics of DHT, the proxy server automatically redistributes and recovers the tasks of failed nodes to ensure the continuity of collective communication operations.

[0069] (4) Message routing and forwarding: The proxy server is responsible for communicating with the client and the server, and communicating with the proxy server.

[0070] a) The proxy server is responsible for receiving the collective communication request from the client and routing and forwarding the message to the corresponding node according to the information of the collective communication group.

[0071] b) The proxy servers communicate with each other through the HTTP protocol to ensure that the messages can be correctly delivered to the target node.

[0072] (5) Execute collective communication: a) Broadcast: Send data from one node to all other nodes.

[0073] b) Reduce: Reduce the data of all nodes and send the result to one node.

[0074] c) Gather: collect data from all nodes into one node.

[0075] d) Scatter: Distribute data from one node to all other nodes.

[0076] In a specific embodiment, the specific steps of the client, the proxy server and the server in the message communication are: a) Client: 1) Send an HTTP request to the sending proxy server and embed the proxy server information in the HTTP message.

[0077] 2) Receive the response from the sending proxy server.

[0078] b)Proxy Server: 1) Sending proxy server: receives client requests, processes collective communication operations, and forwards data, embedding proxy server information in HTTP messages.

[0079] 2) Relay proxy server: receives data, processes collective communication operations, embeds proxy server information in HTTP messages, and forwards them to the next proxy server or server.

[0080] 3) Receiving proxy server: receives data and forwards it to the server.

[0081] c) Server: 1) Receive data from the proxy server.

[0082] 2) Process the request and return a response.

[0083] d) Collective communication primitives: implement operations such as broadcast, reduce, gather, and scatter.

[0084] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0085] In one embodiment, a heterogeneous dynamic fault-tolerant distributed collective communication system is provided, the system comprising: a client, a proxy server, and a server; The client is a process or node, which is the sending side of collective communication and is used to send HTTP requests to the proxy server; The proxy server includes: the proxy service includes a sending proxy server, a relay proxy server and a receiving proxy server; each proxy server joins the global distributed hash table DHT to maintain the global state required for collective communication; the sending proxy server is used to receive client requests, process collective communication operations, and forward data; the relay proxy server is used to receive data, process collective communication operations, and forward to the next proxy server or server; the receiving proxy server is used to receive data and forward it to the server; The server is a process or node that is the response side of collective communication, used to process requests and return responses to clients; The client participating in collective communication sends an HTTP request to the proxy server to apply for registration of collective communication group records; the collective communication group records are encoded in dictionary form, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1,..., node n}; the proxy server establishes a collective communication group based on the unique identifier, and implements the storage of globally consistent collective communication group status through the distributed hash table DHT, and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations; the proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and exit of nodes; the proxy server receives the client's collective communication request, and routes and forwards the message to the corresponding node according to the information of the collective communication group, and the proxy servers communicate with each other through the HTTP protocol; the server executes collective communication to implement broadcast, protocol, collection, and dispersion operations.

[0086] For the specific limitations of the heterogeneous dynamic fault-tolerant distributed collective communication system, please refer to the limitations of the heterogeneous dynamic fault-tolerant distributed collective communication method above, which will not be repeated here. Each module in the above-mentioned heterogeneous dynamic fault-tolerant distributed collective communication system can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A heterogeneous dynamic fault-tolerant distributed collective communication method, characterized in that: The method comprises: The process or node participating in the collective communication sends an HTTP request to the proxy server to apply for registration of the collective communication group record; the collective communication group record is encoded in the form of a dictionary, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1, ..., node n}; The proxy server establishes a collective communication group according to a unique identifier, and implements storage of a globally consistent collective communication group state through a distributed hash table DHT, and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations; The proxy server dynamically manages the collective communication group according to the collective communication ID and handles the joining and leaving of nodes; The proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group. The proxy servers communicate with each other through the HTTP protocol; The server performs collective communication to implement broadcast, protocol, collection, and dispersion operations.

2. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 1, characterized in that: The proxy service includes a sending proxy server, a relay proxy server and a receiving proxy server; each proxy server joins the global distributed hash table DHT to maintain the global state required for collective communication; The sending proxy server is used to receive client requests, process collective communication operations, and forward data; The relay proxy server is used to receive data, process collective communication operations, and forward to the next proxy server or server; The receiving proxy server is used to receive data and forward it to the server.

3. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 2, characterized in that: The specific process of broadcast operation includes: The sending node prepares the data to be broadcast and sends the data to be broadcast to the proxy server through an HTTP POST request; The proxy server receives the HTTP POST request, queries the DHT to obtain the target set information according to the communication group, searches the set communication group in the DHT by the unique identifier, and obtains the address and status information of the target node; The proxy server forwards the data to be broadcast to the proxy servers of all target nodes or recursively delivers it to the relay proxy server; The proxy server of the target node receives the data to be broadcast, and sends the data to be broadcast to the target node. The target node processes the data after receiving it.

4. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 2, characterized in that: The specific process of protocol operation includes: All nodes prepare the data to be reduced and send the data to be reduced to the sending proxy server through HTTP POST request; The sending proxy server receives the HTTP POST request, queries and obtains the target address in the DHT (distributed hash table) according to the unique identifier of the communication group, and forwards the data to the proxy server of the target node or recursively delivers it to the relay proxy server to achieve multi-level forwarding; The proxy server of the target node receives the data and forwards the data to be reduced to the target node; The target node receives all data to be reduced, performs reduction operations, and then returns the reduction results to the client.

5. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 2, characterized in that: The specific process of the collection operation includes: All nodes prepare the data to be collected and send the data to the sending proxy server through HTTP POST request; The sending proxy server receives the request, queries and obtains the address and status of the target node in the DHT according to the unique identifier of the communication group, and forwards the data to be collected to the proxy server of the target receiving node, or recursively delivers it to the relay proxy server to achieve multi-level forwarding; The proxy server of the target node receives the data to be collected and forwards the data to be collected to the target node; The target node receives all the data to be collected, performs the collection operation, and then returns the collection result to the client.

6. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 2, characterized in that: The specific process of decentralized operation includes: The sending node prepares the data to be distributed and divides the data to be distributed into multiple data blocks, each data block corresponds to a target node; The sending node sends each data block to the proxy server of the corresponding target node through an HTTP POST request; The proxy server receives the HTTP POST request, queries and obtains the target set information in the DHT according to the unique identifier of the communication group, including the address and status of the target node, and forwards the data to be dispersed to the proxy servers of all target nodes, or recursively delivers it to the relay proxy server; The proxy server of the target node receives the data to be dispersed from the proxy server of the sending node and forwards it to the target node; The destination node receives the data block and processes and saves it.

7. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 1, characterized in that: The process or node participating in collective communication sends an HTTP request to the proxy server to apply for registration of collective communication group records, including: The processes or nodes participating in collective communication register collective communication group records with the proxy server; Each node registers a proxy server for receiving or sending, and the corresponding proxy server provides transparent delivery of data; Each proxy server registers a relay proxy server, and the corresponding proxy server provides data delivery to the destination node. At the same time, each relay proxy server can register subsequent proxy servers to achieve recursive data delivery; Each new process or node dynamically queries the proxy server's communication group record and joins the existing collective communication group.

8. The heterogeneous dynamic fault-tolerant distributed collective communication method according to claim 1, characterized in that: The proxy server dynamically manages the collective communication group according to the collective communication ID, and processes the joining and exiting of nodes including: The proxy server establishes and maintains a collective communication group according to the collective communication ID and records information of participating nodes; Update the collective communication group according to the joining and leaving of nodes; Through the self-healing characteristics of DHT, the proxy server automatically redistributes and recovers the tasks of failed nodes, ensuring the continuity of collective communication operations.

9. A heterogeneous dynamic fault-tolerant distributed collective communication system, characterized in that: The system comprises: a client, a proxy server and a server; The client is a process or node, which is the sending side of collective communication and is used to send HTTP requests to the proxy server; The proxy server includes: the proxy service includes a sending proxy server, a relay proxy server and a receiving proxy server; each proxy server joins the global distributed hash table DHT to maintain the global state required for collective communication; the sending proxy server is used to receive client requests, process collective communication operations, and forward data; the relay proxy server is used to receive data, process collective communication operations, and forward to the next proxy server or server; the receiving proxy server is used to receive data and forward it to the server; The server is a process or node, which is the response side of collective communication, used to process requests and return responses to clients; The client participating in collective communication sends an HTTP request to the proxy server to apply for registration of collective communication group records; the collective communication group records are encoded in dictionary form, using the collective communication ID as the key and the participating node list as the value; the node list represents each node identifier through a globally unique string; the key-value pair of a collective communication group includes: collective communication ID: {node 1,..., node n}; the proxy server establishes a collective communication group based on the unique identifier, and implements the storage of globally consistent collective communication group status through the distributed hash table DHT, and maintains the collective communication group and information through dynamic addition, deletion, query and modification operations; the proxy server dynamically manages the collective communication group according to the collective communication ID, and handles the joining and exit of nodes; the proxy server receives the collective communication request from the client, and routes and forwards the message to the corresponding node according to the information of the collective communication group, and the proxy servers communicate with each other through the HTTP protocol; the server executes collective communication to implement broadcast, protocol, collection, and dispersion operations.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the heterogeneous dynamic fault-tolerant distributed collective communication method described in any one of claims 1 to 8 is implemented.

Images