Collective communication method, network device, server, computer storage medium, and computer program product

Abstract

Embodiments of the present disclosure provide a collective communication method, a network device, a server, a computer storage medium, and a computer program product. The collective communication method comprises: parsing a packet header of the current data packet forwarded by a neighboring upstream node to obtain a node identifier code and a superimposed node identifier code of the current node, wherein the superimposed node identifier code of the current node indicates a superimposed result of the node identifier code of the current node and a node identifier code of a downstream node; when the current node completes a local operation, removing the node identifier code of the current node from the superimposed node identifier code of the current node to obtain a superimposed node identifier code of a neighboring downstream node of the current node; and when the current node completes a forwarding operation of a downlink data packet, removing the superimposed node identifier code of the neighboring downstream node from the superimposed node identifier code of the current node.

Description

[0001] This disclosure claims priority to Chinese patent application number 202410182433.7, filed with the China Patent Office on February 18, 2024, entitled "Collective Communication Method, Network Device, Server, Computer Storage Medium, and Computer Program Product," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a collective communication method, network device, server, computer storage medium, and computer program product. Background: Collective communication, also known as group communication or multicast communication, is a communication method that simultaneously transmits the same message to multiple destination nodes in a network. The purpose of collective communication is to deliver messages to one or more designated recipients at once, thereby improving communication efficiency and bandwidth utilization. Collective communication can be built using the In-Network Computing & Aggregation Protocol (INCAP) to facilitate collective communication and data acceleration in high performance computing (HPC) / artificial intelligence (AI) scenarios.

[0002] INECAP is primarily used for offloading and accelerating communication primitives such as multicast (Beast) and reduce (Reduce) in collective communication, thereby ensuring the computational efficiency of collective communication. However, in current collective communication, communication reliability and efficiency still need to be improved. In view of this, embodiments of the present disclosure provide a collective communication method, network device, server, computer storage medium, and computer program product to at least partially address the aforementioned issues. According to a first aspect of an embodiment of the present disclosure, a collective communication method is provided, comprising: parsing a message header of a current data message forwarded by an adjacent upstream node to obtain a node identification code and a superimposed node identification code of the current node, wherein the superimposed node identification code of the current node indicates a superposition result of the node identification code of the current node and the node identification code of a downstream node; when the current node completes a local operation, removing the node identification code of the current node from the superimposed node identification code of the current node to obtain a superimposed node identification code of a neighboring downstream node; and when the current node completes a forwarding operation of a downstream data message, removing the superimposed node identification code of the neighboring downstream node from the superimposed node identification code of the current node. According to a second aspect of an embodiment of the present disclosure, a collective communication device is provided, comprising: a parsing module parsing a message header of a current data message forwarded by an adjacent upstream node to obtain a node identification code and a superimposed node identification code of the current node, wherein the superimposed node identification code of the current node indicates a superimposition result of the node identification code of the current node and the node identification code of a downstream node. A first state management module is configured to remove the node identification code of the current node from the superimposed node identification code of the current node when the current node completes a local operation, thereby obtaining the superimposed node identification code of the current node's adjacent downstream node. A second state management module is configured to remove the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node when the current node completes a forwarding operation of a downstream data message. According to a third aspect of an embodiment of the present disclosure, a network device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is configured to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect. According to a fourth aspect of an embodiment of the present disclosure, a server is provided, comprising: the network device described in the second aspect and a host, wherein the network device is mounted to the host.According to a fifth aspect of an embodiment of the present disclosure, a computer storage medium is provided, on which a computer program is stored. When executed by a processor, the program implements the method described in the first aspect. According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instructions. When executed by a processor, the computer program / instructions implement the method described in the first aspect. In the scheme of the embodiment of the present disclosure, a node's local operation and forwarding operation constitute the core operations of the communication primitives of collective communication. The superimposed node identification code of the current node indicates the superposition result of the node identification code of the current node and the node identification code of the downstream node, which facilitates effectively reflecting the status of the local operation and forwarding operation of the current node. In addition, when the current node completes the local operation, the node identification code of the current node is removed from the superimposed node identification code of the current node to obtain the superimposed node identification code of the current node's adjacent downstream node. This enables efficient and reliable management of the local operation of the current node. When the current node completes the forwarding operation of a downstream data message, the superimposed node identification code of the adjacent downstream node is removed from the superimposed node identification code of the current node. This enables efficient and reliable management of the forwarding operation of the current node, thereby improving the efficiency and reliability of collective communication. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the embodiments of the present disclosure. Those skilled in the art can also obtain other drawings based on these drawings. Figure 1 is a schematic architecture diagram of a collective communication network according to some embodiments of the present disclosure; Figure 2 is a step flow chart of a collective communication method according to other embodiments of the present disclosure; Figure 3A is a schematic diagram of node identification coding according to other embodiments of the present disclosure; Figure 3B is a schematic diagram of the message header format of a data message according to other embodiments of the present disclosure; Figure 4 is a structural block diagram of an apparatus according to other embodiments of the present disclosure; Figure 5 is a schematic diagram of a data message according to other embodiments of the present disclosure; Figure 6A is a schematic architecture diagram of a collective communication network according to other embodiments of the present disclosure; Figure 6B is a schematic architecture diagram of a collective communication network according to other embodiments of the present disclosure; Figure 7 is a schematic interaction diagram of each node in the collective communication network of the embodiment of Figure 2 in a multicast scenario; Figure 8 is a schematic interaction diagram of each node in the collective communication network of the embodiment of Figure 2 in a reduction scenario; Figure 9 is a schematic block diagram of a collective communication apparatus according to other embodiments of the present disclosure; Figure 10 is a structural schematic diagram of a network device according to other embodiments of the present disclosure.Detailed Description of the Preferred Embodiments: To help those skilled in the art better understand the technical solutions in the embodiments of the present disclosure, the following will provide a clear and complete description of the technical solutions in the embodiments of the present disclosure, in conjunction with the accompanying drawings. Obviously, the described embodiments represent only a portion of the embodiments of the present disclosure, and not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure should fall within the scope of protection of the embodiments of the present disclosure. The following will further illustrate the specific implementation of the embodiments of the present disclosure, in conjunction with the accompanying drawings. Figure 1 is a schematic architecture diagram of a collective communication network according to some embodiments of the present disclosure. As shown in Figure 1, collective communication can be based on the In-Network Computing & Aggregation Protocol (INCA). OThe collective network includes nodes, for example, node A, node B, node C, node D, and node E. Each node can be a network device such as a network card or a switch. Each network card is mounted on a server host, and the network cards can communicate directly or through a switch. The collective communication network can be based on a network such as Ethernet. Each node is configured with a network address, such as an IP address, based on an Ethernet transmission protocol. Data transmission between nodes is then achieved using its own network address. Furthermore, each node can have local memory. Each node's local memory can be memory capable of performing direct memory access (DMA), such as the memory of the server host to which the node (e.g., network card) is mounted, or memory mounted on a node (e.g., switch). As shown in Figure 1, the data exchanged between the nodes (AE) in the collective communication network is based on a unified shared virtual address space 0xFFOO. The local memory address spaces (e.g., physical address spaces) corresponding to nodes A, B, C, D, and E are 0xAOOO, 0xBOOO, 0xCOOO, 0xDOOO, and 0xEOOO, respectively. In addition, each node may also include a local cache. Before using the cache space in the local cache, the node must send a cache space request instruction. After the cache space is used, the node can send a cache space release instruction. The collective communication method according to other embodiments of the present disclosure will be described in detail below with reference to Figure 2. The collective communication method of Figure 2 can be applied to communication networks such as collective communication networks. For example, the collective communication method can be applied to the collective communication network exemplified in Figure 1 and executed by any node in the collective communication network. Specifically, the collective communication method includes:

[0003] S210: Parse the header of the current data message forwarded by the adjacent upstream node to obtain the node identification code of the current node and the superimposed node identification code. The superimposed node identification code of the current node indicates the superposition of the node identification code of the current node and the node identification code of the downstream node. It should be understood that the header of the current data message may be a protocol header of the transport layer or a layer above it, for example, a UDP layer protocol header. The current data message is obtained by decapsulating the protocol of a layer below the transport layer. It should also be understood that the node identification code identifies each node through encoding to facilitate state management, such as the operating state or communication state of each node. For example, each node may be identified by an encoded bit sequence. The superimposed node identification code may be the superposition of the node identification code of the current node and the node identification code of the downstream node. The superimposed node identification code is used to manage the state of the current node and its downstream nodes.

[0004] S220: When the current node completes the local operation, the node identification code of the current node is removed from the superimposed node identification code of the current node to obtain the superimposed node identification codes of the current node's adjacent downstream nodes. It should be understood that when a node has no downstream nodes, the superimposed node identification code of the node is the node identification code of the node. A node's downstream nodes include adjacent downstream nodes and non-adjacent downstream nodes. When an adjacent downstream node has no downstream nodes, the node's downstream nodes only include the adjacent downstream nodes of the node. It should also be understood that the removal of the current node's node identification code from the superimposed node identification code of the current node depends on the encoding scheme of the node identification codes. If the encoding scheme is binary, the node identification code of the current node is subtracted from the superimposed node identification code of the current node. If the encoding scheme is bit masking, the representative bit position of the node identification code of the current node in the superimposed node identification code of the current node is inverted. For example, if the representative bit position is "0001," the "0001 bit" is set to zero.

[0005] S230: When the current node completes forwarding the downstream data message, the superimposed node identification code of the adjacent downstream node is removed from the superimposed node identification code of the current node. It should be understood that a node is the upstream node of each downstream node of the node, and a node is the adjacent upstream node of the adjacent downstream node of the node. In other words, the upstream and downstream relationships of nodes are inverse. It should also be understood that the removal of the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node depends on the encoding mode of the node identification code. If the encoding mode is binary, the superimposed node identification code of the adjacent downstream node is subtracted from the superimposed node identification code of the current node. If the encoding mode is bit masking, the representative bit position of the superimposed node identification code of the adjacent downstream node in the superimposed node identification code of the current node is not inverted. For example, if the representative bit position is "0001", the "0001 bit" is set to zero. In the embodiments of the present disclosure, a node's local and forwarding operations constitute the core operations of the collective communication primitives. The superimposed node identification code of the current node indicates the superposition of the node identification code of the current node and the node identification code of the downstream node, effectively reflecting the status of the current node's local and forwarding operations. Furthermore, when the current node completes its local operation, the node identification code of the current node is removed from its superimposed node identification code to obtain the superimposed node identification code of the current node's adjacent downstream node. This enables efficient and reliable management of the current node's local operation. When the current node completes forwarding of a downstream data message, the superimposed node identification code of the adjacent downstream node is removed from its superimposed node identification code. This enables efficient and reliable management of the current node's forwarding operation, thereby improving the efficiency and reliability of collective communication. Regarding the node identification code encoding scheme, the node identification code of each node in the collective communication network is determined based on a bit sequence of the same length, and different nodes have different node identification codes. This processing method further enables efficient management of superimposed node identification codes. The node identification codes of multiple nodes can be superimposed to form a superimposed node identification code. For example, the node identification codes of the four nodes 0001, 0010, 0100, and 1000 can be superimposed to obtain 1 1 1 1 as the superimposed node identification code. In this case, each node identification code is obtained by encoding based on a bit mask, and the representative bit values ​​of the node identification codes of different nodes are located in different positions in the bit mask. It should be understood that in the above example, "1" is the representative feature value of the node, and "0" may also be used as the representative bit value.Furthermore, in the above example, the number of characterization bits is 1, but an integer greater than 1 may also be used to characterize a specific node. It should also be understood that if each node identification code is encoded based on binary encoding rather than bit masking, different bit values ​​at the same bit position also form different node identification codes. Accordingly, the superimposed node identification code formed by multiple node identification codes is the sum of the multiple node identification codes. Furthermore, when the node identification code is encoded based on a bit mask, removing at least one node identification code from the superimposed node identification code yields higher computational efficiency. When the node identification code is encoded based on binary encoding, the bit sequence of the node identification code is shorter, occupying fewer data packet header fields, which is beneficial for improving data packet communication efficiency. Without loss of generality, the bit sequence includes a main bit field and an extended bit field. The main bit field is encoded based on a bit mask, and the extended bit field is encoded based on binary encoding. The characterization bit values ​​of the node identification codes of different nodes are located at different positions in the bit mask, or the node identification codes of different nodes have different bit values ​​in the extended bit field. Through this processing method, the main bit field is obtained by encoding based on the bit mask, further efficiently managing the superimposed node identification code. In addition, the extended bit field includes a bit string of a preset length. When encoded based on binary, different bit strings represent different extended bit fields. In other words, as long as there are different bit values ​​at the same bit position of two bit strings, the two bit strings are different. For example, 1 1 1 1 and 1 1 10 are different bit strings. Therefore, the extended bit field is obtained by encoding based on binary, making full use of the number of bit combinations to generate more bit strings, thereby reducing the length of the node identification code. In some examples, the node identification code can be represented as<Set I D : B i tStr i ng> , where BitString (i.e., bit string) is an example of a primary bit field; SetID is an example of an extended bit field. Furthermore, the bit sequence also includes a network domain identification field of the collective communication network, which is encoded based on binary. Through this processing method, data packets can be reliably transmitted between different network domains, expanding the transmission range of data packets in collective communication. In some examples, the node identification code can be represented as<Subdoma i n : Set I D : B i tSt r i ng> , where Subdoma in is a subdomain, representing an example of a network domain identification field.More specifically, Subdomin can be used in I NCA domains, with a range of [0, 255]. Subdomin's design can reduce inter-domain communication traffic and support a variety of hierarchical collective communications and algorithm topologies. The Bit String Length (BSL) is variable, and a single I NCA domain can support 1024 nodes. When the bit string length is long, the extended bit field may be Set 1 D*2"BSL. For example, FIG3A shows a node identification encoding based on the <Subdomin: Set ID: BitStrin> format. In this example, P0D-0 and P0D-1 are different I NCA domains. The node identification encoding of node A is 1:0:0001 and 0:0:0001, belonging to two I NCA domains. Node B, node D, node E, node F, and node G belong to POD-0, and the Subdomin of each node is 0. For the main bit field, node A (0001), node B (0010), node D (0100), and node E (1000) respectively occupy the respective bit positions of the main bit field encoded based on the bit mask. Therefore, the bit value (1) of the extended bit field of node F and node G is the same as that of node A, node B, node D, The bit value (0) of the extended bit field of node E is different. Correspondingly, node C, node H, node I, node J and node K belong to POD-1. The Subdomano in of each node is 0. For the main bit field, node A (0001), node C (0010), node H (0100), and node I (1000) respectively occupy the respective bit positions of the main bit field encoded based on the bit mask. Therefore, the bit value (1) of the extended bit field of node J and node K is different from the bit value (0) of the extended bit field of node A, node C, node H, and node I. The header format of the data message The network data packet transmitted by the network layer includes a network layer protocol header and a data message. The data message includes a message header (for example, a transport layer protocol header) and data (payload). As shown in FIG3B , the message header includes a field for representing the node encoding, The virtual address field of the virtual address space and the sequence number field of the data message.Before forwarding, the current node can assign attribute data from various fields of the transport layer header of the downstream data message to corresponding fields to obtain the transport layer header of the downstream data message. The current node then encapsulates the data to be encapsulated based on the transport layer header to obtain the downstream data message. Furthermore, the current node encapsulates the downstream data message based on the network layer protocol header to obtain a network data packet. Specifically, "Protocol Version" indicates the protocol evolution version; "Reserved Field" indicates a reserved field for later protocol extension of "Operation Type" or "Bit String Length" or other protocol definitions; "Operation Type" indicates the operation type information of the data message, for example, 8 bits can be used to support 255 operators; "Data Type" indicates a data type that supports multiple data type identifiers such as FP32 / FP16; "Subfield" is used to define the INCA field of the data message; "Extended Bit Field" indicates the aforementioned extended bit field; "Superposition Node Representation Code" indicates the superposition node representation code; "Message Sequence Number" indicates the sequence number of the data message, which is used for rapid location in the local cache space and for constructing a sliding window process based on the sequence number of the data message; "Virtual Address" indicates a virtual address in the virtual address space; and "Bit String" indicates a main bit field. In some embodiments, in the collective communication method, a network data packet sent by an adjacent upstream node may also be received, and then, based on the network layer protocol, the network data packet may be decapsulated to obtain the current data message. Through this processing method, while being compatible with the network layer protocol, Reliable management of data packets at the transport layer is achieved. The forwarding table configuration process is shown in Figure 3A. The communication primitives of collective communication rely on the upstream and downstream relationships of nodes in the topology graph formed by each node. In the disclosed embodiment, the superimposed node identification code of the current node indicates the superposition of the node identification code of the current node and the node identification code of the downstream node. Thus, the superimposed node identification code represents the upstream and downstream forwarding relationship of the node. For example, a forwarding table can be pre-configured for each node in the collective communication network. The entries in the forwarding table are the superimposed node identification codes of the adjacent downstream nodes. As an example, the entries in the forwarding table are the correspondence between the identifiers (e.g., network addresses) of the adjacent downstream nodes and the superimposed node identification codes. As shown in Figure 4, Nbr indicates the node identifier, and F-BM indicates the node identification code (e.g., the primary bit field). When the adjacent downstream node itself has no downstream node, the superimposed node identification code of the adjacent downstream node is the node identification code of the adjacent downstream node.Specifically, regarding node downstream relationships, as shown in Figure 4, node A's downstream nodes include node B, node C, node D, node E, node F, and node G, where node A's adjacent downstream nodes are nodes B and node C. Accordingly, the entries in node A's forwarding table include: B-1 1 1 1; C-101 1. Node B's downstream nodes include node D, node E, and node F, where node B's adjacent downstream nodes are nodes D and node F. Accordingly, the entries in node B's forwarding table include: D-0101 : F-0010. Node C's downstream node and adjacent downstream node are both node G0. Node D's downstream node and adjacent downstream node are both node E. Accordingly, the entries in node D's forwarding table include: E-0001 . It should be understood that node upstream relationships and node downstream relationships are inversely related and will not be further elaborated here. In general scenarios, for node interaction in collective communication (e.g., multicast scenarios (Beast) or reduce scenarios (Reduce)), without loss of generality, when the current node completes the forwarding operation of a downstream data message, the superimposed node identification code of the neighboring downstream node is removed from the superimposed node identification code of the current node. This can be specifically implemented as follows: upon receiving an operation completion response for a downstream data message from the neighboring downstream node, the superimposed node identification code of the neighboring downstream node is removed from the superimposed node identification code of the current node. This processing approach improves the reliability of the forwarding operation of the downstream data message. Furthermore, after the superimposed node identification code of the current node is cleared, an operation completion response can be returned to the neighboring upstream node. This processing approach improves the reliability of various node operations, including forwarding operations and local operations. Regarding the node interaction process in collective communication in the multicast scenario (Beast), in some embodiments, write operations are an important operation type for the multicast scenario (Beast). Accordingly, in the collective communication method disclosed herein, the header of the current data message can be parsed to obtain the operation type information of the current node. Then, based on the operation type information of the current node, the local operation of the current node is executed. This processing approach allows the node's operation type information to be configured in the message header, improving the node's overall computing and communication efficiency. Specifically, the node identification code, the superimposed node identification code, and the current node's operation type information are all encapsulated in the message header of the current data message.When parsing the header of the current data message, the node identification code, the superimposed node identification code, and the operation type information of the current node can be obtained. Accordingly, based on the read or write operation indicated by the operation type information of the current node, a local read operation or a local write operation is performed on the current data message. Furthermore, local cache space can be pre-allocated to store the input to be written in the parsed result of the current data message in the local cache space of the current node. Furthermore, as an example of performing a local operation on the current node based on the operation type information of the current node, the data to be written can be written from the local cache space to the local memory of the current node based on the write operation indicated by the operation type information of the current node. This processing approach improves the node's write operation efficiency, thereby improving the node's overall computing and communication efficiency. It should be understood that the current node can release the local cache space after the operation response is completed. As a specific implementation of the write operation, the first virtual address of the data to be written in the shared storage space of the collective communication network can be obtained when parsing the header of the current data message. Then, to write the pending data in the current data packet to the local memory of the current node, the locally cached address mapping table can be searched to determine the first physical address corresponding to the first virtual address in the local memory. The pending data is then written to the first physical address. Specifically, the locally cached address mapping table indicates the mapping relationship between the unified shared virtual address space and the physical address space of the local memory. By searching the locally cached address mapping table, the first physical address corresponding to the first virtual address in the local memory is determined. This processing approach utilizes shared storage space in collective communication and, through the address mapping table, is compatible with the node's local physical storage space, improving overall node computing and communication efficiency. In some embodiments, the node interaction process in collective communication under the reduce scenario (Reduce) is an important operation type. Accordingly, in the collective communication method of the present disclosure, to execute a local operation of the current node based on the operation type information of the current node, local data may be read from the local memory of the current node based on the reduction operation indicated by the operation type information of the current node, and at least one downstream reduction result sent by at least one adjacent downstream node may be received. A local reduction calculation may then be performed on the local data and the at least one downstream reduction result, and the local reduction result of the local reduction calculation may then be returned to the adjacent upstream node. In other words, the local reduction calculation indicates a reduction calculation performed on the local data and the at least one downstream reduction result.Through this processing approach, each node performs a unified and consistent reduction operation, improving the node's reduction operation efficiency and, in turn, overall node computing and communication efficiency. It should be understood that the current node can receive a receipt response for a data message sent by a neighboring downstream node and, based on the receipt response, remove the neighboring downstream node's superimposed node identification code (e.g., set the neighboring downstream node's superimposed node identification code to zero). Alternatively, the current node can receive an operation completion response from a neighboring downstream node and, based on the operation completion response, remove the neighboring downstream node's superimposed node identification code (e.g., set the neighboring downstream node's superimposed node identification code to zero). Furthermore, to read local data from the current node's local memory based on the reduction operation indicated by the current node's operation type information, local cache space can be allocated for the current node. Then, based on the reduction operation indicated by the current node's operation type information, local data is read from the current node's local memory into the local cache space, and at least one downstream reduction result is stored in the local cache space. The local cache space is then released after the local reduction result of the local reduction computation is returned to the neighboring upstream node. Furthermore, in the collective communication method, the node identification code and superimposed node identification code of the adjacent downstream node can be configured into the header of the downstream data message, and then the downstream data message is forwarded to the adjacent downstream node. Specifically, before the current node performs forwarding, the attribute data in each field of the transport layer header of the downstream data message can be configured into the corresponding field to obtain the transport layer header of the downstream data message. The data to be encapsulated is then encapsulated based on the transport layer header to obtain the downstream data message. Furthermore, the downstream data message is encapsulated based on the network layer protocol header to obtain a network data packet. In other words, this processing method only requires minor modifications to the header of the current data message to obtain the header of the downstream data message, improving the node's message encapsulation efficiency and, in turn, overall, improving the node's computing and communication efficiency. Furthermore, the node's processor or controller can use the forwarding table to perform data message forwarding operations. For example, in a collective communication method, a preconfigured forwarding table for the current node can be queried to determine the superimposed node identification code of a downstream neighboring node. The forwarding table indicates the correspondence between the destination address of the neighboring downstream node and the superimposed node identification code of the neighboring downstream node. This processing approach improves the efficiency of obtaining the superimposed node identification code of the neighboring downstream node based on the node preconfiguration. More specifically, the destination address of the neighboring downstream node is the network address of the neighboring downstream node in the collective communication network.For example, the destination address can be obtained by parsing the network layer protocol header. In this case, when forwarding a downstream data message to a neighboring downstream node, the downstream data message can be forwarded to the neighboring downstream node based on the network address. This processing approach facilitates efficient data message transmission and reception, further improving message transmission management efficiency. In other embodiments, in the collective communication method, the current data message can be arranged into the data message sequence of the current node based on its message sequence number. Then, the current data message can be identified within the data message sequence using a sliding window. The superimposed node identification code of the current node can then be updated based on the operation completion status of the current node. The operation completion status indicates the completion of the local operation and / or forwarding operation of the current node. Specifically, upon completion of the local operation and / or forwarding operation of the current node, the superimposed node identification code of the current node can be updated. In other words, upon receiving a reception response indicating the completion of a forwarding operation for the current node, or an operation completion response indicating the completion of a local operation for the current node, the superimposed node identification code of the current node can be updated, placing the superimposed node identification code of the current node in a pending state for subsequent local and / or forwarding operations. Furthermore, after the update of the superimposed node identification code of the current node is completed, the next data message in the data message sequence is determined using a sliding window. This processing approach ensures the reliability of node operations when a node sends and receives a large number of data messages, while reducing the complexity of processing such a large number of data messages. Specifically, as shown in FIG5 , the data messages can be arranged into a data message sequence according to the message sequence. State information of each data message in the data message sequence in the local cache space can be managed. The state information includes the superimposed state of the node identification codes in the data messages, namely, the superimposed node identification code of the current node. The state information also includes the message sequence number of the data message. The state information also includes the virtual address of the data message. It should be understood that the sliding window slides in a specific order to select a current data packet and cause the current data packet to enter a management execution state. The management execution state instructs the processor or controller of the current node to update the superimposed state of the node identification code of the current data packet (for example, removing the superimposed node identification code of the adjacent downstream node). Data packets not selected by the sliding window are in a non-management execution state, which indicates that the superimposed state of the node identification code of the data packet is maintained.That is, when a data message is in a non-management execution state, any receive response, operation completion response, or data message received by the current node is stored in a local cache. This is done until the data message enters a management execution state, at which point the processor or controller of the current node reads the receive response, operation completion response, or data message. The collective communication network according to other embodiments of the present disclosure will be described in detail below with reference to FIG. The collective communication method of FIG. 6A includes the following:

[0006] S610: Parse the message header of the current data message forwarded by the adjacent upstream node to obtain the node identification code and superimposed node identification code of the current node, and operation type information of the current node.

[0007] S620: Allocate local cache space of the current node, and write the data to be written in the current data message into the local cache space.

[0008] S630: Based on the write operation indicated by the operation type information of the current node, write the data to be written from the local cache space to the local memory of the current node.

[0009] S640: Upon receiving an operation completion response indicating completion of the write operation, release the local cache space. It should be understood that the description and explanation of the execution process of each step of this embodiment can be found in the various embodiments of FIG. 2 , and are not further elaborated here. The collective communication network according to other embodiments of the present disclosure will be described in detail below in conjunction with FIG. The collective communication method of FIG. 6B includes:

[0010] S650: Parse the message header of the current data message forwarded by the adjacent upstream node to obtain the node identification code and superimposed node identification code of the current node, and operation type information of the current node.

[0011] S660: Allocate local cache space of the current node, and write the data to be written in the current data message into the local cache space.

[0012] S670: Based on the reduction operation indicated by the operation type information of the current node, read local data from the local memory of the current node, and receive at least one downstream reduction result sent by at least one adjacent downstream node.

[0013] S680: Perform local reduction calculation on local data and at least one downstream reduction result.

[0014] S690: Return the local reduction result of the local reduction calculation to the adjacent upstream node. It should be understood that the description and explanation of the execution process of each step of this embodiment can be referred to the various embodiments of FIG. 2 and will not be repeated here. The interaction process of a specific scenario in collective communication will be described in detail below in conjunction with FIG. 7 and FIG. 8. FIG. 7 is a schematic diagram of the interaction between nodes in the collective communication network of the embodiment of FIG. 2 in a multicast scenario. As shown in FIG. 7, the collective communication method includes the following steps:

[0015] S710: Node A sends data to be written to node B, and writes node B's superimposed node identification code into the data message of the data to be written. For example, node A carries INCA-header: 0:11111 in the downstream data message, and records 0:11111 locally.

[0016] S720: Node B parses the data message and stores the data to be written and the superimposed node identification code in the local cache. For example, after receiving the data to be written, Node B stores the data to be written in the local cache and records 0:11111 in the cache.

[0017] S730: Node B performs a local operation on the data to be written. For example, node B performs direct memory access (DMA) on local memory. After completing the local operation (for example, after receiving a write completion response from the local memory), node B's node identification code (bits 0:1000) is reset to zero. S740: Node B queries the forwarding table to determine the superimposed node identification codes of nodes D and F. In this example, since node F has no downstream nodes, F's superimposed node identification code is F's node identification code itself. Specifically, node B queries the INCA-FT (Forwarding Table), transfers the data to be written to node B's adjacent downstream nodes D and F, and sets the corresponding superimposed node identification codes BitMask.

[0018] S750: Node B sends the data to be written to node D and configures the data message with node B's superimposed node identification code. For example, node B sends the data to be written with the INCA header: 0:0101, but the data is lost (for example, due to a retransmission timeout (RTO) event: that is, the operation completion response from node D is not received within the unit time).

[0019] S760: Node B sends the data to be written to node F and adds node F's superimposed node identification code to the data message. For example, node B sends the data to be written with the INCA header: 0:0010. After receiving the data to be written, node F returns a response to node B. Node B sets node F's superimposed node identification code (bits 0:0010) to zero.

[0020] S770: After the transmission timeout, node B resends the data to be written to node D and configures node B's superimposed node identification code in the data message. For example, node B resends the data message to node D based on node D's superimposed node identification code in its local cache. After receiving the data message, node D performs a local operation and resets node D's node identification code (0:0100) to zero. If node D then returns an operation completion response to node B after receiving the data message, it resets node D's superimposed node identification code in its local cache to zero. If node D does not receive an operation completion response from node B, it retransmits the data message again. After node B's superimposed node identification code is all zero, node B releases local cache space. It should be understood that node D's operation completion response may also carry node D's superimposed node identification code (0:0101).

[0021] S780: Node D sends the data to be written to node E and configures node E's node identification code into the data message. For example, node D sends a data message carrying the data to be written to node E. Upon receiving an operation completion response from node E, node D releases local cache space. It should be understood that the operation completion response from node E may include node E's node identification code (0:0001). Furthermore, Figure 8 is a schematic diagram illustrating the interactions between nodes in the collective communication network of the embodiment of Figure 2 in a reduction scenario. As shown in Figure 8, the collective communication method includes the following steps:

[0022] S810: Node A sends a data message to node B, and configures the superimposed node identification code of node B in the data message. For example, node A sends a reduction request with INCA-Header 0:1111 o

[0023] S820: After receiving the data packet, node B allocates local cache. For example, after receiving the reduce request, node B allocates local cache and records INCA-Header 0:1111.

[0024] S830: Node B queries the forwarding table to determine adjacent downstream nodes D and F. For example, node B queries the INCA forwarding table to determine nodes D and F. o Node B forwards the data message to node D, and carries the node identification code of node D in the data message. Node B transmits the data message to node F, and carries the node identification code of node F in the data message.

[0025] S840: Node B performs a local operation and removes the node identification code of node B. For example, node B performs a direct memory access on the local memory to obtain local data, and then sets the node identification code of B (bits 0:1000) to zero.

[0026] S850: Node B receives the downstream reduction result sent by node F. In this example, since node F has no downstream nodes, the downstream reduction result from node F is node F's local data. Specifically, after receiving the reduce request, node F sends node F's local data as node B's downstream reduction result to node B. After receiving the downstream reduction result from node F, node B performs the reduction operation and sets node F's node identification code (bits 0:0010) to zero.

[0027] S860: Node D receives the downstream reduction result sent by node E. For example, after receiving the reduction request, node D allocates local cache and queries the forwarding table to determine node D's adjacent downstream node E. Node D then reads local data and sends a reduction request to node E. Since node E has no downstream nodes, the downstream reduction result from node E is node E's local data.

[0028] S870: Node D returns the local reduction result to node B. For example, after node D receives the downstream reduction result from node E, node D performs a local reduction calculation on the downstream reduction result from node E and local data. It should be understood that since node E has no downstream nodes, the downstream reduction result from node E is node E's local data. Node D then performs a local reduction calculation and sends the local reduction result to node B. Node D can then remove the node identification code of node E. Without loss of generality, node D can be used as the current node. Based on the reduction operation indicated by node D's operation type information, node D reads local data from its local memory and receives at least one downstream reduction result sent by at least one adjacent downstream node. Node D then performs a local reduction calculation on the local data and the at least one downstream reduction result, and then returns the local reduction result of the local reduction calculation to node B.

[0029] S880: Node B performs a local reduction calculation on the downstream reduction result from node D and node B's local data. For example, after receiving the downstream reduction result from node D, node B removes the superimposed node identification code of node D. When the superimposed node identification code of node B is cleared, node B sends the local reduction result to the adjacent upstream node and releases local cache space.

[0030] S890: After node A completes local data reduction, node A initiates a multicast operation to node B. Figure 9 is a schematic block diagram of a collective communication device according to other embodiments of the present disclosure. The collective communication device of Figure 9 corresponds to the collective communication method of Figure 2 and includes: a parsing module 910 that parses the header of a current data message forwarded by an adjacent upstream node to obtain a node identification code and a superimposed node identification code of the current node, wherein the superimposed node identification code of the current node indicates the superposition of the node identification code of the current node and the node identification code of the downstream node. A first state management module 920 that, when the current node completes a local operation, removes the node identification code of the current node from the superimposed node identification code of the current node to obtain the superimposed node identification code of the adjacent downstream node of the current node. A second state management module 930 that, when the current node completes a forwarding operation of a downstream data message, removes the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node. In the embodiments of the present disclosure, a node's local and forwarding operations constitute the core operations of the collective communication primitives. The superimposed node identification code of the current node indicates the superposition of the node identification code of the current node and the node identification code of the downstream node, which effectively reflects the status of the local and forwarding operations of the current node. Furthermore, when the current node completes a local operation, the node identification code of the current node is removed from the superimposed node identification code of the current node to obtain the superimposed node identification code of the current node's adjacent downstream node. This enables efficient and reliable management of the current node's local operations. When the current node completes a forwarding operation for a downstream data message, the superimposed node identification code of the adjacent downstream node is removed from the superimposed node identification code of the current node. This enables efficient and reliable management of the current node's forwarding operations, thereby improving the efficiency and reliability of collective communication. In other embodiments of the present disclosure, the node identification code of each node in the collective communication network is determined based on a bit sequence of the same length, and different nodes have different node identification codes. In other embodiments of the present disclosure, the bit sequence includes a main bit field and an extended bit field. The main bit field is encoded based on a bit mask, and the extended bit field is encoded based on a binary system. The bit values ​​representing the node identifiers of different nodes are located in different positions in the bit mask, or the node identifiers of different nodes have different bit values ​​in the extended bit field. In other embodiments of the present disclosure, the bit sequence also includes a network domain identifier field for the collective communication network, and the network domain identifier field is encoded based on a binary system.In other embodiments of the present disclosure, the parsing module is further configured to: parse the header of the current data packet to obtain operation type information of the current node. The collective processing device also includes a local operation module, which performs local operations on the current node based on the operation type information of the current node. In other embodiments of the present disclosure, the local operation module is specifically configured to: allocate local cache space for the current node; write the to-be-written data in the current data packet into the local cache space; and write the to-be-written data from the local cache space to the local memory of the current node based on the write operation indicated by the operation type information of the current node. In other embodiments of the present disclosure, the parsing module is further configured to: parse the header of the current data packet to obtain a first virtual address of the to-be-written data in the shared memory space of the collective communication network. The local operation module is specifically configured to: search the address mapping table of the current node to determine the first physical address corresponding to the first virtual address in the local memory; and write the to-be-written data to the first physical address. In other embodiments of the present disclosure, the local operation module is specifically configured to: read local data from the local memory of the current node based on a reduction operation indicated by the operation type information of the current node; receive at least one downstream reduction result sent by at least one adjacent downstream node; perform a local reduction calculation on the local data and the at least one downstream reduction result; and return the local reduction result of the local reduction calculation to the adjacent upstream node. In other embodiments of the present disclosure, the local operation module is specifically configured to: allocate local cache space for the current node; read local data from the local memory of the current node into the local cache space based on the reduction operation indicated by the operation type information of the current node; store the at least one downstream reduction result in the local cache space; and release the local cache space after returning the local reduction result of the local reduction calculation to the adjacent upstream node. In other embodiments of the present disclosure, the collective communication device further includes a forwarding operation module configured to assign a node identification code and a superimposed node identification code of the adjacent downstream node to a message header of the downstream data message; and forward the downstream data message to the adjacent downstream node. In some other embodiments of the present disclosure, the forwarding operation module is specifically used to: query a forwarding table pre-configured for the current node to determine the superimposed node identification code of the downstream adjacent node, where the forwarding table indicates the correspondence between the sending destination address of the adjacent downstream node and the superimposed node identification code of the adjacent downstream node.In other embodiments of the present disclosure, the destination address of the transmission of the adjacent downstream node is the network address of the adjacent downstream node in the collective communication network. The forwarding operation module is specifically configured to: forward the downstream data packet to the adjacent downstream node based on the network address. In other embodiments of the present disclosure, the parsing module is specifically configured to: receive a network data packet sent by the adjacent upstream node; and decapsulate the network data packet based on a network layer protocol to obtain the current data packet. In other embodiments of the present disclosure, the second state management module is specifically configured to: upon receiving an operation completion response for the downstream data packet from the adjacent downstream node, remove the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node. In other embodiments of the present disclosure, the local operation module is further configured to: after the superimposed node identification code of the current node is cleared, return an operation completion response to the adjacent upstream node. In other embodiments of the present disclosure, the local operation module is further configured to: allocate local cache space for the current node; and after returning the operation completion response to the adjacent upstream node, release the local cache space. In other embodiments of the present disclosure, the local operation module is further configured to: arrange the current data packet into a data packet sequence of the current node based on the message sequence number of the current data packet; determine the current data packet in the data packet sequence using a sliding window; and update the superimposed node identification code of the current node based on the operation completion status of the current node, where the operation completion status indicates the completion of the local operation and / or forwarding operation of the current node. Referring to FIG10 , a schematic structural diagram of a network device according to another embodiment of the present disclosure is shown. The specific embodiments of the present disclosure do not limit the specific implementation of the network device. As shown in FIG10 , the network device may include: a processor 602 for executing a program 610, a communication interface 604, a memory 606, and a communication bus 608. The processor, communication interface, and memory communicate with each other via the communication bus. The communication interface is configured to communicate with other electronic devices or a server. The processor is configured to execute the program and may specifically perform the relevant steps of the above-described method embodiments. Specifically, the program may include program code, and the program code includes computer operation instructions.The processor may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure. The one or more processors included in the smart device may be of the same type, such as one or more CPUs, or different types, such as one or more CPUs and one or more ASICs. The memory is used to store programs. The memory may include high-speed RAM or non-volatile memory, such as at least one disk drive. The program may include multiple computer instructions. Specifically, the program may cause a processor to execute the following through the multiple computer instructions: parsing a message header of a current data message forwarded by an adjacent upstream node to obtain a node identification code and a superimposed node identification code of the current node, wherein the superimposed node identification code of the current node indicates a superposition of the node identification code of the current node and the node identification code of a downstream node; when the current node completes a local operation, removing the node identification code of the current node from the superimposed node identification code of the current node to obtain the superimposed node identification code of the adjacent downstream node of the current node; and when the current node completes a forwarding operation of a downstream data message, removing the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node. The specific implementation of each step in the program can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned method embodiments, and corresponding beneficial effects are achieved, and are not further described here. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes of the devices and modules described above can refer to the corresponding process descriptions in the above-mentioned method embodiments, and are not further described here. Embodiments of the present disclosure also provide a computer storage medium storing a computer program that, when executed by a processor, implements the method described in any of the aforementioned method embodiments. The computer storage medium includes, but is not limited to, a compact disc (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk. Embodiments of the present disclosure also provide a computer program product comprising computer instructions that instruct a computing device to execute the aforementioned collective communication method.Furthermore, it should be noted that all user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for model training, data used for analysis, stored data, and displayed data, etc.) involved in the embodiments of this disclosure are authorized by the user or fully authorized by all parties. The collection, use, and processing of these data must comply with relevant regulations and standards, and corresponding entry points are provided for users to choose to authorize or deny. It should be noted that, depending on implementation needs, the various components / steps described in the embodiments of this disclosure may be split into more components / steps, or two or more components / steps or partial operations of components / steps may be combined into new components / steps to achieve the objectives of the embodiments of this disclosure. The above-mentioned method according to the embodiment of the present disclosure may be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)). It will be understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods described herein are implemented. Furthermore, when a general-purpose computer accesses code for implementing the methods described herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods described herein. Those skilled in the art will appreciate that the various exemplary units and method steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The above embodiments are intended only to illustrate the embodiments of the present disclosure and are not intended to limit them. Persons skilled in the relevant technical fields may make various changes and modifications without departing from the spirit and scope of the embodiments of the present disclosure. Therefore, all equivalent technical solutions are also within the scope of the embodiments of the present disclosure. The scope of patent protection for the embodiments of the present disclosure shall be defined by the claims.

Claims

Claims 1. A collective communication method, comprising: Parse the message header of the current data message forwarded by the adjacent upstream node to obtain the node identification code and the superimposed node identification code of the current node, wherein the superimposed node identification code of the current node indicates the superposition result of the node identification code of the current node and the node identification code of the downstream node; when the current node completes the local operation, remove the node identification code of the current node from the superimposed node identification code of the current node to obtain the superimposed node identification code of the adjacent downstream node of the current node; when the current node completes the forwarding operation of the downstream data message, remove the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node.

2. The method according to claim 1, wherein: The node identification code of each node in the collective communication network is determined based on a bit sequence of the same length, and the node identification codes of different nodes are different.

3. The method according to claim 2, wherein: The bit sequence includes a main bit field and an extended bit field, the main bit field is obtained by encoding based on a bit mask, and the extended bit field is obtained by encoding based on binary. The representation bit values ​​of the node identification codes of different nodes are in different positions in the bit mask, or the node identification codes of different nodes have different bit values ​​in the extended bit field.

4. The method according to claim 3, wherein: The bit sequence further includes a network domain identification field of the collective communication network, and the network domain identification field is obtained by encoding based on binary.

5. The method according to any one of claims 1 to 4, wherein: The method further includes: parsing a message header of the current data message to obtain operation type information of the current node; and performing a local operation of the current node based on the operation type information of the current node.

6. The method according to claim 5, wherein: The method also includes: allocating local cache space for the current node; writing the data to be written in the current data message into the local cache space; and performing local operations of the current node based on the operation type information of the current node, including: writing the data to be written from the local cache space to the local memory of the current node based on the write operation indicated by the operation type information of the current node.

7. The method according to claim 6, wherein: The method also includes: parsing the message header of the current data message to obtain the first virtual address of the data to be written in the shared storage space of the collective communication network; writing the data to be written in the current data message to the local memory of the current node, including: searching the address mapping table of the current node to determine the first physical address corresponding to the first virtual address in the local memory; writing the data to be written to the first physical address.

8. The method according to claim 5, wherein: Executing a local operation of the current node based on the operation type information of the current node includes: Based on the reduction operation indicated by the operation type information of the current node, read local data from the local memory of the current node; receive at least one downstream reduction result sent by at least one adjacent downstream node; performing a local reduction calculation on the local data and the at least one downstream reduction result; Returning a local reduction result of the local reduction calculation to the adjacent upstream node.

9. The method according to claim 8, wherein: Reading local data from the local memory of the current node based on a reduce operation indicated by the operation type information of the current node includes: allocating local cache space of the current node; reading local data from the local memory of the current node into the local cache space based on the reduce operation indicated by the operation type information of the current node; the method further includes: storing the at least one downstream reduction result in the local cache space; and releasing the local cache space after returning the local reduction result of the local reduce calculation to the adjacent upstream node.

10. The method according to any one of claims 1 to 9, wherein: The method further includes: configuring the node identification code and the superimposed node identification code of the adjacent downstream node into a message header of the downstream data message; and forwarding the downstream data message to the adjacent downstream node.

11. The method according to claim 10, wherein: The method further includes querying a forwarding table pre-configured for the current node to determine the superimposed node identification code of the downstream adjacent node, wherein the forwarding table indicates a correspondence between a sending destination address of the adjacent downstream node and the superimposed node identification code of the adjacent downstream node.

12. The method according to claim 11, wherein: The destination address of the transmission of the adjacent downstream node is the network address of the adjacent downstream node in the collective communication network; Forwarding the downstream data message to the adjacent downstream node includes: forwarding the downstream data message to the adjacent downstream node based on the network address.

13. The method according to any one of claims 1 to 12, wherein: The method further includes: receiving a network data packet sent by the adjacent upstream node; and decapsulating the network data packet based on a network layer protocol to obtain the current data message.

14. The method according to any one of claims 1 to 13, wherein: When the current node completes the forwarding operation of the downstream data message, removing the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node includes: when receiving an operation completion response of the downstream data message from the adjacent downstream node, removing the superimposed node identification code of the adjacent downstream node from the superimposed node identification code of the current node.

15. The method according to claim 14, wherein: The method further comprises: After the superimposed node identification code of the current node is cleared, an operation completion response is returned to the adjacent upstream node.

16. The method according to claim 15, wherein: The method further includes: allocating local cache space of the current node; and releasing the local cache space after returning an operation completion response to the adjacent upstream node.

17. The method according to any one of claims 1 to 16, wherein: The method also includes: arranging the current data packet into a data packet sequence of the current node based on a message sequence number of the current data packet; determining the current data packet in the data packet sequence through a sliding window; and updating the superimposed node identification code of the current node based on an operation completion status of the current node, wherein the operation completion status indicates completion of a local operation and / or forwarding operation of the current node.

18. A network device comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, where the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1-17.

19. A server, comprising: host; The network device according to claim 18, wherein the network device is mounted to the host.

20. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

21. A computer program product, comprising a computer program / instruction, which implements the method according to any one of claims 1 to 17 when executed by a processor. 19

Images