A communication method and apparatus
By registering the same memory address in multiple communication nodes of a distributed system, the memory addressing problem in RDMA point-to-multipoint communication is solved, achieving efficient communication and load balancing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2021-12-27
- Publication Date
- 2026-08-04
AI Technical Summary
Existing technologies in point-to-multipoint communication based on RDMA suffer from memory addressing problems, which prevent efficient communication between multiple communication nodes.
Point-to-multipoint communication is achieved by registering the same memory address in multiple communication nodes of a distributed system and using the same memory address to access the MapReduce (MR) nodes in multiple communication nodes.
It improves communication efficiency, reduces the load on switching equipment, and achieves load balancing and efficient RDMA communication.
Smart Images
Figure CN116367092B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technology, and in particular to a communication method and apparatus. Background Technology
[0002] Remote direct memory access (RDMA) is a technology that bypasses the operating system kernel of a remote host to access data in its memory. When writing directly to the destination node's memory using RDMA, the destination node's network interface card (NIC) can directly write the received packet payload into the destination node's memory based on the received packet header information, without the destination node's operating system being aware of the process. Because RDMA memory writing does not require operating system intervention, it not only saves significant CPU resources but also avoids the CPU becoming a communication bottleneck, thereby improving system throughput and reducing network communication latency. RDMA communication technology is widely used in distributed computing clusters such as HPC and AI, as well as distributed storage systems.
[0003] In addition, point-to-multipoint communication is often used in distributed systems. For example, common point-to-multipoint communication methods include Anycast or Multicast.
[0004] With the widespread application of RDMA technology in distributed systems, how to efficiently perform point-to-multipoint communication based on RDMA is a problem that needs to be solved. Summary of the Invention
[0005] This application provides a communication method and apparatus that can efficiently perform point-to-multipoint communication based on RDMA.
[0006] Firstly, a communication method is provided, applied to a distributed system comprising multiple communication nodes. The method includes: each of the multiple communication nodes registering a first memory region (MR) according to a first memory address; and after determining that the multiple communication nodes have successfully registered their respective MRs, the first communication node determines that the first MR among the multiple communication nodes is permitted to perform Remote Direct Memory Access (RDMA) communication. This communication method, by registering MRs with the same memory address among the multiple communication nodes in the distributed system, enables the use of the same memory address to access the MRs in multiple communication nodes during RDMA-based point-to-multipoint communication, thereby achieving efficient RDMA-based point-to-multipoint communication.
[0007] In one possible design, the method further includes: a first communication node obtaining a first memory address; the first communication node sending a first registration instruction to multiple communication nodes respectively, indicating that a first memory region (MR) should be registered according to the first memory address; the first registration instruction carrying descriptive information of the first memory address; and multiple communication nodes registering the first memory region MR according to the first memory address, including: after obtaining the first registration instruction, the multiple communication nodes registering the first memory region MR according to the first memory address respectively. In the above design, by having the first communication node determine the first memory address first, and then having multiple communication nodes register MRs according to the first memory address, it is possible to register MRs with the same memory address among multiple communication nodes.
[0008] In one possible design, the method further includes: after determining that one of the multiple communication nodes has failed to register the first MR, the first communication node determines a second memory address; the first communication node sends a second registration instruction to each of the multiple communication nodes, instructing them to register the second MR according to the second memory address; the second registration instruction carries descriptive information of the second memory address; and the multiple communication nodes register the second MR according to the second memory address. In the above design, after one of the multiple communication nodes fails to register the first MR (e.g., the first memory address has already been occupied by a previously registered MR), the first communication node re-determines the second memory address and triggers the multiple communication nodes to register the second MR according to the second memory address, thereby ensuring the successful registration of the MR.
[0009] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0010] Secondly, a communication method is provided, comprising: a first communication node acquiring a first memory address; and after determining that multiple communication nodes have successfully registered a first memory region (MR) according to the first memory address, the first communication node determining that the first MR among the multiple communication nodes is allowed to perform remote direct memory access (RDMA) communication.
[0011] In one possible design, the method further includes: a first communication node sending a first registration instruction to multiple communication nodes respectively, for instructing the registration of a first memory address; the first registration instruction carries descriptive information of the first memory address.
[0012] In one possible design, the method further includes: after determining that one of the multiple communication nodes has failed to register the first MR, the first communication node determines a second memory address; the first communication node sends a second registration instruction to the multiple communication nodes respectively, indicating that the second MR is registered according to the second memory address; the second registration instruction carries description information of the second memory address.
[0013] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0014] Thirdly, a communication method is provided, comprising: a second communication node receiving a first registration instruction from a first communication node; the first registration instruction carrying descriptive information of a first memory address; and the second communication node registering a first memory region (MR) according to the first memory address. In one possible design, the first memory address includes: a first identifier for identifying the first MR and a virtual memory address of the first MR in memory.
[0015] Fourthly, a communication method is provided, comprising: a first communication node acquiring a first memory address; the first communication node sending a first message to a plurality of second communication nodes; the first message instructing the plurality of second communication nodes to write first data into a first memory address; the first message carrying first data and descriptive information of the first memory address of the first data.
[0016] In one possible design, the first message also includes a second identifier; the second identifier is used to indicate that the memory address of the first MR is shared by multiple communication nodes.
[0017] Fifthly, a communication method is provided, comprising: a first communication node acquiring a first memory address; the first memory address being a memory address shared by first data in multiple second communication nodes; the first communication node sending a second message to a third communication node among the multiple second communication nodes; the second message being used to instruct the reading of the first data in the first memory address and its return to the first communication node; the second message carrying descriptive information of the first memory address; wherein the second message can be executed by each of the multiple second communication nodes.
[0018] In a sixth aspect, a communication node is provided, comprising: an acquisition unit for acquiring a first memory address; and a processing unit for determining, after determining that multiple communication nodes have successfully registered a first memory region (MR) according to the first memory address, that the first MR among the multiple communication nodes is allowed to perform remote direct memory access (RDMA) communication.
[0019] In one possible design, the processing unit is further configured to send a first registration instruction to each of the plurality of communication nodes, indicating that the first MR is registered according to the first memory address; the first registration instruction carries description information of the first memory address.
[0020] In one possible design, the acquisition unit is further configured to determine a second memory address after the first communication node determines that one of the plurality of communication nodes has failed to successfully register the first MR; the processing unit is further configured to send a second registration instruction to the plurality of communication nodes respectively, indicating that the second MR is registered according to the second memory address; the second registration instruction carries description information of the second memory address.
[0021] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0022] A seventh aspect provides a communication node, comprising: an acquisition unit for receiving a first registration instruction from a first communication node; the first registration instruction carrying description information of the first memory address; and a registration unit for registering a first memory region MR according to the first memory address.
[0023] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0024] Eighthly, a communication node is provided, comprising: an acquisition unit for acquiring a first memory address; a sending unit for sending a first message to a plurality of second communication nodes; the first message being used to instruct the plurality of second communication nodes to write first data into a first memory address; the first message carrying the first data and descriptive information of the first memory address of the first data.
[0025] In one possible design, the first message also includes a second identifier; the second identifier is used to indicate that the memory address of the first MR is shared by multiple communication nodes.
[0026] A ninth aspect provides a communication node, comprising: an acquisition unit for acquiring a first memory address; the first memory address being a memory address shared by first data in a plurality of second communication nodes; and a sending unit for sending a second message to a third communication node among the plurality of second communication nodes; the second message being used to instruct the reading of the first data in the first memory address and its return to the first communication node; the second message carrying descriptive information of the first memory address; wherein the second message can be executed by each of the plurality of second communication nodes.
[0027] In a tenth aspect, a communication node is provided, comprising: a processor and an interface, wherein the processor receives or transmits data through the interface, and the processor is configured to implement a method as described in the second aspect or any of the designs in the second aspect, or to implement a method as described in the third aspect or any of the designs in the third aspect, or to implement a method as described in the fourth aspect or any of the designs in the fourth aspect, or to implement a method as described in the fifth aspect or any of the designs in the fifth aspect.
[0028] Eleventhly, a chip is provided, including a memory and a processor, the memory for storing computer instructions, and the processor for calling and executing the computer instructions from the memory to implement a method as described in the second aspect or any of the designs in the second aspect, or to implement a method as described in the third aspect or any of the designs in the third aspect, or to implement a method as described in the fourth aspect or any of the designs in the fourth aspect, or to implement a method as described in the fifth aspect or any of the designs in the fifth aspect.
[0029] In a twelfth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, which, when executed by a processor, implements a method as described in the second aspect or any of the designs in the second aspect, or implements a method as described in the third aspect or any of the designs in the third aspect, or implements a method as described in the fourth aspect or any of the designs in the fourth aspect, or implements a method as described in the fifth aspect or any of the designs in the fifth aspect.
[0030] In a thirteenth aspect, a computer program product is provided, the computer program product including instructions that, when executed on a processor, implement a method as designed in the second aspect or any of the second aspects, or implement a method as designed in the third aspect or any of the third aspects, or implement a method as designed in the fourth aspect or any of the fourth aspects, or implement a method as designed in the fifth aspect or any of the fifth aspects.
[0031] Fourteenth aspect, a distributed system is provided; the distributed system includes multiple communication nodes; the multiple communication nodes include a first communication node; the multiple communication nodes are used to perform a method as designed in the first aspect or any of the first aspect, or to perform a method as designed in the second aspect or any of the second aspect, or to perform a method as designed in the third aspect or any of the third aspect, or to perform a method as designed in the fourth aspect or any of the fourth aspect, or to perform a method as designed in the fifth aspect or any of the fifth aspect.
[0032] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0033] Figure 1A schematic diagram illustrating a point-to-multipoint communication method provided in an embodiment of this application;
[0034] Figure 2 This application provides a schematic diagram of the structure of a distributed system according to an embodiment of the present application.
[0035] Figure 3 One of the flowcharts of a communication method provided in this application embodiment;
[0036] Figure 4 A second schematic flowchart illustrating a communication method provided in an embodiment of this application;
[0037] Figure 5 This application provides a schematic diagram of a registration MR process.
[0038] Figure 6 A schematic diagram of an MR registration function provided in an embodiment of this application;
[0039] Figure 7 A third schematic flowchart illustrating a communication method provided in an embodiment of this application;
[0040] Figure 8 A schematic diagram of a data access function provided in an embodiment of this application;
[0041] Figure 9 A fourth schematic flowchart illustrating a communication method provided in an embodiment of this application;
[0042] Figure 10 This is one of the structural schematic diagrams of a communication node provided in the embodiments of this application;
[0043] Figure 11 This is a second schematic diagram of the structure of a communication node provided in an embodiment of this application;
[0044] Figure 12 This is the third schematic diagram of the structure of a communication node provided in the embodiments of this application;
[0045] Figure 13 This is the fourth schematic diagram of the structure of a communication node provided in the embodiments of this application;
[0046] Figure 14 This is the fifth schematic diagram of the structure of a communication node provided in the embodiments of this application. Detailed Implementation
[0047] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.
[0048] First, the relevant technologies involved in the embodiments of this application will be introduced:
[0049] A host channel adapter (HCA) is a network card that supports RDMA transmission.
[0050] Queue pair (QP): Before communication, the two ends of the RDMA protocol need to establish a send queue and a receive queue, which are collectively referred to as QP.
[0051] Memory region (MR): Before communication, the RDMA protocol needs to register the memory where the data to be communicated is located. The registered memory region is called MR.
[0052] Point-to-multipoint communication (PPC) refers to communication between an accessing node and multiple accessed nodes in a network. Common PPC modes include Anycast and Multicast. In Anycast mode, such as... Figure 1 As shown in (a), after the message sent by the accessing node reaches the switching device, the switching device selects one of the multiple accessed nodes (accessed node a, accessed node b, and accessed node c in the figure) according to a certain algorithm to forward the message. All of the accessed nodes can correctly process the message. Anycast mode can be used for on-network scheduling of switching devices to achieve load balancing among multiple nodes. In Multicast mode, as shown... Figure 1 As shown in (b), after the message sent by the accessing node reaches the switching device, the switching device replicates the message multiple times and sends them to multiple accessed nodes (accessed node d, accessed node e, and accessed node f in the figure). Multicast mode can be used for multi-replica replication in distributed systems or for multi-node data synchronization.
[0053] The architecture of the distributed system applying the technical solution provided in this embodiment will be described below with reference to the accompanying drawings. Figure 2 The diagram shown is a schematic representation of a distributed system provided in this embodiment. Specifically, the distributed system 10 can be a storage system, a high-performance computing (HPC) system, etc.
[0054] Specifically, the distributed system 10 includes multiple communication nodes (three communication nodes are shown exemplary in the figure: communication node 111, communication node 112, and communication node 113). In some application scenarios, each of the multiple communication nodes can be a single device (e.g., a single server), meaning the functionality of a communication node can be implemented by a single device. In other application scenarios, each of the multiple communication nodes can also include a device cluster consisting of multiple devices, where the functionality of the communication nodes can be implemented by running virtual machines on the device cluster. Each of the multiple communication nodes can provide computing resources or storage resources to the distributed system.
[0055] In addition, Figure 2 Multiple communication nodes in the system access the distributed system through switching device 121 in the switching layer. It is understood that in practical applications, the switching layer may include one or more switching devices. Figure 2 The example described herein uses switching device 121 as an example only. In practical applications, the switching layer may also include multiple switching devices, and the Academy of Sciences does not impose any restrictions on this in this embodiment.
[0056] In distributed system 10, each communication node can bypass the processors of other communication nodes based on RDMA technology, thereby accessing the memory of other communication nodes through the network interface card. For example, Figure 2 Communication node 11 can directly access memory area MR2 in memory 122 of communication node 12 via network card 121, or access memory area MR3 in memory 132 of communication node 13 via network card 131, based on RDMA technology. Similarly, communication node 12 can directly access memory area MR1 in memory 111 of communication node 11 via network card 111, or access memory area MR3 in memory 132 of communication node 13 via network card 131, based on RDMA technology.
[0057] In order to achieve direct memory writing to the network card, related technologies require that, before RDMA communication, the communication node to which the memory mapping node (MR) belongs needs to initiate a MR registration operation, registering at least a portion of the communication node's memory as the MR so that the network card can access it. Figure 2 Taking MR1 in communication node 11 as an example, the registration process of MR1 includes the following steps:
[0058] 1) In communication node 11, the relevant application requests memory space for communication, and the operating system allocates a virtual memory address (VA) for this space. Here, VA is the address carried by the remote node when accessing MR 1.
[0059] 2) The relevant application calls the MR registration function to initiate MR registration.
[0060] 3) Network card 111 pins the specified memory area, preventing that part of the memory from being swapped out to the disk.
[0061] 4) Network interface card 111 writes the mapping relationship between the physical address (PA) and the VA of the specified memory region into the memory translate table (MTT). Among them, PA is the address used by the memory management unit (MMU) when reading and writing memory space.
[0062] 5) The network interface card (NIC) generates L_key and R_key and returns them to the application, indicating successful MR registration. L_key is the local key for the MR, and R_key is the remote key for the MR. Multiple MRs registered on the same NIC may have different L_keys and R_keys.
[0063] After MR1 is successfully registered, before actual RDMA communication begins, communication node 11 will send the R_key and VA of MR1 to other communication nodes so that they can access it. During RDMA communication, the R_key and VA are filled into the message header by the sending end (e.g., communication node 12) and sent to communication node 11. The network card 111 of communication node 11 accesses MR1 based on the R_key and VA in the message header.
[0064] Furthermore, in related technologies, since the registration process of the MR used for RDMA communication in each communication node is relatively independent, the MRs in each communication node use different R_keys and VAs. For example, Figure 2 The R_keys and VAs used by MR1, MR2, and MR3 are all different. Therefore, memory addressing problems are encountered in point-to-multipoint communication scenarios based on RDMA. For example, when communication node 11 multicasts data to MR2 and MR3, because MR2 and MR3 use different R_keys and VAs, the message sent by communication node 11 can only be correctly parsed and processed by a single node, thus failing to achieve point-to-multipoint communication.
[0065] To address the aforementioned issues, a method for implementing reliable multicast in RDMA networks based on switching equipment has been proposed in related technologies. For example... Figure 3 As shown, the method includes: S201, the accessing node packages the memory addresses (specifically, VA and R_key) of multiple accessed nodes (accessed node 1 and accessed node 2 in the figure) into a control message Envelope, and sends it to the switching device before the data message. The Envelope message may also carry the network addresses of the multiple accessed nodes. S202, after receiving the Envelope message, the switching device records these memory addresses and network addresses. S203, upon the arrival of a data message, the switching device copies the data message multiple times and replaces the corresponding fields in the message header with the recorded memory addresses and network addresses, and sends them to the multiple accessed nodes respectively, so that each accessed node can correctly parse and process the received message.
[0066] The above Figure 3 While the related technologies described can achieve point-to-multipoint communication based on RDMA, each transmitted message requires sending a control message and updating the VA and R_key entries recorded by the switching device based on the control message. For example, with a bandwidth of 100Gbps, a 10KB message only requires a transmission delay of about 1µs. This means that for a 10KB message, the switching device needs to refresh the recorded VA and R_key every 1µs, which puts a lot of pressure on the hardware implementation of the switching device.
[0067] To address the aforementioned problems, this embodiment provides a communication method. This method involves registering MRs (Memory Providers) with the same memory address in multiple communication nodes of a distributed system. This allows for the use of the same memory address to access MRs in multiple communication nodes during RDMA-based point-to-multipoint communication. Figure 2 For example, when establishing MR1, MR2 and MR3, MR1, MR2 and MR3 can be made to use the same memory address, and then MR1, MR2 and MR3 can be accessed using that memory address, thereby achieving the effect of efficiently realizing point-to-multipoint communication based on RDMA.
[0068] The following is based on Figure 2 Taking the distributed system shown as an example, the communication method provided in this embodiment will be described in detail. Figure 4 As shown, the method may include:
[0069] S301. Select n communication nodes from the distributed system 10 to form a communication group. Where n is a positive integer greater than 1.
[0070] Specifically, a communication group can be formed by selecting n communication nodes from the distributed system 10 that participate in the same distributed computing / storage task through a preset method (e.g., selection by a technician or based on the current distributed computing / storage task). This allows each communication node in the communication group to register a MapReduce (MR) using the same memory address (hereinafter referred to as the first memory address). The following explanation uses a communication group consisting of communication nodes 11, 12, and 13 as an example.
[0071] In the process of selecting n communication nodes from distributed system 10, a communication node (hereinafter referred to as communication node 11) can be selected from the communication group as the master node. In one implementation, the communication node with the smallest IP address can be selected as the master node in the communication group. After the master node is determined, it can execute the process of selecting n communication nodes from distributed system 10.
[0072] In addition, to facilitate the distinction between the MRs that share the same memory address and other MRs mentioned above, the MR that shares the same memory address will be referred to as the first MR.
[0073] S302, Each communication node in the communication group negotiates with each other to determine the first memory address.
[0074] In one implementation, the first memory address may include a first identifier for identifying the first MR (hereinafter also referred to as Group_key, or G_key) and the virtual memory address of the first MR in memory (i.e. the address carried by the remote node when accessing the first MR, hereinafter also referred to as Group address, or GA).
[0075] The specific negotiation process for determining the first memory address among the communication nodes within the communication group can be implemented by the upper-layer application. This embodiment does not impose restrictions on the specific implementation of the negotiation process.
[0076] Specifically, after negotiating and determining the first memory address, the first memory address can be stored in the master node (such as communication node 11). The method then further includes:
[0077] S303, The master node sends the first registration instruction to each communication node in the communication group.
[0078] The first registration instruction is used to instruct the receiving communication node to register the first MR according to the first memory address. In this embodiment, the specific form of the first registration instruction is not limited. Specifically, the first registration instruction carries descriptive information about the first memory address. For example, the first memory address can be directly carried in the first registration instruction, meaning the descriptive information can be the first memory address itself; or, for another example, the first registration instruction can carry an algorithm for generating the first memory address, meaning the descriptive information can be the algorithm for generating the first memory address, so that each receiving communication node can generate the first memory address according to the algorithm.
[0079] It should be noted that the master node, as a node in the communication group, does not need to send the first registration instruction to itself. In this case, the master node sending the first registration instruction to each communication node in the communication group can be understood as including: the master node internally generating a registration instruction to instruct the registration of the first memory address (MR).
[0080] It is understandable that in some other implementations, each communication node in the communication group can directly register the MR according to the first memory address after negotiating and determining the first memory address, without needing to trigger other communication nodes to register the first MR by obtaining the first registration instruction. Therefore, when other communication nodes in the communication group directly register the first MR according to the first memory address after negotiating and determining the first memory address, the content of S303 may not be executed.
[0081] S304. Each communication node in the communication group registers its first memory address according to the first memory address.
[0082] The following example illustrates S304 using the process of communication node 12 registering the first MR after receiving the first registration instruction. Specifically, as follows... Figure 5 As shown, S304 may include:
[0083] S3041, Communication Node 12 runs the relevant application for establishing MR (hereinafter referred to as the first application).
[0084] S3042. The first application calls the MR registration function to send a registration request carrying the first memory address to the network card 121.
[0085] For example, the MR registration function interface is as follows: Figure 6The function `ibv_reg_group_mr()` is shown in the diagram. In the input fields, `pd` represents the protection domain; `addr` represents the PA of the memory region to be registered; `length` represents the length of the memory region to be registered (MR); and `access` represents the access permissions of the MR. These four items can be found in the MR registration function documentation in related technologies. Additionally, in `ibv_reg_group_mr()`, `G_addr` represents the virtual memory address (i.e., the GA in the first memory address) of the first MR included in the aforementioned first memory address; and `G_key` represents the first identifier of the first MR included in the aforementioned first memory address. The output is the MR registration result.
[0086] After receiving the registration request, S3043 and NIC121 check whether the first memory address has already been occupied.
[0087] For example, network card 121 can check whether the G_key and / or GA included in the first memory address has been occupied. If it is determined that the G_key and / or GA has been occupied, a registration failure is returned to the first application.
[0088] S3044. After confirming that the first memory address is not occupied, network card 121 registers the first MR.
[0089] Specifically, network card 121 pins the specified memory region, preventing that portion of memory from being swapped out to disk. It then writes the mapping relationship between the PA and GA of this memory region and the G_key into the MTT, and marks the first MR as groupMR in the MTT, indicating that the first MR shares a memory address with the memory region.
[0090] In this embodiment, through the above S301-S304 process, MRs with the same memory address can be registered in multiple communication nodes of the distributed system, so that when performing point-to-multipoint communication based on RDMA, the same memory address can be used to access the MRs in multiple communication nodes.
[0091] In one implementation, the method further includes:
[0092] S305. After each communication node in the communication group successfully registers the first MR, it sends a registration success command to the master node.
[0093] In this process, after the master node determines that each communication node in the communication group has successfully registered its first MR, the first MR in each communication node is allowed to perform RDMA communication.
[0094] It should be noted that the master node, as a node in the communication group, does not need to send a registration success command to itself. In this case, after each communication node in the group successfully registers its first Registered Message (MR), it sends a registration success command to the master node. This can be understood as the master node internally generating a registration success command to indicate the successful registration of the first MR.
[0095] In another implementation, the method also includes:
[0096] S306. After determining that a communication node in the communication group has failed to register the first MR, the communication nodes in the communication group renegotiate and determine the second memory address.
[0097] For example, if the first memory address conflicts with the memory address of an existing MR in a communication node, it is determined that the communication node has not successfully registered the first MR.
[0098] The process of renegotiation among the communication nodes in the communication group to determine the second memory address can be referred to in S302 above.
[0099] Furthermore, after the communication nodes within the communication group renegotiate and determine the second memory address, the first MR can be registered according to the second memory address using the content of S303-S304 described above. Repeated points will not be elaborated upon here.
[0100] The following section provides a detailed explanation of the process of accessing the first MR based on RDMA, using examples. Specifically, it will be illustrated with... Figure 2 Taking the writing of data from communication node 11 to the first MR in communication nodes 12 and 13 as an example, Figure 7 As shown, the method may further include:
[0101] S401, Communication Node 11 sends the first message to Communication Node 12 and Communication Node 13 respectively.
[0102] The first message instructs the receiving node to write the data to be written (hereinafter referred to as "first data") into the first memory location (MR) of the receiving node. Specifically, the first message may carry the memory address where the first data is to be written (which may include a virtual memory address and the identifier of the MR), or the first message may carry descriptive information about the memory address where the first data is to be written.
[0103] In one implementation, when communication nodes communicate through a switching device (i.e.) Figure 2 In the scenario shown, S401 may specifically include the following steps:
[0104] S4011, Communication Node 11 calls the function for accessing data (hereinafter referred to as the first function) to write a work queue element (WQE) to QP.
[0105] For example, the first function interface is as follows: Figure 8 As shown in ibv_post_send(). Here, wr.rdma.remote_addr represents the virtual memory address where the first data is about to be written (e.g., GA in the first memory address above); wr.rdma.rkey represents the identifier of the MR where the first data is about to be written (e.g., G_key in the first memory address above); wr.rdma.isgMR indicates that the MR where the first data is about to be written is a shared memory address among multiple communication nodes.
[0106] S4012, Communication node 11 sends one or more data packets carrying first data to switching device 14.
[0107] Among them, one or more of the aforementioned data packets can be understood as the first message.
[0108] In one implementation, in one or more data packets carrying first data, the first data packet carries the virtual memory address to which the first data is to be written and the identifier of the MR.
[0109] For example, the memory address to which the first data will be written and the identifier of the MR can be carried in the Remote Extended Transport Header (RETH) header of the first data packet. For example, as shown in Table 1 below, bytes 0-3 in the RETH header are used to represent the last 32 bits of the virtual memory address to which the first data will be written, bytes 4-7 are used to represent the first 32 bits of the virtual memory address to which the first data will be written, bytes 8-11 are used to represent the identifier of the MR to which the first data will be written, and bytes 12-15 are used to represent the length of the first data.
[0110] Table 1
[0111]
[0112] In another implementation, the first message also carries a preset identifier (hereinafter referred to as the "second identifier" for easy distinction). The second identifier is used to indicate that the MR to which the first data is about to be written is an MR with a shared memory address among multiple communication nodes.
[0113] For example, the second identifier can be carried in a reserved field (e.g., Reserve6) in the base transport header (BTH) of one or more data packets.
[0114] By carrying a second identifier in the first message, the MR with a shared memory address and the ordinary MR can be distinguished, so that the MR with a shared memory address and the ordinary MR can use the same virtual memory address.
[0115] S4013 After receiving a data packet carrying the first data, the switching device 14 forwards one or more data packets to the communication node 12 and the communication node 13 respectively according to the network addresses of the communication node 12 and the communication node 13.
[0116] In one implementation, before executing S4013, in order for the switching device 14 to know the network addresses of communication node 12 and communication node 13, the method may further include:
[0117] S4014, Communication device 11 sends a control message to switching device 14.
[0118] The control message carries the network addresses of communication node 12 and communication node 13. For example, the control message can be an Envelope message.
[0119] S4015 After receiving a data packet carrying the first data, the switching device 14 adds the recorded network addresses of communication node 12 and communication node 13 to one or more data packets.
[0120] Specifically, after adding the recorded network addresses of communication node 12 and communication node 13 to one or more data packets, the switching device 14 forwards one or more data packets to communication node 12 and communication node 13 respectively.
[0121] S402, after receiving the first message, communication node 12 and communication node 13 respectively write the first data into the first MR.
[0122] Taking communication node 12 as an example, S402 may specifically include:
[0123] S4021, Communication node 12 determines the virtual memory address where the first data is about to be written, the identifier of MR, and the second identifier based on the first message.
[0124] Specifically, communication node 12 can determine the virtual memory address to which the first data is about to be written and the identifier of the MR based on the RETH header of the first data packet in the first message; and determine the second identifier based on the BTH header of each data packet (that is, determine that the first MR to be written is the MR of the shared memory address of multiple communication nodes).
[0125] S4022, Communication Node 12 addresses and writes the first data according to the virtual memory address to which the first data is to be written, the identifier of MR, and the second identifier.
[0126] The process of communication node 13 writing the first data into the first MR can be referred to the contents of S4021-S4022 above, and will not be repeated here.
[0127] Compared to related technologies that use memory addresses of multiple accessed nodes to package into control packets and have the switching device replace the corresponding fields in the packet header with memory addresses to achieve reliable multicast in RDMA networks, the method in this embodiment offers several advantages. First, since the memory addresses of the first MR (Memory Address Register) are the same in multiple accessed nodes (i.e., communication nodes 12 and 13), the accessing node (i.e., communication node 11) can directly add the memory address of the first MR to the data packet, eliminating the need for the switching device to replace the corresponding fields in the packet header with memory addresses, thus reducing the load on the switching device. Furthermore, because the network address update frequency of communication nodes is very low, without needing to replace the corresponding fields in the packet header with memory addresses on the switching device side, the technical solution provided in this embodiment allows the switching device to update control packets (such as Envelope packets) at a lower frequency compared to the aforementioned related technologies, further reducing the load on the switching device.
[0128] Additionally, when communication node 11 reads the first data from the remote node (i.e., communication node 12 or communication node 13), such as Figure 9 As shown, the method may further include:
[0129] S501, Communication node 11 selects one of communication nodes, communication node 12 and communication node 13, to send the second message.
[0130] The second message is used to instruct the receiving node to send the first data back to the communication node 11. Specifically, the first message may carry the virtual memory address (or the identifier of the virtual memory address) of the first data in the first MR, the identifier of the first MR, and a second identifier of the MR used to indicate that the first MR is a shared memory address.
[0131] In one implementation, communication node 11 can select the node with the least load from communication nodes 12 and 13 to send the second message.
[0132] In scenarios where communication nodes communicate through switching devices (i.e.) Figure 2 In the scenario shown, S501 may specifically include the following steps:
[0133] S5011, Communication Node 11 calls the first function for accessing data to write WQE to QP.
[0134] Specifically, the first function can be an interface such as Figure 8 The ibv_post_send() function is shown in the image. For a detailed description of the ibv_post_send() function, please refer to the corresponding content in S4011 above; it will not be repeated here.
[0135] S5012, Communication node 11 sends a second message to switching device 14.
[0136] S5013 After receiving the second message, the switching device 14 forwards the second message to the communication node 12 or the communication node 13 according to the network address of the communication node 12 or the communication node 13.
[0137] In one possible design, the switching device 14 selects the node with the least load from communication nodes 12 and 13 to forward the second message based on the load status of communication nodes 12 and 13.
[0138] In one implementation, before executing S5013, in order for the switching device 14 to know the network addresses of communication node 12 and communication node 13, the method may further include:
[0139] S5014, Communication device 11 sends a control message to switching device 14.
[0140] The control message carries the network addresses of communication node 12 and communication node 13. For example, the control message can be an Envelope message.
[0141] S5015 After receiving a data packet carrying the first data, the switching device 14 adds the recorded network address of the communication node 12 or the communication node 13 to the second message.
[0142] Specifically, after adding the recorded network address of communication node 12 or communication node 13 to the second message, the switching device 14 forwards the second message to communication node 12 or communication node 13.
[0143] Taking the second message received by communication node 12 as an example, the method further includes:
[0144] S502, after receiving the second message, communication node 12 reads the first data and sends it back to communication node 11.
[0145] Specifically, the communication node 12 can address and write the first data based on the virtual memory address in the first MR, the identifier of the first MR, and the second identifier of the MR used to indicate that the first MR is a shared memory address.
[0146] The method described in this embodiment can achieve direct read of RDMA memory and can better achieve load balancing among multiple communication nodes.
[0147] like Figure 10 The diagram shown is a structural schematic of a communication node provided in this embodiment. The communication node includes an acquisition unit 601 and a processing unit 602. The communication node 60 can be used to implement the above-described... Figure 4 The functions of communication node 11. Specifically:
[0148] Acquisition unit 601 is used to acquire the first memory address;
[0149] The processing unit 602 is configured to determine, after determining that the first MR among the multiple communication nodes is allowed to perform remote direct memory access (RDMA) communication, after determining that the multiple communication nodes have successfully registered the first memory area (MR) according to the first memory address.
[0150] In one possible design, the processing unit 602 is further configured to send a first registration instruction to each of the plurality of communication nodes, indicating that the first MR is registered according to the first memory address; the first registration instruction carries description information of the first memory address.
[0151] In one possible design, the acquisition unit 601 is further configured to determine the second memory address after the first communication node determines that one of the plurality of communication nodes has failed to register the first MR.
[0152] The processing unit 602 is further configured to send a second registration instruction to the plurality of communication nodes respectively, for instructing the registration of the second MR according to the second memory address; the second registration instruction carries description information of the second memory address.
[0153] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0154] like Figure 11 The diagram shown illustrates the structure of another communication node provided in this embodiment. This communication node includes an acquisition unit 701 and a registration unit 702. The communication node 70 can be used to implement the above-described... Figure 4 The functions of communication node 12 or communication node 13. Specifically:
[0155] The acquisition unit 701 is used to receive a first registration instruction from the first communication node; the first registration instruction carries description information of the first memory address;
[0156] Registration unit 702 is used to register the first memory region MR according to the first memory address.
[0157] In one possible design, the first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
[0158] like Figure 12 The diagram shown illustrates the structure of another communication node provided in this embodiment. This communication node includes an acquisition unit 801 and a transmission unit 802. This communication node 80 can be used to implement the above-described... Figure 7 The functions of communication node 11. Specifically:
[0159] Acquisition unit 801 is used to acquire the first memory address;
[0160] The sending unit 802 is used to send a first message to a plurality of second communication nodes; the first message is used to instruct the plurality of second communication nodes to write first data into a first memory location (MR); the first message carries the first data and description information of the first memory address of the first data.
[0161] In one possible design, the first message also includes a second identifier; the second identifier is used to indicate that the memory address of the first MR is shared by multiple communication nodes.
[0162] like Figure 13 The diagram shown illustrates the structure of another communication node provided in this embodiment. The communication node 90 includes an acquisition unit 901 and a transmission unit 902. This communication node 90 can be used to implement the above-described... Figure 9 The functions of communication node 11. Specifically:
[0163] The acquisition unit 901 is used to acquire a first memory address; the first memory address is a memory address shared by the first data in multiple second communication nodes.
[0164] The sending unit 902 is used to send a second message to a third communication node among a plurality of second communication nodes; the second message is used to instruct the reading of first data from a first memory address and sending it back to the first communication node; the second message carries description information of the first memory address; wherein the second message can be executed by each of the plurality of second communication nodes.
[0165] It should be understood that the communication nodes 60, 70, 80, or 90 in the embodiments of this application can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figure 4 , Figure 5 , Figure 7 or Figure 9 In the communication method shown, the communication node and its various modules can also be software modules.
[0166] Figure 14 This is a schematic diagram of another communication node provided in this embodiment. The communication node 100 can be a chip or a system-on-a-chip. The communication node 100 may include some or all of the following components: a processor 1001, a communication line 1008, a memory 1003, and at least one communication interface 1002.
[0167] The processor 1001 is used to execute the communication method provided in this embodiment.
[0168] Specifically, processor 1001 may include a general-purpose central processing unit (CPU), and processor 1001 may also include a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), or an application-specific integrated circuit (ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0169] In a specific implementation, as one example, the processor 1001 may include one or more CPUs, for example... Figure 14 CPU0 and CPU1 in the CPU.
[0170] In a specific implementation, as one embodiment, device 100 may include multiple processors, for example... Figure 14 Processors 1001 and 1007 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process, for example, data (computer program instructions).
[0171] Additionally, memory 1003 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). Memory 1003 may exist independently and be connected to processor 1001 via communication line 1008. Memory 1003 may also be integrated with processor 1001.
[0172] The memory 1003 stores computer instructions. The processor 1001 can execute the communication method provided in this embodiment by executing the computer instructions stored in the memory 1003.
[0173] Optionally, the computer execution instructions in this embodiment may also be referred to as application code, and this embodiment does not specifically limit this.
[0174] In addition, the communication interface 1002 uses any transceiver-like device for communicating with other communication nodes, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.
[0175] In addition, communication line 1008 is used to connect the various components in communication device 100. Specifically, communication line 1008 may include data bus, power bus, control bus, and status signal bus, etc. However, for clarity, all buses are labeled as communication line 1008 in the figure.
[0176] In a specific implementation, as one embodiment, the communication device 100 may further include an output device 1005 and an input device 1006. The output device 1005 can communicate with the processor 1001 and can display information in various ways. For example, the output device 1005 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1006 can communicate with the processor 1001 and can receive user input in various ways. For example, the input device 1006 may be a mouse, keyboard, touchscreen device, or sensing device, etc.
[0177] Additionally, the communication device 100 may also include a storage medium 1004. The storage medium 1004 stores computer instructions and various data for implementing the technical solutions of this embodiment. This allows the communication device 100 to load the computer instructions and various data stored in the storage medium 1004 into the memory 1003 when executing the communication method described above in this embodiment, so that the processor 1001 can execute the communication method provided in this embodiment by executing the computer instructions stored in the memory 1003.
[0178] It should be understood that the communication device 100 according to this embodiment may correspond to the communication device 30 in this embodiment, and may correspond to the corresponding subject executing the communication method according to this embodiment, and the above and other operations and / or functions of each module in the communication device 100 are respectively for implementing Figure 4 , Figure 5 , Figure 7 or Figure 9 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.
[0179] As one possible embodiment, this application also provides a distributed system composed of multiple communication devices 100 described above. Specifically, the multiple communication devices 100 can be... Figure 2 This refers to multiple communication nodes or portions of hardware devices within multiple communication nodes. Furthermore, the structure of the distributed system provided in this embodiment can be as follows: Figure 1 As shown.
[0180] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a network device or terminal device. Of course, the processor and storage medium can also exist as discrete components in the network device or terminal device.
[0181] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this embodiment are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as an SSD.
[0182] In this embodiment, unless otherwise specified or there is a logical conflict, the terms and / or descriptions of different implementations are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationships.
[0183] In this embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. Other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Furthermore, for elements appearing in the singular forms "a," "an," and "the," unless explicitly specified by the context, it does not mean "one or only one," but rather "one or more than one." For example, "a device" means one or more such devices. Moreover, "at least one of..." means one or any combination of subsequent related objects. For example, "at least one of A, B, and C" includes A, B, C, AB, AC, BC, or ABC. In the textual description of this embodiment, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formula of this embodiment, the character " / " indicates a "division" relationship between the preceding and following related objects.
[0184] It is understood that the various numerical designations used in this embodiment are merely for descriptive convenience and are not intended to limit the scope of this embodiment. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.
Claims
1. A communication method characterized by comprising: The method is applied to a distributed system; the distributed system includes multiple communication nodes; the method includes: The plurality of communication nodes respectively register the first memory area MR according to the first memory address; After determining that the plurality of communication nodes have successfully registered the first memory region (MR), the first communication node determines that the first MR among the plurality of communication nodes is allowed to perform remote direct memory access (RDMA) communication; wherein, the first communication node is any one of the plurality of communication nodes.
2. The method of claim 1, wherein, The method further includes: The first communication node obtains the first memory address; The first communication node sends a first registration instruction to each of the plurality of communication nodes, indicating that the first MR should be registered according to the first memory address; the first registration instruction carries description information of the first memory address; The plurality of communication nodes respectively register a first memory region (MR) according to a first memory address, including: After receiving the first registration instruction, the multiple communication nodes register the first memory area (MR) according to the first memory address.
3. The method according to claim 1 or 2, characterized in that, The method further includes: After determining that one of the plurality of communication nodes has failed to register the first MR, the first communication node determines the second memory address; The first communication node sends a second registration instruction to the plurality of communication nodes respectively, indicating that the second MR is registered according to the second memory address; the second registration instruction carries description information of the second memory address; The multiple communication nodes each register a second MR according to the second memory address.
4. The method according to any one of claims 1 to 3, characterized in that, The first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
5. A communication method characterized by comprising: include: The first communication node obtains the first memory address; After determining that multiple communication nodes have successfully registered the first memory region (MR) according to the first memory address, the first communication node determines that the first MR among the multiple communication nodes is allowed to perform remote direct memory access (RDMA) communication; wherein, the first communication node is any one of the multiple communication nodes.
6. The method of claim 5, wherein, The method further includes: The first communication node sends a first registration instruction to each of the plurality of communication nodes, indicating that the first MR is registered according to the first memory address; the first registration instruction carries description information of the first memory address.
7. The method according to claim 5 or 6, characterized in that, The method further includes: After determining that one of the plurality of communication nodes has failed to register the first MR, the first communication node determines the second memory address; The first communication node sends a second registration instruction to each of the plurality of communication nodes, indicating that the second MR is registered according to the second memory address; the second registration instruction carries description information of the second memory address.
8. The method according to any one of claims 5-7, characterized in that, The first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
9. A communication method characterized by comprising: include: The second communication node receives the first registration instruction from the first communication node; The first registration instruction carries a description of the first memory address; The second communication node registers the first memory region (MR) according to the first memory address; wherein, the first communication node and the second communication node are any two communication nodes among a plurality of communication nodes included in the distributed system, and the plurality of communication nodes respectively register the first MR according to the first memory address.
10. The method of claim 9, wherein, The first memory address includes: a first identifier for identifying the first MR and the virtual memory address of the first MR in memory.
11. A communication method, comprising: The method includes: The first communication node obtains the first memory address; The first communication node sends a first message to a plurality of second communication nodes; the first message is used to instruct the plurality of second communication nodes to write first data into a first memory location (MR); the first message carries the first data and a description of the first memory address of the first data; wherein, the first MR is registered by the first communication node and the plurality of second communication nodes respectively according to the first memory address.
12. The method of claim 11, wherein, The first message also includes a second identifier; the second identifier is used to indicate that the memory address of the first MR is shared by multiple communication nodes.
13. A method of communication, comprising: The method includes: The first communication node obtains the first memory address; the first memory address is the memory address shared by the first data in multiple second communication nodes; The first communication node sends a second message to a third communication node among the plurality of second communication nodes; the second message is used to instruct the reading of first data from a first memory address and sending it back to the first communication node; the second message carries description information of the first memory address; wherein, the second message can be executed by each of the plurality of second communication nodes; the first communication node and the plurality of second communication nodes respectively register a first memory region MR according to the first memory address.
14. A communication node, characterized in that, include: The acquisition unit is used to acquire the first memory address; The processing unit is configured to, after determining that multiple communication nodes have successfully registered the first memory region (MR) according to the first memory address, determine that the first MR among the multiple communication nodes is allowed to perform remote direct memory access (RDMA) communication; wherein, the communication node is any one of the multiple communication nodes.
15. A communication node, characterized by include: The acquisition unit is used to receive a first registration instruction from the first communication node; The first registration instruction carries a description of the first memory address; A registration unit is configured to register a first memory region (MR) according to the first memory address, wherein the communication node and the first communication node are any two communication nodes among a plurality of communication nodes included in the distributed system, and the communication node and the plurality of communication nodes respectively register the first MR according to the first memory address.
16. A communication node, characterized by include: The acquisition unit is used to acquire the first memory address; The sending unit is used to send a first message to multiple second communication nodes; The first message is used to instruct the plurality of second communication nodes to write the first data into the first MR; The first message carries the first data and a description of the first memory address of the first data, wherein the first MR is registered by the communication node and the plurality of second communication nodes respectively according to the first memory address.
17. A communication node, characterized by include: The acquisition unit is used to acquire the first memory address; The first memory address is the memory address shared by the first data in multiple second communication nodes; A sending unit is configured to send a second message to a third communication node among a plurality of second communication nodes; the second message is configured to instruct the reading of first data from a first memory address and sending it back to the first communication node; the second message carries description information of the first memory address; wherein the second message can be executed by each of the plurality of second communication nodes; the communication node and the plurality of second communication nodes respectively register a first memory region MR according to the first memory address.
18. A communication node, characterized by include: A processor and an interface, wherein the processor receives or transmits data through the interface, the processor is configured to implement the method as described in any one of claims 5-8, or the processor is configured to implement the method as described in claim 9 or 10, or the processor is configured to implement the method as described in claim 11 or 12, or the processor is configured to implement the method as described in claim 13.
19. A chip, characterized by The device includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to call and execute the computer instructions from the memory to implement the method as described in any one of claims 5-8, or to implement the method as described in claim 9 or 10, or to implement the method as described in claim 11 or 12, or the processor being used to implement the method as described in claim 13.
20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 5-8, or the method as described in claim 9 or 10, or the method as described in claim 11 or 12, or the processor is used to implement the method as described in claim 13.
21. A computer program product, characterised in that, The computer program product includes instructions that, when executed on a processor, implement the method as claimed in any one of claims 5-8, or implement the method as claimed in claim 9 or 10, or implement the method as claimed in claim 11 or 12, or the processor is configured to implement the method as claimed in claim 13.
22. A distributed system, characterized by It includes multiple communication nodes; the multiple communication nodes are used to perform the method as described in any one of claims 1-13.