Distributed system, distributed token processing method and apparatus

Abstract

The present disclosure relates to a distributed system, a distributed token processing method and device, and relates to the technical field of distribution. The method comprises: for the intelligent network card in the single proposer node of the distributed system, in response to receiving the application command sent by the CPU of the proposer, generating an ID request; storing the ID request in the cache queue of the proposer node; in response to the intelligent network card of the proposer node receiving the distributed token from the previous proposer node, adjusting the distributed token according to the ID request stored in the cache queue; and sending the adjusted distributed token to the next proposer node. The present disclosure can support multiple proposer nodes in the distributed system, and does not require an additional independent system to maintain the ID increment service of the distributed token, thereby saving server resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of distributed technology, and in particular to a distributed system, a distributed token processing method, and an apparatus. Background Technology

[0002] Distributed consensus is the core of distributed systems. The core idea of ​​distributed consensus algorithms is: the proposer submits a proposal within the cluster, the acceptors vote on the proposal, and if the majority of the parties reach a consensus on the same proposal, then the proposal becomes a consensus.

[0003] In distributed consensus algorithms, proposals require a monotonically increasing proposal number. Current engineering solutions include: supporting only one proposer and supporting multiple proposers. In the single-proposer solution, the proposer itself can maintain the monotonically increasing proposal number. In the multi-proposer solution, a separate system is often needed to provide a monotonically increasing ID service, with each proposer accessing the ID service over the network. This separate system typically uses a dual-machine primary / backup configuration, with the primary machine providing the service and the backup machine providing backup, resulting in high resource consumption. Summary of the Invention

[0004] Embodiments of this disclosure provide a distributed system, a distributed token processing method, and an apparatus.

[0005] In a first aspect, embodiments of this disclosure provide a distributed system, comprising: multiple proposer nodes, each proposer node being equipped with a smart network interface card (NIC); each smart NIC including a transmitter and a receiver; for a single proposer node, the transmitter of the smart NIC of that proposer node is connected to the receiver of the smart NIC of the next proposer node, forming a unidirectional physical ring; the smart NICs of each proposer node are connected via network communication according to the connection order of each proposer node in the unidirectional physical ring, forming a unidirectional logical ring.

[0006] Secondly, embodiments of this disclosure provide a distributed token processing method applied to a distributed system as described in the first aspect. The method includes: for a smart interface card in a single proposer node of the distributed system, in response to receiving an application command sent by the CPU of the proposer node, generating an ID request; storing the ID request in a cache queue of the proposer node; in response to the smart interface card of the proposer node receiving a distributed token from a previous proposer node, adjusting the distributed token according to the ID request stored in the cache queue; and sending the adjusted distributed token to a subsequent proposer node.

[0007] Thirdly, embodiments of this disclosure provide a distributed token processing apparatus disposed in a proposer node of a distributed system as described in the first aspect. The apparatus includes: a request generation unit configured to generate an ID request for a smart network interface card (NIC) in a single proposer of the distributed system in response to receiving a request command sent by the proposer's CPU; a request caching unit configured to store the ID request in a cache queue of the proposer node; a token adjustment unit configured to adjust the distributed token according to the ID request stored in the cache queue in response to the smart NIC of the proposer node receiving a distributed token from a previous proposer node; and a token transmission unit configured to send the adjusted distributed token to a subsequent proposer node.

[0008] By applying the technical solution disclosed herein, multiple proposer nodes can be supported in a distributed system without the need for an additional independent system to maintain the distributed token's auto-incrementing ID service, thus saving server resources.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram of the structure of one embodiment of the distributed system disclosed herein;

[0012] Figure 2 This is a schematic diagram of the physical ring of the distributed system disclosed herein;

[0013] Figure 3 This is a schematic diagram of the logical loop of the distributed system disclosed herein;

[0014] Figure 4 This is a flowchart illustrating one embodiment of the distributed token processing method disclosed herein;

[0015] Figure 5 This is a flowchart illustrating another embodiment of the distributed token processing method disclosed herein;

[0016] Figure 6 A delay diagram for the existing independent auto-incrementing token processing method;

[0017] Figure 7 This is a schematic diagram illustrating the delay of the distributed token processing method disclosed herein;

[0018] Figure 8This is a schematic diagram of one embodiment of the distributed token processing device disclosed herein. Detailed Implementation

[0019] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0021] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0022] To make the technical solutions and advantages of this disclosure clearer, the following description, in conjunction with the accompanying drawings and specific embodiments, will provide a more detailed account of this disclosure.

[0023] Figure 1 A schematic diagram of the structure of one embodiment of the distributed system of this disclosure is shown. Figure 1 As shown, the distributed system 100 in this embodiment may include multiple proposer nodes, which form a ring communication structure. Specifically, each proposer node may be equipped with a smart network interface card (NIC). Each smart NIC may include a transmitter and a receiver. For a single proposer node, the transmitter of its smart NIC is connected to the receiver of the smart NIC of the next proposer node. It is understood that this connection uses a transmission medium to connect the transmitter and receiver of two adjacent proposer nodes, resulting in a unidirectional physical ring. Furthermore, the smart NICs of each proposer node can be connected via network communication to form a unidirectional logical ring. It should be noted that the connection order of the proposer nodes in the unidirectional logical ring is the same as the connection order of the proposer nodes in the unidirectional physical ring.

[0024] In some optional implementations of this embodiment, in the above distributed system, the transmitting and receiving ends of the smart network cards of two adjacent proposer nodes are connected by optical fiber.

[0025] See Figure 2When establishing a physical ring in a distributed system, the sender of each proposer node can connect to the receiver of the next proposer node, forming a unidirectional physical ring. It should be noted that when using a physical ring, the token protocol message does not need to consider the Layer 2 or Layer 3 address of the proposer node. Thus, when sending a token message, it can be directly sent from the sender to the receiver of the next proposer node.

[0026] See Figure 3 When establishing a logical ring, smart network interface cards (NICs) are not limited to optical ports; they can physically connect to the network via Layer 2 or Layer 3. When using a logical ring, each smart NIC needs to know the information of the subsequent proposer nodes within the ring, specifically their Layer 2 or Layer 3 addresses. Token protocol messages only need to focus on the Layer 2 or Layer 3 addresses; each time a token protocol message is sent, the source or destination address of the message is modified according to the Layer 2 or Layer 3 address.

[0027] It should be noted that, whether it is a physical ring or a logical ring, each proposer node within the ring is generally bound to a unique identifier, usually represented by a number.

[0028] Figure 4 The flowchart 400 illustrates an embodiment of the distributed token processing method of this disclosure. The distributed token processing method of this disclosure can be applied to... Figure 1 In the distributed system shown, furthermore, the distributed token processing method in this embodiment can be applied to the smart network interface card (NIC) of the proposer node in the distributed system, that is, the execution subject of this method is the smart NIC. For example... Figure 4 As shown, the distributed token processing method in this embodiment may include the following steps:

[0029] Step 401: For the smart network interface card in a single proposer node of the distributed system, in response to receiving the request command sent by the CPU of the proposer node, an ID request is generated.

[0030] In this embodiment, the distributed system may include multiple proposer nodes, each capable of making proposals. A token is generated for each proposal. For the smart network interface card (NIC), if it receives a request command from the CPU of the proposer node, it generates an ID request. Here, the request command can be a PCIe command used to request an ID. This embodiment does not limit the format of the PCIe command, which may include ID request information, such as a request sequence number and a write-back address. The request sequence number is a numeric value maintained by the proposer node for matching response messages. The write-back address can be a memory address on the CPU side, used to store the distributed token. When the smart NIC receives the above request command, it can extract the token request information and generate an ID request based on it.

[0031] Step 402: Store the ID request in the cache queue of the proposer node.

[0032] In this embodiment, after generating an ID request, the smart network interface card (NIC) can store the ID request in the cache queue of the proposer node. Specifically, the smart NIC can cache each ID request in a local FIFO (First Input First Out) queue.

[0033] Step 403: In response to the smart card of the proposer node receiving the distributed token from the previous proposer node, the distributed token is adjusted according to the ID request stored in the cache queue.

[0034] If the proposer node's smart card receives the distributed token from the previous proposer node, it can adjust the distributed token based on the ID requests stored in the cache queue. Specifically, it can increase the value carried by the distributed token by the aforementioned amount based on the number of ID requests stored in the cache queue. Alternatively, it can retrieve the earliest generated ID request from the cache queue and increase the value carried by the distributed token by a preset value. Or, it can retrieve ID requests generated outside of a preset time period and adjust the value carried by the distributed token based on the number of retrieved ID requests.

[0035] Step 404: Send the adjusted distributed token to the next proposer node.

[0036] After adjusting the distributed token, the smart network interface card (NIC) can send the adjusted distributed token to the next proposing node. This enables the transmission of the distributed token within the distributed system.

[0037] The distributed token processing method provided by the above embodiments of this disclosure can support multiple proposer nodes in a distributed system and does not require an additional independent system to maintain the distributed token's auto-incrementing ID service, thus saving server resources.

[0038] See also Figure 5 This illustrates a flow 500 of another embodiment of the distributed token processing method according to this disclosure. Figure 5 As shown, the method in this embodiment may include the following steps:

[0039] Step 501: For the smart network interface card in a single proposer node of the distributed system, in response to receiving the request command sent by the CPU of the proposer node, an ID request is generated.

[0040] Understandably, a smart network interface card (NIC) generates an ID request for each request it receives.

[0041] Step 502: Store the ID requests sequentially in the cache queue according to their generation time.

[0042] In this embodiment, the generated ID requests can be stored sequentially in a cache queue according to their generation time. This way, when retrieving them, they can be retrieved based on their generation time.

[0043] Step 503: In response to the smart card of the proposer node receiving the distributed token from the previous proposer node, the earliest generated ID request is retrieved from the cache queue, and the value carried by the distributed token is increased by a preset value.

[0044] In this embodiment, if the smart card of the proposer node receives a distributed token from the previous proposer node, it can retrieve the ID request from the cache queue and then increment the value carried by the distributed token by a preset value. In some specific practices, the value of the distributed token can be incremented by 1. It is understood that if the cache queue is empty, it means that the current proposer node has no ID request, and the distributed token can be directly sent to the next proposer node.

[0045] In some specific practices, the cache queue is a FIFO queue. If the FIFO queue is not empty, it means that the current proposer node has an ID request, and the value carried by the distributed token, namely TOKEN_ID, can be read. One request is taken from the FIFO queue, the value carried by the distributed token is modified to (TOKEN_ID+1), and the adjusted distributed token is sent to the next proposer node.

[0046] Step 504: In response to the smart card of the proposer node receiving the distributed token from the previous proposer node, the adjusted value of the distributed token is determined based on the number of ID requests stored in the cache queue and the value carried by the distributed token.

[0047] In this embodiment, the distributed token can also be adjusted via step 504. Specifically, the adjusted value of the distributed token can be determined based on the number of ID requests stored in the cache queue and the value of the distributed token. For example, if there are N ID requests in the cache queue, all N ID requests can be retrieved, and the value carried by the distributed token can be adjusted according to the value of N.

[0048] Step 505: Send the adjusted distributed token to the next proposer node.

[0049] Step 506: Periodically check the status of the distributed token; if the status of the distributed token is found to be abnormal, regenerate the distributed token.

[0050] In this embodiment, the status of the distributed token can also be checked periodically. Specifically, this can be achieved by setting a scheduled task. The status of the distributed token can be normal or abnormal. An abnormal status can manifest as loss. If the status of the distributed token is determined to be abnormal, it can be regenerated. Specifically, the value of a distributed token can be obtained from an external source as the regenerated distributed token.

[0051] In some optional implementations of this embodiment, each proposer node in the distributed system can be controlled to generate an initial distributed token, and the generated initial distributed token and the corresponding identifiers of each proposer node can be transmitted within the physical ring; each proposer node in the distributed system can select a specified token from the received distributed tokens.

[0052] In this implementation, when regenerating the distributed token, each proposer node in the distributed system can be controlled to generate an initial distributed token. Then, the generated initial distributed token and the corresponding identifiers of each node are transmitted within the physical ring. During transmission, each proposer node in the distributed system can select a specific token from the received distributed tokens. It is understood that the distributed tokens selected by each proposer node should be consistent.

[0053] In some optional implementations of this embodiment, a proposer node can be designated as the master proposer node from the distributed system. The distributed token generated by the master proposer node is then used as the regenerated distributed token.

[0054] Step 507: Write the adjusted distributed token into the CPU-side memory according to the write-back address.

[0055] The request command sent by the CPU to the smart network interface card (NIC) may include a write-back address, which represents the address in memory used to store the distributed token. After adjusting the distributed token, the smart NIC can write the adjusted distributed token into the CPU's memory based on the write-back address.

[0056] Step 508: In response to determining that the writing of the distributed token is complete, initiate an interrupt transaction and notify the CPU; or in response to receiving a polling request from the CPU, check whether the distributed token has been received according to the write-back address in the polling request. If the distributed token has been received, send the distributed token to the CPU.

[0057] In this embodiment, after writing the distributed token to the write-back address, the smart network interface card (NIC) can initiate an interrupt transaction to notify the CPU. The CPU can then read the distributed token based on the write-back address. Alternatively, the smart NIC can choose not to initiate an interrupt transaction and wait for CPU polling. That is, upon receiving a CPU polling request, it determines the write-back address in the request and checks whether a distributed token has been received based on that address. If a distributed token is received, it sends it to the CPU.

[0058] See also Figure 6 and Figure 7 This analysis examines the latency generated by existing distributed token processing methods and the distributed token processing method disclosed herein. Figure 6 This refers to an independent, auto-incrementing distributed token value for each existing individual proposer. Since CPU processing speed is high, latency caused by CPU processing is not considered. For example... Figure 6 As shown, the proposer's PCIe command is sent to the smart network interface card (NIC) (arrow 1). The proposer's NIC then transmits the command over the network to the ID service host's NIC (arrow 2). The ID service host's NIC interacts with the ID service host's CPU (including writing and responding to the distributed token value) (arrows 3 and 4). The ID service host's NIC then sends backup information over the network to the ID service standby NIC (arrow 5). The standby NIC interacts with the standby CPU (including writing and responding to the distributed token value) (arrows 6 and 7). The standby NIC then interacts with the ID service host's NIC over the network (arrow 8). The ID service host's NIC interacts with the ID service host's CPU again (arrows 9 and 10). Finally, the ID service host's NIC interacts with the proposer's NIC over the network (arrow 11), and then with the proposer's CPU (arrow 12). Thus, the entire interaction process includes 8 PCIe transactions and 4 network transmissions.

[0059] and Figure 7 In this process, Proposer 1's CPU sends a request command to the smart network interface card (NIC) (arrow 1). Then, Proposer 1's NIC sends the distributed token to Proposer 2's NIC via the network (arrow 2). Proposer 2's NIC then sends the distributed token to Proposer 3's NIC via the network (arrow 3). Proposer 3's NIC then sends the distributed token to Proposer 1's NIC via the network (arrow 4). Finally, Proposer 1's NIC interacts with the CPU (arrow 5). In total, the entire process involves two PCIe transactions and four network transmissions.

[0060] Therefore, it can be seen that the distributed token processing method disclosed herein can effectively reduce latency in the distributed token processing process. Simultaneously, by transferring control of the distributed token within the ring through the token mechanism, distributed data contention is avoided. Furthermore, since only the proposer has PCIe operations, end-to-end latency can be reduced. Moreover, by controlling the number of token link nodes in this disclosure, the number of network transmissions can be effectively reduced, thereby further reducing end-to-end latency.

[0061] Further reference Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a distributed token processing device, which is similar to... Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to the smart network interface card of the proposer node in a distributed system.

[0062] like Figure 8 As shown, the distributed token processing device 800 of this embodiment includes: a request generation unit 801, a request caching unit 802, a token adjustment unit 803, and a token transmission unit 804.

[0063] The request generation unit 801 is configured to generate an ID request for a smart NIC in a single proposer of a distributed system in response to receiving a request command sent by the proposer's CPU.

[0064] Request caching unit 802 is configured to store ID requests in the cache queue of the proposer node.

[0065] The token adjustment unit 803 is configured to adjust the distributed token in response to the smart card of the proposer node receiving the distributed token from the previous proposer node, based on the ID request stored in the cache queue.

[0066] The token delivery unit 804 is configured to send the adjusted distributed token to the next proposer node.

[0067] In summary, the technical solution disclosed herein can support multiple proposers in a distributed system without requiring an additional independent system to maintain distributed tokens, thus saving server resources.

[0068] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distributed token processing method applied to a distributed system, the distributed system including multiple proposer nodes, each proposer node being equipped with a smart network interface card (NIC), each smart NIC including a sender and a receiver, for a single proposer node, the sender of the smart NIC of that proposer node is connected to the receiver of the smart NIC of the next proposer node, forming a unidirectional physical ring, the smart NICs of each proposer node being connected via network communication according to the connection order of each proposer node in the unidirectional physical ring, forming a unidirectional logical ring, the method comprising: For the smart NIC in a single proposer node of the distributed system, an ID request is generated in response to receiving an application command sent by the CPU of the proposer node; The ID request is stored in the cache queue of the proposer node; The smart card of the proposer node receives the distributed token from the previous proposer node and adjusts the distributed token according to the ID request stored in the cache queue; The adjusted distributed token is sent to the next proposer node.

2. The method according to claim 1, wherein, The step of storing the ID request in the proposer node's cache queue includes: The ID requests are stored sequentially in the cache queue according to their generation time.

3. The method according to claim 2, wherein, The step of adjusting the distributed token based on the ID requests stored in the cache queue includes: The earliest generated ID request is retrieved from the cache queue, and the value carried by the distributed token is increased by a preset value.

4. The method according to claim 1, wherein, The step of adjusting the distributed token based on the ID requests stored in the cache queue includes: The adjusted value of the distributed token is determined based on the number of ID requests stored in the cache queue and the value carried by the distributed token.

5. The method according to claim 1, wherein, The method further includes: The status of the distributed token is checked periodically; If the distributed token is found to be in an abnormal state, the distributed token is regenerated.

6. The method according to claim 5, wherein, The regeneration of the distributed token includes: Control each proposer node in the distributed system to generate an initial distributed token, and transmit the generated initial distributed token and the corresponding identifier of each proposer node within the physical ring. Control each proposer node in the distributed system to select a specified distributed token from the received distributed tokens.

7. The method according to claim 5, wherein, The regeneration of the distributed token includes: The main proposer node is determined from the distributed system; The main proposer node will regenerate the distributed token.

8. The method of claim 1, wherein the application command includes a request sequence number and a write-back address; and The method further includes: Based on the write-back address, the adjusted distributed token is written into the CPU-side memory.

9. The method according to claim 8, wherein, The method further includes: In response to determining that the writing of the distributed token is complete, an interrupt transaction is initiated, and the CPU is notified; or In response to receiving a polling request from the CPU, check whether a distributed token has been received based on the write-back address in the polling request; if the distributed token is received, send the distributed token to the CPU.

10. A distributed token processing device, disposed in a proposer node of a distributed system, the distributed system including multiple proposer nodes, each proposer node being equipped with a smart network interface card (NIC), each smart NIC including a transmitter and a receiver, wherein for a single proposer node, the transmitter of the smart NIC of that proposer node is connected to the receiver of the smart NIC of the next proposer node, forming a unidirectional physical ring, and the smart NICs of each proposer node are connected via network communication according to the connection order of each proposer node in the unidirectional physical ring, forming a unidirectional logical ring, the device comprising: The request generation unit is configured to generate an ID request for a smart network interface card in a single proposer of the distributed system in response to receiving a request command sent by the proposer's CPU. The request caching unit is configured to store the ID request in the cache queue of the proposer node; The token adjustment unit is configured to adjust the distributed token in response to the smart card of the proposer node receiving the distributed token from the previous proposer node, based on the ID request stored in the cache queue. The token delivery unit is configured to send the adjusted distributed token to the next proposer node.

Images