A multi-shard method and device based on network P2P communication and random broadcast

Abstract

The application discloses a kind of based on network P2P communication and random broadcast's multi-slice method and device, this method is through on the basis of Kademlia and Gossip protocol, propose a through to P2P network is sliced, prevent message from other slice network nodes when spreading, and through the way of multi-path random selection neighborhood node and the amount of configurable redundant information, construct the P2P network with higher stability, lower latency, higher transmission efficiency.To solve the problems of wasting network bandwidth and nodes in the related art that the conventional Gossip protocol usually needs to transmit a large amount of redundant information when transmitting information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sharding network P2P communication, and particularly relates to a multi-sharding method and device based on network P2P communication and random broadcast. BACKGROUND

[0002] P2P network (Peer-to-peer networking) is a decentralized network structure, which plays an important role in the information transmission of distributed systems. It allows nodes distributed in different geographical locations to share information without the participation of a centralized server. P2P network is widely used in various distributed scenarios including blockchain due to its decentralization, high scalability, high robustness, and high privacy.

[0003] Common techniques for building P2P networks include Kademlia, Chord protocol, SWIM (Scalable Weakly-consistent Infection-style Membership Protocol), and Gossip protocol: Kademlia is a DHT-based P2P routing protocol that allows nodes to quickly find and locate other nodes through a distributed hash table, thereby improving routing efficiency and network stability. Kademlia is widely used to build P2P networks such as BitTorrent. Chord is another DHT protocol for P2P networks. It provides more reliable data lookup and distributed storage, and achieves distributed data management through a consistent hash table. SWIM is a protocol for member detection and node management, which aims to improve the stability and scalability of P2P networks. SWIM reduces noise in the network by reducing error state information in the network. The Gossip protocol randomly selects a number of neighbor nodes and propagates messages to these nodes, which in turn propagate to their neighbors, and so on. This way ensures that messages eventually propagate throughout the network.

[0004] Traditional Gossip protocols have some performance and efficiency limitations in certain situations. These limitations include but are not limited to: First, regarding network topology instability: Gossip protocols usually randomly select adjacent nodes between nodes for information transmission, which may cause delayed transmission or loss of information when the network topology is unstable or there are many node failures. Second, regarding high latency: Due to the random nature of the Gossip protocol, some information may take a long time to propagate throughout the network, resulting in high latency. Finally, regarding resource waste: Traditional Gossip protocols usually need to transmit a large amount of redundant information during information transmission, thereby wasting network bandwidth and nodes. SUMMARY

[0005] An embodiment of the present application aims to provide a multi-shard method and device based on network P2P communication and random broadcast, to solve the technical problem of wasting network bandwidth and nodes in the prior art, as the traditional Gossip protocol usually needs to transmit a large amount of redundant information when transmitting information.

[0006] According to a first aspect of an embodiment of the present application, a multi-shard method based on network P2P communication and random broadcast is provided, comprising the following steps:

[0007] (1) Network shard: the first 32 bits of the 256-bit address space are taken as the shard ID of the network space, and the last 224 bits are obtained by Hash256 truncation on the node public key, as the address of the network node in a specific network shard;

[0008] (2) Network initialization: the node is initialized to join the P2P network, and a random value is independently generated, which is transmitted to all neighbor nodes through the heartbeat after being added to the network;

[0009] (3) Message source node: according to the range of the node ID, i.e. 0~MaxInt, K neighbor nodes are randomly selected;

[0010] (4) Send the message and the range K_i distributed by the current node to the K nodes, and add the node and the sent node random value to the Bloom filter;

[0011] (5) Each node receiving the message calculates a new range according to the range K_i of the last node,

[0012] (6) From the calculated range, K nodes are randomly selected, and if K is less than K, all nodes are selected, and the Bloom filter is used to filter out the nodes that have been sent, and the message is continuously broadcasted;

[0013] (7) Continue the hierarchical broadcast according to step (4) until each node cannot find a node from the neighbor table, i.e. cannot pass the Bloom filter, and then terminate the transmission.

[0014] Specifically, K in step (3) refers to how many neighbor nodes each node transmits a message to.

[0015] Further, the step (5) calculates a new range, specifically: if it is the first node, the range is 0~K_i+theta; if it is the last node, the range is: K_(i-1)~MaxInt; the range of the intermediate node is: K_(i-1)~K_i+theta; wherein theta refers to a redundancy factor, used to solve the problem of blocking message propagation by unreliable nodes in a decentralized network, according to the value of theta, the same node has the probability to receive repeated messages from multiple other nodes, if theta is equal to 0, the message propagation efficiency is 1, that is, each node will only receive 1 message.

[0016] Specifically, the selection of the value of the redundancy factor theta is based on the experience value, 1024 nodes are 0.5*(K_(i+1)-K_i), and the number of other clusters can be adjusted according to the test scene and determined after verifying the effect.

[0017] Specifically, in order to improve the scalability and efficiency of the network, sometimes the entire network is divided into multiple smaller sub-networks (sharded network), when cross-shard network-wide broadcast of messages is needed, only need to ensure that each shard network has a message source, and then run a random broadcast shard algorithm in a 32-bit space.

[0018] Specifically, by the method of message filtering and random layered broadcast, the number of times that the same message is received by the node is reduced, and the network broadcast efficiency of 1024 nodes is reduced from 1:800 to 1:5.

[0019] According to the second aspect of the embodiment of the application, a multi-shard device based on network P2P communication and random broadcast comprises the following modules:

[0020] Network sharding module: the first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are obtained by Hash256 truncation on the node public key, which is used as the address of the network node in a specific network shard;

[0021] Network initialization module: the node is initialized to join the P2P network, a random value is independently generated, and the random value is transmitted to all neighbor nodes through the heartbeat after being added to the network;

[0022] Message source node module: according to the range of the node ID, that is, 0~MaxInt, randomly select K neighbor nodes;

[0023] Bloom filter module: send the message and the range K_i distributed by the current node to the K nodes, and add the node and the random value of the sent node to the Bloom filter;

[0024] Node calculation module: each node receiving the message calculates a new range according to the range K_i of the last node,

[0025] Filtering node module: from the calculated range, randomly select K nodes, if less than K, select all nodes, and filter out the nodes that have been sent through the Bloom filter, continue to broadcast the message;

[0026] Hierarchical broadcast module: continue hierarchical broadcast according to the operation of the Bloom filter module, until each node cannot find nodes from the neighbor table, i.e., cannot pass through the Bloom filter, then terminate the sending.

[0027] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors;

[0028] a memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0030] According to a third aspect of the embodiments of the present application, a computer readable storage medium having stored thereon computer instructions, characterized in that the instructions, when executed by a processor, implement the steps of the method.

[0031] The technical solutions provided by the embodiments of the present application can include the following beneficial effects:

[0032] The present application proposes an algorithm and device for building a P2P network with higher stability, lower latency and higher transmission efficiency by fragmenting the P2P network, randomly selecting neighbor nodes in multiple ways and configuring the amount of redundant information, on the basis of Kademlia and Gossip protocols. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0034] Figure 1 is a network fragmentation method schematic diagram of the present application;

[0035] Figure 2 is a random hierarchical broadcast method schematic diagram of the present application;

[0036] Figure 3 is a schematic diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0037] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to any embodiment of the application, unless specifically stated otherwise. The following description relates to the drawings in which:

[0038] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0039] The application is implemented by the following method:

[0040] (a) When a node joins a P2P network, a node ID with uniqueness and identifiability needs to be generated for each node, i.e., node ID, the traditional method may be random generation or generation based on certain attributes (such as IP address, MAC address, etc.), and the method of generating node ID in the application is replaced by the technical solution of 32-bit fragment ID + 224-bit fragment ID generation method.

[0041] (b) When a node sends a message as a message sender, the traditional way uses Gossip algorithm: when a node needs to send a message, it will randomly select a certain number of neighbor nodes (for example, K) for message passing, and these neighbor nodes will continue to randomly select other neighbor nodes for forwarding, forming the "propagation" effect of information. Due to the random selection between nodes, the same message may be sent and received multiple times in the network, causing bandwidth waste and delay, and at the same time, due to the random propagation strategy, some nodes may have a very small probability of not receiving the broadcasted message. The application will replace the Gossip algorithm with the message propagation algorithm in the application.

[0042] (c) When a node receives information as a message receiver, it needs to implement the message filtering mechanism in the application solution.

[0043] The application divides the network into fragments, so that messages will not receive messages from other fragment network nodes when propagating. In the message propagation algorithm, instead of randomly selecting adjacent nodes in all network spaces, the application randomly selects the next hop node broadcast range through the bisection method.

[0044] (1) Network fragmentation:

[0045] For example, Figure 1As shown. The first 32 bits of the 256-bit address space are used as the network space shard ID. The last 224 bits are obtained by Hash256 truncating the last 224 bits of the node public key, as the address of the network node under a specific network shard.

[0046] (2) Random layer broadcast method:

[0047] As shown in Figure 2 , the method flow is as follows:

[0048] Step 201: Network initialization: node joins network initialization, independently generates a random value, and transmits it to all neighbor nodes through heartbeat after joining the network.

[0049] Step 202: Message source node, randomly select K neighbor nodes according to the range of node ID (such as 0~MaxInt), K refers to how many neighbor nodes each node will propagate a message to.

[0050] Step 203: Send the message and the current node distribution range K_i to the K nodes, and add the node and the sent node random value to the Bloom filter.

[0051] Step 204: Each node that receives the message calculates the new range according to the range K_i of the previous node. If it is the first node, the range is 0~K_i+θ; if it is the last node, the range is: K_(i-1)~MaxInt; the range of intermediate nodes is: K_(i-1)~K_i+θ. (Where θ refers to the redundancy factor, used to solve the problem of blocking message propagation by unreliable nodes in a decentralized network. Each node has a chance to repeatedly deliver messages to the same node. If θ is equal to 0, the message propagation efficiency is 1, i.e. each node will only receive 1 message.)

[0052] Step 205: From the calculated range, randomly select K nodes (if less than K, select all nodes), and filter out the nodes that have been sent through the Bloom filter, and continue to broadcast the message.

[0053] Step 206: Continue layer broadcast according to the operation of step 203 until each node cannot find nodes from the neighbor table (cannot pass the Bloom filter) and terminate sending.

[0054] The selection of redundancy factor θ value is based on empirical value. For nodes with 1024, it is generally 0.5*(K_(i+1)-K_i). For other cluster quantities, you can adjust and verify the effect after testing the scene to determine the value.

[0055] If the message needs to be broadcasted across the network, then only need to run this algorithm in 32-bit space, ensure that each network has a message source.

[0056] The technical solution of the present application divides the network into different network slices, and logically isolates different network slices, which can effectively reduce the number of broadcast nodes and improve the P2P communication performance of the nodes. On the other hand, the message filtering and random layered broadcast algorithm reduces the number of times that a node receives the same message, effectively reducing the network broadcast efficiency of 1024 nodes from 1:800 to about 1:5, thereby significantly improving the broadcast efficiency of a multi-node network.

[0057] The present application also provides a multi-slice device based on network P2P communication and random broadcast. The device comprises the following modules:

[0058] The network slicing module: the first 32 bits of the 256-bit address space are used as the network space slice ID, and the last 224 bits are obtained by Hash256 intercepting the node public key, which is used as the address of the network node in a specific network slice;

[0059] The network initialization module: the node is initialized to join the P2P network, a random value is independently generated, and the random value is transmitted to all neighbor nodes through the heartbeat after being added to the network;

[0060] The message source node module: according to the range of the node ID, i.e. 0~MaxInt, K neighbor nodes are randomly selected;

[0061] The Bloom filter module: the message and the range K_i of the current node are sent to the K nodes, and the node and the random value of the sent node are added to the Bloom filter;

[0062] The node calculation module: each node receiving the message calculates a new range according to the range K_i of the previous node,

[0063] The filter node module: from the calculated range, K nodes are randomly selected, if less than K, all nodes are selected, and the Bloom filter is used to filter out the nodes that have been sent, and the message is continuously broadcasted;

[0064] The layered broadcast module: according to the operation of the Bloom filter module, the layered broadcast is continuously performed until each node cannot find a node from the neighbor table, i.e. cannot pass the Bloom filter, and then the sending is terminated.

[0065] As to the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0066] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The apparatus embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed to multiple network units. Some or all of the modules can be selected to achieve the purpose of the application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0067] Correspondingly, the application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a multi-shard method based on network P2P communication and random broadcast as described above. As Figure 3 As shown in the figure, a hardware structure diagram of a deep learning data set access system provided by the embodiment of the application is in any data processing capable device. In addition to the processor, the memory and the network interface shown in the figure, the data processing capable device in the embodiment can also include other hardware according to the actual function of the data processing capable device, which will not be described here. Figure 3 As shown in the figure, a hardware structure diagram of a deep learning data set access system provided by the embodiment of the application is in any data processing capable device. In addition to the processor, the memory and the network interface shown in the figure, the data processing capable device in the embodiment can also include other hardware according to the actual function of the data processing capable device, which will not be described here.

[0068] Correspondingly, the application also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement a multi-shard method based on network P2P communication and random broadcast as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any data processing capable device and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0069] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application embrace any and all variations, modifications, and adaptations of the application described herein, which are within the scope of the general inventive concept and include those expressly described herein, as well as other combinations of features, functions, and concepts included in the present disclosure or that are inherent in this field of technology.

[0070] It is to be understood that the application is not limited to the precise construction here described and as shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A multi-shard method based on network P2P communication and random broadcast, characterized in that, The method comprises the following steps: (1) network slicing: the first 32 bits of the 256-bit address space are taken as the network space slice ID, and the last 224 bits are taken as the address of the network node in a specific network slice by intercepting the last 224 bits of the Hash256 of the node public key; (2) network initialization: the node is added to the P2P network initialization, an independent random value is generated, and the random value is added to the network and then transmitted to all neighbor nodes through heartbeat; (3) message source node: K neighbor nodes are randomly selected according to the range of the node ID, i.e. 0 ~ MaxInt; K refers to the number of neighbor nodes to which each node transmits a message; (4) the message and the range K_i of the current node are sent to the K nodes, and the node and the sent node random value are added to the Bloom filter; (5) each node receiving the message calculates a new range according to the range K_i of the previous node, (6) K nodes are randomly selected from the calculated range, and if K is less than K, all nodes are selected, and the Bloom filter is used to filter out the nodes that have been sent, and the message is continuously broadcasted; (7) the step (4) is continuously operated to perform hierarchical broadcasting until each node cannot find a node from the neighbor table, i.e. the Bloom filter cannot be passed, and then the sending is terminated.

2. The multi-shard method based on network P2P communication and random broadcast according to claim 1, characterized in that, In the step (5), the new range is calculated, specifically: if it is the first node, the range is 0 ~ K_i+ Ө; if it is the last node, the range is: K_(i-1) ~ MaxInt; and the range of the intermediate node is: K_(i-1) ~ K_i+ Ө; wherein Ө refers to a redundancy factor for solving the problem that the unreliable node in the decentralized network blocks the message transmission; according to the size of the Ө value, the same node has a probability of receiving repeated messages from multiple other nodes; if Ө is equal to 0, the message transmission efficiency is 1, i.e. each node only receives the message once.

3. The multi-shard method based on network P2P communication and random broadcast according to claim 2, characterized in that, The selection of the redundancy factor Ө value is based on an empirical value, and for a node of 1024, the value is 0.5 * (K_(i+1)- K_i); for other cluster quantities, the value can be adjusted and verified according to the test scene.

4. The multi-shard method based on network P2P communication and random broadcast of claim 1, wherein, The method divides the entire network into multiple smaller sub-networks; when cross-slice network broadcasting of a message is required, it is only necessary to ensure that each slice network has a message source, and then a random broadcast slicing algorithm is run in the 32-bit space, thereby improving the scalability and efficiency of the network.

5. The method of claim 1, wherein, Through the message filtering and random hierarchical broadcasting method, the number of times that a node receives the same message is reduced, and the network broadcasting efficiency of 1024 nodes is reduced from 1: 800 to 1:

5.

6. A multi-shard device based on network P2P communication and random broadcast, characterized in that, The method comprises the following modules: a network slicing module: the first 32 bits of the 256-bit address space are taken as the network space slice ID, and the last 224 bits are taken as the address of the network node in a specific network slice by intercepting the last 224 bits of the Hash256 of the node public key; a network initialization module: the node is added to the P2P network initialization, an independent random value is generated, and the random value is added to the network and then transmitted to all neighbor nodes through heartbeat; Message source node module: randomly select K neighbor nodes according to the range of node ID, i.e. 0 ~ MaxInt; the K refers to how many neighbor nodes each node will propagate a message to; Bloom filter module: send the message and the range K_i of the current node to the K nodes, and add the node and the random value of the sent node to the Bloom filter; Node calculation module: each node that receives the message calculates a new range according to the range K_i of the last node, Filter node module: from the calculated range, randomly select K nodes, if less than K, select all nodes, and through the Bloom filter, filter out the nodes that have been sent, and continue to broadcast the message; Hierarchical broadcast module: continue hierarchical broadcast according to the operation of the Bloom filter module until each node cannot find a node from the neighbor table, i.e. cannot pass the Bloom filter, then terminate the sending.

7. An electronic device, comprising: Comprise: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1-5.

Images