Multi-fragmentation method and device based on network P2P communication and random broadcast

Abstract

The invention discloses a multi-fragmentation method and a multi-fragmentation device based on network P2P communication and random broadcast, which are characterized in that on the basis of Kademlia and Gossip protocols, a method for fragmenting a P2P network to prevent messages from receiving messages from other fragmentation network nodes during propagation is provided, and the message transmission efficiency is improved. And the P2P network with higher stability, lower latency and higher transmission efficiency is constructed in a way of randomly selecting neighborhood nodes in multiple paths and configuring redundant information. The problem that in the related technology, when information is transmitted through a traditional Gossip protocol, a large amount of redundant information usually needs to be transmitted, and consequently network bandwidth and nodes are wasted is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of shard network P2P communication, and in particular relates to a multi-shard method and device based on network P2P communication and random broadcasting. Background Art

[0002] P2P network (Peer-to-peer networking) is a decentralized network structure that plays an important role in the information transmission of distributed systems. It allows nodes distributed in different geographical locations to share information without the participation of a centralized server. P2P network is widely used in various distributed scenarios including blockchain due to its decentralization, high scalability, high robustness, and high privacy.

[0003] Common technologies for building P2P networks include: Kademlia, Chord protocol, SWIM (Scalable Weakly-consistent Infection-style Membership Protocol) and Gossip protocol: Kademlia is a DHT-based P2P routing protocol that allows nodes to quickly find and locate other nodes through distributed hash tables, thereby improving routing efficiency and network stability. Kademlia is widely used to build P2P networks, such as BitTorrent. Chord is another DHT protocol used in P2P networks. It provides more reliable data search and distributed storage, and implements distributed data management through consistent hash tables. SWIM is a protocol for member detection and node management, which aims to improve the stability and scalability of P2P networks. SWIM reduces noise in the network by reducing erroneous state information in the network. The Gossip protocol randomly selects a number of neighbor nodes and propagates messages to these nodes, which then propagate to their neighbors, and so on. This method ensures that the message is eventually propagated to the entire network.

[0004] The traditional Gossip protocol has some performance and efficiency limitations in some cases. These limitations include but are not limited to: First, the problem of network topology instability: the Gossip protocol usually randomly selects adjacent nodes between nodes for information transmission. When the network topology is unstable or there are many node failures, this may lead to delayed transmission or loss of information. Secondly, it is about the high latency problem: due to the random nature of the Gossip protocol, some information may take a long time to propagate to the entire network, resulting in high latency. Finally, there is also the problem of resource waste: when transmitting information, the traditional Gossip protocol usually needs to transmit a large amount of redundant information, which wastes network bandwidth and nodes. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a multi-sharding method and device based on network P2P communication and random broadcasting, so as to solve the technical problem that the traditional Gossip protocol in the related technology usually needs to transmit a large amount of redundant information when transmitting information, thereby wasting network bandwidth and nodes.

[0006] According to a first aspect of an embodiment of the present application, a multi-sharding method based on network P2P communication and random broadcast is provided, comprising the following steps:

[0007] (1) Network sharding: The first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are truncated by Hash256 of the node public key as the address of the network node in a specific network shard;

[0008] (2) Network initialization: Initialize the node by adding it to the P2P network, generate a random value independently, add the random value to the network, and pass it to all neighboring nodes through heartbeats;

[0009] (3) Message source node: randomly select K neighbor nodes according to the range of node ID, i.e., 0 to MaxInt;

[0010] (4) Send the message and the range K_i distributed by the current node to these K nodes, and add the random values ​​of this node and the sent nodes to the Bloom filter;

[0011] (5) Each node that receives the message calculates a new range based on the range K_i of the previous node.

[0012] (6) Randomly select K nodes from the calculated range. If the number is less than K, select all nodes. At the same time, filter out the nodes that have already sent the message through the Bloom filter and continue to broadcast the message.

[0013] (7) Follow step (4) and continue the hierarchical broadcast until each node can no longer find a node from the neighbor table, that is, it cannot pass the Bloom filter, and then terminate the transmission.

[0014] Specifically, K in step (3) refers to how many neighbor nodes each node propagates a message to.

[0015] Furthermore, the new range is calculated in step (5), specifically: if it is the first node, the range is 0 to K_i+θ; if it is the last node, the range is: K_(i-1) to MaxInt; the intermediate node range is: K_(i-1) to K_i+θ; where θ refers to the redundancy factor, which is used to solve the problem of unreliable nodes blocking message propagation in a decentralized network. Depending on the value of θ, the same node has a probability of receiving duplicate messages from multiple other nodes. If θ is equal to 0, the message propagation efficiency is 1, that is, each node will only receive one message.

[0016] Specifically, the selection of the redundancy factor θ value is based on empirical values. For 1024 nodes, it is 0.5*(K_(i+1)-K_i). The number of other clusters can be adjusted according to the test scenario and determined after verifying the effect.

[0017] Specifically, in order to improve the scalability and efficiency of the network, the entire network is sometimes divided into multiple smaller sub-networks (shard networks). When a message needs to be broadcast across the entire shard network, you only need to ensure that each shard network has a message source, and then run the random broadcast sharding algorithm on the 32-bit space.

[0018] Specifically, the method of message filtering and random layered broadcasting is used to reduce the number of times a node receives the same message, and reduce the network broadcast efficiency of 1024 nodes from 1:800 to 1:5.

[0019] According to a second aspect of an embodiment of the present application, a multi-sharding device based on network P2P communication and random broadcasting includes the following modules:

[0020] Network sharding module: The first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are intercepted by Hash256 of the node public key as the address of the network node in a specific network shard;

[0021] Network initialization module: Initialize the node by adding it to the P2P network, generate a random value independently, add the random value to the network and pass it to all neighboring nodes through heartbeats;

[0022] Message source node module: randomly select K neighbor nodes according to the range of node ID, i.e. 0 to MaxInt;

[0023] Bloom filter module: sends the message and the range K_i distributed by the current node to the K nodes, and adds the random values ​​of the current node and the sent nodes to the Bloom filter;

[0024] Node calculation module: Each node that receives the message calculates the new range based on the range K_i of the previous node.

[0025] Filter node module: randomly select K nodes from the calculated range. If the number is less than K, select all nodes. At the same time, filter out the nodes that have been sent through the Bloom filter and continue to broadcast the message.

[0026] Hierarchical broadcast module: According to the operation of the Bloom filter module, the hierarchical broadcast is continued until each node can no longer find a node from the neighbor table, that is, it cannot pass the Bloom filter, and then the transmission is terminated.

[0027] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: one or more processors;

[0028] A memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the described method.

[0030] According to a third aspect of an embodiment of the present application, a computer-readable storage medium stores computer instructions thereon, wherein the instructions implement the steps of the method when executed by a processor.

[0031] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0032] Based on Kademlia and Gossip protocols, the present invention proposes an algorithm and device for constructing a P2P network with higher stability, lower latency and higher transmission efficiency by sharding the P2P network, multi-path random selection of neighboring nodes and configurable redundant information volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 It is a schematic diagram of the network slicing method of the present invention;

[0035] Figure 2 is a schematic diagram of the random layered broadcasting method of the present invention;

[0036] Figure 3 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0038] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes one or more.

[0039] The present invention is implemented by the following method:

[0040] (a) When a node joins a P2P network, it is necessary to generate a unique and identifiable node identifier, i.e., a node ID, for each node. The traditional method may be to generate it randomly or based on certain attributes (such as IP address, MAC address, etc.). The present invention replaces the method of generating a node ID with the 32-bit shard ID + 224-bit shard ID generation method in the technical solution.

[0041] (b) When a node sends a message as a message sender, the traditional method uses the Gossip algorithm: when a node needs to send a message, it will randomly select a certain number of neighbor nodes (for example, K) to transmit the message. These neighbor nodes will then continue to randomly select other neighbor nodes to forward, forming a "propagation" effect of information. Due to the random selection between nodes, the same message may be sent and received multiple times in the network, causing bandwidth waste and delay. At the same time, due to the random propagation strategy, some nodes will have a very small probability of not receiving the broadcasted message. The present invention will replace the Gossip algorithm with the message propagation algorithm of the present invention.

[0042] (c) When a node receives information as a message receiver, it is necessary to implement the message filtering mechanism in the solution of the present invention.

[0043] The present invention divides the network into slices so that the message will not receive messages from other slice network nodes when it is propagated. In the algorithm of message propagation, the present invention does not randomly select adjacent nodes in all network spaces, but randomly selects the range of the next hop node broadcast by binary selection.

[0044] (1) Network sharding:

[0045] like Figure 1As shown in the figure, the first 32 bits of the 256-bit address space are used as the shard ID of the network space. The last 224 bits are intercepted by Hash256 of the node public key and used as the address of the network node in a specific network shard.

[0046] (2) Random layered broadcast method:

[0047] like Figure 2 As shown, the method flow is as follows:

[0048] Step 201: Network initialization: When a node joins the network, it will independently generate a random value and pass it to all neighboring nodes through heartbeats after joining the network.

[0049] Step 202: The message source node randomly selects K (K refers to the number of neighbor nodes to which each node propagates a message) neighbor nodes according to the range of node ID (eg, 0 to MaxInt).

[0050] Step 203: Send the message and the range K_i distributed by the current node to the K nodes, and add the random values ​​of the current node and the sent nodes to the Bloom filter.

[0051] Step 204: Each node that receives the message calculates a new range based on the range K_i of the previous node. If it is the first node, the range is 0 to K_i+θ; if it is the last node, the range is: K_(i-1) to MaxInt; the range of intermediate nodes is: K_(i-1) to K_i+θ. (θ refers to the redundancy factor, which is used to solve the problem of unreliable nodes blocking message propagation in decentralized networks. Each node has a chance to repeatedly transmit messages to the same node. If θ is equal to 0, the message propagation efficiency is 1, that is, each node will only receive 1 message.)

[0052] Step 205: Randomly select K nodes from the calculated range (if it is less than K, select all nodes), and filter out the nodes that have already sent the message through the Bloom filter, and continue to broadcast the message.

[0053] Step 206: According to the operation of step 203, the hierarchical broadcast is continued until each node can no longer find a node from the neighbor table (cannot pass the Bloom filter), and then the transmission is terminated.

[0054] The selection of the redundancy factor θ value is based on empirical values. For 1024 nodes, it is generally 0.5*(K_(i+1)-K_i). The number of other clusters can be adjusted according to the test scenario and determined after verifying the effect.

[0055] If you need to broadcast the message across the entire shard network, you only need to run this algorithm on a 32-bit space to ensure that each shard network has a message source.

[0056] The technical solution of the present invention slices the network and realizes logical isolation for different sliced ​​networks, which can effectively reduce the number of broadcast nodes and improve the node P2P communication performance. On the other hand, the number of times the node receives the same message is reduced through message filtering and random layered broadcast algorithm, effectively reducing the network broadcast efficiency of 1024 nodes from 1:800 to about 1:5. The broadcast efficiency of multi-node network is significantly improved.

[0057] The present invention also provides a multi-slice device based on network P2P communication and random broadcasting. The device includes the following modules:

[0058] Network sharding module: The first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are intercepted by Hash256 of the node public key as the address of the network node in a specific network shard;

[0059] Network initialization module: Initialize the node by adding it to the P2P network, generate a random value independently, add the random value to the network and pass it to all neighboring nodes through heartbeats;

[0060] Message source node module: randomly select K neighbor nodes according to the range of node ID, i.e. 0 to MaxInt;

[0061] Bloom filter module: sends the message and the range K_i distributed by the current node to the K nodes, and adds the random values ​​of the current node and the sent nodes to the Bloom filter;

[0062] Node calculation module: Each node that receives the message calculates the new range based on the range K_i of the previous node.

[0063] Filter node module: randomly select K nodes from the calculated range. If the number is less than K, select all nodes. At the same time, filter out the nodes that have been sent through the Bloom filter and continue to broadcast the message.

[0064] Hierarchical broadcast module: According to the operation of the Bloom filter module, the hierarchical broadcast is continued until each node can no longer find a node from the neighbor table, that is, it cannot pass the Bloom filter, and then the transmission is terminated.

[0065] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0066] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0067] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-sharding method based on network P2P communication and random broadcasting. Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability in which a deep learning data set access system provided by an embodiment of the present invention is located, except Figure 3 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0068] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, a multi-sharding method based on network P2P communication and random broadcasting as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0069] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0070] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A multi-sharding method based on network P2P communication and random broadcasting, characterized in that: The following steps are involved: (1) Network sharding: The first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are intercepted by Hash256 of the node public key as the address of the network node in a specific network shard; (2) Network initialization: Initialize the node by adding it to the P2P network, generate a random value independently, add the random value to the network, and pass it to all neighboring nodes through heartbeats; (3) Message source node: randomly select K neighbor nodes according to the range of node ID, i.e. 0 ~ MaxInt; (4) Send the message and the range K_i distributed by the current node to these K nodes, and add the random values ​​of this node and the sent nodes to the Bloom filter; (5) Each node that receives the message calculates a new range based on the range K_i of the previous node. (6) Randomly select K nodes from the calculated range. If the number is less than K, select all nodes. At the same time, filter out the nodes that have already sent the message through the Bloom filter and continue to broadcast the message. (7) Follow step (4) and continue hierarchical broadcasting until each node can no longer find a node from the neighbor table, that is, it cannot pass the Bloom filter, and then terminate the transmission.

2. A multi-sharding method based on network P2P communication and random broadcasting according to claim 1, characterized in that: The K in step (3) refers to how many neighbor nodes each node propagates a message to.

3. A multi-sharding method based on network P2P communication and random broadcasting according to claim 1, characterized in that: The new range calculated in step (5) is as follows: if it is the first node, the range is 0 ~ K_i+ Ө; if it is the last node, the range is: K_(i-1) ~ MaxInt; the intermediate node range is: K_(i-1) ~ K_i+ Ө; where Ө refers to the redundancy factor, which is used to solve the problem of unreliable nodes blocking message propagation in a decentralized network. Depending on the value of Ө, the same node has a probability of receiving duplicate messages from multiple other nodes. If Ө is equal to 0, the message propagation efficiency is 1, that is, each node will only receive one message.

4. A multi-sharding method based on network P2P communication and random broadcasting according to claim 3, characterized in that: The selection of the redundancy factor Ө value is based on the empirical value. For 1024 nodes, it is 0.5 * (K_(i+1)- K_i). The number of other clusters can be adjusted according to the test scenario and determined after the effect is verified.

5. The multi-sharding method based on network P2P communication and random broadcasting according to claim 1, characterized in that: This method divides the entire network into multiple smaller sub-networks. When a message needs to be broadcast across the entire shard network, you only need to ensure that each shard network has a message source, and then run a random broadcast sharding algorithm on a 32-bit space to improve the scalability and efficiency of the network.

6. A multi-sharding method based on network P2P communication and random broadcasting according to claim 1, characterized in that: By using message filtering and random layered broadcasting methods, the number of times a node receives the same message is reduced, and the network broadcast efficiency of 1024 nodes is reduced from 1:800 to 1:

5.

7. A multi-sharding device based on network P2P communication and random broadcasting, characterized in that: Includes the following modules: Network sharding module: The first 32 bits of the 256-bit address space are used as the shard ID of the network space, and the last 224 bits are intercepted by Hash256 of the node public key as the address of the network node in a specific network shard; Network initialization module: Initialize the node by adding it to the P2P network, generate a random value independently, add the random value to the network and pass it to all neighboring nodes through heartbeats; Message source node module: randomly select K neighbor nodes according to the range of node ID, i.e. 0 ~ MaxInt; Bloom filter module: sends the message and the range K_i distributed by the current node to the K nodes, and adds the random values ​​of the current node and the sent nodes to the Bloom filter; Node calculation module: Each node that receives the message calculates the new range based on the range K_i of the previous node. Filter node module: randomly select K nodes from the calculated range. If the number is less than K, select all nodes. At the same time, filter out the nodes that have been sent through the Bloom filter and continue to broadcast the message. Hierarchical broadcast module: According to the operation of the Bloom filter module, the hierarchical broadcast is continued until each node can no longer find a node from the neighbor table, that is, it cannot pass the Bloom filter, and then the transmission is terminated.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images