A P2P system load perception method in Byzantine environment

Abstract

The present invention discloses a load perception method for a P2P system in a Byzantine environment, which uses consistent hashing to perform load balancing distribution and load estimation, and solves the problem of how to perceive the system load in a Byzantine environment where nodes in a P2P network do not trust each other and some nodes have the possibility of doing evil. The method uses consistent hashing to achieve unique distribution of requests, avoids repeated load reporting by multiple nodes, and uses consistent hashing to achieve balanced distribution of requests. Through the mapping of the hash algorithm in consistent hashing, the server node is evenly anchored to a certain endpoint on the hash ring, ensuring that the load reported by each node is as close as possible, and reducing the impact of malicious reporting or non-reporting of load by Byzantine nodes on the main estimated value.

Description

Technical Field

[0001] The invention belongs to the technical field of distributed networks and relates to a P2P system load perception method in a Byzantine environment. Background Art

[0002] After the storage and computing capabilities of a single machine reached their limits, distributed storage systems and distributed computing systems came into being. Several computers interact with each other through the network to form a P2P system. Large tasks are reasonably assigned to each machine for parallel execution, and the overall processing efficiency is improved. But at the same time, the problem is "how can a distributed system effectively allocate tasks and manage machines?" If the manager node can understand the status and load of each node machine in the P2P network in real time, it can reasonably allocate tasks and solve this problem. Therefore, the problem can be transformed into "how to perceive the load of each node in the P2P system." In a normal network environment, a manager node can be used to regularly interact with other nodes, and the manager node collects their load information for decision-making.

[0003] The Byzantine environment refers to an unsafe distributed network environment with malicious nodes. It originates from a real scenario during the Byzantine Roman Empire War. In this scenario, generals rely only on messengers to deliver messages. The problem of how to make the generals reach the same strategic decision in the presence of interference from traitorous generals is called the Byzantine Generals Problem. Therefore, in the Byzantine environment of the distributed field, Byzantine nodes (different from normal nodes) will do evil and forge false load information to interfere with the decision-making of manager nodes. In this case, it is difficult for manager nodes to obtain the real overall system load information from all server nodes through simple network interactions.

[0004] Consistent hashing is a special hashing algorithm that is mainly used to solve the problem of dynamic expansion and contraction of distributed systems. When a cluster adds or removes server nodes, consistent hashing technology can change the mapping relationship between existing service requests and request processing nodes as little as possible. Consistent hashing is scalable, ensuring that when adding or reducing server nodes, the data storage changes minimally, which greatly saves the cost of data migration compared to traditional hashing algorithms. At the same time, consistent hashing can also maintain good load balancing performance when data grows rapidly. By introducing virtual nodes, an actual node can be mapped to multiple virtual nodes. When data continues to grow, some virtual nodes may contain a lot of data, resulting in uneven distribution on virtual nodes. At this time, the virtual node containing more data can be split into two virtual nodes without affecting other virtual nodes, so there is no need to re-hash and divide all data. In this way, the number of physical nodes can be dynamically expanded as data grows, and the cost is much smaller than the remapping of traditional hashing algorithms. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method for perceiving a load of a P2P system in a Byzantine environment. The method perceives the system load through consistent hashing, which is mainly divided into two aspects: first, using consistent hashing to achieve unique allocation of requests to avoid repeated reporting of requests by multiple nodes; second, using consistent hashing to achieve balanced allocation of requests to ensure that the load reported by each node is as close as possible, reducing the impact of malicious reporting or non-reporting of load by Byzantine nodes on the main estimate.

[0006] The specific technical solution to achieve the purpose of the present invention is: a P2P system load perception method in a Byzantine environment based on consistent hashing, comprising the following steps:

[0007] Step 1: Build a P2P cluster network, configure node identity information, and start the cluster machine. Identity information includes node role (server node or manager node), node IP, node ID (used to uniquely identify server nodes), and public and private keys (used to encrypt messages and identity authentication). A cluster is a collection of server nodes with the same responsibilities in a network. In the present invention, a cluster includes several server nodes, one of which serves as a manager node (the manager node itself is also a server node). If the manager node goes down, the cluster network will quickly elect a manager node from the remaining server nodes. The server node is mainly responsible for processing requests from clients, counting the load processed by this node, and reporting to the manager node.

[0008] The manager node has the following responsibilities:

[0009] First, the manager node is responsible for monitoring and managing the entire system. There is only one manager node at a time, which maintains the configuration information such as the ID, IP, and port of the nodes in the entire cluster network.

[0010] Second, the manager node regularly maintains connection with the server node through heartbeats: first, the manager node sends a heartbeat request packet to all server nodes. After receiving the heartbeat request packet, the server node updates the identity of the current cluster manager node and replies with a heartbeat response packet to inform the manager node of its current status. After receiving it, the manager node will also update the current status of the server node.

[0011] Third, the manager node is responsible for initializing and updating the consistent hashing load balancing algorithm, and negotiates with other server nodes to reach a consensus to ensure consistency in the allocation of requests by all nodes in the cluster.

[0012] Fourth, the manager node is responsible for the expansion and reduction of cluster nodes and provides node discovery services. The node discovery service provides an interface for querying cluster information on a default port, so that server nodes that subsequently enter the cluster can obtain the address port information of other nodes through this interface and establish connections.

[0013] The server node includes a hash query module, a request processing module, and a load statistics module.

[0014] The hash query module is used to determine the attribution of the request sent by the client to the current server node. Through hash query, it is located which server node should actually serve the request. If it is itself, it is handed over to the request processing module for service and response. Otherwise, it is forwarded to the corresponding server node for request processing.

[0015] The request processing module is used to process the client's request and return the result to the client. The module provides an abstract interface, and its functions are customizable. Different functions can be set and implemented as needed, such as the common key-value database service: updating the database by parsing read and write requests and returning the corresponding query results. At the same time, it is necessary to call the load statistics module to record the load consumed in processing the request.

[0016] The load statistics module is used to record the total load information received by the current server node and the load information in each stage. The load information specifically includes parameters such as load, CPU execution time consumed for processing requests (specifically divided into kernel execution time and user state execution time), I / O throughput and various delays. These parameters can be used to analyze the performance of the current server node and evaluate the load.

[0017] The manager node includes a discovery service module, a load estimation module, and a load balancing module.

[0018] The discovery service module is used for the server nodes that subsequently enter the cluster to obtain the address port information of other nodes and establish a connection. Specifically, a default listening port is set when the cluster is started. Other server nodes will send requests to query cluster information, and the manager node returns the query results after the authority check;

[0019] The load estimation module is used to collect and estimate the load information of all server nodes. The module will avoid estimation deviation caused by the forged load information of Byzantine nodes through median sorting. The number of Byzantine nodes does not exceed one-third of the total number of clusters, assuming that there are f nodes. To avoid malicious behavior of these nodes, the server node will collect 2f+1 load information, and then sort and take the median, that is, the load information of the f+1th node. There are f load information on the left and right of the median to avoid f Byzantine nodes maliciously returning too low or too high load information. If they return load information similar to that of normal nodes, it is equivalent to performing the same as normal nodes, which will not interfere with the estimation of the final load information. The relevant detailed explanation is explained in step 5.

[0020] The load balancing module is used to re-divide the load attribution area of ​​the consistent hash when the manager node increases or expands the server node. Then the manager node will reach a consensus on the newly divided consistent hash allocation scheme in the cluster through a Byzantine Fault Tolerance (BFT) consensus algorithm such as the Practical Byzantine Fault Tolerance (PBFT) algorithm.

[0021] Step 2: The server node sends a registration request message to the public port monitored by the manager node. The manager node generates a consistent hash allocation table based on the cluster node information and broadcasts it to all nodes through the Byzantine fault-tolerant consensus. Specifically, the server node sends a registration request message to the manager node, the content of which is the identity information mentioned in step 1. After collecting all the registration request messages of the server nodes, the manager node sets the consistent hash, allocates a specified number of virtual nodes, and binds them to the actual nodes in the P2P network cluster. The relationship between the actual node and the virtual node is one-to-one and / or one-to-many mapping. An actual node can be mapped to one and / or multiple virtual nodes. The client's request will first find the virtual node to which it belongs, and then further locate the actual node. The P2P network cluster first starts the manager node, provides a public listening port, waits for other server nodes to register information or obtain cluster information on the port, and sets the initial number of nodes. Then start other server nodes, register information on the manager node, and obtain the assigned ID.

[0022] The consistent hashing forms a ring, and each request will be processed by a unique server node. Each server node is responsible for requests within a section of the arc in the counterclockwise direction of the ring, and these arcs are approximately equally divided by the hash of the server node. When the number of registrations reaches the initial number of nodes, the manager node starts to generate and initialize the consistent hash allocation table based on all the node information of the cluster. The underlying data structure of the allocation table is a red-black tree, which is a binary balanced search tree with the characteristics of efficient node search and node addition and deletion. The nodes of the red-black tree store the mapping relationship from request to virtual node. The consistent hashing algorithm used in the present invention is Ketama Hash, and its allocation steps are mainly as follows:

[0023] Step 2-1: Obtain the address information and memory, CPU, and disk resource status of all server node lists from the registration configuration information. The latter (memory, CPU, and disk resource status) will be used as the weight of the number of virtual nodes allocated to the server node;

[0024] Step 2-2: Each server node calculates how many virtual nodes are needed according to the weight. The baseline configuration is to set 160 virtual nodes for each server node. Each server node generates 40 strings, 10.0.1.1:11211-1, 10.0.1.1:11211-2 to 10.0.1.1:11211-40, and calculates 40 16-byte hash values ​​according to the MD5 hash digest algorithm. Each hash is divided into 4 4-byte hash values, for a total of 160 hash values, i.e. 160 virtual nodes.

[0025] Step 2-3: Store all hash values ​​and corresponding node addresses into the aforementioned consistent hash allocation table. A 4-byte hash means that the hash value range is [0, 2 32 -1], the subsequent search uses the red-black tree binary search algorithm, the complexity is O(log(vn)), where n is the number of actual nodes and v is the number of virtual nodes corresponding to each actual node;

[0026] Step 2-4: The manager node digitally signs the consistent hash allocation table and sends it to all server nodes. After receiving it, the server node checks the signature and accepts the hash allocation plan after confirming that it is correct. The initialization phase of the entire cluster ends and begins to accept client requests and provide services.

[0027] Step 3: The client sends a request, and the server processes it accordingly and counts the load information. The client randomly sends the request to the neighboring server node. If the client does not have the server node information locally, it will first obtain the server node configuration information of the cluster from the manager node. After testing the connection status of each server node, the client selects the server node with the smallest round-trip delay to send the request. After receiving the request, the server node performs MD5 hashing on the request to obtain a 16-byte hash value. The last 4 bytes are used as the consistent hash value of the request, recorded as K. Then, a binary search is used in the consistent hash allocation table to locate the hash value S of the first virtual node after K (each virtual node has a unique hash value), and then the actual node ID corresponding to the virtual node with a hash value of S is determined. If the actual node ID is the current server node ID, it will be processed by the current server node. Otherwise, the client request is forwarded to the server node to which the request is correctly assigned, and the query and processing process is repeated. When the correct server node processes the request, the load statistics module will be called to record the load status of the current request.

[0028] Specifically, the specific steps of querying the consistent hash allocation table to obtain the ID of the corresponding server are:

[0029] Step 3-1: First, cluster nodes register with the administrator and discover each other to build a P2P network and configure Figure 2 In the consistent hash allocation table, each server node maintains a copy of the consistent hash allocation table locally. After initialization is completed, the cluster provides services to the outside world.

[0030] Step 3-2: A server node receives a client request, serializes the request into binary form, performs MD5 hashing on the request to obtain a hash value K, and then searches the hash table for the first value S that is greater than or equal to K (i.e. Figure 2 The first server node position clockwise in the table), and obtains the bound server node ID with index S from the consistent hash allocation table;

[0031] Step 3-3: If the ID is the ID of this server node, the request will be processed by this node and the load of this node will be increased. Otherwise, it will be forwarded to the corresponding server node ID to ensure that the request is processed by a unique server node.

[0032] Step 4: During the client request processing phase of the P2P network cluster, the manager node starts the load collection and statistics tasks in parallel. The manager node performs the load statistics task every epoch. Epoch refers to a simple repetitive time interval, usually 10 to 20 seconds, which can be customized according to business needs. The manager node determines the time to start load collection by setting a timer, and initiates a load collection request to all server nodes at the beginning of each epoch. After receiving the load collection request, the server node (including the manager node) starts to count the load information from the last statistics to the current time. The load information includes parameters such as load, CPU execution time consumed by processing requests (specifically, it can be divided into kernel execution time and user state execution time), I / O throughput, and various delays. These parameters can be used to analyze the performance of the current server node and comprehensively evaluate the load. After the server node has counted the load information, it encapsulates the above load information into a message body format, digitally signs it, and sends it to the manager node. The digital signature is to prevent others from deliberately forging false load statistics information. The manager node can verify whether the message is sent by the server node through the digital signature to avoid forged information from unknown sources.

[0033] Step 5: After the manager node receives replies from 2f+1 server nodes (including the manager node itself as a server node), it can start to count the load. The total number of nodes in the cluster is 3f+1, of which f are Byzantine nodes. First, sort the load of each server node in non-descending order, take the median of the load in the sorted replies (i.e., the load of the f+1th reply) as the average load of the entire system, and multiply it by the total number of server nodes in the system to obtain an estimated value of the overall system load.

[0034] Here we further explain "collecting 2f+1 server nodes can be used to perform load statistics" and "estimate the total load with the load of the median node": In addition to forging load information, Byzantine nodes may also remain silent and not respond to the load collection requests of the manager node. When the Byzantine node is in the latter state, the manager node cannot determine whether it is due to malicious intent or due to network delays and network partitions. Therefore, when there are f Byzantine nodes and n cluster nodes (n>f), the manager node needs to enter the subsequent statistical stage when it receives nf replies. It cannot be in a waiting state for the f Byzantine nodes that may all be deliberately not replying. This ensures the activity of the cluster, that is, the business can continue without hovering somewhere; but receiving nf replies also has another possibility: f of the replies are forged by Byzantine nodes, which may forge too large or too small load information. In order to avoid the influence of these f false messages, the loads of these nf replies need to be sorted and the median is taken to avoid the influence of the f smaller median values ​​and the f larger load values ​​(if the load is in the middle and the actual load does not deviate much, the correct total valuation can be obtained). Therefore, nf>=2f+1, and n>=3f+1 can be obtained. In order to maximize the use of resources, the total number of cluster points is usually 3f+1, which ensures the security of the cluster and prevents malicious nodes from affecting the normal execution of the business.

[0035] Step 6: The manager node distributes the estimated value of the overall system load and the replies to the 2f+1 load collection requests used for the estimation to all server nodes. The entire cluster reaches a consensus on the estimated value, and the load statistics are completed. The replies to the 2f+1 load collection requests have the electronic signature of their source server node to verify the correctness of the source of the reply.

[0036] Compared with the prior art, the beneficial effect of the present invention is to solve the problem of load balancing and estimating system load in a Byzantine environment. The prior art did not take into account that in an untrusted environment, malicious nodes would falsely report their own load conditions, such as being too large or too small, which would lead to deviations in the overall load estimate. At the same time, in order to make the median of each node load approximate to the average, consistent hashing technology is used to achieve balanced and unique distribution of requests. Through the mapping of the hash algorithm in consistent hashing, the server nodes are evenly anchored to a certain endpoint on the hash ring (see Appendix Figure 2 ), the client's requests will also be evenly distributed on the hash ring. The first server node is found clockwise, which is the server node to which it belongs. In this way, each client request will be uniquely assigned to a server node, thus ensuring the balanced and unique distribution of client requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1This is a system diagram of the present invention based on consistent hashing load awareness.

[0038] Figure 2 This is a schematic diagram of the consistent hashing configuration of the present invention.

[0039] Figure 3 This is a flow chart of consistent hashing load balancing and load estimation of the present invention. DETAILED DESCRIPTION

[0040] The invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0041] The present invention proposes a method for perceiving the load of a P2P system in a Byzantine environment, which solves the problem of how to perceive the system load in a Byzantine environment where nodes in a P2P network do not trust each other and some nodes may be malicious. The content of the method includes: first, using consistent hashing to achieve unique allocation of requests to avoid repeated load reporting by multiple nodes; second, using consistent hashing to achieve balanced allocation of requests to ensure that the load reported by each node is as close as possible, and reduce the impact of malicious reporting or non-reporting of load by Byzantine nodes on the main estimate.

[0042] Figure 1 This is a system diagram of the present invention based on consistent hash load perception, the system mainly includes a manager node and several server nodes; the manager node includes a discovery service module, a load balancing module, and a load estimation module; the server node includes a hash query module, a request processing module, and a load statistics module.

[0043] Figure 2 It shows how to use consistent hashing for balanced and unique distribution, with a hash value range of [0, 2 32-1], that is, the number field composed of all the values ​​of four bytes and 32 bits, constitutes the ring shown in the figure. There are three server nodes in the figure. First, the configuration information of the node is binary serialized, and the 128-bit hash summary value is obtained through MD5 hashing. The last 32 bits are cut off to get the hash values ​​S1, S2 and S3, which are anchored to the corresponding positions on the ring. Based on the completely random characteristics of hash mapping, the positions of these server nodes on the ring are usually evenly distributed. In the diagram, when request 1 arrives, the server node binary serializes request 1 and performs MD5 hash digest to obtain hash value K1, then searches clockwise from K1 to reach the first server node S2, and the request will be processed by server node 2; similarly, after request 2 arrives, request 2 is binary serialized and performed MD5 hash digest to obtain hash value K2, searches clockwise from K2 to reach the first server node S3, and is processed by server node 3; after request 3 arrives, request 3 is binary serialized and performed MD5 hash digest to obtain hash value K3, searches clockwise from K3 to reach the next server node S1, and is processed by server node 1 (for K3, it starts at 0 after exceeding the maximum value). According to the above process, it is not difficult to find that each request will be processed by a unique server node, and each server node is responsible for requests within a section of the arc in the counterclockwise direction on the ring, that is, server node 2 is responsible for requests within the arc between server node 1 and server node 2, server node 3 is responsible for requests within the arc between server node 2 and server node 3, and server node 1 is responsible for requests within the arc between server node 3 and server node 1. These arcs are approximately equally divided by the hash of the server nodes, thereby ensuring the load balance of each server node.

[0044] Figure 3The flowchart of the system for load balancing and load estimation is shown. The number of nodes in the cluster is 3f+1, f of which are Byzantine nodes (including malicious nodes and faulty nodes), and 2f+1 are normal nodes. First, build a P2P cluster network, configure node identity information, and start the cluster machine. The server node sends a registration request message to the public port monitored by the manager node. The manager node generates a consistent hash allocation table based on the cluster node information and broadcasts it to all nodes through the Byzantine fault-tolerant consensus. The server node waits for the client's request to arrive. After receiving the request, the server node performs MD5 hashing on the request to obtain the hash value K. The server node uses binary search to locate the hash value S of the first virtual node after K (each virtual node has a unique hash value) in the consistent hash allocation table, and then determines the actual node ID corresponding to the virtual node. If the ID is not the server node ID, the client request is forwarded to the server node to which the request is correctly attributed, and the query and processing process is re-performed. If the actual node ID is the current server node ID, it is processed by the current server node, and the load statistics module is called to record the load of processing the current request. If this node is a manager node and enters a new epoch, the manager node starts the load collection and statistics tasks in parallel and initiates a load collection request to all server nodes. If this node is a server node and receives a load collection request from a manager node, the load information from the last statistics to the current time is counted, encapsulated into a message body format, and digitally signed and sent to the manager node. After receiving the replies from 2f+1 server nodes (including the manager node itself as a server node), the manager node starts to count the load, that is, sort the load of each server node, take the median of the load in the sorted reply (that is, the load of the f+1th reply) as the average load of the entire system, and multiply it by the total number of server nodes in the system to obtain the estimated value of the overall load of the system. The manager node distributes the estimated value of the overall load of the system and the replies to the 2f+1 load collection requests used for estimation to all server nodes. The entire cluster reaches a consensus on the estimated value, and the load statistics are completed. The system continues to cycle the above business process.

[0045] The main application scenario of the present invention is the load statistics in the decentralized alliance chain. The alliance chain is a permissioned blockchain with threshold entry and exit restrictions. It has the characteristics of decentralization, tamper-proof and traceability, and can act as an intermediary for different organizations. In the alliance chain, each organization maintains a server node, and some business transaction requests will be submitted to different server nodes for processing. At this time, it is necessary to make an overall estimate of the overall load in the alliance chain. Because it is difficult for different organizations to reach trust, the alliance chain scenario is usually designed based on the Byzantine environment. The load statistics technology proposed in this invention can be applied to this scenario.

[0046] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A P2P system load perception method in a Byzantine environment, characterized in that: The method uses consistent hashing to perform load balancing and load estimation, and includes the following steps: Step 1: Build a P2P cluster network, configure node identity information, and start the cluster machine; Step 2: The server node sends a registration request to the public port monitored by the manager node. The manager node generates a consistent hash distribution table based on the cluster node information and broadcasts it to all nodes through the Byzantine fault-tolerant consensus. In step 2, the registration request information includes node role, node IP, node ID and public and private keys; the node roles include server node and manager node, the node ID is used to uniquely identify the server node, and the public and private keys are used to encrypt messages and identity authentication; the manager node collects all server node registration request information and sets a consistent hash, allocates virtual nodes, and binds them to actual nodes in the P2P network cluster. The relationship between actual nodes and virtual nodes is one-to-one and / or one-to-many mapping, that is, one actual node can be mapped to one and / or multiple virtual nodes; the client's request will first find the virtual node to which it belongs, and then further locate the actual node; In step 2, the steps of allocating virtual nodes by consistent hashing include: Step 2-1: Obtain the address information and memory, CPU, and disk resource status of all server node lists from the registration request information, and use the memory, CPU, and disk resource status as the weight for allocating virtual nodes to the server nodes; Step 2-2: Each server node calculates how many virtual nodes are needed according to the weight. The baseline configuration is to set 160 virtual nodes for each server node. Each server node generates 40 strings, 10.0.1.1:11211-1, 10.0.1.1:11211-2 to 10.0.1.1:11211-40, and calculates 40 16-byte hash values ​​according to the MD5 hash digest algorithm. Each hash is divided into 4 4-byte hash values, for a total of 160 hash values, i.e. 160 virtual nodes. Step 2-3: Store all hash values ​​and corresponding node addresses into the consistent hash allocation table. The range of hash values ​​corresponding to 4-byte hash is [0, 2 32 -1], the subsequent search uses the red-black tree binary search algorithm, the complexity is O(log(vn)), where n is the number of actual nodes and v is the number of virtual nodes corresponding to each actual node; Step 2-4: The manager node digitally signs the consistent hash allocation table and sends it to all server nodes. After receiving it, the server node checks the signature and accepts the hash allocation plan after confirming that it is correct. The initialization phase of the entire cluster ends and starts to accept client requests and provide services. Step 3: The client sends a request to the server node. After judging the server node ID, the corresponding server node processes it and records the load situation. In step 3, the client randomly sends the request to the adjacent server node. If the client does not have the server node information locally, it will first obtain the server node configuration information of the cluster from the manager node. After testing the connection status of each server node, the client selects the server node with the smallest round-trip delay to send the request. After receiving the request, the server node performs MD5 hashing on the request to obtain a 16-byte hash value, and the last 4 bytes are used as the consistent hash value of the request, recorded as K. Then, a binary search is used in the consistent hash allocation table to locate the hash value S of the first virtual node after K, and then the actual node ID corresponding to the virtual node with the hash value S is determined. If the actual node is the current server node ID, it will be processed by the current server node. Otherwise, the client request is forwarded to the server node to which the request correctly belongs, and the query and processing process is repeated. When the correct server node processes the request, the load statistics module will be called to record the load status of the current request. The specific steps to query the consistent hash allocation table to obtain the ID of the corresponding server are: Step 3-1: Cluster nodes register with the manager and discover each other to build a P2P network and configure a consistent hash table. Each server node maintains a copy of the consistent hash table locally. After initialization, the cluster provides services to the outside world. Step 3-2: A server node receives a client request, serializes the request into binary form, performs MD5 hash digest on the request, obtains a hash value K, then searches the hash table for the first value S greater than or equal to K, and obtains the bound server node ID with index S from the consistent hash allocation table; Step 3-3: If the ID is the ID of this server node, the request will be processed by this node and the load of this node will be increased. Otherwise, it will be forwarded to the corresponding server node ID to ensure that the request is processed by a unique server node. Step 4: The manager node starts the load collection and statistics tasks in parallel and initiates a load collection request to all server nodes. After receiving the load collection request, the server node counts and encapsulates the load information and sends the encapsulated load information to the manager node. Step 5: After receiving the replies from 2f+1 server nodes, the manager node performs load statistics, sorts the load of each server node, and multiplies the median of the sorted load by the total number of server nodes in the system to obtain the estimated value of the overall system load; Step 6: The manager node distributes the estimated value of the overall system load and the replies to the 2f+1 load collection requests used for the estimation to all server nodes. The entire cluster reaches a consensus on the estimated value, and the load statistics are completed.

2. The load sensing method according to claim 1, characterized in that: In step 1, the cluster is a collection of server nodes with the same responsibilities in the network; the node identity information includes node role, node IP, node ID and public and private keys; the node roles include server nodes and manager nodes, the node ID is used to uniquely identify the server node, and the public and private keys are used to encrypt messages and identity authentication.

3. The load sensing method according to claim 1, characterized in that: In step 1, the cluster includes several server nodes, one of which serves as a manager node; when the manager node goes down, the cluster network will elect an administrator node from the remaining server nodes.

4. The load sensing method according to claim 2, characterized in that: The server node is responsible for processing requests from the client, counting the load processed by the node, and reporting to the manager node. The server node includes a hash query module, a request processing module, and a load statistics module; The manager node is responsible for monitoring and managing the entire system, maintaining connection with the server node through heartbeats regularly, initializing and updating the consistent hash load balancing algorithm to ensure consistency in the request division and attribution of all nodes in the cluster, expanding and shrinking the cluster nodes, and providing node discovery services. The manager node includes a discovery service module, a load estimation module, and a load balancing module.

5. The load sensing method according to claim 1, characterized in that: In step 4, the manager node performs load statistics tasks every epoch, and sets a timer to determine the time to start load collection. At the beginning of each epoch, it initiates a load collection request to all server nodes. After receiving the load collection request, the server nodes including the manager node start to collect statistics on the load information from the last statistics to the current time. The load information includes the load, the CPU execution time consumed to process the request, the IO throughput and various delays. After the server node completes the statistics, it encapsulates the load information into a message body format, digitally signs it and sends it to the manager node.

6. The load sensing method according to claim 1, characterized in that: In step 5, after the manager node receives replies from 2f+1 server nodes including itself, it starts to count the load. The total number of nodes in the cluster is 3f+1, of which f are Byzantine nodes. First, the load of each server node is sorted, and the median of the load in the sorted replies is taken as the average load of the entire system, and multiplied by the total number of server nodes in the system to obtain an estimated value of the overall system load.

Images