A large-scale cluster management method based on P2P
Through P2P technology, subclusters are built, distributed and merged task results in large-scale clusters, and network blocking and high server performance requirements in large-scale cluster management under traditional Client-Server mode are solved, and efficient data processing and transmission are achieved.
Patent Information
- Application Number
- CN202111501826.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-09
AI Technical Summary
The traditional Client-Server model causes network blockage and high server performance requirements in large-scale cluster management, and cannot effectively handle the problems of big data files.
Using P2P technology, select the cluster main node and computing node, build a subcluster through P2P technology, the main node issues test tasks and data processing tasks, and the nodes in the subcluster themselves distribute and merge the task results to reduce the number of server processing and network transmission.
Through early data aggregation and merging, the number of server processing is reduced, network blockage is avoided, and the efficiency of large-scale cluster management is improved.
Smart Images

Figure CN114237882B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a large-scale cluster management method based on P2P. Background Art
[0002] In traditional terminal control clusters, the Client-Server model is often used. In this CS model, the server usually sends a new request, which is transmitted to the corresponding server through the network. The client internally parses the accepted request and then sends the execution result to the server using the same network channel. Finally, when the server determines that all machines in the cluster have completed execution, it performs unified aggregation, storage and other business logic.
[0003] The disadvantages of this CS mode are not so obvious in the management of small-scale clusters, but for large-scale clusters (more than 300,000 machines), initiating hundreds of thousands of requests and receiving hundreds of thousands of data at the same time may cause the entire network to crash, and the network requirements for the entire cluster are very high. On the other hand, the server needs to parse the reported data in this large-scale cluster at the same time, and the performance requirements of the server are also high.
[0004] The traditional CS model is not only unsuitable for large-scale cluster management, but also unsuitable for processing tasks with large amounts of data.
[0005] The mature development of P2P (peer to peer) networks enables multiple terminal devices to connect to each other and share the required data, which reduces the pressure on the server and improves the speed at which terminals obtain data to a certain extent.
[0006] Therefore, how to use P2P technology to solve large-scale cluster management and how to use P2P technology to solve the problem of sending large data files in the cluster have become technical problems that need to be solved urgently in the industry. Summary of the invention
[0007] The present invention aims at the technical problems existing in the prior art and provides a large-scale cluster management method based on P2P. The P2P technology is used to reduce the network congestion of the main server, thereby solving the large-scale cluster management problem.
[0008] The technical solution of the present invention to solve the above technical problems is as follows:
[0009] A large-scale cluster management method based on P2P, comprising:
[0010] Select one node from the node cluster as the cluster master node, and the other nodes as computing nodes; the master node sends test tasks to all nodes in the node cluster at a specified time interval T to test the node status. At the same time, the master node is also used to send data processing tasks and collect data uploaded by other nodes;
[0011] Other computing nodes use P2P technology to find nearby nodes and build at least one sub-cluster;
[0012] The cluster master node and computing nodes are started synchronously. The computing node that receives the test task earliest in the sub-cluster is the master computing node of the sub-cluster, and the other computing nodes are slave computing nodes. The master computing node receives and distributes data processing tasks and collects and merges the calculation results of all computing nodes in the sub-cluster and uploads them to the cluster master node.
[0013] Furthermore, a time threshold T1 is set. If the time interval from when the master computing node of a sub-cluster starts to when it receives a test task is greater than the time threshold T1, the master computing node of the sub-cluster is connected to the cluster master node through other sub-clusters.
[0014] Furthermore, the cluster master node determines the computing nodes involved in executing the data processing task based on the node status and data processing task requirements, writes the node information of the computing nodes involved in the execution into the data processing task data packet, and then sends the data processing task data packet to each sub-cluster master computing node.
[0015] Furthermore, after all computing nodes in each sub-cluster receive the data processing task data packet, they parse and determine whether there are tasks in the data packet that their own nodes need to process, retain the tasks that their own nodes need to process, and pass the tasks that need to be processed by other nodes to other computing nodes connected to themselves.
[0016] Furthermore, it is determined whether a computing node has a leaf node. If so, the tasks that need to be processed by other nodes are passed to its leaf node. If there is no leaf node, other tasks except those that need to be processed by its own node are ignored.
[0017] Furthermore, if the current computing node participates in task execution, it waits for T2 time after the task is completed, receives the task execution results of its leaf nodes within the T2 time period and merges them with the task execution results of its own node. After T2 time expires, the merged results are passed to the upper-level node; if the current computing node does not participate in task execution, it waits for T3 time. After T3 time expires, the task execution results of its leaf nodes are passed to the upper-level node.
[0018] Furthermore, the master node has a fixed address.
[0019] Furthermore, if there is a computing node in the node cluster that does not belong to any sub-cluster and has not received a test task issued by the cluster master node after T1 time, the computing node establishes a connection with the master node according to the fixed address of the cluster master node.
[0020] Furthermore, when each computing node uses P2P technology to search for nearby nodes, it uses a pre-built-in private key to verify the legitimacy of the connectable computing node.
[0021] The beneficial effects of the present invention are as follows: all nodes of a node cluster are divided into multiple sub-clusters through P2P technology, and the data in the entire cluster can be divided into small block data sets by adopting the method of pre-aggregation and pre-merger of data, which greatly reduces the amount of data that the server needs to process and also avoids the transmission of large data in the network, resulting in network congestion. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of node cluster division provided in an embodiment of the present invention;
[0023] Figure 2 A schematic diagram of the internal structure of a sub-cluster and its relationship with the master node provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.
[0025] The embodiment of the present invention provides a large-scale cluster management method based on P2P. First, a node is selected from the node cluster as the cluster master node, and the other nodes are used as computing nodes. The other computing nodes use P2P technology to find nearby nodes to build at least one sub-cluster. The cluster master node and the computing nodes are started synchronously. The computing node that receives the test task earliest in the sub-cluster is the master computing node of the sub-cluster, and the other computing nodes are slave computing nodes. The network structure is as follows: Figure 1 Figure 2 shown.
[0026] The cluster master node has a fixed address. It is the request issuer in the entire system and the final convergence point of all information. It is used to issue test tasks or data processing tasks, collect data uploaded by other nodes, and is responsible for connecting with external business logic.
[0027] The master node sends test tasks to all nodes in the node cluster at a specified time interval T to test the node status.
[0028] A time threshold T1 is set. If the time interval from when the master computing node of a sub-cluster is started to when it receives a test task is greater than the time threshold T1, the master computing node of the sub-cluster is connected to the master node of the cluster through other sub-clusters.
[0029] If there is a computing node in the node cluster that does not belong to any sub-cluster and has not received a test task issued by the cluster master node after T1 time, the computing node establishes a connection with the master node according to the fixed address of the cluster master node.
[0030] When all nodes are started, the computing nodes use P2P technology in the local area network to search for other nearby nodes. When a new connectable node is found, the pre-built-in private key is used to verify its legitimacy to prevent malicious nodes from connecting.
[0031] The above method ensures that all started nodes in the entire cluster can connect to the master node. The method of automatically searching for nearby nodes through P2P can solve the problem of network congestion caused by all machines in the cluster requesting the server.
[0032] When receiving a task from the master node, the computing node will first determine whether it is within the scope of the task. If not, it will pass the task to other machine nodes in the local area network. If it is, it will execute the corresponding task. After the task is completed, it will determine whether it has received the result information passed by other nodes. If so, it will merge the result information it has generated with the existing structure and pass it to other upper-level nodes. If not, it will directly pass the data to the upper-level node.
[0033] Specifically, assume that there are 4 clusters, a master node (node M), and a single node (node N1). The entire architecture is as follows Figure 1 As shown, the P2P network can be used for node communication within a cluster, and can also be used for communication between clusters.
[0034] Master node: You need to establish a master node, such as Figure 1 The master node is the request sender in the entire system and the final convergence point of all information. The master node is responsible for connecting with the external business logic. When a new task needs to be sent, the master node first calculates which nodes in the entire cluster need to execute the task and which nodes can ignore the task; when receiving the result returned by the node, the master node is also responsible for calculating the progress of the current task, which nodes have errors, and which nodes have not responded. The master node has a fixed address and sends a test task every T.
[0035] Single node: In the entire cluster, there may be some nodes that do not belong to any local area network, which will cause them to not receive any requests. Because it is known that the master node will send a test task every T interval, if no task is received after the waiting time threshold T1, it will directly connect according to the fixed address of the master node (node M).
[0036] Cluster: In Figure 1 In the example, each cluster uses P2P to communicate with each other. When the cluster communicates with node M, it will connect to the node with the lowest communication delay between each node in the cluster and node M. This connection is responsible for receiving tasks issued by node M and distributing them to the cluster; as well as aggregating the results in the cluster and sending them to node M. Assume that there are 5 nodes in cluster 1, and its internal architecture is as follows: Figure 2 shown.
[0037] When all nodes are located in the same LAN, P2P technology can easily form the entire LAN into a small cluster. In this small cluster, the most suitable node in the entire cluster is selected to be responsible for external communication based on the current load or communication delay with the main node.
[0038] The steps for processing tasks within the small cluster and on each node are as follows:
[0039] 1) Node M sends a new task Task1
[0040] 2) Node N2 receives Task 1 sent by Node M, and passes Task 1 to Node N3 and Node N4;
[0041] 3) Node N2 also analyzes task Task1, and then determines whether it (node N2) needs to execute the task.
[0042] 4) If node N2 needs to execute, it will wait for a period of time T2 after completing the task; after time T2 times out, it will summarize the results received during this period and finally send them to node M.
[0043] 5) If node N2 does not need to execute, it will wait for a period of time T3 after receiving the results sent by other receivers. After time T3 times out, it will summarize the results received during this period and finally send them to node M.
[0044] 6) The processing of tasks by node N3 is similar to that by node N2.
[0045] 7) After receiving the task, nodes N4 and N5 find that there is no other node connected to them (leaf nodes), so they only need to determine whether they need to execute it. If not, they ignore the task. If they need to execute it, after the execution is completed, they directly pass the result to the node of the next level.
[0046] This method uses the method of pre-aggregation and pre-merger of data to divide the data in the entire cluster into small blocks of data sets, greatly reducing the amount of data that the server needs to process and also avoiding the transmission of large data in the network, which may cause network congestion.
[0047] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0048] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A large-scale cluster management method based on P2P, characterized in that: include: Select one node from the node cluster as the cluster master node, and the other nodes as computing nodes; The cluster master node sends test tasks to all nodes in the node cluster at a specified time interval T to test the node status. At the same time, the cluster master node is also used to send data processing tasks and collect data uploaded by other nodes; Other computing nodes use P2P technology to find nearby nodes and build at least one sub-cluster; The cluster master node and computing nodes are started synchronously. The computing node that receives the test task earliest in the sub-cluster is the master computing node of the sub-cluster, and the other computing nodes are slave computing nodes. The master computing node receives and distributes data processing tasks and collects and merges the calculation results of all computing nodes in the sub-cluster and uploads them to the cluster master node. The cluster master node determines the computing nodes that participate in the data processing task according to the node status and data processing task requirements, writes the node information of the computing nodes involved in the execution into the data processing task data packet, and then sends the data processing task data packet to the master computing nodes of each sub-cluster; After receiving the data processing task data packet, all computing nodes in each sub-cluster parse and determine whether there are tasks that their own nodes need to process in the data packet, retain the tasks that their own nodes need to process, and pass the tasks that need to be processed by other nodes to other computing nodes connected to themselves; Determine whether a computing node has a leaf node. If so, pass the tasks that need to be processed by other nodes to its leaf node. If there is no leaf node, ignore other tasks except those that need to be processed by the node itself. If the current computing node participates in the task execution, it will wait for T2 time after the task is completed, receive the task execution results of its leaf nodes within the T2 time period and merge them with the task execution results of its own node. After T2 time expires, the merged results will be passed to the upper-level node; if the current computing node does not participate in the task execution, it will wait for T3 time. After T3 time expires, the task execution results of its leaf nodes will be passed to the upper-level node; A time threshold T1 is set. If the time interval from when the master computing node of a sub-cluster is started to when it receives a test task is greater than the time threshold T1, the master computing node of the sub-cluster is connected to the master node of the cluster through other sub-clusters.
2. The method according to claim 1, characterized in that: The cluster master node has a fixed address.
3. The method according to claim 2, characterized in that If there is a computing node in the node cluster that does not belong to any sub-cluster and has not received a test task sent by the cluster master node after T1 time, the computing node establishes a connection with the cluster master node according to the fixed address of the cluster master node.
4. The method according to claim 1, characterized in that: When each computing node uses P2P technology to find nearby nodes, it uses a pre-built-in private key to verify the legitimacy of the connectable computing nodes.
