Multi-node file aggregation load balancing method and system oriented to edge cloud collaboration

Abstract

The invention provides a multi-node file aggregation load balancing method and system oriented to edge cloud collaboration, and the method comprises the steps: obtaining a load coefficient of each node in a multi-layer tree structure for file aggregation, and summarizing the operation information of each node to a corresponding sub-tree management node, adjusting the load coefficient of the node according to the operation information by using the sub-tree management node; obtaining an optimal path for file convergence by using the load coefficients of all the nodes passed by the nodes when the nodes can reach the destination node, wherein the multi-layer tree structure comprises a plurality of sub-tree management nodes; according to the method and the system, the load coefficients of the nodes are dynamically adjusted through the sub-tree management nodes, and the optimal path is determined according to the adjusted load coefficients for file aggregation, so that the effect of global load balancing is achieved, the transmission efficiency of the whole system is effectively improved, and the transmission efficiency is improved by utilizing node transmission of a multi-layer tree structure. And the data throughput is improved.

Description

Technical Field

[0001] The present invention relates to the field of load balancing technology, and in particular to a multi-node file aggregation load balancing method and system for edge-cloud collaboration. Background Art

[0002] With the continuous deepening of Internet technology, the demand for safe and fast access to digital assets distributed around the world is increasing day by day. Usually, a centralized approach is adopted for file aggregation. However, with the expansion of organizational management scope and business services, the demand for data traffic and computing power has grown exponentially, resulting in excessive load on the central node for file aggregation, affecting the operating efficiency of the overall system.

[0003] Usually in edge-cloud collaborative environments, the method of multi-node step-by-step aggregation is used to solve the problem of excessive load on centralized nodes. This method can effectively reduce the load on central nodes and improve network performance. In the multi-node file aggregation method, in order to avoid excessive load on the same node, load balancing technology is used to determine which node to store and process the file.

[0004] Commonly used methods of load balancing are polling and weighting. The polling algorithm will evenly distribute requests to each node, but if the performance of a node server is poor, the server will be overloaded, affecting system performance. The weighting algorithm can allocate different requests according to the server of each node, but if the network is congested or there are problems with the server in the edge-cloud collaborative environment, it cannot be adaptively adjusted and requires manual intervention. Therefore, conventional load balancing technology cannot effectively handle complex multi-level network environments. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention proposes a multi-node file aggregation load balancing method for edge-cloud collaboration, comprising:

[0006] Obtain the load factor of each node in the multi-level tree structure for file aggregation, aggregate the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load factor of the node according to the operation information;

[0007] The load coefficients of all nodes that a node passes through when reaching the destination node are used to obtain the optimal path for file aggregation;

[0008] The multi-level tree structure includes multiple subtree management nodes.

[0009] Preferably, the step of aggregating the operation information of each node to the corresponding subtree management node includes:

[0010] If the parent node of a node in the subtree is not the subtree management node, then the operation information is sent to the parent node of a reachable subtree management node every preset period, and the operation information sent by the subordinate node is forwarded to the parent node;

[0011] Using the parent node of the reachable subtree management node, the received operation information of the node and the subordinate nodes are sent to the subtree management node;

[0012] If the parent node of a node in the subtree is the subtree management node, the operation information of the node itself is sent to the subtree management node in each preset period, and the operation information sent by the subordinate node is forwarded to the subtree management node.

[0013] The process of obtaining the load factor of each node in the file aggregation process includes:

[0014] After each node is connected to a node inside the file aggregation system, the load factor of the node is calculated based on the load parameters of the node within a preset time period;

[0015] The load parameters of the node include one or more of the following: the average round-trip time from the node to each parent node, the packet loss rate of the node within a preset time period, the number of files currently to be processed by the node, the number of server CPU cores of the node, the current CPU operating frequency, the current CPU occupancy rate, the server bandwidth, and the current network bandwidth occupancy rate of the node.

[0016] Preferably, the calculation formula of the node load factor is:

[0017]

[0018] Where LBC represents the load factor of the node; RTT represents the average round-trip time from the node to each parent node; PDR represents the packet loss rate of the node in the preset time period; Q represents the number of files currently to be processed by the node; N cpu Indicates the number of CPU cores of the node server, C cpu Indicates the CPU operating frequency, U cpu Indicates the current CPU usage of the node; C bw Indicates the server bandwidth, U bw represents the current network bandwidth occupancy rate of the node; k1 represents the weight of the node's RTT; k2 represents the weight of the number of files to be processed Q at the node; k3 represents the weight of the node's PDR; k4 represents The weight of The weights of each load parameter k1, k2, k3, k4 and k5 are determined according to the user's business needs.

[0019] Preferably, the using the subtree management node to adjust the load factor of the node according to the operation information includes:

[0020] If the subtree management node finds that the load of a node is abnormal based on the operation information, it adjusts the weight corresponding to the load parameter of the node according to the following formula:

[0021]

[0022] Among them, k ja represents the weight corresponding to the jth load parameter of the node after the subtree management node is adjusted; k jf represents the weight corresponding to the jth load parameter of the node before the subtree management node is adjusted; α represents the learning rate set by the subtree management node; W j Indicates the weight corresponding to the jth load parameter set by the user; U j represents the value of the jth load parameter; T j Indicates the preset j-th load parameter threshold; the value of j is 1, 2, 3, 4, 5.

[0023] Preferably, obtaining the optimal path by using the load coefficients of all nodes that the node passes through when reaching the destination node includes:

[0024] Get the path load coefficient table of all parent nodes of the node;

[0025] If the destination node of the packet to be forwarded in the node is the parent node, the optimal path is direct forwarding;

[0026] Otherwise, the packet to be forwarded is placed in a waiting queue, and the initial optimal path to the destination node is obtained according to the path load coefficient table of the node where the packet is forwarded;

[0027] If there are no nodes with abnormal loads among all the nodes passed by the initial optimal path, the initial optimal path will be taken as the optimal path;

[0028] Otherwise, multiple selection coefficients of the node are obtained by using the load coefficient of each parent node of the node's superior that can reach the destination node of the packet to be forwarded, and the load coefficient of the path from each parent node to the destination node;

[0029] The path consisting of the parent node corresponding to the node when the node obtains the minimum selection coefficient and the path from the parent node to the destination node is taken as the optimal path.

[0030] Preferably, obtaining the initial optimal path and path load coefficient for each ancestor node from the node includes:

[0031] Traverse all paths from a node to its ancestor node, add up the load coefficients of all nodes on each path, and get the path load coefficient of each path;

[0032] The path with the smallest path load coefficient is taken as the initial optimal path.

[0033] Preferably, the calculation formula of the selection coefficient of the node is:

[0034] S i =q 1i *LBC i +q 2i *LBC iid

[0035] Among them, S i Indicates the selection coefficient of the i-th parent node of the current node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; q 1i represents the selection weight corresponding to the load factor of the i-th parent node; q 2i represents the selection weight corresponding to the path load coefficient from the i-th parent node to the destination node;

[0036] The weight q 1i ,q 2i Determined by the corresponding subtree management node.

[0037] Preferably, the selection weight q 1i ,q 2i The corresponding subtree management node is determined according to the following formula:

[0038]

[0039]

[0040] Among them, q 1i represents the selection weight corresponding to the updated load factor of the i-th parent node; q 2i represents the selection weight corresponding to the updated path load coefficient from the i-th parent node to the destination node; q 1if represents the selection weight corresponding to the preset load factor of the i-th parent node; q 2if represents the selection weight corresponding to the preset path load coefficient from the i-th parent node to the destination node; α represents the learning rate set by the subtree management node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; n represents the number of transit nodes between the i-th parent node and the destination node.

[0041] Based on the same inventive concept, the present invention provides a multi-node file aggregation load balancing system for edge-cloud collaboration, including:

[0042] Load coefficient determination module: used to obtain the load coefficient of each node in the multi-level tree structure for file aggregation, aggregate the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load coefficient of the node according to the operation information;

[0043] Path determination module: Utilizes the load coefficients of all nodes that a node passes through when reaching the destination node to obtain the optimal path for file aggregation.

[0044] In the load factor determination module, the operation information of each node is aggregated to the corresponding subtree management node, including:

[0045] If the parent node of a node in the subtree is not the subtree management node, then the operation information is sent to the parent node of a reachable subtree management node every preset period, and the operation information sent by the subordinate node is forwarded to the parent node;

[0046] Using the parent node of the reachable subtree management node, the received operation information of the node and the subordinate nodes are sent to the subtree management node;

[0047] If the parent node of a node in the subtree is the subtree management node, the operation information of the node itself is sent to the subtree management node in each preset period, and the operation information sent by the subordinate node is forwarded to the subtree management node.

[0048] Preferably, the load factor of each node in the process of acquiring the file aggregation includes:

[0049] After each node is connected to a node inside the file aggregation system, the load factor of the node is calculated based on the load parameters of the node within a preset time period;

[0050] The load parameters of the node include one or more of the following: the average round-trip time from the node to each parent node, the packet loss rate of the node within a preset time period, the number of files currently to be processed by the node, the number of server CPU cores of the node, the current CPU operating frequency, the current CPU occupancy rate, the server bandwidth, and the current network bandwidth occupancy rate of the node.

[0051] Preferably, the calculation formula of the node load factor is:

[0052]

[0053] Where LBC represents the load factor of the node; RTT represents the average round-trip time from the node to each parent node; PDR represents the packet loss rate of the node in the preset time period; Q represents the number of files currently to be processed by the node; Ncpu Indicates the number of CPU cores of the node server, C cpu Indicates the CPU operating frequency, U cpu Indicates the current CPU usage of the node; C bw Indicates the server bandwidth, U bw represents the current network bandwidth occupancy rate of the node; k1 represents the weight of the node's RTT; k2 represents the weight of the number of files to be processed Q at the node; k3 represents the weight of the node's PDR; k4 represents The weight of The weights of each load parameter k1, k2, k3, k4 and k5 are determined according to the user's business needs.

[0054] Preferably, the using the subtree management node to adjust the load factor of the node according to the operation information includes:

[0055] If the subtree management node finds that the load of a node is abnormal based on the operation information, it adjusts the weight corresponding to the load parameter of the node according to the following formula:

[0056]

[0057] Among them, k ja represents the weight corresponding to the jth load parameter of the node after the subtree management node is adjusted; k jf represents the weight corresponding to the jth load parameter of the node before the subtree management node is adjusted; α represents the learning rate set by the subtree management node; W j Indicates the weight corresponding to the jth load parameter set by the user; U j represents the value of the jth load parameter; T j Indicates the preset j-th load parameter threshold; the value of j is 1, 2, 3, 4, 5.

[0058] Preferably, obtaining the optimal path by using the load coefficients of all nodes that the node passes through when reaching the destination node includes:

[0059] Get the path load coefficient table of all parent nodes of the node;

[0060] If the destination node of the packet to be forwarded in the node is the parent node, the optimal path is direct forwarding;

[0061] Otherwise, the packet to be forwarded is placed in a waiting queue, and the initial optimal path to the destination node is obtained according to the path load coefficient table of the node where the packet is forwarded;

[0062] If there are no nodes with abnormal loads among all the nodes passed by the initial optimal path, the initial optimal path will be taken as the optimal path;

[0063] Otherwise, multiple selection coefficients of the node are obtained by using the load coefficient of each parent node of the node's superior that can reach the destination node of the packet to be forwarded, and the load coefficient of the path from each parent node to the destination node;

[0064] The path consisting of the parent node corresponding to the node when the node obtains the minimum selection coefficient and the path from the parent node to the destination node is taken as the optimal path.

[0065] In the path determination module, the initial optimal path and path load coefficient of each ancestor node that can be reached by the node are obtained, including:

[0066] Traverse all paths from a node to its ancestor node, add up the load coefficients of all nodes on each path, and get the path load coefficient of each path;

[0067] The path with the smallest path load coefficient is taken as the initial optimal path.

[0068] Preferably, the calculation formula of the selection coefficient of the node is:

[0069] S i =q 1i *LBC i +q 2i *LBC iid

[0070] Among them, S i Indicates the selection coefficient of the i-th parent node of the current node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; q 1i represents the selection weight corresponding to the load factor of the i-th parent node; q 2i represents the selection weight corresponding to the path load coefficient from the i-th parent node to the destination node;

[0071] The weight q 1i ,q 2i Determined by the corresponding subtree management node.

[0072] Preferably, the selection weight q 1i ,q 2i The corresponding subtree management node is determined according to the following formula:

[0073]

[0074]

[0075] Among them, q 1i represents the selection weight corresponding to the updated load factor of the i-th parent node; q 2irepresents the selection weight corresponding to the updated path load coefficient from the i-th parent node to the destination node; q 1if represents the selection weight corresponding to the preset load factor of the i-th parent node; q 2if represents the selection weight corresponding to the preset path load coefficient from the i-th parent node to the destination node; α represents the learning rate set by the subtree management node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; n represents the number of transit nodes between the i-th parent node and the destination node.

[0076] Compared with the closest prior art, the present invention has the following beneficial effects:

[0077] The present invention provides a multi-node file aggregation load balancing method and system for edge-cloud collaboration. The multi-level tree structure of file aggregation includes multiple subtree management nodes. The load coefficient of each node in the file aggregation process is obtained, and the operation information of each node is aggregated to the corresponding subtree management node. The subtree management node is used to adjust the load coefficient of the node according to the operation information; the load coefficient of all nodes passed by the node when reaching the destination node is used to obtain the optimal path for file aggregation; the subtree management node is used to dynamically adjust the load coefficient of the node, and the optimal path to the destination node is determined according to the adjusted load coefficient to perform file aggregation, thereby achieving the effect of global load balancing, effectively improving the transmission efficiency of the entire system, and utilizing the node transmission of the multi-level tree structure to improve data throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flow chart of a multi-node file aggregation load balancing method for edge-cloud collaboration provided by the present invention;

[0079] Figure 2 A schematic diagram of a multi-level tree structure for file aggregation of the present invention;

[0080] Figure 3 A detailed flow chart of the document collection for the present invention;

[0081] Figure 4 A schematic diagram of updating a path load coefficient table in a file aggregation process of the present invention;

[0082] Figure 5 A structural schematic diagram of a multi-node file aggregation load balancing system for edge-cloud collaboration provided by the present invention. DETAILED DESCRIPTION

[0083] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0084] Embodiment 1:

[0085] The present invention provides a multi-node file aggregation load balancing method for edge-cloud collaboration, such as Figure 1 As shown, including:

[0086] S1. Obtain the load factor of each node in the multi-level tree structure for file aggregation, aggregate the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load factor of the node according to the operation information;

[0087] The multi-level tree structure includes multiple subtree management nodes.

[0088] S2. Utilize the load coefficients of all nodes that a node passes through when reaching the destination node to obtain the optimal path for file aggregation.

[0089] In step S1, the multi-level tree structure for file aggregation includes multiple subtree management nodes, specifically including:

[0090] The nodes for file aggregation are multi-level tree structures, such as Figure 2 As shown, multiple subtree management nodes are established in the multi-level tree structure according to the actual architecture of the enterprise to manage the nodes within the corresponding subtrees. For example, node 101 is the root node, that is, the headquarters management page in the enterprise, and node 303 is the subtree management node, that is, a regional management page in the enterprise.

[0091] File aggregation modes include push aggregation mode and pull aggregation mode. The push aggregation mode means that the lower-level node actively sends files to the upper-level node under set circumstances; the pull aggregation mode means that the upper-level node achieves aggregation by initiating a file query request to the lower-level node in its subtree and obtaining a file response.

[0092] In step S1, the load factor of each node in the multi-level tree structure for file aggregation is obtained, the operation information of each node is aggregated to the corresponding subtree management node, and the subtree management node is used to adjust the load factor of the node according to the operation information, such as Figure 3 As shown, specifically including:

[0093] S11. Obtain the load factor of each node in the file aggregation process.

[0094] After each node is connected to a node inside the file aggregation system, the load factor of the node is calculated based on the load parameters of the node within a preset time period;

[0095] All parent nodes of the node are used to send a path load coefficient table to the node, wherein the path load coefficient table includes an initial optimal path to each ancestor node, a load coefficient of the parent node, and a path load coefficient to the corresponding ancestor node.

[0096] The load parameters of the node include one or more of the following: the average round-trip time from the node to each parent node, the packet loss rate of the node within a preset time period, the number of files currently to be processed by the node, the number of server CPU cores of the node, the current CPU operating frequency, the current CPU occupancy rate, the server bandwidth, and the current network bandwidth occupancy rate of the node. The load parameters are parameters used to measure the degree of system load.

[0097] The load factor of the node is calculated using the data of multiple load parameters obtained within a preset time period. The length of the time period is set according to the specific implementation situation. The calculation formula of the node load factor is:

[0098]

[0099] Where LBC represents the load factor of the node; RTT represents the average round-trip time from the node to each parent node; PDR represents the packet loss rate of the node in the preset time period; Q represents the number of files currently to be processed by the node; N cpu Indicates the number of CPU cores of the node server, C cpu Indicates the CPU operating frequency, U cpu Indicates the current CPU usage of the node; C bw Indicates the server bandwidth, U bw represents the current network bandwidth occupancy rate of the node; k1 represents the weight of the node's RTT; k2 represents the weight of the number of files to be processed Q at the node; k3 represents the weight of the node's PDR; k4 represents The weight of The weight of .

[0100] Among them, the weights k1, k2, k3, k4 and k5 of each parameter are determined according to the user's business needs and adjusted through the subtree management node of the subtree where the current node is located. For example, for business scenarios with higher latency requirements, a higher weight can be set for RTT, and k1, k2, k3, k4 and k5 can be set to [0.5, 0.1, 0.1, 0.1, 0.2] respectively.

[0101] S12. Use the subtree management node to obtain the operation information of each node in the subtree.

[0102] If the parent node of a node in the subtree is not the subtree management node, then the operation information is sent to the parent node of a reachable subtree management node every preset period, and the operation information sent by the subordinate node is forwarded to the parent node;

[0103] Using the parent node of the reachable subtree management node, the received operation information of the node and the subordinate nodes are sent to the subtree management node;

[0104] If the parent node of a node in the subtree is the subtree management node, the operation information of the node itself is sent to the subtree management node in each preset period, and the operation information sent by the subordinate node is forwarded to the subtree management node.

[0105] Among them, the operation information of the node includes one or more of the following: the operation status of the node, the weight of the path load coefficient calculation and the stored digital asset status; the operation status of the node includes the operation status of the server CPU, memory, network and other operation load messages, and the stored digital asset status refers to the stored files.

[0106] The subtree management node summarizes the operation information reported by each subordinate node within the subtree. The summarized operation messages include the node operation status, node queue, weight calculated by the path load coefficient and the stored digital asset status. This information is summarized to form a subtree knowledge base.

[0107] After the subtree management node obtains the operation information of each node within the subtree to form a word count knowledge base, it displays the subtree knowledge base to the outside with intuitive icons, uses the subtree management node to prune the subtree, hides the details within the subtree to the outside, reduces the amount of upstream data, and improves the system operation efficiency.

[0108] The subtree management node obtains the overall load of the subtree by calculating the median and average of various load parameters of each node within the subtree, and then sends it to the upper-level node.

[0109] If the subtree management node does not receive the operation information of a node inside the subtree for two consecutive cycles, the node will be judged as a lost node. The subtree management node will issue an immediate update command to remove the information of the node in the path load coefficient table, update the path load coefficient table, and bypass the lost node when issuing the path load coefficient table.

[0110] S13, using the subtree management node to adjust the load factor of the node according to the operation information, specifically including:

[0111] If the subtree management node obtains that the load of a node is abnormal according to the operation information, the subtree management node adjusts the load factor of the node by adjusting the weight corresponding to the load parameter of the node; otherwise, there is no need to adjust the load factor of the node.

[0112] The weights k1, k2, k3, k4 and k5 of each load parameter corresponding to the load coefficient of the computing node are unified by k j express;

[0113] If the subtree management node receives information about abnormal load of a node, the subtree management node adjusts the weight corresponding to the load parameter of the node according to the following formula:

[0114]

[0115] Among them, k ja represents the weight corresponding to the jth load parameter of the node after the subtree management node is adjusted; k jf represents the weight corresponding to the jth load parameter of the node before the subtree management node is adjusted; α represents the learning rate set by the subtree management node; W j Indicates the weight corresponding to the jth load parameter set by the user; U j represents the value of the jth load parameter; T j Indicates the preset j-th load parameter threshold; the value of j is 1, 2, 3, 4, 5. If the value of the load parameter does not satisfy the formula for weight adjustment, no adjustment is made.

[0116] S2. Utilize the load coefficients of all nodes that a node passes through when reaching the destination node to obtain the optimal path for file aggregation.

[0117] After the current node receives the file, it segments the file according to the segment size set by the user, and puts each segment of the file into waiting queues at different levels. The file segments in different waiting queues are transmitted according to different load balancing strategies, that is, the initial optimal path is selected for transmission.

[0118] The optimal path is obtained by using the load coefficients of all nodes that a node passes through when reaching the destination node, including:

[0119] Get the path load coefficient table of all parent nodes of the node;

[0120] If the destination node of the packet to be forwarded in the node is the parent node, the optimal path is direct forwarding;

[0121] Otherwise, the packet to be forwarded is placed in a waiting queue, and the initial optimal path to the destination node is obtained according to the path load coefficient table of the node where the packet is forwarded;

[0122] If there are no nodes with abnormal loads among all the nodes passed by the initial optimal path, the initial optimal path will be taken as the optimal path;

[0123] Otherwise, multiple selection coefficients of the node are obtained by using the load coefficient of each parent node of the node's superior that can reach the destination node of the packet to be forwarded, and the load coefficient of the path from each parent node to the destination node;

[0124] The path consisting of the parent node corresponding to the node when the node obtains the minimum selection coefficient and the path from the parent node to the destination node is taken as the optimal path.

[0125] Among them, whether there is a node with abnormal load among all the nodes passed through in the initial optimal path is judged by the subtree management node corresponding to each node, and the subtree management node can obtain the operating status of each node in the corresponding subtree.

[0126] The path load coefficient table of the node includes the initial optimal path for the node to reach each ancestor node, the path load coefficient and the load coefficient of the node.

[0127] Get the initial optimal path and path load factor of each ancestor node reachable by the node, including:

[0128] Traverse all paths from a node to its ancestor node, add up the load coefficients of all nodes on each path, and get the path load coefficient of each path;

[0129] The path with the smallest path load coefficient is taken as the initial optimal path.

[0130] The calculation formula for the selection coefficient of node selection is:

[0131] S i =q 1i *LBC i +q 2i *LBC iid

[0132] Among them, S i Indicates the selection coefficient of the i-th parent node of the current node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; q 1i represents the selection weight corresponding to the load factor of the i-th parent node; q 2i represents the selection weight corresponding to the path load coefficient from the i-th parent node to the destination node; the selection weight q 1i ,q 2i Determined by the corresponding subtree management node.

[0133] The selection weight q 1i ,q 2i The corresponding subtree management node is determined according to the following formula:

[0134]

[0135]

[0136] Among them, q 1irepresents the selection weight corresponding to the updated load factor of the i-th parent node; q 2i represents the selection weight corresponding to the updated path load coefficient from the i-th parent node to the destination node; q 1if represents the selection weight corresponding to the preset load factor of the i-th parent node; q 2if represents the selection weight corresponding to the preset path load coefficient from the i-th parent node to the destination node; α represents the learning rate set by the subtree management node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; n represents the number of transit nodes between the i-th parent node and the destination node.

[0137] After the current node obtains the optimal path for file aggregation, it updates the load coefficient of each node that the file passes through during aggregation. The update formula of the node load coefficient is:

[0138] LBC a =LBC f +p1*SIZE pkt +p2*RTT pkt

[0139] Among them, LBC a Indicates the load factor of the node after file aggregation; LBC f Indicates the load factor of the node before file aggregation; SIZE pkt Indicates the size of the forwarded packet; RTT pkt It represents the round trip time between nodes when the forwarding packet is transmitted; p1 and p2 represent the weights corresponding to the size of the forwarding packet and the round trip time respectively, and the weights p1 and p2 are determined by the subtree management node of the subtree.

[0140] When files are aggregated, the next node will respond successfully after completing packet reception or packet forwarding, and then the temporary copy of the packet of the current node can be deleted; if it fails, a retransmission exception is thrown, and the current node that receives the retransmission exception retransmits the content. If the load factor of a node with excessive load still does not decrease after multiple load balancing adjustments (the number of times is preset by yourself) using the subtree management node, then the node is a vulnerable point of the system, and the subtree management node is used for alarm processing.

[0141] Subtree management nodes can achieve reliable access control. Since file aggregation is performed from bottom to top, subtree management nodes at all levels can maintain the internal node status of the subtree by themselves, control the request scope of the subtree internal pull mode and other permission settings, and will not cause confidentiality threats.

[0142] Embodiment 2:

[0143] The present invention provides a multi-node file aggregation load balancing method for edge-cloud collaboration, and an example of specific implementation steps is as follows: Figure 2 , Figure 4 As shown, including:

[0144] Root node 101, subtree management node 303, edge nodes 301, 302, 501, node 301 receives the file and needs to forward it upward.

[0145] The root node 101 is used to send down the path load coefficient table step by step. The first row of the path load coefficient table of each node represents the load coefficient of the node itself at this time. Then it is sent down step by step. The path load coefficient of each node is calculated according to the algorithm to obtain the path load coefficient table.

[0146] Node 301 receives the large file A whose destination node is the root node 101. It splits the large file A into two fragments, a and b, and puts the two fragments into the 1-layer queue. The parent node of the optimal path for node 301 to reach the root node is 201, so it sends fragment a to node 201, and then updates the path load coefficient table.

[0147] For shard 2, Figure 3 As shown, since the path load coefficient from node 201 to destination node 101 is greater than the path load coefficient from node 202 to destination node 101 after updating, fragment 2 selects parent node 202 for sending.

[0148] like Figure 2 As shown, node 303 is the subtree management node of nodes 303, 401, 402, and 501. Subtree management node 303 manages the nodes inside the subtree with it as the root, obtains the operation information of all nodes inside the subtree, uniformly summarizes and builds a subtree knowledge base, hides the internal details of the subtree from the outside through pruning operations, and sends the summarized operation information inside the subtree to the upper-level node, reducing the amount of uplink data.

[0149] The pruning operation means, for example, that node 401 has file B and node 402 has file C. Then, when 303 sends the operation information to the superior, it informs the superior node that node 303 has files B and C. The operation status is the same.

[0150] The subtree management node 303 dynamically adjusts the calculation weight of the selection coefficient in real time according to the load conditions of each path, so that the file aggregation reaches a local optimal solution and implements a global load balancing strategy.

[0151] For example, if node 501 forwards all files to node 401 for some reason, resulting in abnormal load, subtree management node 302 will issue a command to make node 401 discard the files at the end of its queue, thereby increasing node 501's selection weight q for going to node 401. 1i, so that the obtained selection coefficient becomes larger, and finally other paths are selected for file aggregation.

[0152] Embodiment 3:

[0153] Based on the same concept, the present invention also provides a multi-node file aggregation load balancing system for edge-cloud collaboration, such as Figure 5 As shown, including:

[0154] Load coefficient determination module: used to obtain the load coefficient of each node in the multi-level tree structure for file aggregation, summarize the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load coefficient of the node according to the operation information; the multi-level tree structure includes multiple subtree management nodes.

[0155] Path determination module: Utilizes the load coefficients of all nodes that a node passes through when reaching the destination node to obtain the optimal path for file aggregation.

[0156] Specifically, in the load factor determination module, the step of aggregating the operation information of each node to the corresponding subtree management node includes:

[0157] If the parent node of a node in the subtree is not the subtree management node, then the operation information is sent to the parent node of a reachable subtree management node every preset period, and the operation information sent by the subordinate node is forwarded to the parent node;

[0158] Using the parent node of the reachable subtree management node, the received operation information of the node and the subordinate nodes are sent to the subtree management node;

[0159] If the parent node of a node in the subtree is the subtree management node, the operation information of the node itself is sent to the subtree management node in each preset period, and the operation information sent by the subordinate node is forwarded to the subtree management node.

[0160] The operation information includes one or more of the following: the operation status of the node, the weight calculated by the path load coefficient, and the stored digital asset status.

[0161] The process of obtaining the load factor of each node in the file aggregation process includes:

[0162] After each node is connected to a node inside the file aggregation system, the load factor of the node is calculated based on the load parameters of the node within a preset time period;

[0163] The load parameters of the node include one or more of the following: the average round-trip time from the node to each parent node, the packet loss rate of the node within a preset time period, the number of files currently to be processed by the node, the number of server CPU cores of the node, the current CPU operating frequency, the current CPU occupancy rate, the server bandwidth, and the current network bandwidth occupancy rate of the node.

[0164] The calculation formula of the node load factor is:

[0165]

[0166] Where LBC represents the load factor of the node; RTT represents the average round-trip time from the node to each parent node; PDR represents the packet loss rate of the node in the preset time period; Q represents the number of files currently to be processed by the node; N cpu Indicates the number of CPU cores of the node server, C cpu Indicates the CPU operating frequency, U cpu Indicates the current CPU usage of the node; C bw Indicates the server bandwidth, U bw represents the current network bandwidth occupancy rate of the node; k1 represents the weight of the node's RTT; k2 represents the weight of the number of files to be processed Q at the node; k3 represents the weight of the node's PDR; k4 represents The weight of The weight of .

[0167] The weights k1, k2, k3, k4 and k5 of the load parameters for calculating the load coefficient are determined according to the user's business needs.

[0168] The using the subtree management node to adjust the load factor of the node according to the operation information includes:

[0169] If the subtree management node obtains that the load of a certain node is abnormal according to the operation information, the subtree management node is used to adjust the load factor of the node by adjusting the weight corresponding to the load parameter of the node;

[0170] Otherwise, there is no need to adjust the load factor of the node.

[0171] The weight corresponding to the node load parameter is adjusted according to the following formula:

[0172]

[0173] Among them, k ja represents the weight corresponding to the jth load parameter of the node after the subtree management node is adjusted; k jf represents the weight corresponding to the jth load parameter of the node before the subtree management node is adjusted; α represents the learning rate set by the subtree management node; Wj Indicates the weight corresponding to the jth load parameter set by the user; U j represents the value of the jth load parameter; T j Indicates the preset j-th load parameter threshold; the value of j is 1, 2, 3, 4, 5.

[0174] In the path determination module, the optimal path is obtained by using the load coefficients of all nodes that the node passes through when reaching the destination node, including:

[0175] Get the path load coefficient table of all parent nodes of the node;

[0176] If the destination node of the packet to be forwarded in the node is the parent node, the optimal path is direct forwarding;

[0177] Otherwise, the packet to be forwarded is placed in a waiting queue, and the initial optimal path to the destination node is obtained according to the path load coefficient table of the node where the packet is forwarded;

[0178] If there are no nodes with abnormal loads among all the nodes passed by the initial optimal path, the initial optimal path will be taken as the optimal path;

[0179] Otherwise, multiple selection coefficients of the node are obtained by using the load coefficient of each parent node of the node's superior that can reach the destination node of the packet to be forwarded, and the load coefficient of the path from each parent node to the destination node;

[0180] The path consisting of the parent node corresponding to the node when the node obtains the minimum selection coefficient and the path from the parent node to the destination node is taken as the optimal path.

[0181] The path load coefficient table of the node includes the initial optimal path for the node to reach each ancestor node, the path load coefficient and the load coefficient of the node.

[0182] Get the initial optimal path and path load factor of each ancestor node reachable by the node, including:

[0183] Traverse all paths from a node to its ancestor node, add up the load coefficients of all nodes on each path, and get the path load coefficient of each path;

[0184] The path with the smallest path load coefficient is taken as the initial optimal path.

[0185] The calculation formula of the node selection coefficient is:

[0186] S i =q 1i *LBC i +q 2i *LBC iid

[0187] Among them, S i Indicates the selection coefficient of the i-th parent node of the current node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; q 1i represents the selection weight corresponding to the load factor of the i-th parent node; q 2i represents the selection weight corresponding to the path load coefficient from the i-th parent node to the destination node;

[0188] The weight q 1i ,q 2i Determined by the corresponding subtree management node.

[0189] The selection weight q 1i ,q 2i The corresponding subtree management node is determined according to the following formula:

[0190]

[0191]

[0192] Among them, q 1i represents the selection weight corresponding to the updated load factor of the i-th parent node; q 2i represents the selection weight corresponding to the updated path load coefficient from the i-th parent node to the destination node; q 1if represents the selection weight corresponding to the preset load factor of the i-th parent node; q 2if represents the selection weight corresponding to the preset path load coefficient from the i-th parent node to the destination node; α represents the learning rate set by the subtree management node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; n represents the number of transit nodes between the i-th parent node and the destination node.

[0193] After obtaining the optimal path for file aggregation, the method further includes: updating the load coefficient of each node passed by the file during aggregation.

[0194] The update formula of node load factor is:

[0195] LBC a =LBC f +p1*SIZE pkt +p2*RTT pkt

[0196] Among them, LBC a Indicates the load factor of the node after file aggregation; LBCf Indicates the load factor of the node before file aggregation; SIZE pkt Indicates the size of the forwarded packet; RTT pkt It represents the round trip time between nodes when the forwarding packet is transmitted; p1 and p2 represent the weights corresponding to the size of the forwarding packet and the round trip time respectively, and the weights p1 and p2 are determined by the subtree management node of the subtree.

[0197] After the subtree management node is used to obtain the operation information of each node in the subtree, the method further includes:

[0198] Use the subtree management node to prune the subtree and hide the internal details of the subtree from the outside.

[0199] The modes of file aggregation include push aggregation and pull aggregation.

[0200] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A multi-node file aggregation load balancing method for edge-cloud collaboration, characterized in that: include: Obtain the load factor of each node in the multi-level tree structure for file aggregation, aggregate the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load factor of the node according to the operation information; The load coefficients of all nodes that a node passes through when reaching the destination node are used to obtain the optimal path for file aggregation; The multi-level tree structure includes multiple subtree management nodes.

2. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 1, characterized in that: The step of aggregating the operation information of each node to the corresponding subtree management node includes: If the parent node of a node in the subtree is not the subtree management node, then the operation information is sent to the parent node of a reachable subtree management node every preset period, and the operation information sent by the subordinate node is forwarded to the parent node; Using the parent node of the reachable subtree management node, the received operation information of the node and the subordinate nodes are sent to the subtree management node; If the parent node of a node in the subtree is the subtree management node, the operation information of the node itself is sent to the subtree management node in each preset period, and the operation information sent by the subordinate node is forwarded to the subtree management node.

3. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 1, characterized in that: The step of obtaining the load factor of each node in the multi-level tree structure for file aggregation includes: After each node is connected to a node inside the file aggregation system, the load factor of the node is calculated based on the load parameters of the node within a preset time period; The load parameters of the node include one or more of the following: the average round-trip time from the node to each parent node, the packet loss rate of the node within a preset time period, the number of files currently to be processed by the node, the number of server CPU cores of the node, the current CPU operating frequency, the current CPU occupancy rate, the server bandwidth, and the current network bandwidth occupancy rate of the node.

4. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 3, characterized in that: The calculation formula of the node load factor is: Where LBC represents the load factor of the node; RTT represents the average round-trip time from the node to each parent node; PDR represents the packet loss rate of the node in the preset time period; Q represents the number of files currently to be processed by the node; N cpu Indicates the number of server CPU cores of the node, C cpu Indicates the CPU operating frequency, U cpu Indicates the current CPU usage of the node; C bw Indicates the server bandwidth, U bw represents the current network bandwidth occupancy rate of the node; k1 represents the weight of the node's RTT; k2 represents the weight of the number of files to be processed Q at the node; k3 represents the weight of the node's PDR; k4 represents The weight of The weights of each load parameter k1, k2, k3, k4 and k5 are determined according to the user's business needs.

5. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 1, characterized in that: The using the subtree management node to adjust the load factor of the node according to the operation information includes: If the subtree management node finds that the load of a node is abnormal based on the operation information, it adjusts the weight corresponding to the load parameter of the node according to the following formula: Among them, k ja represents the weight corresponding to the jth load parameter of the node after the subtree management node is adjusted; k jf represents the weight corresponding to the jth load parameter of the node before the subtree management node is adjusted; α represents the learning rate set by the subtree management node; W j Indicates the weight corresponding to the jth load parameter set by the user; U j represents the value of the jth load parameter; T j Indicates the preset j-th load parameter threshold; the value of j is 1, 2, 3, 4, 5.

6. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 1, characterized in that: The method of obtaining the optimal path by utilizing the load coefficients of all nodes that a node passes through when reaching a destination node includes: Get the path load coefficient table of all parent nodes of the node; If the destination node of the packet to be forwarded in the node is the parent node, the optimal path is direct forwarding; Otherwise, the packet to be forwarded is placed in a waiting queue, and the initial optimal path to the destination node is obtained according to the path load coefficient table of the node where the packet is forwarded; If there are no nodes with abnormal loads among all the nodes passed by the initial optimal path, the initial optimal path will be taken as the optimal path; Otherwise, multiple selection coefficients of the node are obtained by using the load coefficient of each parent node of the node's superior that can reach the destination node of the packet to be forwarded, and the load coefficient of the path from each parent node to the destination node; The path consisting of the parent node corresponding to the node when the node obtains the minimum selection coefficient and the path from the parent node to the destination node is taken as the optimal path.

7. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 6, characterized in that: Get the initial optimal path and path load factor of each ancestor node reachable by the node, including: Traverse all paths from a node to its ancestor node, add up the load coefficients of all nodes on each path, and get the path load coefficient of each path; The path with the smallest path load coefficient is taken as the initial optimal path.

8. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 6, characterized in that: The calculation formula of the node selection coefficient is: S i =q 1i *LBC i +q 2i *LBC iid Among them, S i Indicates the selection coefficient of the i-th parent node of the current node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; q 1i represents the selection weight corresponding to the load factor of the i-th parent node; q 2i Indicates the selection weight corresponding to the path load coefficient from the i-th parent node to the destination node; The weight q 1i ,q 2i Determined by the corresponding subtree management node.

9. A multi-node file aggregation load balancing method for edge-cloud collaboration as claimed in claim 8, characterized in that: The selection weight q 1i ,q 2i The corresponding subtree management node is determined according to the following formula: Among them, q 1i represents the selection weight corresponding to the updated load factor of the i-th parent node; q 2i represents the selection weight corresponding to the updated path load coefficient from the i-th parent node to the destination node; q 1if represents the selection weight corresponding to the preset load factor of the i-th parent node; q 2if represents the selection weight corresponding to the preset path load coefficient from the i-th parent node to the destination node; α represents the learning rate set by the subtree management node; LBC i Indicates the load factor of the i-th parent node; LBC iid represents the path load coefficient from the i-th parent node to the destination node; n represents the number of transit nodes between the i-th parent node and the destination node.

10. A multi-node file aggregation load balancing system for edge-cloud collaboration, characterized in that: include: Load coefficient determination module: used to obtain the load coefficient of each node in the multi-level tree structure for file aggregation, aggregate the operation information of each node to the corresponding subtree management node, and use the subtree management node to adjust the load coefficient of the node according to the operation information; Path determination module: Utilizes the load coefficients of all nodes that a node passes through when reaching the destination node to obtain the optimal path for file aggregation.

Images