Dynamic capacity expansion and contraction method and system for decentralized chain-type acquisition structure
By dynamically adjusting the number of computing nodes and task allocation in the decentralized chain acquisition structure, the problem of excessive load of the central node is solved, and efficient processing of complex network environments and system stability is improved.
Patent Information
- Application Number
- CN202410892691.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-26
AI Technical Summary
When the number of resource users and business services are expanded, traditional centralized file aggregation technology leads to excessive load on the central node, affecting system efficiency. The existing decentralized chain acquisition technology is difficult to effectively deal with complex multi-level network environments.
The dynamic scaling method of decentralized chain acquisition structure is adopted to calculate the estimated total load and current load capacity through the management node, dynamically adjust the number of calculated nodes, and monitor the load situation in real time, allocate tasks, and use arrays to recursively calculate the estimated total load. Prioritize the simple configuration of compute nodes.
It improves network performance and reliability, effectively deals with complex multi-level network environments, reduces data tilt problems, and improves data acquisition efficiency and system stability.
Smart Images

Figure CN119030929A_ABST
Abstract
Description
Technical Field
[0001] The present invention specifically relates to a method and system for dynamically expanding and contracting a decentralized chain collection structure, and belongs to the field of computer resource allocation. Background Art
[0002] With the continuous deepening of Internet technology, enterprises have put forward new requirements for the informatization of assets, especially the increasing demand for safe and fast access to digital assets distributed in various places. Traditional file aggregation technology usually adopts a centralized approach to file aggregation. However, with the expansion of organizational management scope and business services, the number of resource users is also increasing rapidly, resulting in exponential growth in data traffic and computing power requirements, which will lead to excessive load on the central node and affect the overall system operation efficiency.
[0003] The decentralized chain collection technology can effectively solve the problem of excessive load on the central node through the dynamic expansion and contraction of the chain collection structure. It uses a multi-level collection algorithm to determine which node the traffic or data is stored and processed on, which can effectively reduce the load on the central node and improve the performance of the network. It has a high degree of reliability and fault tolerance, allowing the network to cope with failures. However, the current decentralized chain collection technology distributes traffic by using polling or weight-based methods, which often cannot effectively handle complex multi-level network environments. Summary of the invention
[0004] The purpose of the present invention is to propose a new method and system for dynamic expansion and contraction of a decentralized chain collection structure, which can reliably and quickly converge multi-node data and status to a designated node under the chain structure, improve network performance and reliability, and effectively handle complex multi-level network environments.
[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is as follows:
[0006] A method for dynamically scaling up and down a decentralized chain collection structure comprises the following steps:
[0007] Organize the data collection nodes into a chain structure, and divide the collection nodes into management nodes and computing nodes;
[0008] The management node calculates the estimated total load based on the collection target and calculates the current total load capacity for the computing node;
[0009] The management node dynamically expands or shrinks the computing nodes based on the relationship between the estimated total load and the current total load capacity of the computing nodes. Expansion refers to adding computing nodes, and shrinking refers to reducing computing nodes.
[0010] When there is no need to expand or shrink capacity, or after expansion or shrinking, the management node assigns tasks to the computing nodes and monitors the load of the computing nodes in real time;
[0011] The computing nodes collect data from the collection targets according to the assigned tasks.
[0012] Furthermore, the management node calculates the estimated total load according to the number of collection targets, target indicators and indicator calculation complexity, and the calculation formula is:
[0013]
[0014] Where T represents the expected total load, n represents the number of acquisition targets; M i represents the target index, that is, the number of indicators that need to be collected on each collection target; C i Indicates the computational complexity of the indicator.
[0015] Furthermore, an array recursive method is used to calculate the estimated total load.
[0016] Furthermore, the current total load capacity is calculated according to the number of nodes, node performance and node load rate of the computing nodes, including calculating the upper limit and lower limit of the current total load capacity, and the calculation formula is:
[0017] C max =Sum(P i )*r max
[0018] C min =Sum(P i )*r min
[0019] Among them, C max and C min Indicates the upper and lower limits of the current total load capacity; P i Indicates the node performance of each computing node, specifically the number of indicators that can be collected; Sum represents the sum of the node performance of each computing node; r max and r min Indicates the upper and lower thresholds of the node load rate.
[0020] Furthermore, dynamically expanding or shrinking capacity according to the relationship between the estimated total load and the current total load capacity includes:
[0021] If the estimated total load is greater than the upper limit of the current total load capacity, capacity expansion is performed;
[0022] If the estimated total load is less than the lower limit of the current total load capacity, capacity reduction is performed.
[0023] Furthermore, the method for assigning tasks to a single computing node is: when the current load of the computing node is greater than 90% of the current maximum load, the task is assigned to other computing nodes that are not greater than 90%; when assigning tasks, the computing nodes are selected in order of the remaining load amount, and the maximum load of the computing node is not exceeded after assignment.
[0024] Furthermore, when selecting computing nodes, computing nodes with simple configurations are given priority, provided that the total load capacity meets the load computing requirements.
[0025] A dynamic expansion and contraction system of a decentralized chain collection structure includes collection nodes constituting the chain structure, wherein the collection nodes include:
[0026] Management nodes are responsible for calculating the estimated total load based on the acquisition target and calculating the current total load capacity for the computing nodes; dynamically expanding or shrinking the computing nodes based on the relationship between the estimated total load and the current total load capacity of the computing nodes. Expanding refers to adding computing nodes, while shrinking refers to reducing computing nodes; and allocating tasks to computing nodes and monitoring the load of computing nodes in real time;
[0027] Computing nodes are responsible for collecting data from collection targets according to the assigned tasks.
[0028] The present invention has the advantages that:
[0029] 1) Load balancing strategy based on decentralized chain: The expected total load is estimated in advance according to the collection target, and the total load capacity of the computing nodes is obtained through multi-dimensional measurement of each computing node in the link. The data collection nodes are dynamically selected according to the relationship between the expected total load and the total load capacity, which solves the research gap of decentralized load balancing and greatly improves the efficiency and stability of data collection.
[0030] 2) Based on link self-management: Link management nodes are set up in the link according to actual business needs. These nodes are responsible for managing the nodes within the link they are in, maintaining the global balance of the link, reducing the problem of data skew in the link, and ensuring system reliability. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 The present invention is a flowchart of a method for dynamically scaling a decentralized chain collection structure according to an embodiment of the present invention.
[0032] Figures 2 to 5 It is a statistical chart of CPU usage and memory usage in experimental tests. DETAILED DESCRIPTION
[0033] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.
[0034] The embodiment of the present invention specifically provides a method for dynamically scaling up and down a decentralized chain collection structure. Based on this method, a system is constructed in which data collection nodes are organized according to a chain structure. The collection nodes can be computers, servers, virtual machines, etc., and then the collection nodes are divided into management nodes and computing nodes according to their functions and dynamically adjusted. The process is as follows: Figure 1 The method comprises the following processing steps:
[0035] S1: During the system initialization and deployment process, all collection nodes in the link need to be registered for service. After the system is started, the current estimated total load will be calculated based on the number of collection targets, target indicators, and indicator calculation complexity.
[0036] Number of Targets: Indicates the number of targets that need to be collected.
[0037] Target Metrics: Indicates the number of metrics that need to be collected on each collection target, using the array M i Indicates that, for example, M=[4,5,10,6] means that the number of indicators to be collected on the four collection targets are 4 indicators, 5 indicators, 10 indicators, and 6 indicators respectively.
[0038] Metrics Complexity: Indicates the complexity of the metrics of each collection target, which can be data volume, processing time, etc., represented by array C. For example, C = [1,3,8,4] means that the complexity of the metrics on the four collection targets is 1, 3, 8, and 4 respectively.
[0039] Total Load: It is the estimated value of the computational load required to collect data of all collection targets, represented by T. The calculation formula for the total load is:
[0040]
[0041] Where n represents the number of acquisition targets, M i represents the target indicator, C i Indicates the complexity of indicator calculation, and T indicates the expected total load. For example, T = 4*1+5*3+10*8+6*4=123.
[0042] Taking into account the differences in hardware levels in different environments, the system uses array recursion to calculate during the initialization process to estimate the total load required by the system during operation (i.e., the expected total load). Array recursion is a flexible method that can dynamically adjust the estimation of the total load according to actual conditions. The advantage of using array recursion is that it can make a refined evaluation of the total load based on different hardware configurations and performance characteristics. This method can take into account factors such as hardware processing power, memory capacity, and parallel computing capabilities to determine the workload that the system can withstand. During the total load estimation process, the system abstracts different tasks or indicators into elements in an array and processes the array recursively. During the recursive process, the system dynamically adjusts the recursive depth and calculation method based on factors such as task complexity, data volume, and computing requirements to reflect the situation in the actual environment. In this way, the system can more accurately estimate the required computing resources and performance requirements, and make corresponding resource allocation and optimization at the beginning of deployment. This helps ensure that the system can run stably in different environments and meet the expected workload requirements. In summary, using array recursion to calculate the estimated total load can better take into account the differences in hardware levels in different environments and provide more accurate system performance evaluation and resource management to ensure system stability and scalability.
[0043] S2: Before assigning tasks to computing nodes, the management node will calculate the current total load capacity of each computing node based on the number of computing nodes, node performance, node load rate and other indicators. The current total load capacity can be used to understand the resource status and processing capacity of each computing node to measure and determine whether the link needs to be expanded or reduced.
[0044] Number of Nodes: Indicates the number of computing nodes in the system.
[0045] Node performance: indicates the performance of the number of indicators that each computing node can collect, using the array P i For example, P = [30, 70, 50] means that the number of indicators that can be collected by the three computing nodes is 30, 70, and 50.
[0046] Node load rate (Rate of Loading): includes the upper threshold and lower threshold of the node load rate. The upper threshold is represented by r. For example, r=90% means that the upper threshold is 90% and the lower threshold is 10%.
[0047] Total load (Node Load Capacity): indicates the current total load capacity of all computing nodes, represented by C, including the upper limit C max and the lower limit C min, the current total load capacity of the computing nodes is measured by calculating the number of computing nodes, node performance, and node load rate. Specifically, the node performance of the current computing nodes is added and multiplied by the node load rate. The calculation formula for the current total load capacity is as follows:
[0048] C max =Sum(P i )*r max
[0049] C min =Sum(P i )*r min
[0050] Among them, P i represents the node performance, r max and r min Indicates the upper and lower thresholds of the node load rate, C max and C min Indicates the upper and lower limits of the total load capacity of the node. For example, the upper limit C max =(30+70+50)*0.9=135, lower limit C min =(30+70+50)*0.1=15.
[0051] S4: By comparing the estimated total load with the upper and lower limits of the total load capacity, it is determined whether the computing nodes in the link need to be expanded or reduced, as shown below:
[0052] If T>C max , then expand the capacity and add computing nodes;
[0053] If T <C min , then reduce the capacity and reduce the computing nodes.
[0054] For example, if the total load is expected to be 123, the upper limit of the total load capacity is 135, and the lower limit is 15, there is no need to scale up or down, and tasks can be assigned. If the total load is expected to be 140, which exceeds the upper limit of 135, then capacity expansion is required. If the total load is expected to be 10, which is lower than the lower limit of 15, then capacity reduction is required.
[0055] Among them, the method of allocating tasks to a single computing node is: when the current load of the computing node is greater than 90% of the current maximum load, the task is allocated to other computing nodes that are not greater than 90%; when allocating, the computing nodes can be selected by sorting according to the size of the remaining load quota, and the maximum load of the computing node will not be exceeded after allocation.
[0056] In the above method, when allocating to computing nodes, in addition to considering the total load capacity of the computing nodes, the ease of use is also considered. Computing nodes with simple configurations are given priority under the premise of satisfying the computing capacity. This means that the system not only pays attention to the hardware capabilities of the computing nodes, but also considers the ease of use and operability of the computing nodes. For example, a node with strong performance may have complex settings and configuration requirements, while a node with weaker performance but easy to use may be more suitable for specific tasks. Based on the comprehensive consideration of the total load capacity and usage of the nodes, the differences between different nodes are fully utilized, and the roles and tasks are automatically allocated. According to the characteristics and resource conditions of the computing nodes, appropriate tasks are allocated to appropriate computing nodes to ensure that the tasks can be effectively processed on the appropriate acquisition nodes to achieve the best system performance and resource utilization, which can improve the efficiency, scalability and stability of the system, and make the operation and management of the system more convenient and flexible.
[0057] Experimental test:
[0058] In the comparative experiment on the occupancy of hardware resources, this experiment compared the method of the present invention with the distributed collection technology based on Kubernetes (K8S). The results of the comparative experiment are as follows:
[0059] 1. The CPU occupancy rate of node 1 of the method of the present invention is 58%, and the memory occupancy rate is 0.1%. Figure 2 shown.
[0060] 2. The CPU occupancy rate of node 2 of the method of the present invention is 45%, and the memory occupancy rate is 0.1%. Figure 3 shown.
[0061] 3. The CPU usage of node 1 based on K8S distributed collection technology is 163%, and the memory usage is 0.18%. Figure 4 shown.
[0062] 4. The CPU usage of node 2 based on K8S distributed collection technology is 181%, and the memory usage is 0.2%. Figure 5 shown.
[0063] The comparison of the above experimental results proves that under the same test environment, the method of the present invention has significantly lower CPU occupancy and relatively lower memory occupancy compared with the distributed acquisition technology based on K8S.
[0064] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.
Claims
1. A method for dynamic expansion and contraction of a decentralized chain collection structure, characterized in that: The following steps are involved: Organize the data collection nodes into a chain structure, and divide the collection nodes into management nodes and computing nodes; The management node calculates the estimated total load based on the collection target and calculates the current total load capacity for the computing node; The management node dynamically expands or shrinks the computing nodes based on the relationship between the estimated total load and the current total load capacity of the computing nodes. Expansion refers to adding computing nodes, and shrinking refers to reducing computing nodes. When there is no need to expand or shrink capacity, or after expansion or shrinking, the management node assigns tasks to the computing nodes and monitors the load of the computing nodes in real time; The computing nodes collect data from the collection targets according to the assigned tasks.
2. The method according to claim 1, characterized in that The management node calculates the estimated total load based on the number of collection targets, target indicators, and indicator calculation complexity. The calculation formula is: Where T represents the expected total load, n represents the number of acquisition targets; M i represents the target index, that is, the number of indicators that need to be collected on each collection target; C i Indicates the computational complexity of the indicator.
3. The method according to claim 2, characterized in that The estimated total load is calculated using an array recursive method.
4. The method according to claim 1, characterized in that The current total load capacity is calculated based on the number of computing nodes, node performance, and node load rate, including the upper and lower limits of the current total load capacity. The calculation formula is: C max =Sum(P i )*r max C min =Sum(P i )*r min Among them, C max and C min Indicates the upper and lower limits of the current total load capacity; P i Indicates the node performance of each computing node, specifically the number of indicators that can be collected; Sum represents the sum of the node performance of each computing node; r max and r min Indicates the upper and lower thresholds of the node load rate.
5. The method according to claim 4, characterized in that Dynamic expansion or reduction based on the relationship between the expected total load and the current total load capacity includes: If the estimated total load is greater than the upper limit of the current total load capacity, capacity expansion is performed; If the estimated total load is less than the lower limit of the current total load capacity, capacity reduction is performed.
6. The method according to claim 1, characterized in that The method for assigning tasks to a single computing node is: when the current load of the computing node is greater than 90% of the current maximum load, the task is assigned to other computing nodes that are not greater than 90%; when assigning tasks, the computing nodes are selected in order of the remaining load quota, and the maximum load of the computing node is not exceeded after assignment.
7. The method according to claim 6, characterized in that When selecting computing nodes, give priority to computing nodes with simple configurations, provided that the total load capacity meets the load computing requirements.
8. A dynamic expansion and contraction system of a decentralized chain collection structure, characterized in that: The collection nodes forming a chain structure include: Management nodes are responsible for calculating the estimated total load based on the acquisition target and calculating the current total load capacity for the computing nodes; Dynamically expand or shrink the computing nodes based on the relationship between the estimated total load and the current total load capacity of the computing nodes. Expanding refers to adding computing nodes, while shrinking refers to reducing computing nodes. It is also responsible for assigning tasks to computing nodes and monitoring the load of computing nodes in real time. Computing nodes are responsible for collecting data from collection targets according to the assigned tasks.
Citation Information
Cited By
Dynamic capacity expansion and contraction system and method for order dispatching service of business opportunity platform
CN119960975A
A dynamic scaling system and method for an order dispatch service of a business opportunity platform
CN119960975B
