Collection scheduling method and device based on decentration of super-large-scale IT equipment
Through the decentralized acquisition and scheduling method, multiple acquisition tasks are used to perform nodes in parallel processing and allocate tasks according to server performance indicators, the problem of resource imbalance and centralized scheduling bottlenecks in large-scale networks is solved, and efficient data collection and System stability.
Patent Information
- Application Number
- CN202411000964.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
It is difficult for the existing technology to effectively coordinate multiple nodes and make full use of the acquisition server resources in large-scale network operations and maintenance, especially in super-large-scale communication networks, there are problems of resource imbalance and centralized scheduling bottlenecks.
Adopt the decentralized acquisition and scheduling method based on super-large-scale IT equipment, and use multiple acquisition task execution nodes to process in parallel, and reasonably allocate tasks according to server performance indicators by obtaining network element inventory, splitting tasks, generating scheduling strategies and executing tasks. , ensure system stability and performance.
It realizes effective coordination of multi-node resources, improves data acquisition efficiency and system stability, has elasticity and fault tolerance, is suitable for complex communication networks or large data centers, and has flexibility and adaptability.
Smart Images

Figure CN118939399A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a collection and scheduling method and device based on the decentralization of ultra-large-scale IT equipment. Background Art
[0002] With the advancement of the digital society, the scale of today's networks is getting larger and larger, and the types of equipment in various communication networks are increasing. Therefore, in order to ensure the normal operation of communication networks, network management systems play a vital role in network operation and maintenance. The network management system relies on the data collection of infrastructure for network detection and network equipment monitoring. Data collection has become the basis of network monitoring, operation and management. Therefore, in large-scale network monitoring, how to more effectively schedule data collection and monitoring tasks, and how to more stably collect data from various OMCs and network elements in the network, higher requirements are put forward.
[0003] Data collection includes two categories: single-node server collection and multi-node server collection.
[0004] The collection process of single-node server collection is mainly completed by regularly or periodically executing collection tasks to complete task scheduling, but this type of method can generally only complete the periodic task scheduling of a single node on the server, such as scheduling at intervals of specified time periods, starting scheduling after providing a certain delay, and starting scheduling at a specific time. However, in large-scale network operation and maintenance monitoring, single-node server collection tasks cannot meet the monitoring requirements of the entire network. This type of method only realizes task scheduling, and does not use the load of the collection node as the basis for the allocation of collection tasks. Therefore, when the resources of the collection nodes are uneven or the load is uneven, there will be problems such as the inability to smoothly allocate the collection tasks when the load of the collection nodes is high, or the inability to effectively execute the collection tasks after allocation. Such problems are particularly prominent in the scenario of large-scale data collection task allocation.
[0005] The multi-node task allocation method in multi-node server collection requires a centralized task allocation center. The task allocation of all nodes is uniformly managed and scheduled by the centralized task allocation center, and then combined with regular or periodic task allocation strategies to allocate tasks to multiple nodes and coordinate and send them to each collection node for task scheduling. However, this type of method cannot effectively coordinate collection tasks in the case of complex ultra-large-scale communication networks, where there may be network isolation between the collected devices (as shown in the attached figure of the specification). Figure 2 At the same time, due to the existence of centralized nodes, once problems occur in centralized scheduling, such as process hangs and service pseudo-death, the entire task allocation mechanism will fail; and when ultra-large-scale collection task scheduling scenarios occur, the centralized task scheduling allocation solution also has performance bottlenecks.
[0006] Therefore, the existing technologies cannot effectively coordinate multiple nodes and fully utilize the acquisition server resources. Summary of the invention
[0007] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and propose a decentralized collection scheduling method and device based on ultra-large-scale IT equipment. The method can effectively coordinate multiple nodes and fully utilize the collection server resources to achieve scheduling.
[0008] In a first aspect, the present invention provides a collection scheduling method based on the decentralization of ultra-large-scale IT equipment, the method comprising the following steps:
[0009] Step S1: obtaining a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list;
[0010] Step S2: splitting the network element list into multiple collection tasks; and saving the collection tasks into a collection task pool;
[0011] Step S3: Generate a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators;
[0012] Step S4: acquiring the target collection task from the collection task pool through the collection task execution node according to the collection task scheduling strategy;
[0013] Step S5: executing the target collection task to complete the decentralized collection scheduling based on ultra-large-scale IT equipment;
[0014] There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
[0015] Furthermore, in step S1, the list of network elements to be collected is obtained through the CMDB system, which specifically includes the following steps:
[0016] Step S11: Log in to the CMDB system;
[0017] Step S12: According to the query function or search function provided by the CMDB system, the list of network elements to be collected is searched and screened, that is, the list of network elements to be collected is obtained.
[0018] Furthermore, in step S2, the network element list is split into multiple collection tasks, which specifically includes the following steps:
[0019] Step A1: preset the collection task splitting cycle;
[0020] Step A2: based on the collection task splitting cycle, split the network element list into multiple collection tasks according to the attribute characteristics of the collection tasks;
[0021] In S2, the collection task is saved in a collection task pool, which specifically includes the following steps:
[0022] Step B1: Create different collection queues in the task pool according to the network identification feature code;
[0023] The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications with the MD5 algorithm;
[0024] Step B2: Put the split collection tasks into corresponding collection queues according to the network identification feature codes of the network elements.
[0025] Further, after step S4 and before step S5, the method further includes step K;
[0026] Step K: Determine whether the server free memory of the current collection task execution node is greater than the startup memory of the target collection task:
[0027] If the server free memory of the current collection task execution node is greater than the startup memory of the target collection task, then go to step S5, otherwise the target collection task is put back into the collection task pool for recycling, waiting to be acquired by the next collection task execution node.
[0028] Furthermore, the server performance indicators include CPU utilization, number of CPU cores, memory utilization, and disk IO utilization;
[0029] The step S3 generates a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators, and specifically includes the following steps:
[0030] Step S31: Calculate the first delay time of acquiring the acquisition task according to the CPU utilization and the number of CPU cores;
[0031] The calculation formula of the first delay time of the acquisition task is as follows:
[0032] t1 delay =p cpuavg t design
[0033] in,
[0034] t1 delay is the first delay time of the acquisition task;
[0035] p cpuavgis the average CPU utilization;
[0036] t design To design the delay time;
[0037] p cpuavg The calculation formula is as follows:
[0038]
[0039] p cpuused The current CPU utilization rate ranges from 0 to 1.
[0040] c cpunum is the number of CPU cores, which must be a positive integer;
[0041] Step S32: Calculate the second delay time for acquiring the acquisition task according to the memory utilization rate;
[0042] The calculation formula of the second delay time of the acquisition task is as follows:
[0043] t2 delay =p memused t design
[0044] in,
[0045] t2 delay is the second delay time of the acquisition task;
[0046] p memused is the memory utilization;
[0047] Memory utilization p memused The calculation formula is as follows:
[0048]
[0049] c memused is the size of the used memory; c memtotle is the total memory size;
[0050] Step S33: Calculate the third delay time for acquiring the acquisition task according to the disk IO utilization rate;
[0051] t3 delay =p ioused t design
[0052] t3 delay is the third delay time of the acquisition task;
[0053] p ioused is the disk IO utilization;
[0054] Step S34: generating a collection task scheduling strategy according to the collection tasks in the collection task pool according to the first delayed task time, the second delayed task time, and the third delayed task time.
[0055] In a second aspect, the present invention provides a decentralized collection and scheduling device based on ultra-large-scale IT equipment, the device comprising:
[0056] A first acquisition unit is used to acquire a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list;
[0057] a first processing unit, connected to the first acquisition unit, configured to split the network element list into a plurality of collection tasks; and further configured to save the collection tasks into a collection task pool;
[0058] A second processing unit, connected to the first processing unit, is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the server performance index;
[0059] A second acquisition unit, connected to the second processing unit, and configured to acquire a target acquisition task from the acquisition task pool through an acquisition task execution node according to the acquisition task scheduling strategy;
[0060] a third processing unit, connected to the second acquisition unit, and configured to execute the target acquisition task to complete the decentralized acquisition scheduling based on ultra-large-scale IT equipment;
[0061] There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
[0062] Furthermore, the first acquiring unit includes:
[0063] Login module, used to log in to the CMDB system;
[0064] The acquisition module is connected to the login module and is used to search and filter the list of network elements to be collected according to the query function or search function provided by the CMDB system, that is, to obtain the list of network elements to be collected.
[0065] Further, the first processing unit includes a first processing subunit and a second processing subunit; the first processing subunit includes splitting the network element list into multiple collection tasks, and the second processing subunit includes saving the collection task into a collection task pool;
[0066] The first processing subunit comprises:
[0067] Preprocessing module, used to preset the collection task splitting cycle;
[0068] A splitting module, connected to the preprocessing module, for splitting the network element list into multiple collection tasks based on the collection task splitting cycle and the attribute characteristics of the collection tasks;
[0069] The second processing subunit comprises:
[0070] A creation module is used to create different collection queues in the task pool according to the network identification signature code;
[0071] The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications with the MD5 algorithm;
[0072] The placement module is connected to the creation module and is used to place the split collection tasks into corresponding collection queues according to the network identification feature codes where the network elements are located.
[0073] Furthermore, the device further comprises a determination unit, and the determination unit is connected to the second acquisition unit and the third processing unit respectively;
[0074] The determination unit is used to determine whether the server free memory of the current acquisition task execution node is greater than the startup memory of the target acquisition task;
[0075] Wherein, the third processing unit executes the target collection task when the idle memory of the server of the current collection task execution node is greater than the startup memory of the target collection task according to the determination result of the determination unit.
[0076] Furthermore, the server performance indicators include CPU utilization, number of CPU cores, memory utilization, and disk IO utilization;
[0077] The second processing unit comprises:
[0078] A first calculation module, used for calculating a first delay time of acquiring a collection task according to a CPU utilization rate and a number of CPU cores;
[0079] The calculation formula of the first delay time of the acquisition task is as follows:
[0080] t1 delay =p cpuavg t design
[0081] in,
[0082] t1 delay is the first delay time of the acquisition task;
[0083] p cpuavg is the average CPU utilization;
[0084] tdesign To design the delay time;
[0085] p cpuavg The calculation formula is as follows:
[0086]
[0087] p cpuused The current CPU utilization rate ranges from 0 to 1.
[0088] c cpunum is the number of CPU cores, which must be a positive integer;
[0089] A second calculation module, connected to the first calculation module, used to calculate a second delay time for acquiring the acquisition task according to the memory utilization;
[0090] The calculation formula of the second delay time of the acquisition task is as follows:
[0091] t2 delay =p memused t design
[0092] in,
[0093] t2 delay is the second delay time of the acquisition task;
[0094] p memused is the memory utilization;
[0095] Memory utilization p memused The calculation formula is as follows:
[0096]
[0097] c memused is the size of the used memory; c memtotle is the total memory size;
[0098] A third calculation module, connected to the second calculation module, is used to calculate a third delay time for acquiring the acquisition task according to the disk IO utilization rate;
[0099] t3 delay =p ioused t design
[0100] t3 delay is the third delay time of the acquisition task;
[0101] p ioused is the disk IO utilization;
[0102] The strategy generation module is connected to the first calculation module, the second calculation module and the third calculation module respectively, and is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the first delayed task time, the second delayed task time and the third delayed task time.
[0103] Beneficial effects of the present invention:
[0104] 1. The present invention can realize decentralized collection scheduling based on ultra-large-scale IT equipment, thereby effectively coordinating multiple nodes and making full use of collection server resources to achieve scheduling.
[0105] 2. The present invention splits the collection task into multiple execution nodes for parallel processing, fully utilizing the computing resources of IT equipment and improving data collection efficiency.
[0106] 3. The present invention can reasonably distribute tasks according to the load data of the node server and improve the system stability and performance.
[0107] 4. The present invention has elasticity and fault tolerance. The existence of multiple execution nodes ensures the elasticity and fault tolerance of the system. Even if some nodes fail, other nodes can continue to execute tasks, ensuring the continuity of data collection.
[0108] 5. The present invention has universal applicability and can be applied to various IT equipment in complex communication networks or large data centers. It can realize the collection and management of performance indicators of these devices. At the same time, it can customize relevant collection modules according to specific application scenarios, and has high flexibility and adaptability.
[0109] 6. The collection task execution nodes of the present invention are diverse and flexible, and deployment and maintenance become relatively simple. At the same time, the nodes can be dynamically adjusted according to actual conditions to improve the stability and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0110] Figure 1 It is a schematic diagram of a decentralized collection and scheduling method based on ultra-large-scale IT equipment in an embodiment of the present invention;
[0111] Figure 2 A schematic diagram of a peer-to-peer network collection task allocation mechanism in an embodiment of the present invention;
[0112] Figure 3 A schematic diagram of a task allocation mechanism in an embodiment of the present invention;
[0113] Figure 4 The overall data flow diagram of the acquisition task scheduling in the embodiment of the present invention;
[0114] Figure 5 This is a flow chart of the acquisition task scheduling business in an embodiment of the present invention;
[0115] Figure 6 A business flow chart for splitting a collection task in an embodiment of the present invention;
[0116] Figure 7 A flow chart of a collection task acquisition strategy business process in an embodiment of the present invention;
[0117] Figure 8 It is a schematic diagram of a decentralized collection and scheduling device based on ultra-large-scale IT equipment in an embodiment of the present invention;
[0118] Among them, the figure numerals are: 10, first acquisition unit, 20, first processing unit, 30, second processing unit, 40, second acquisition unit, 50, third processing unit. DETAILED DESCRIPTION
[0119] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0120] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.
[0121] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.
[0122] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.
[0123] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.
[0124] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.
[0125] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.
[0126] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.
[0127] Embodiment 1:
[0128] like Figure 1 As shown, this embodiment provides a collection scheduling method based on the decentralization of ultra-large-scale IT equipment, and the method includes the following steps:
[0129] Step S1: Obtain a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list.
[0130] For ease of management and maintenance, the network element list usually lists the information of each network element device in the form of a network element list, including device name, model, serial number, IP address, location, etc. In addition to listing the information of each network element device in the form of a network element list, the network element list can also be presented in the form of a topology diagram, a table, a tree structure, etc. Different forms are suitable for different scenarios. For example, a topology diagram can intuitively display the connection relationship between devices, a table format is more suitable for quickly finding specific information, and a tree structure can clearly display the hierarchical relationship between devices. Choose the appropriate form to present the network element list information according to specific needs and usage scenarios.
[0131] As a specific implementation, in step S1, the list of network elements to be collected is obtained through the CMDB system, which specifically includes the following steps:
[0132] Step S11: Log in to the CMDB system;
[0133] Step S12: According to the query function or search function provided by the CMDB system, the list of network elements to be collected is searched and screened, that is, the list of network elements to be collected is obtained.
[0134] The full name of CMDB is "Configuration Management Database". CMDB (Configuration Management Database) system is usually used to record the information of all configuration items of an organization (including hardware, software, documents, etc.), so the list of network elements to be collected can be obtained through the CMDB system. Through the CMDB system, it is easy to find and manage the acquisition of network element lists for each network element device. Using the Configuration Management Database (CMDB) system is a common method. CMDB is a database used to store information about an organization's IT equipment and configurations, which can help organizations manage their IT infrastructure. Through the CMDB system, the network element list can be effectively collected and managed, including the information, configuration and relationships of each network element device. Obtaining the network element list through the CMDB system can improve the accuracy and real-time nature of the data, which is beneficial to the management and maintenance of network equipment.
[0135] Step S2: splitting the network element list into multiple collection tasks; and saving the collection tasks into a collection task pool.
[0136] As a specific implementation, in step S2, the network element list is split into multiple collection tasks, which specifically includes the following steps:
[0137] Step A1: preset the collection task splitting cycle;
[0138] Step A2: Based on the collection task splitting cycle, the network element list is split into multiple collection tasks according to the attribute characteristics of the collection tasks.
[0139] Specifically, the splitting cycle of the collection task needs to be determined according to factors such as network scale, number of devices, and update frequency, that is, how many subtasks the network element list is split into, and the collection frequency and cycle of each subtask are specified. This can be set according to factors such as network scale, number of devices, and update frequency. Then, according to the attribute characteristics of the collection task, the network element list is split into multiple subtasks. Specifically, it includes the following aspects:
[0140] (1) Device type splitting: Classify the network element list according to device type (switch, router, firewall, etc.). Each category can be used as a separate collection task.
[0141] (2) Regional / location splitting: Split the equipment according to its geographical location or deployment location, and assign equipment in different locations to different collection tasks.
[0142] (3) Functional splitting: The equipment is split according to its function (core equipment, aggregation equipment, access equipment, etc.). Each functional type can be used as an independent collection task.
[0143] (4) Importance / priority splitting: Group devices into different collection tasks based on their importance or priority to ensure that important devices can be collected in a timely manner.
[0144] In S2, the collection task is saved in a collection task pool, which specifically includes the following steps:
[0145] Step B1: Create different collection queues in the task pool according to the network identification feature code;
[0146] The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications using the MD5 algorithm.
[0147] MD5 (Message Digest Algorithm 5) is a commonly used hash function that can map data of any length to a 128-bit hash value. This embodiment uses the MD5 algorithm because the MD5 algorithm has the characteristics of fixed-length output, irreversibility, uniqueness, and fast calculation.
[0148] Step B2: Put the split collection tasks into corresponding collection queues according to the network identification feature codes of the network elements.
[0149] Specifically, first, different collection queues are created in the task pool according to the network identification feature code (obtained by encrypting the network identification of different network elements in the CMDB using the MD5 algorithm). These queues can be distinguished according to the hash value of the feature code or other rules, so that the split collection tasks can be placed in the corresponding queues later. Then the split collection tasks are placed in the corresponding collection queues according to the network identification feature code of the network element. This requires placing each collection task in the corresponding queue according to the network identification feature code associated with it. This ensures that collection tasks with the same network identification are placed in the same queue, which is convenient for subsequent scheduling and execution.
[0150] Step S3: Generate a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators.
[0151] As a specific implementation method, the server performance indicators include CPU utilization, CPU core number, memory utilization, and disk IO utilization;
[0152] The step S3 generates a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators, and specifically includes the following steps:
[0153] Step S31: Calculate the first delay time of acquiring the acquisition task according to the CPU utilization and the number of CPU cores;
[0154] The calculation formula of the first delay time of the acquisition task is as follows:
[0155] t1 delay =p cpuavg t design
[0156] in,
[0157] t1 delay is the first delay time of the acquisition task;
[0158] p cpuavg is the average CPU utilization;
[0159] t design To design the delay time;
[0160] p cpuavg The calculation formula is as follows:
[0161]
[0162] p cpuused The current CPU utilization rate ranges from 0 to 1.
[0163] c cpunum is the number of CPU cores, which must be a positive integer;
[0164] Step S32: Calculate the second delay time for acquiring the acquisition task according to the memory utilization rate;
[0165] The calculation formula of the second delay time of the acquisition task is as follows:
[0166] t2 delay =p memused t design
[0167] in,
[0168] t2 delay is the second delay time of the acquisition task;
[0169] p memused is the memory utilization;
[0170] Memory utilization p memused The calculation formula is as follows:
[0171]
[0172] c memusedis the size of the used memory; c memtotle is the total memory size;
[0173] Step S33: Calculate the third delay time for acquiring the acquisition task according to the disk IO utilization rate;
[0174] t3 delay =p ioused t design
[0175] t3 delay is the third delay time of the acquisition task;
[0176] p ioused is the disk IO utilization;
[0177] Step S34: generating a collection task scheduling strategy according to the collection tasks in the collection task pool according to the first delayed task time, the second delayed task time, and the third delayed task time.
[0178] Specifically, the priority of the scheduled tasks is set by comprehensively considering different delayed task times, and the execution order and time of the tasks are dynamically adjusted during actual scheduling, including the following steps:
[0179] (1) Three delayed task times need to be considered: the first delayed task time, the second delayed task time, and the third delayed task time. A weighted average method can be used to calculate a comprehensive delay index, or the weighted average method can be not used, or the execution can be performed sequentially, first executing the first delayed task time, then executing the second delayed task time, and finally executing the third delayed task time. Assuming that the three delay times are T1, T2, and T3, and the weights are W1, W2, and W3, respectively, the calculation formula for the comprehensive delay index T is:
[0180] T=(T1*W1+T2*W2+T2*W2) / (W1+W2+W3).
[0181] (2) According to the calculated comprehensive delay index, the collection tasks are divided into different priorities. The specific setting is based on the actual situation: the task with a shorter delay time has a higher priority; or the task with a longer delay time is executed first.
[0182] (3) When scheduling tasks, it is necessary to consider the current resource utilization and avoid scheduling too many collection tasks in the same time period to avoid resource competition and performance issues. For example, high-priority tasks can be scheduled in different time periods instead of being concentrated at the same time.
[0183] (4) According to dynamic factors such as network load and system resources, the execution order and time of each collection task are dynamically adjusted during actual scheduling.
[0184] Step S4: According to the collection task scheduling strategy, the target collection task is obtained from the collection task pool through the collection task execution node.
[0185] The collection task execution node selects the target collection task from the collection task pool according to the collection task scheduling strategy. The collection task scheduling strategy is to effectively manage a large number of collection tasks and reasonably utilize system resources. By formulating the collection task scheduling strategy, the task execution order can be optimized and the task execution efficiency can be improved. At the same time, the task execution order can be reasonably arranged according to the task priority and delay time to ensure the effective utilization of system resources.
[0186] This step specifically includes:
[0187] (1) Obtaining task information: The collection task execution node obtains relevant information of the task to be executed from the collection task pool, such as task type, priority, delay time, etc.
[0188] (2) Task selection based on scheduling strategy: According to the formulated collection task scheduling strategy, the execution node selects the next target collection task based on factors such as task priority and delay time.
[0189] (3) Task execution: The execution node starts to execute the selected target collection task, including data collection, processing, and result submission.
[0190] (4) Update task status: After the task is completed, update the status of the task in the task pool and mark it as completed or update related information.
[0191] Step S5: executing the target collection task to complete the decentralized collection scheduling based on ultra-large-scale IT equipment;
[0192] There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
[0193] For decentralized collection scheduling based on ultra-large-scale IT equipment, when starting to execute the target collection task, the following methods are used to implement decentralized collection task execution:
[0194] (1) Parallel execution: Use multiple collection task execution nodes to execute the target collection task in parallel to increase execution efficiency and reduce task execution time.
[0195] (2) Load balancing: A load balancing strategy is used to ensure that the loads of various collection task execution nodes are basically balanced, avoiding performance problems caused by excessive load on a single node.
[0196] (3) Task self-dispatching: Each collection task execution node has the ability to obtain tasks from the collection task pool, select the target collection task and start execution according to the scheduling strategy, thus realizing decentralized task dispatching and execution.
[0197] (4) Resource monitoring: Each execution node needs to monitor the usage of its own resources to avoid excessive resource consumption or load imbalance.
[0198] (5) Unified management: Although the task execution nodes are decentralized, a unified task management and monitoring system is still required to manage the task pool, monitor node status and task execution, and coordinate task allocation and scheduling among different nodes.
[0199] In a specific implementation, after step S4 and before step S5, the method further includes step K;
[0200] Step K: Determine whether the server free memory of the current collection task execution node is greater than the startup memory of the target collection task:
[0201] If the server free memory of the current collection task execution node is greater than the startup memory of the target collection task, then go to step S5, otherwise the target collection task is put back into the collection task pool for recycling, waiting to be acquired by the next collection task execution node.
[0202] Specifically, the execution of this step includes the following aspects:
[0203] (1) Obtaining the current server free memory: The collection task execution node needs to obtain the current server free memory status to ensure that there is enough free memory to start a new target collection task.
[0204] (2) Determine the startup memory of the target acquisition task: The startup memory of the target acquisition task refers to the memory size required to execute the task. This value should be determined in advance and match the actual memory situation.
[0205] (3) Comparison between server free memory and startup memory: Compare the free memory of the current server with the startup memory of the target collection task to determine whether there is enough free memory to start the target collection task.
[0206] (4) Decision to execute or recycle tasks: If the free memory of the current server is larger than the startup memory of the target collection task, the target collection task is executed (proceeding to step S5). Otherwise, the target collection task is returned to the collection task pool for recycling, and waits for the next collection task execution node to obtain the target collection task.
[0207] Through the above steps, the node can effectively schedule and execute the collection tasks according to the memory situation, avoiding unstable or abnormal task execution due to insufficient memory.
[0208] The multi-node task allocation methods in the prior art all require a centralized task allocation center, and the task allocation of all nodes is uniformly managed and scheduled by the centralized task allocation center. However, if there is network isolation, resources cannot be effectively coordinated. Figure 2 shown.
[0209] like Figure 3 As shown, this embodiment takes stateless tasks as a starting point and can solve the drawbacks of the above traditional task allocation from the mechanism of the method itself, and mainly solves the following problems:
[0210] (1) This embodiment implements a completely decentralized communication network equipment data collection task scheduling implementation method. There is no need for a central task allocation node to uniformly allocate tasks. All task scheduling and allocation strategies are completed by the nodes. Therefore, in theory, it has the characteristics of unlimited horizontal expansion in terms of task scale.
[0211] (2) This embodiment takes into account the performance load of the task execution node, so that the communication network equipment data collection task has a more efficient task execution capability after being assigned.
[0212] This embodiment can take into account the ability of the collection node to normally obtain the collection task when the network between different collected devices is not connected in a complex communication network.
[0213] Specific as Figure 3 As shown in the figure, it is a schematic diagram of the task allocation mechanism, which includes a task pool and multiple network nodes for managing and executing collection tasks.
[0214] (1)Task Pool:
[0215] The task pool is where all pending tasks are stored. Different collection task queues are created in the task pool according to different network identification feature codes.
[0216] The figure shows multiple, at least two task queues: Network A Task Queue and Network B TaskQueue, which store tasks belonging to network A and network B respectively.
[0217] (2) Network Nodes:
[0218] The network node is responsible for getting tasks from the task pool and executing them. There are multiple, at least two network nodes in the picture: Network A Node and Network B Node, corresponding to Network A and Network B respectively.
[0219] These nodes obtain tasks from the corresponding task queues and execute them.
[0220] (3) Network Isolation:
[0221] The picture shows two dotted boxes, labeled Network A and Network B, indicating that the two networks are isolated and task scheduling and execution are performed in their respective independent network environments.
[0222] (4) Network Element (NE):
[0223] The task performed by each network node is to act on a network element (NE), and each network node can act on multiple network elements.
[0224] Each network node in the figure is connected to multiple network elements, which are the target devices for performing specific collection tasks.
[0225] This embodiment provides a decentralized data collection and scheduling method based on ultra-large-scale IT equipment, that is, a decentralized data collection and scheduling method, which can effectively allocate data collection tasks while effectively considering the resource occupancy of each node, ensure the execution of data collection tasks without duplication and omission, and provide stable and reliable basic data support for large-scale network operation and maintenance. The overall data flow diagram of this embodiment is as follows: Figure 4 As shown in the figure, the overall business process diagram is as follows Figure 5 The specific steps include:
[0226] Step 1: Use the list of collected network elements saved in CMDB as the basis for splitting the collection tasks, split according to the collection frequency of equipment network operation and maintenance monitoring, and generate multiple collection task lists for the collection cycle of each collected device, and save them to the task pool. See the collection task splitting business process diagram for details. Figure 6 .
[0227] Step 1.1: The collection task producer (Task Maker) obtains the list of network elements to be collected through the network element resource management library (CMDB).
[0228] Step 1.2: Read the pre-configured collection task splitting cycle strategy, and periodically determine whether the task splitting time has arrived. If the task splitting time has arrived, execute step 1.3; otherwise, wait for the next collection task splitting cycle.
[0229] Step 1.3: The collection task producer generates split collection tasks (Task) based on the collection frequency cycle and collection characteristics of the collected network elements and the collecting network elements, and records the basic resource collection information such as the collection network element information, priority, collection time frequency, task startup memory size, etc. recorded in the CMDB in the collection task into the collection task parameters for use in subsequent collection task execution.
[0230] Step 1.4: The collection task producer saves the split collection tasks to different collection queues (Task Queue) of the task pool (Task Pool) according to the actual network isolation conditions of different network elements.
[0231] Step 1.5: Read the network IDs of different network elements in the CMDB, encrypt the network IDs using the MD5 algorithm to obtain the network ID signature code, create different collection queues (Task Queue) in the task pool (Task Pool) according to the network ID signature code, and put the collection tasks split in step 1.3 into the collection queues of the corresponding network ID signature codes according to the network ID signature codes of the network elements.
[0232] Step 2: The collection node obtains the performance of the corresponding node server and generates a collection task acquisition strategy. The collection strategy generation business process diagram is shown in Figure 7 .
[0233] Step 2.1: The collection node obtains the CPU utilization of the server using the symbol p cpuused Indicates the number of CPU cores using the symbol c cpunum Indicates the total memory size using the symbol c memtotle Indicates the size of memory usage using the symbol c memused Indicates that the constant time t is set in advance design Used to indicate the maximum delay time in each processing step.
[0234] In Linux
[0235] The CPU utilization is obtained by executing the command top -b -n 2-d 1|grep Cpu|grep id|grep wa|tr',"'|tr'%"'|awk'{print$8}'.
[0236] According to the instruction
[0237] p cpuused =98.4%
[0238] The number of CPU cores can be obtained by executing the command cat / proc / cpuinfo | grep
[0239] "processor"|wc -l after obtaining
[0240] According to the instruction
[0241] c cpunum =8
[0242] The used memory is obtained by executing the command free -m | awk '{print$3}' | sed -n '2p'
[0243] According to the instruction
[0244] c memused =9120
[0245] The total memory is obtained by executing the command free -m | awk '{print$2}' | sed -n '2p'
[0246] According to the instruction
[0247] c memtotle =16046
[0248] Disk IO utilization is obtained by executing the command iostat -x
[0249] According to the instruction
[0250] p ioused =0.07%
[0251] Obtained through WMI service in Windows system.
[0252] Assuming the constant t design , the maximum delay time is 5 seconds.
[0253] Step 2.2: The collection node calculates the average single-core CPU utilization of the server and calculates the delay to obtain the collection task time.
[0254] The formula for calculating the average CPU single core utilization is as follows:
[0255]
[0256] Substituting the data, we get the average single-core CPU utilization:
[0257]
[0258] In the above formula, p cpuavg is the average single-core CPU utilization, p cpuused is the overall CPU utilization of the server, c cpunum The number of server cores.
[0259] The delayed acquisition task time is shown in the following formula:
[0260] t delay =p cpuavg t design
[0261] t delay =12.3%×5s=0.615s=615ms
[0262] In the above formula, t delay To delay the acquisition task time.
[0263] Step 2.3: The collection node calculates the memory utilization of the server and calculates the delay to obtain the collection task time.
[0264] For Windows systems, the memory utilization calculation formula is as follows:
[0265]
[0266] Substitute the data from step 2.1 to get the memory utilization
[0267]
[0268] In the above formula, p memused is the server memory utilization, c memused The server has used memory, c memtotle The total amount of server memory.
[0269] The delayed acquisition task time is shown in the following formula:
[0270] t delay =p memused t design
[0271] t delay =56.84%×5s=2.842s=2842ms
[0272] Step 2.4: The collection node calculates the IO utilization of the server and calculates the delay to obtain the collection task time.
[0273] t delay =p ioused t design
[0274] t delay =0.07%×5s=0.0035s≈4ms.
[0275] Step 3: The collection node obtains tasks from the collection task pool.
[0276] Step 4: After the collection node obtains the task, it obtains the idle content size of the current server and compares it with the startup memory size in the collection task parameters (the startup memory size is the startup memory size put in the collection task parameters). If the server idle memory is larger than the task startup memory, it checks whether the overall memory requirements are met. If so, the collection task is started. If the requirements are not met and the server idle memory is smaller than the task startup memory, the obtained collection task is put back into the collection task pool for recycling, waiting for other collection nodes to obtain this collection task.
[0277] Embodiment 2:
[0278] like Figure 8 As shown, this embodiment provides a decentralized collection and scheduling device based on ultra-large-scale IT equipment, and the device includes:
[0279] A first acquisition unit 10 is used to acquire a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list;
[0280] The first processing unit 20 is connected to the first acquisition unit 10, and is used to split the network element list into multiple collection tasks; and is also used to save the collection tasks into a collection task pool;
[0281] The second processing unit 30 is connected to the first processing unit 20 and is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the server performance index;
[0282] A second acquisition unit 40, connected to the second processing unit 30, is used to acquire a target acquisition task from the acquisition task pool through an acquisition task execution node according to the acquisition task scheduling strategy;
[0283] The third processing unit 50 is connected to the second acquisition unit 40 and is used to execute the target acquisition task to complete the decentralized acquisition scheduling based on ultra-large-scale IT equipment;
[0284] There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
[0285] As a specific implementation, the first acquisition unit 10 includes:
[0286] Login module, used to log in to the CMDB system;
[0287] The acquisition module is connected to the login module and is used to search and filter the list of network elements to be collected according to the query function or search function provided by the CMDB system, that is, to obtain the list of network elements to be collected.
[0288] As a specific implementation, the first processing unit 20 includes a first processing subunit and a second processing subunit; the first processing subunit includes splitting the network element list into multiple collection tasks, and the second processing subunit includes saving the collection task into a collection task pool;
[0289] The first processing subunit comprises:
[0290] Preprocessing module, used to preset the collection task splitting cycle;
[0291] A splitting module, connected to the preprocessing module, for splitting the network element list into multiple collection tasks based on the collection task splitting cycle and the attribute characteristics of the collection tasks;
[0292] The second processing subunit comprises:
[0293] A creation module is used to create different collection queues in the task pool according to the network identification signature code;
[0294] The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications with the MD5 algorithm;
[0295] The placement module is connected to the creation module and is used to place the split collection tasks into corresponding collection queues according to the network identification feature codes where the network elements are located.
[0296] As a specific implementation, the device further includes a determination unit, and the determination unit is connected to the second acquisition unit 40 and the third processing unit 50 respectively;
[0297] The determination unit is used to determine whether the server free memory of the current acquisition task execution node is greater than the startup memory of the target acquisition task;
[0298] Wherein, the third processing unit 50 executes the target collection task when the free memory of the server of the current collection task execution node is greater than the startup memory of the target collection task according to the determination result of the determination unit.
[0299] As a specific implementation method, the server performance indicators include CPU utilization, CPU core number, memory utilization, and disk IO utilization;
[0300] The second processing unit 30 includes:
[0301] A first calculation module, used for calculating a first delay time of an acquisition task according to CPU utilization and the number of CPU cores;
[0302] The calculation formula of the first delay time of the acquisition task is as follows:
[0303] t1 delay =p cpuavg t design
[0304] in,
[0305] t1 delay is the first delay time of the acquisition task;
[0306] p cpuavg is the average CPU utilization;
[0307] t design To design the delay time;
[0308] p cpuavg The calculation formula is as follows:
[0309]
[0310] p cpuused The current CPU utilization rate ranges from 0 to 1.
[0311] c cpunum is the number of CPU cores, which must be a positive integer;
[0312] A second calculation module, connected to the first calculation module, used to calculate a second delay time for acquiring the acquisition task according to the memory utilization;
[0313] The calculation formula of the second delay time of the acquisition task is as follows:
[0314] t2 delay =p memused t design
[0315] in,
[0316] t2 delay is the second delay time of the acquisition task;
[0317] p memused is the memory utilization;
[0318] Memory utilization p memused The calculation formula is as follows:
[0319]
[0320] c memused is the size of the used memory; c memtotle is the total memory size;
[0321] A third calculation module, connected to the second calculation module, is used to calculate a third delay time for acquiring the acquisition task according to the disk IO utilization rate;
[0322] t3 delay =p ioused t design
[0323] t3 delay is the third delay time of the acquisition task;
[0324] p ioused is the disk IO utilization;
[0325] The strategy generation module is connected to the first calculation module, the second calculation module and the third calculation module respectively, and is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the first delayed task time, the second delayed task time and the third delayed task time.
[0326] The device in this embodiment can execute the method in embodiment 1.
[0327] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A collection and scheduling method based on the decentralization of ultra-large-scale IT equipment, characterized in that: The method comprises the following steps: Step S1: obtaining a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list; Step S2: splitting the network element list into multiple collection tasks; And, saving the acquisition task into an acquisition task pool; Step S3: Generate a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators; Step S4: acquiring a target collection task from the collection task pool through a collection task execution node according to the collection task scheduling strategy; Step S5: executing the target collection task to complete the decentralized collection scheduling based on ultra-large-scale IT equipment; There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
2. The collection and scheduling method based on the decentralization of ultra-large-scale IT equipment according to claim 1 is characterized in that: In step S1, the list of network elements to be collected is obtained through the CMDB system, which specifically includes the following steps: Step S11: Log in to the CMDB system; Step S12: According to the query function or search function provided by the CMDB system, the list of network elements to be collected is searched and screened, that is, the list of network elements to be collected is obtained.
3. The collection and scheduling method based on the decentralization of ultra-large-scale IT equipment according to claim 1 is characterized in that: In step S2, the network element list is split into multiple collection tasks, which specifically includes the following steps: Step A1: preset the collection task splitting cycle; Step A2: based on the collection task splitting cycle, split the network element list into multiple collection tasks according to the attribute characteristics of the collection tasks; In S2, the collection task is saved in a collection task pool, which specifically includes the following steps: Step B1: Create different collection queues in the task pool according to the network identification feature code; The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications with the MD5 algorithm; Step B2: Put the split collection tasks into corresponding collection queues according to the network identification feature codes of the network elements.
4. The collection and scheduling method based on the decentralization of ultra-large-scale IT equipment according to claim 1 is characterized in that: After step S4 and before step S5, the method further includes step K; Step K: Determine whether the server free memory of the current collection task execution node is greater than the startup memory of the target collection task: If the server free memory of the current collection task execution node is greater than the startup memory of the target collection task, then go to step S5, otherwise the target collection task is put back into the collection task pool for recycling, waiting to be acquired by the next collection task execution node.
5. The decentralized collection and scheduling method based on ultra-large-scale IT equipment according to any one of claims 1 to 4, characterized in that: The server performance indicators include CPU utilization, number of CPU cores, memory utilization, and disk IO utilization; The step S3 generates a collection task scheduling strategy based on the collection tasks in the collection task pool according to the server performance indicators, and specifically includes the following steps: Step S31: Calculate the first delay time of acquiring the acquisition task according to the CPU utilization and the number of CPU cores; The calculation formula of the first delay time of the acquisition task is as follows: t1 delay =p cpuavg t design in, t1 delay is the first delay time of the acquisition task; p cpuavg is the average CPU utilization; t design To design the delay time; p cpuavg The calculation formula is as follows: p cpuused The current CPU utilization rate ranges from 0 to 1. c cpunum is the number of CPU cores, which must be a positive integer; Step S32: Calculate the second delay time for acquiring the acquisition task according to the memory utilization rate; The calculation formula of the second delay time of the acquisition task is as follows: t2delay=p memused t design in, t2 delay is the second delay time of the acquisition task; p memused is the memory utilization; Memory utilization p memused The calculation formula is as follows: c memused is the size of the used memory; c memtotle is the total memory size; Step S33: Calculate the third delay time for acquiring the acquisition task according to the disk IO utilization rate; t3 delay =p ioused t design t3 delay is the third delay time of the acquisition task; p ioused is the disk IO utilization; Step S34: generating a collection task scheduling strategy according to the collection tasks in the collection task pool according to the first delayed task time, the second delayed task time, and the third delayed task time.
6. A decentralized collection and scheduling device based on ultra-large-scale IT equipment, characterized in that: The device comprises: A first acquisition unit is used to acquire a list of network elements to be collected, wherein the list of network elements includes information of each network element device listed in the form of a network element list; A first processing unit, connected to the first acquisition unit, configured to split the network element list into a plurality of collection tasks; and further configured to save the collection tasks into a collection task pool; A second processing unit, connected to the first processing unit, is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the server performance index; A second acquisition unit, connected to the second processing unit, and configured to acquire a target acquisition task from the acquisition task pool through an acquisition task execution node according to the acquisition task scheduling strategy; a third processing unit, connected to the second acquisition unit, and configured to execute the target acquisition task to complete the decentralized acquisition scheduling based on ultra-large-scale IT equipment; There are multiple collection task execution nodes, and any collection task execution node can collect tasks from the collection task pool.
7. The decentralized collection and scheduling device based on ultra-large-scale IT equipment according to claim 6 is characterized in that: The first acquiring unit includes: Login module, used to log in to the CMDB system; The acquisition module is connected to the login module and is used to search and filter the list of network elements to be collected according to the query function or search function provided by the CMDB system, that is, to obtain the list of network elements to be collected.
8. The decentralized collection and scheduling device based on ultra-large-scale IT equipment according to claim 6 is characterized in that: The first processing unit includes a first processing subunit and a second processing subunit; the first processing subunit includes splitting the network element list into multiple collection tasks, and the second processing subunit includes saving the collection task into a collection task pool; The first processing subunit comprises: Preprocessing module, used to preset the collection task splitting cycle; A splitting module, connected to the preprocessing module, for splitting the network element list into multiple collection tasks based on the collection task splitting cycle and the attribute characteristics of the collection tasks; The second processing subunit comprises: A creation module is used to create different collection queues in the task pool according to the network identification signature code; The network identification feature code is obtained by reading the network identifications of different network elements in the CMDB and then encrypting these network identifications with the MD5 algorithm; The placement module is connected to the creation module and is used to place the split collection tasks into corresponding collection queues according to the network identification feature codes where the network elements are located.
9. The decentralized collection and scheduling device based on ultra-large-scale IT equipment according to claim 6 is characterized in that: The device further comprises a determination unit, wherein the determination unit is connected to the second acquisition unit and the third processing unit respectively; The determination unit is used to determine whether the server free memory of the current acquisition task execution node is greater than the startup memory of the target acquisition task; Wherein, the third processing unit executes the target collection task when the idle memory of the server of the current collection task execution node is greater than the startup memory of the target collection task according to the determination result of the determination unit.
10. The decentralized collection and scheduling device based on ultra-large-scale IT equipment according to any one of claims 6 to 9, characterized in that: The server performance indicators include CPU utilization, number of CPU cores, memory utilization, and disk IO utilization; The second processing unit comprises: A first calculation module, used for calculating a first delay time of an acquisition task according to CPU utilization and the number of CPU cores; The calculation formula of the first delay time of the acquisition task is as follows: t1 delay =p cpuavg t design in, t1 delay is the first delay time of the acquisition task; p cpuavg is the average CPU utilization; t design To design the delay time; p cpuavg The calculation formula is as follows: p cpuused The current CPU utilization rate ranges from 0 to 1. c cpunum is the number of CPU cores, which must be a positive integer; A second calculation module, connected to the first calculation module, used to calculate a second delay time for acquiring the acquisition task according to the memory utilization; The calculation formula of the second delay time of the acquisition task is as follows: t2 delay =p memused t design in, t2 delay is the second delay time of the acquisition task; p memused is the memory utilization; Memory utilization p memused The calculation formula is as follows: c memused is the size of the used memory; c memtotle is the total memory size; A third calculation module, connected to the second calculation module, is used to calculate a third delay time for obtaining the acquisition task according to the disk IO utilization rate; t3 delay =p ioused t design t3 delay is the third delay time of the acquisition task; p ioused is the disk IO utilization; The strategy generation module is connected to the first calculation module, the second calculation module and the third calculation module respectively, and is used to generate a collection task scheduling strategy according to the collection tasks in the collection task pool based on the first delayed task time, the second delayed task time and the third delayed task time.
