A resource scheduling method based on intelligent monitoring
Through intelligent monitoring technology, real-time monitoring and management of physical and virtual resources of the cloud system, the problem of low resource scheduling efficiency is solved, and the intelligent management and stable and secure operation of cloud platform resources are realized.
Patent Information
- Application Number
- CN202011075333.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-10-09
AI Technical Summary
It is difficult for existing cloud systems to achieve intelligent monitoring of physical and virtual resources, resulting in low resource scheduling efficiency and the inability to promptly detect and deal with resource overload or insufficient problems.
Using a resource scheduling method based on intelligent monitoring, the CPU, memory, disk and load conditions of physical machines and virtual machines are monitored in real time through node collectors, cluster controllers and aggregation systems, and the status of physical machines is judged based on preset thresholds. Filtering is not available. Resources, and data statistics and alarms are performed through data collectors and virtual machine monitors, realizing intelligent allocation and management of resources.
It realizes intelligent dynamic management of cloud platform resources, improves resource utilization, ensures the stability and security of the cloud computing environment, provides system management related functions, dynamically displays the operating status of all levels of the cloud system, and enhances operation management controllability.
Smart Images

Figure CN114327849B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for scheduling cloud system resources, and in particular to a method for scheduling resources based on intelligent monitoring. Background Art
[0002] With the widespread application of cloud systems, it is becoming increasingly urgent to provide operational, monitorable, manageable and measurable services to ensure the stable and secure operation of cloud computing users' resources. Summary of the invention
[0003] The purpose of the present invention is to provide a resource scheduling method based on intelligent monitoring, which is to monitor physical resources and virtualized resources of a cluster;
[0004] The total number of CPU cores occupied by the virtual machines on the physical machine + the CPU setting value >= the number of cores of the physical machine, and the node collector (AC for short) is reported. The node collector reports to the aggregation system (Aggregator) client to confirm that the physical machine status is unavailable. When the cluster controller requests to query resources, it will filter out the physical machine for use;
[0005] The total memory usage of the virtual machines on the physical machine + the memory setting value >= the total memory of the physical machine, and the node collector is reported. The node collector will report to the aggregation system client to confirm that the physical machine status is unavailable. When the cluster controller requests to query resources, the physical machine is filtered out for use;
[0006] The total number of virtual machine disks on the physical machine + the disk setting value >= the total number of physical machine disks, and the node collector system is reported. The node collector will report to the aggregation system client to confirm that the physical machine status is unavailable. When the cluster controller requests to query resources, the physical machine is filtered out for use;
[0007] The total number of virtual machines on the physical machine is greater than or equal to the load setting value. The node collector system is reported. The node collector will report to the aggregation system client to confirm that the physical machine status is unavailable. When the cluster controller requests to query resources, the physical machine is filtered out.
[0008] The aggregation system client synchronizes the physical machine status information to the aggregation system server;
[0009] The cluster controller (Central Controller, CC for short) obtains physical machine status information from the aggregation system server and provides it for external calls through the application program interface; it can select physical resources and create resources according to external requests.
[0010] The data collector module of physical resource monitoring is responsible for collecting data on physical resources through the protocol and reporting it to the node collector system; virtualization resource monitoring connects to the virtual machine monitors on each physical machine through the network, identifies the virtual machines on the physical machine one by one, and summarizes the allocated capacity and usage of the virtual network, CPU, and memory; the virtual machine monitor monitors the working status of the virtual machine in heartbeat mode and quantifies the indicators.
[0011] The data collection of physical resources is divided into two modes: active collection and passive reception. Active collection collects information from the equipment through periodic polling; passive reception receives notifications from asynchronous receiving equipment and third-party systems.
[0012] Both active and passive collection require the collector to have a unified interface, expand supported device types through plug-ins, and expand device properties through configuration files.
[0013] Select physical resources to create the resource path as follows:
[0014] The first step is to find a matching resource pool based on the model and configuration you want to apply for;
[0015] The second step is to obtain all physical machines in the resource pool that meet the requirements for filtering.
[0016] The third step is to calculate the number of disks, memory, and CPU cores of the physical machine in the database.
[0017] The fourth step is to determine whether the remaining capacity is greater than the application amount;
[0018] Step 5: Sort the physical machines in the set in descending order according to memory, disk, and CPU;
[0019] Step 6: Obtain the physical machine parameters of the monitoring platform and filter out online physical machines;
[0020] Step 7: Obtain the sum of the CPU usage, disk usage, and memory usage of the physical machine in the calculation monitoring data and sort them in ascending order;
[0021] Step 8: Obtain the current load status of the physical machine from the monitoring data and sort it in ascending order;
[0022] Step 9: Get the first piece of data from the above sorting results to create resources.
[0023] This method implements the corresponding functions through node collector, cluster controller and aggregation system:
[0024] The cluster controller is responsible for the resource management within the cluster, including the addition, deletion, and modification of physical machines, the addition, deletion, modification, start and stop, and shutdown of virtual machines, the resource usage of the resource pool, the distribution of virtual machines, the scheduling of virtual machines, and the application or recovery of resources; the aggregation system is divided into a client (Agent) and a server (Server). The client is installed on each physical machine, collects the status and other information of all running subsystems on the physical machine and reports it to the server, as well as responds to queries and service requests issued by applications; the server collects the status information of the entire system and responds to the query requests of the cluster controller; the node collector is the information collection system of the virtual machine and the physical machine, including the performance data and log data of the machine; and communicates with the aggregation system client to report the status, performance, log and other information of the physical machine.
[0025] Beneficial effects achieved by the technical solution of the present invention:
[0026] The present invention is designed to intelligently and dynamically display the operating status of each level, each module, and each function of the cloud system for cloud platform operation and management personnel, and provide related functions for system management. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a schematic diagram of a resource scheduling method based on intelligent monitoring of the present invention.
[0028] Figure 2 The invention discloses a flow chart of creating resources in a resource scheduling method based on intelligent monitoring. DETAILED DESCRIPTION
[0029] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are only used to illustrate the technical solution of the present invention but not to limit the protection scope of the present invention.
[0030] like Figure 1 As shown, the present invention provides a resource scheduling method based on intelligent monitoring, which is based on physical resource monitoring and virtualized resource monitoring; physical resource monitoring refers to the cloud computing environment running on traditional IT infrastructure, and the infrastructure includes physical servers, switches, routers, firewalls and other equipment; the operation of the cloud computing environment needs to ensure the normal operation of the IT infrastructure, and the data collector module of the cloud computing monitoring system is responsible for collecting data on physical resources and reporting it to the node collector system through various protocols: SNMP, TRAP.SYSLOG, XFLOW, IPMI, ICMP, TELNET, etc.
[0031] Data collection should be divided into two methods: active collection and passive reception:
[0032] a. Active collection: periodically polling the information of the collection device;
[0033] b. Passive reception: asynchronous reception of notifications from devices and third-party systems.
[0034] Both methods require the collector to have a unified interface, expand supported device types through plug-ins, and expand device properties through configuration files.
[0035] Virtualization resource monitoring connects to the VMM (virtual machine monitor) on each physical machine through the network, identifies each virtual machine on each physical machine one by one, and summarizes the statistics of the virtual network, CPU, and memory allocation capacity and usage. Monitor the working status of each virtual machine in heartbeat mode to detect common anomalies. Quantify the relevant indicators, provide API access interfaces to the scheduling and management levels, and report to the node collector system.
[0036] The node collector algorithm calculation rules are as follows:
[0037] CPU: When the total number of cores occupied by virtual machines on the physical machine + 2 >= the number of cores of the physical machine, an alarm will be triggered and reported to the node collector system. The node collector will report to the aggregation system client to inform it that the physical machine status is unavailable. After that, the cluster controller will filter out the physical machine when requesting resource query.
[0038] Memory: When the total memory usage of the virtual machines on the physical machine + 1G >= the total memory of the physical machine, an alarm will be triggered and reported to the node collector system. The node collector will report to the aggregation system client to inform it that the physical machine status is unavailable. After that, the cluster controller will filter out the physical machine when requesting resource query.
[0039] Disk: When the total number of virtual machine disks on the physical machine + 20G >= the total number of physical machine disks, an alarm will be triggered and reported to the node collector system. The node collector will report to the aggregation system client to inform it that the physical machine status is unavailable. When the cluster controller requests to query resources, it will filter out the physical machine for use.
[0040] Load: When the total number of virtual machines on a physical machine is greater than or equal to 50, an alarm will be triggered and reported to the node collector system. The node collector will report to the aggregation system client to inform it that the physical machine is unavailable. After that, the cluster controller will filter out the physical machine when requesting to query resources. Through physical resource monitoring and virtualization resource monitoring, the CPU, memory, disk, and load of the physical machine are calculated according to the physical resource algorithm calculation rules, and physical resources are selected for allocation and new resources are created.
[0041] The calculation rules of the physical resource algorithm are as follows:
[0042] CPU: The database initializes the total number of CPU cores of the physical machine and reserves 2 physical cores for each physical machine;
[0043] Memory: The database initializes the total memory of the physical machine and reserves 1024M space for each physical machine;
[0044] Disk: The total number of physical machine disks for database initialization, and 100G space is reserved for each physical machine;
[0045] Physical machine busy factor: The maximum concurrency factor of the database initialization physical machine is 3, that is, a physical machine can issue heavy task operations to a maximum of 3 machines at the same time, and the others wait in line.
[0046] Here we take the application machine size: 2-core CPU\4G memory\100G disk as an example.
[0047] CPU core count calculation rule: Requested core count = Physical core count / 2
[0048] The disk calculation rules are as follows:
[0049] Apply for memory ≤ 8G, disk space = system disk*2+data disk+2;
[0050] Windows system: swap=0;
[0051] Linux system: apply for memory ≤ 8G, swap = 2;
[0052] 16G=>applied memory>86, swap=4;
[0053] Request memory>16G, swap=6;
[0054] The final disk capacity occupied by the memory requested = requested system disk capacity * 2 + requested data disk system capacity + swap;
[0055] The memory calculation rule is as follows: applied memory = physical memory.
[0056] Select physical resources for allocation and create resource paths as follows:
[0057] Find the matching resource pool based on the model and configuration you want to apply for;
[0058] Get all physical machines in the resource pool that meet the requirements for filtering;
[0059] Calculate the number of disks, memory, and CPU cores of these physical machines in the database respectively;
[0060] Determine whether the remaining capacity is greater than the application amount;
[0061] Sort the physical machines in the collection in descending order by memory, disk, and CPU;
[0062] Obtain the physical machine parameters of the monitoring platform and filter out online physical machines;
[0063] Obtain the sum of the CPU usage + disk usage + memory usage of each physical machine in the calculation monitoring data and sort them in ascending order;
[0064] Finally, obtain the current load status of each physical machine from the monitoring data and sort them in ascending order;
[0065] Get the first piece of data from the above sorting results to create resources.
[0066] Data collection of physical resources is divided into two modes: active collection and passive reception. Active collection periodically polls the information of the collection device; passive reception asynchronously receives notifications from devices and third-party systems. Both active and passive collection require the collector to have a unified interface, expand the supported device types through plug-ins, and expand device properties through configuration files.
[0067] This method realizes corresponding functions through node collectors, cluster controllers, and aggregation systems.
[0068] Cluster controller: responsible for the resource management within the cluster, including the management of physical machines (addition, deletion, modification, etc.), virtual machine management (addition, deletion, modification, start and stop, shutdown, etc.), resource pool management (resource usage, virtual machine distribution, virtual machine scheduling management, resource application and recovery, etc.). The cluster controller is the core of the computing resource system. All request operations such as "generate", "delete", "restart", and "search" from the API need to be coordinated by the cluster controller.
[0069] Aggregation system: The aggregation system is divided into a client (Agent) and a server (Server). The client is installed on each physical machine, collects the status and other information of all subsystems running on the physical machine and reports it to the server, as well as responds to queries and service requests issued by applications; the server collects the status information of the entire system and responds to query requests from the cluster controller.
[0070] The node collector is an information collection system for virtual machines and physical machines, including the performance data and log data of the machines. It communicates with the aggregation system client to report the status, performance, log and other information of the physical machines.
[0071] Resource monitoring management and service monitoring provide operational, monitorable, manageable and measurable services for resources and services to ensure stable and secure operation of cloud computing users' resources. It will display the operating status of each level, module and function of the cloud system to cloud platform operation and management personnel, and provide related functions for system management, including collecting performance, alarm and configuration information of monitored objects on the cloud platform, presenting this information in an appropriate form, notifying operation and management personnel when an alarm occurs, etc., and providing the function of sending specified performance, alarm and configuration information to external systems.
Claims
1. A resource scheduling method based on intelligent monitoring, characterized in that: Perform physical resource monitoring and virtualized resource monitoring on the cluster; the total number of CPU cores occupied by virtual machines on the physical machine + CPU setting value >= the number of physical machine cores, report to the node collector, the node collector reports to the aggregation system client, confirms that the state of the physical machine is unavailable, and the cluster controller will filter out the use of the physical machine when requesting resource query; The total number of virtual machine disks on the physical machine + the disk setting value >= the total number of physical machine disks, and report to the node collector system. The node collector will report to the aggregation system client to confirm that the physical machine status is unavailable, and the cluster controller will filter out the physical machine when requesting to query resources; The total number of virtual machines on the physical machine is greater than or equal to the load setting value, and the node collector system is reported. The node collector will report to the aggregation system client to confirm that the state of the physical machine is unavailable, and the cluster controller will filter out the use of the physical machine when requesting to query resources; The aggregation system client synchronizes the physical machine status information to the aggregation system server; The cluster controller obtains the physical machine status information from the aggregation system server, and provides the physical machine status information for external invocation through an application program interface; Select physical resources to create resources based on external requests.
2. The resource scheduling method based on intelligent monitoring according to claim 1 is characterized in that The data collector module for physical resource monitoring is responsible for collecting data on physical resources through a communication protocol and reporting it to the node collector system; The virtualized resource monitoring is connected to the virtual machine monitors on each physical machine through the network, identifies the virtual machines on the physical machine one by one, and summarizes the allocated capacity and usage of the virtual network, CPU, and memory; The virtual machine monitor monitors the working status of the virtual machine in a heartbeat mode and quantifies the indicators; The data collection of the physical resources is divided into two modes: active collection and passive reception. The active collection periodically polls the information of the collection device; the passive reception is asynchronous reception of notifications sent by the device and the third-party system; Both active collection and passive collection require the collector to have a unified interface, expand supported device types through plug-ins, and expand device properties through configuration files.
3. The resource scheduling method based on intelligent monitoring according to claim 1 is characterized in that The path for selecting physical resources to create resources is as follows: The first step is to find a matching resource pool based on the model and configuration you want to apply for; The second step is to obtain all physical machines in the resource pool that meet the requirements for filtering. The third step is to calculate the number of disks, memory, and CPU cores of the physical machine in the database respectively; The fourth step is to determine whether the remaining capacity is greater than the application amount; Step 5: Sort the physical machines in the set in descending order according to memory, disk, and CPU; Step 6: Obtain the physical machine parameters of the monitoring platform and filter out online physical machines; Step 7: Obtain the sum of the CPU usage, disk usage, and memory usage of the physical machine in the calculation monitoring data and sort them in ascending order; Step 8: Obtain the current load status of the physical machine from the monitoring data and sort it in ascending order; Step 9: Get the first piece of data from the above sorting results to create resources.
4. The resource scheduling method based on intelligent monitoring according to claim 1 is characterized in that The cluster controller is responsible for the resource management within the cluster, including the addition, deletion, and modification management of the physical machines, the addition, deletion, modification, start and stop, and shutdown management of virtual machines, the resource usage of the resource pool, the distribution of virtual machines, the scheduling management of virtual machines, and the application or recovery of resources; The aggregation system is divided into a client and a server. The client is installed on each physical machine, collects the status and other information of all subsystems running on the physical machine and reports to the server, and responds to queries and service requests issued by applications; The server collects the status information of the entire system and responds to the query request of the cluster controller; The node collector is an information collection system for virtual machines and physical machines, including performance data and log data of the machines; and communicates with the aggregation system client to report the status, performance, and logs of the physical machines.
