Multi-machine resource configuration management method, device, equipment, medium and product

Abstract

The invention discloses a multi-machine resource configuration management method and device, equipment, a medium and a product, and relates to the technical field of multi-machine resource configuration.The method comprises the steps that queue allocation is conducted on satellites according to processing priorities; distributing machine nodes for each job queue according to machine node resources required by the job queues to obtain a plurality of job queues and a plurality of corresponding machine node groups; acquiring state information of each host node in the machine node group, host resource load information and resource information required by each job in the job queue in real time, and matching each job in the job queue to the machine node in the corresponding machine node group according to the state information and the host resource load information of each host node in the machine node group; and activating the jobs meeting the conditions one by one according to the priority of each job in the job queue and executing the jobs in parallel. The whole computer network system is changed into a relatively single virtual host or super computer, so that the effect of providing a relatively single service operation platform for upper-layer conventional service scheduling is realized.

Description

Technical Field

[0001] The present application relates to the technical field of multi-machine resource configuration, and in particular to a multi-machine resource configuration management method, device, equipment, medium and product. Background Art

[0002] The business operation platform is usually a computer and network system with complex business associations, high performance requirements, and large data throughput. As the scheduling management center, the business operation and resource allocation management methods rely on high-performance computer clusters, high-speed networks, and mature and advanced system software. They deeply analyze the resource requirements of each business operation, and reasonably arrange the operation according to the schedule and requirements of the operation control system, data processing flow, and system resource status, combined with the operation characteristics and resource status. However, the existing technology cannot maximize the advantages of system resources. Summary of the invention

[0003] The purpose of this application is to provide a multi-machine resource configuration management method, device, equipment, medium and product that can maximize the advantages of system resources.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a multi-machine resource configuration management method, the multi-machine resource configuration management method comprising:

[0006] Allocate each satellite to a queue according to the processing priority to obtain several operation queues;

[0007] Allocating a machine node to each of the job queues according to the machine node resources required by the job queues, to obtain a plurality of job queues and a corresponding plurality of machine node groups;

[0008] Acquire the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue in real time;

[0009] Matching each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information;

[0010] According to the priority of each job in the job queue, jobs that meet the condition are activated one by one for parallel execution; the condition that meets the condition is that the remaining resources of the machine node matching the job are sufficient to execute the job;

[0011] Monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, set the state domain of the machine node to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, the state domain of the machine node is an execution state.

[0012] In a second aspect, the present application provides a multi-machine resource configuration management device, the multi-machine resource configuration management device comprising:

[0013] Multi-machine node management module, used for:

[0014] Allocate each satellite to a queue according to the processing priority to obtain several operation queues;

[0015] Allocating a machine node to each of the job queues according to the machine node resources required by the job queues, to obtain a plurality of job queues and a corresponding plurality of machine node groups;

[0016] Resource status collection module, used to:

[0017] Acquire the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue in real time;

[0018] Matching each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information;

[0019] According to the priority of each job in the job queue, jobs that meet the condition are activated one by one for parallel execution; the condition that meets the condition is that the remaining resources of the machine node matching the job are sufficient to execute the job;

[0020] The multi-machine load balancing management module is used to monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, the state domain of the machine node is set to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, the state domain of the machine node is set to an execution state.

[0021] Optionally, the multi-machine load balancing management module includes: a master process and a plurality of slave processes;

[0022] The main process is responsible for comprehensively analyzing various input data according to the pre-set scheduling strategy to determine on which node host the job should be executed;

[0023] The slave process runs on the relevant host node; the slave process obtains the execution command parameters of the job and starts the relevant job process to execute the task.

[0024] Optionally, the master process is responsible for comprehensively analyzing various input data according to a pre-set scheduling strategy to determine on which node host the job is to be executed; the slave process running on the relevant host node obtains the execution command parameters of the job and starts the relevant job process to execute the task.

[0025] Optionally, the resource status collection module includes: a main service process and a group of slave processes;

[0026] The main process runs on the business system business scheduling master machine;

[0027] The slave process is on the aggregation host node; the slave process is responsible for collecting system load parameter information of each machine, including r1m, r15s, r15m, ut, mem, io, swp, LinkStat, JidFlag, XntpFlag, etc., and reporting to the main service process on the main server; the main service process on the main server is responsible for summarizing and analyzing the system load status information of the slave processes on each node, and generating relevant operation log information, real-time reports of system resources, and storing the necessary system resource information in the database.

[0028] Optionally, the multi-machine load balancing management module is also used to implement dynamic configuration and management of permanent service processes according to operator commands.

[0029] Optionally, when the multi-machine node management module assigns jobs, it considers the maximum number of jobs that can be submitted by each host, the CPU busyness, memory usage, and paging rate of the host, and other job loads and system resource loads, and performs scheduling and allocation based on the job's definition attributes, including the required number of CPUs, memory space, job running time, job priority, the queue configuration to which the job belongs, and the scheduling strategy defined in advance by the designer.

[0030] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned multi-machine resource configuration management method.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned multi-machine resource configuration management method when executed by a processor.

[0032] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned multi-machine resource configuration management method when executed by a processor.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] The present application provides a multi-machine resource configuration management method, device, equipment, medium and product, the method comprising: allocating queues for each satellite according to processing priority to obtain a plurality of job queues; allocating machine nodes for each job queue according to the machine node resources required by the job queue to obtain a plurality of job queues and a corresponding plurality of machine node groups; acquiring the status information of each host node in the machine node group and the host resource load information and the required resource information of each job in the job queue in real time; matching each job in the job queue to the machine node in the corresponding machine node group according to the required resource information, the status information of each host node and the host resource load information; activating the jobs that meet the conditions for parallel execution one by one according to the priority of each job in the job queue; the remaining resources of the machine node that meets the conditions for matching the job are sufficient to execute the job; monitoring the number information and resource usage value information of the jobs that are executed in parallel by each machine node in real time; when the number information and resource usage value information of the jobs that are executed in parallel reach a threshold value, setting the state field of the machine node to a busy state until the number information and resource usage value information of the jobs that are executed in parallel decrease, and the state field of the machine node to an execution state. The present application achieves the effect of providing a relatively single business operation platform for upper-level conventional business scheduling by turning the entire computer network system into a relatively single virtual host or supercomputer. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a multi-machine resource configuration management method provided in one embodiment of the present application;

[0037] Figure 2 A schematic diagram of the information flow of a multi-machine resource configuration management system and method provided in one embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of a multi-machine resource configuration management device provided in one embodiment of the present application;

[0039] Figure 4 A data flow diagram of a multi-machine node management module provided in an embodiment of the present application;

[0040] Figure 5 A data flow diagram of a multi-machine load balancing scheduling management module provided in an embodiment of the present application;

[0041] Figure 6 A data flow diagram of a resource status collection module provided in an embodiment of the present application;

[0042] Figure 7 A data flow diagram of the operation configuration guard module provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The present application provides a multi-machine resource configuration management method for unified management of all computing resources in a distributed system. The multi-machine resource configuration management method includes: a multi-machine node management module, a multi-machine load balancing management module, a resource status collection module, and an operation configuration guard module. The multi-machine node management module manages each machine node participating in the operation in the system in groups according to a predetermined node scheduling strategy. The multi-machine load balancing management module supports load balancing and dynamic allocation through a multi-level load balancing algorithm; at the same time, the module adopts a priority-based preemptive scheduling strategy to ensure that key tasks are run without constraints and interference; the resource status collection module collects the load parameters of each node host in the cluster in real time and analyzes the system resource usage status of each node host; at the same time, the module summarizes the system resource usage status in the cluster, receives the load status of each node host in the cluster for summary analysis, and provides the necessary real-time system resource information for the multi-machine load balancing manager. The operation configuration guard module monitors the operation status of the permanent service process in the business software deployed on the cluster master server in real time, and realizes the dynamic configuration and management of the permanent service process according to the operator's command. Through the present disclosure, various resources of cluster computers are fully utilized to realize dynamic scheduling of computing loads, improve the acceleration ratio of parallel tasks, shorten the job response time, and maximize the advantages of system resources.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0047] In an exemplary embodiment, Figure 1 As shown, a multi-machine resource configuration management method is provided, and the multi-machine resource configuration management method includes:

[0048] S1. Allocate each satellite to a queue according to the processing priority to obtain several operation queues.

[0049] S2. Allocate a machine node to each of the job queues according to the machine node resources required by the job queue, to obtain a plurality of job queues and a corresponding plurality of machine node groups.

[0050] S3. Acquire in real time the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue.

[0051] S4. Match each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information.

[0052] S5. According to the priority of each job in the job queue, activate the jobs that meet the conditions one by one for parallel execution; the condition that meets the conditions is that the remaining resources of the machine node matching the job are sufficient to execute the job.

[0053] S6. Monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, set the state domain of the machine node to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, and then set the state domain of the machine node to an execution state.

[0054] Specifically, the purpose of this embodiment is to provide a multi-machine resource configuration management method for unified management of all computing resources in a distributed system. Through this embodiment, various resources of cluster computers are fully utilized to achieve dynamic scheduling of computing loads, improve the parallel task acceleration ratio, shorten the job response time, and maximize the advantages of system resources.

[0055] See also Figure 2The multi-machine resource configuration management system and method group nodes according to the predetermined node scheduling strategy and the percentage of resources that different satellite processing operation groups should occupy for real-time operation; data processing of different satellite processing operation groups is completed by their respective node resource groups, realizing priority control and resource utilization strategy for multi-satellite business processing, ensuring reasonable allocation of system resources and coordinated and orderly processing of high-priority satellite data and business processing between multiple satellites.

[0056] The specific processing flow of the overall system is as follows:

[0057] The operation configuration daemon module implements the initialization, startup and termination of business processing of each server application system software (including multi-machine node management module, multi-machine load balancing management module, and resource status collection module), monitors the operation status of each module in the application system server software in real time, and realizes dynamic configuration and management of each module according to system configuration changes.

[0058] The multi-machine node management module allocates each machine node (machine node refers to different servers in the cluster where the program is running) to different computer groups according to the processing priority and resource occupancy of each satellite in the business system; defines the jobs of different satellite groups to be allocated to different job queues (job queues include high, medium and low priority queues for domestic satellites and high, medium and low priority queues for foreign satellites), and binds computer groups to job queues, that is, different satellite job groups run on different node groups; node scheduling strategies are set in the job queue configuration association; different job control and scheduling strategies are defined according to the corresponding job characteristics of each job queue, and different jobs of the same satellite can be allocated to different job queues according to their different characteristics; job queues formulate priority strategies according to the needs of satellite business; normal The job priority strategy is: first process the load data of each star with higher timeliness requirements, and then process the load data with lower timeliness requirements. The job priority strategy can also be dynamically adjusted according to user needs, among which the user can adjust the timeliness requirements; finally, the multi-machine node module sends the job operation distribution information and job queue information to the multi-machine load balancing management module; the multi-machine node management module of each node reports the online information of the node to the main node manager through the inter-node network. When the node manager of a node fails, the main node manager sets the online status of the node to "unavailable" in the system node scheduling association, blocks the node, and another node can automatically switch to take over the work of the multi-machine node management module of the failed node; finally, the multi-machine node management module sends the node status information to the multi-machine load balancing management module.

[0059] The resource status collection module obtains the job information submitted for running in each job queue, sorts the priority of the jobs in the job queue, the usage rate of the reserved resources (CPU, memory), etc. according to the job scheduling execution strategy, collects and monitors the resource usage information of each node in real time, activates the execution of jobs that meet the conditions (satisfying the conditions means that the node running the job currently meets the resources (CPU, memory, etc.) reserved for the job and can be processed), and implements the preemptive scheduling strategy between different queues and different priorities of the same satellite or different satellites on each node; finally, the resource status collection module sends the cluster load information to the multi-machine load balancing management module.

[0060] The multi-machine load balancing management module monitors the number of jobs executed in parallel and the resource information of each node in real time. When it reaches or approaches the threshold value, the node status domain is set to BUSY state, and the job submission state of the job queue is limited to suspended. When the number of jobs executed in parallel and the resource usage value of the node decrease, the job submission state of the job queue is restored to execution. Finally, the multi-machine load balancing management module sends the job management information to the multi-machine node management module.

[0061] In an exemplary embodiment, a multi-machine resource configuration management device is provided, which includes: a multi-machine node management module, a multi-machine load balancing management module, a resource status collection module, and a running configuration guard module. The specific components are as follows: Figure 3 shown.

[0062] Multi-machine node management module, used for:

[0063] Allocate each satellite to a queue according to the processing priority to obtain several operation queues;

[0064] Allocating a machine node to each of the job queues according to the machine node resources required by the job queues, to obtain a plurality of job queues and a corresponding plurality of machine node groups;

[0065] Resource status collection module, used to:

[0066] Acquire the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue in real time;

[0067] Matching each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information;

[0068] According to the priority of each job in the job queue, jobs that meet the condition are activated one by one for parallel execution; the condition that meets the condition is that the remaining resources of the machine node matching the job are sufficient to execute the job;

[0069] The multi-machine load balancing management module is used to monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, the state domain of the machine node is set to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, the state domain of the machine node is set to an execution state.

[0070] Specifically, the specific information of each module is shown in Table 1:

[0071] Table 1 Information table of a multi-machine resource configuration management device

[0072]

[0073]

[0074] See also Figure 4 The function of the multi-machine node management module is to find a mapping relationship between jobs and computers, and evenly distribute a series of jobs in the computer cluster to each computer according to the workload, so as to improve the overall computing efficiency of the system, provide the shortest average task response time, and adapt to changing loads.

[0075] The multi-machine node management module can balance the workload and system resource load. When assigning jobs, the maximum number of jobs that can be submitted by each host, the CPU busyness, memory usage, paging rate and other workloads and system resource loads are considered. The job definition attributes, including the required number of CPUs, memory space, job running time, job priority, job queue configuration, etc., and the scheduling strategy defined in advance by the designer are combined for scheduling and allocation. The currently available scheduling strategies include: First Come First Served FCFS; Fairshare; Preemption;

[0076] Exclusive scheduling Exclusive; host fair scheduling HostParation; resource reservation scheduling ResourceReservation.

[0077] The specific processing flow of the module is as follows:

[0078] Receive the job information to be executed submitted by the main process on the main scheduling server;

[0079] The main process on the main scheduling server queries the status information of each host node and the job pair information obtained by the multi-machine node management module in real time, and then matches the job to be processed with the node host that will run the job according to the job priority and the host resource load information collected in real time by the resource status collection module;

[0080] If a job needs to be scheduled with priority, the corresponding priority will be requested according to the configured priority of each job. After receiving the response information, it proves that the priority request is successful. At this time, the job can be inserted into the corresponding execution sequence;

[0081] The slave process running on the node host analyzes the collected data according to the master process and submits it to the master process. The master process extracts the corresponding command execution parameters and submits the relevant job process to execute the task;

[0082] The slave process on each node host needs to check the job running status on the local machine and report it to the master process on the main scheduling server, update the job distribution running status table at the same time, and submit the job execution results to the resource status collection module;

[0083] Issue corresponding log information and warning information according to the running status.

[0084] See also Figure 5 The multi-machine load balancing management module can balance the workload and system resource load. When assigning jobs, it is necessary to consider the maximum number of jobs that can be submitted by each host, the CPU busyness, memory usage, paging rate and other workloads and system resource loads of the host. Combined with the job definition attributes, including the required number of CPUs, memory space, job running time, job priority, job queue configuration, etc. and the scheduling strategy defined in advance by the designer, the scheduling allocation is performed. The currently available scheduling strategies include: First Come First Served FCFS; Fairshare; Preemption; Exclusive; HostParation; Resource Reservation.

[0085] The multi-machine load balancing management module receives the job queue, host load information transmitted by the multi-machine system resource status collection module, host-level threshold information associated with the node configuration, and job operation status distribution table information, and assigns and executes jobs according to relevant scheduling strategies.

[0086] The multi-machine load balancing management module is a permanent process, which consists of a master process and several slave processes. The master process is responsible for comprehensively analyzing various input data according to the pre-set scheduling strategy to determine which node host the job should be executed on; the slave process running on the relevant host node obtains the execution command parameters of the job and starts the relevant job process to execute the task.

[0087] The multi-machine load balancing management module automatically dispatches the job sequences waiting to be run in the system to different machines based on the host node information collected and analyzed by the multi-machine node management module and the resource load status obtained from the multi-machine resource status collector, ensuring that the job sequences run in an orderly, efficient and timely manner according to the pre-defined scheduling strategy, thus ensuring the efficiency of the system and the timeliness of the data.

[0088] Specifically, the master process is responsible for comprehensively analyzing various input data according to the pre-set scheduling strategy to determine on which node host the job will be executed; the slave process running on the relevant host node obtains the execution command parameters of the job and starts the relevant job process to execute the task.

[0089] The processing flow is as follows:

[0090] Receive the job information to be executed submitted by the job process execution management module;

[0091] The main process on the main scheduling server queries the status information of each host node and the job pair information obtained by the real-time analysis of the multi-machine node management module, and then matches the job to be processed with the node host running the job according to the job priority and the host resource load information collected in real time by the resource status collection module and the resource status collection module;

[0092] If a job needs to be scheduled with priority, the job priority scheduling information is sent to the job running priority manager to request the corresponding priority. After receiving the response information, it proves that the priority request is successful. At this time, the job can be inserted into the corresponding execution sequence.

[0093] The slave process running on the node host extracts the corresponding command execution parameters based on the results of the analysis of the collected data by the master process, and submits the relevant job process to execute the task;

[0094] The slave process on each node host needs to check the job running status on the local machine and report it to the master process on the main scheduling server, update the job distribution running status table at the same time, and submit the job execution results to the job running status collection module;

[0095] Issue corresponding log information and warning information according to the running status.

[0096] See also Figure 6 The resource status collection module collects the host load parameters of each node in the cluster in real time and analyzes the system resource usage status of each node host; at the same time, the module summarizes the system resource usage status in the cluster, receives the host load status of each node in the cluster for summary analysis, and provides the necessary real-time system resource information for the multi-machine load balancing manager. The host load parameters collected in real time include:

[0097] r1m: average CPU utilization per minute;

[0098] r15s: average CPU utilization every 15 seconds;

[0099] r15m: average CPU utilization every 15 minutes;

[0100] ut: CPU utilization at that time;

[0101] mem: available memory size;

[0102] io: input and output values;

[0103] swpspace: swap area size;

[0104] memoryspace: available memory size;

[0105] LinkState: physical link status;

[0106] JidFlag: LSF status;

[0107] XntpFlag: NTP time system daemon status.

[0108] The resource status collection module analysis process is as follows: Resource collection and management: responsible for collecting the resources of the computing nodes, the collected resources include CPU type and utilization, host type, physical memory size and current available memory, available Swap space, active paging, I / O rate, number of interactive login users, interactive idle time, / tmp directory space and other user-defined resources, the computing node resource collection and management process reports the collected resource information to the resource collection and management process of the master node, the master node resource collection and management process is responsible for reporting the summarized resource conditions of each computing node to the job reception, scheduling, fault tolerance, and control process, the job reception, scheduling, fault tolerance, and control process performs job scheduling according to the resource usage of each computing node. When the administrator adds and deletes computing nodes by modifying the configuration file, the resource collection and management process on the master node is responsible for reading the configuration file to add and delete computing nodes. The resource collection and management process on the master node is also responsible for upgrading and downgrading computing resources according to the system resource usage. When computing resources are downgraded, only critical jobs can use the downgraded computing resources to ensure the normal operation of critical jobs in the system.

[0109] Job execution and runtime status collection: Responsible for starting job reception, fault tolerance, and control processes. The job execution and runtime status collection process running on the computing node is responsible for starting user jobs, collecting and reporting the runtime status of the job process. If the job exits abnormally, it is responsible for restarting the job.

[0110] Job resource usage collection: Responsible for collecting the system resources consumed by the job process during operation, such as CPU, memory, Swap, etc., and reporting the collected resource usage. At the same time, this process is also responsible for controlling parallel job processes.

[0111] The resource status collection module consists of a main service process and a group of slave processes. The main process runs on the business system business scheduling master machine. There is a slave process in the aggregation host node, which is responsible for collecting the system load parameter information of each machine, including r1m, r15s, r15m, ut, mem, io, swp, LinkStat, JidFlag, XntpFlag, etc., and reporting to the main process on the main server. The main process on the main server is responsible for summarizing and analyzing the system load status information of the slave processes on each node, generating relevant operation log information, real-time reports of system resources, and storing the necessary system resource information in the database. The processing flow is as follows:

[0112] A) The slave processes running on each host node collect system load parameter information of each machine, including r1m (average CPU utilization per minute), r15s (average CPU utilization per 15 seconds), r15m (average CPU utilization per 15 minutes), ut (CPU utilization at that time), mem (available memory size), io (input and output value), swp (swap area space size), LinkStat (physical link status), JidFlag (LSF status), XntpFlag (ntp time system daemon status), etc., and report it to the master process on the master server for summary analysis by the master process;

[0113] B) The master process receives various load parameter information collected by the slave process and performs summary analysis;

[0114] ① Generate various system resource status reports, the main reports include:

[0115] Cluster CPU utilization daily / hourly report;

[0116] CPU utilization daily / hourly report by machine (or mem / swp / tmp / pg / io / it / r1m / r15s / r15m daily / hourly report;

[0117] Daily / hourly reports on CPU time / running time / memory / SWP usage by machine;

[0118] Satellite-based CPU time / run time / memory / SWP usage daily / hourly reports;

[0119] Daily / hourly reports on job throughput based on satellites / queues / machines;

[0120] Daily / hourly reports on job running time / waiting time based on satellite / queue / machine

[0121] The above reports can be customized according to business operation requirements;

[0122] ②Monitor the storage capacity and collect information of the main working directories of the business system. When the usage rate of a file directory exceeds the warning value, an alarm will be prompted.

[0123] C) Generate system operation log information and issue an alarm when the system is overloaded;

[0124] Generate system resource data and write it to the relevant database table.

[0125] See also Figure 7 The operation configuration daemon module monitors the operation of the permanent service process in the business software deployed on the cluster master server in real time, and implements dynamic configuration and management of the permanent service process according to the operator's command. In this system, the multi-machine node management module, multi-machine load balancing management module, and resource status collection module are all configured, started, suspended, and managed by the operation configuration daemon module. The processing flow is as follows:

[0126] The operating system automatically starts the running configuration daemon module during the initialization process, or the operating personnel starts the running configuration daemon module through a system command;

[0127] Establish and connect the configuration-related shared memory segment, read the permanent process running parameter configuration table, and load the permanent service process description given in the parameter configuration table into the configuration-related shared memory segment;

[0128] Start the corresponding permanent service process in the business system according to the permanent process description information of the configuration-associated shared segment, change the relevant status of the permanent service process in the configuration-associated shared memory segment, and send permanent service process status synchronization messages and log messages;

[0129] Monitor the running status of the permanent service process of the business system. When the permanent service process is abnormally interrupted, restart or stop the process according to the state attribute settings in the configuration-related shared memory segment to play the role of guarding the permanent service process of the system;

[0130] Receive operator command information, modify the content of the configuration-associated shared memory segment according to the command information, and complete corresponding process processing actions, such as terminating the execution of a process or resuming the operation of a process, so as to achieve real-time monitoring of the configuration-associated shared memory segment and permanent service process;

[0131] According to the operator's command to stop the system operation, all running service processes are notified by signal to terminate the business operation, and the follow-up processing of the business system is completed at the same time;

[0132] Corresponding log information and alarm information are issued according to the operating status.

[0133] The business operation platform is usually a computer and network system with complex business associations, high performance requirements, and large data throughput. The present disclosure manages all computing resources in a distributed system in a unified manner, and through full utilization of various resources of cluster computers, realizes dynamic scheduling of computing loads, improves the acceleration ratio of parallel tasks, shortens the job response time, and maximizes the advantages of system resources. By turning the entire computer network system into a relatively single virtual host or supercomputer, the effect of providing a relatively single business operation platform for upper-level conventional business scheduling is achieved.

[0134] The above advantages are derived from the multi-machine load balancing management module.

[0135] The multi-machine load balancing management module can balance the workload and system resource load. When assigning jobs, the maximum number of jobs that can be submitted by each host, the CPU busyness of the host, memory usage, paging rate and other workloads and system resource loads are considered, and the defined attributes of the job, including the required number of CPUs, memory space, job running time, job priority, job queue configuration, etc., and the scheduling strategy defined in advance by the designer are combined for scheduling and allocation.

[0136] The function of the multi-machine load balancing management module is to find a mapping relationship between jobs and computers, and evenly distribute a series of jobs in the computer cluster to each computer according to the workload, so as to improve the overall computing efficiency of the system, provide the shortest average task response time, and adapt to changing loads.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-machine resource configuration management method is implemented.

[0138] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0140] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0141] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-machine resource configuration management method, characterized in that: The multi-machine resource configuration management method comprises: Allocate each satellite to a queue according to the processing priority to obtain several operation queues; Allocating a machine node to each of the job queues according to the machine node resources required by the job queues, to obtain a plurality of job queues and a corresponding plurality of machine node groups; Acquire the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue in real time; Matching each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information; According to the priority of each job in the job queue, jobs that meet the condition are activated one by one for parallel execution; the condition that meets the condition is that the remaining resources of the machine node matching the job are sufficient to execute the job; Monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, set the state domain of the machine node to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, the state domain of the machine node is an execution state.

2. A multi-machine resource configuration management device, characterized in that: The multi-machine resource configuration management device comprises: Multi-machine node management module, used for: Allocate each satellite to a queue according to the processing priority to obtain several operation queues; Allocating a machine node to each of the job queues according to the machine node resources required by the job queues, to obtain a plurality of job queues and a corresponding plurality of machine node groups; Resource status collection module, used to: Acquire the status information of each host node in the machine node group, the host resource load information, and the required resource information of each job in the job queue in real time; Matching each job in the job queue to a machine node in a corresponding machine node group according to the required resource information, the status information of each host node, and the host resource load information; According to the priority of each job in the job queue, jobs that meet the condition are activated one by one for parallel execution; the condition that meets the condition is that the remaining resources of the machine node matching the job are sufficient to execute the job; The multi-machine load balancing management module is used to monitor the number information and resource usage value information of parallel execution jobs of each machine node in real time; when the number information and resource usage value information of the parallel execution jobs reach a threshold value, the state domain of the machine node is set to a busy state, until the number information and resource usage value information of the parallel execution jobs decrease, the state domain of the machine node is set to an execution state.

3. The multi-machine resource configuration management device according to claim 2, characterized in that: The multi-machine load balancing management module includes: a master process and several slave processes; The main process is responsible for comprehensively analyzing various input data according to the pre-set scheduling strategy to determine on which node host the job should be executed; The slave process runs on the relevant host node; the slave process obtains the execution command parameters of the job and starts the relevant job process to execute the task.

4. The multi-machine resource configuration management device according to claim 3, characterized in that: The master process is responsible for comprehensively analyzing various input data according to the preset scheduling strategy to determine on which node host the job is to be executed; the slave process running on the relevant host node obtains the execution command parameters of the job and starts the relevant job process to execute the task.

5. The multi-machine resource configuration management device according to claim 2, characterized in that: The resource status collection module includes: a main service process and a group of slave processes; The main process runs on the business system business scheduling master machine; The slave process is on the aggregation host node; the slave process is responsible for collecting system load parameter information of each machine, including r1m, r15s, r15m, ut, mem, io, swp, LinkStat, JidFlag, XntpFlag, and reporting to the main service process on the main server; the main service process on the main server is responsible for summarizing and analyzing the system load status information of the slave processes on each node, and generating relevant operation log information, real-time reports of system resources, and storing the necessary system resource information in the database.

6. The multi-machine resource configuration management device according to claim 2, characterized in that: The multi-machine load balancing management module is also used to implement dynamic configuration and management of permanent service processes according to the operator's commands.

7. The multi-machine resource configuration management device according to claim 2, characterized in that: When the multi-machine node management module performs job assignment, it considers the maximum number of jobs that each host can submit, the host's CPU busyness, memory usage, and paging rate, as well as other job loads and system resource loads, and performs scheduling and allocation based on the job's definition attributes, including the required number of CPUs, memory space, job running time, job priority, the queue configuration to which the job belongs, and the scheduling strategy defined in advance by the designer.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-machine resource configuration management method described in claim 1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-machine resource configuration management method described in claim 1 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-machine resource configuration management method described in claim 1 is implemented.

Images