System resource management method and device, equipment and medium

Abstract

The invention belongs to the technical field of system resource management, and particularly relates to a system resource management method and device, equipment and a medium. The method comprises the following steps: acquiring the total node resource amount owned by a system and historical business processing data of the system; determining service processing duration and node occupation amount of each historical service according to the historical service processing data; establishing a plurality of resource pools according to the business processing durations of all historical businesses; determining the node allocation quantity of each resource pool according to the node occupation quantity and the node resource total quantity of all historical services; and managing the resources of the system according to all the resource pools and the service queue of each resource pool. The operation isolation of different services is realized, the operation time of the same service is fixed, different users are relatively fair when using cluster resources, and the service stability and the resource use fairness of the earthquake processing system are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of system resource management, and particularly relates to a method, device, equipment and medium for managing system resources. Background Art

[0002] In many fields and industries, a large number of computing problems are involved, and a lot of computing resources are required to calculate these problems. Therefore, a computing platform is configured in these industries, and the computing platform will contain a large number of computing resources. Especially in the field of seismic exploration, a large amount of seismic data needs to be calculated to better understand the underground situation.

[0003] In the field of seismic exploration, some complex operations require a large amount of resources for calculation, while simple operations only require a small amount of resources for calculation. However, in order to ensure the calculation efficiency, the computing resources in this computing platform need to at least meet the normal calculation of complex operations. Of course, in practice, the number of platform users and the requirements for calculation efficiency also need to be considered, and more computing resources will be set. However, in many cases, these calculations will not participate in the calculation, resulting in a large waste of computing resources and unable to improve the overall calculation efficiency well. For example, when performing simple operation calculations, perhaps only a small part of the overall resources is needed to quickly complete the simple operation calculations, and the other unparticipated resources are in an idle state, but the processing efficiency of the entire platform and system is not well improved. Of course, this situation can be solved by choosing multi-task parallel processing. However, in the general division of system resources, this method may lead to competition for resources between tasks or conflicts in resource usage, affecting the progress of each task and making it difficult to improve the overall efficiency of the system and platform. Summary of the Invention

[0004] In view of the above technical problems, the present invention provides a method, apparatus, device and medium for managing system resources. The present application obtains the total amount of node resources owned by the system and the historical service processing data of the system; determines the service processing duration and node occupancy of each historical service according to the historical service processing data; establishes multiple resource pools according to the service processing durations of all historical services; determines the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources; allocates the node resources of the system to each resource pool according to the node allocation amount of each resource pool; establishes a service queue in each resource pool according to the node allocation amount of each resource pool; and manages the resources of the system according to all the resource pools and the service queues of each resource pool. This realizes the operation isolation of different services, the fixed operation time of the same service, and relative fairness for different users when using cluster resources, ensuring the service stability and resource usage fairness of the seismic processing system.

[0005] To solve the above technical problems, the technical solution adopted by the present invention includes five aspects.

[0006] In a first aspect, the present application provides a method for managing system resources, including: obtaining the total amount of node resources owned by the system and the historical service processing data of the system; determining the service processing duration and node occupancy of each historical service according to the historical service processing data; establishing multiple resource pools according to the service processing durations of all historical services; determining the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources; allocating the node resources of the system to each resource pool according to the node allocation amount of each resource pool; establishing a service queue in each resource pool according to the node allocation amount of each resource pool; and managing the resources of the system according to all the resource pools and the service queues of each resource pool.

[0007] In some embodiments, the step of establishing multiple resource pools according to the service processing durations of all historical services includes: obtaining a preset processing duration threshold; determining the service category of each historical service according to the service processing duration of each historical service and the processing duration threshold, where the service category includes complex calculation services and simple calculation services; counting the first service quantity of the complex calculation services and the second service quantity of the simple calculation services in all historical services; determining the number of resource pools of each type of resource pool according to the first service quantity and the second service quantity, where the types of resource pools include complex service resource pools and simple service resource pools; and establishing multiple resource pools in the system according to the number of resource pools of each type of resource pool.

[0008] In some embodiments, determining the number of resource pools of various types according to the first service quantity and the second service quantity includes: obtaining a first preset processing capacity preset for each of the complex service resource pools, and a second preset processing capacity preset for each of the simple service resource pools; determining a first number of resource pools of the complex service resource pools according to the first service quantity and the first preset processing capacity; and determining a second number of resource pools of the simple service resource pools according to the second service quantity and the second preset processing capacity.

[0009] In some embodiments, determining the node allocation quantity of each resource pool according to the node occupancy and the total node resources of all historical services includes: obtaining a first preset processing duration preset for the complex computing service; determining a first node allocation quantity of each of the complex service resource pools according to the node occupancy, the service processing duration, and the first preset processing duration of each complex computing service; and determining a second node allocation quantity of each of the simple service resource pools according to the total node resources, the first node allocation quantity, the first number of resource pools, and the second number of resource pools.

[0010] In some embodiments, establishing a service queue in each resource pool according to the node allocation quantity of each resource pool includes: determining an average occupancy of each simple computing service according to the node occupancy of all simple computing services; determining an average processing duration of each simple computing service according to the service processing duration of all simple computing services; obtaining a second preset processing duration preset for the simple computing service; determining a minimum number of nodes of each service queue according to the average occupancy, the average occupancy duration, and the second preset processing duration; determining the number of queues of the service queues in each simple service resource pool according to the second node allocation quantity and the minimum number of nodes; and establishing a service queue in each simple service resource pool according to the number of queues.

[0011] In some embodiments, establishing a service queue in each simple service resource pool according to the number of queues includes: performing the following operations for each simple service resource pool: obtaining the resource types of the node resources in the simple service resource pool; for each service queue, allocating the node resources with the same resource type to the service queue according to the minimum number of nodes, and determining the queue type of the service queue; and establishing a service queue in the simple service resource pool according to the number of queues and the queue type of each service queue.

[0012] In some embodiments, managing the resources of the system according to all the resource pools and the service queues of each resource pool includes: obtaining the service to be processed uploaded by the user and the corresponding service information, where the service information includes: service type and service category; determining the target resource pool category of the service to be processed according to the service category; determining the target service queue type of the service to be processed according to the service type; obtaining the remaining task duration of each target service queue in each target resource pool, where the target resource pool is a resource pool whose type belongs to the target resource pool category, and the target service queue is a service queue whose queue type belongs to the target service queue type; determining the service execution queue of the service to be processed according to each of the remaining task durations; storing the task to be processed in the target resource pool, and obtaining the current remaining duration of the execution service queue in real time; when the current remaining duration is 0, submitting the task to be executed to the execution service queue so that the task to be processed is executed by the execution service queue.

[0013] In a second aspect, the present application provides a management device for system resources, including: a first acquisition module, configured to acquire the total amount of node resources owned by the system and the historical service processing data of the system; a first determination module, configured to determine the service processing duration and node occupancy of each historical service according to the historical service processing data; a first execution module, configured to establish a plurality of resource pools according to the service processing durations of all historical services; a second determination module, configured to determine the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources; a second execution module, configured to allocate the node resources of the system to each resource pool according to the node allocation amount of each resource pool; a third determination module, configured to establish a service queue in each resource pool according to the node allocation amount of each resource pool; and a third execution module, configured to manage the resources of the system according to all the resource pools and the service queues of each resource pool.

[0014] In a third aspect, the present application proposes an electronic device, including: a memory and a processor, where a computer program is stored on the memory, and when the computer program is executed by the processor, it executes the method according to any one of the first aspect.

[0015] In a fourth aspect, the present application proposes a storage medium, where the computer program stored in the storage medium can be executed by one or more processors, and the computer program can be used to implement the method according to any one of the first aspect.

[0016] Advantages of the present invention: This application obtains the total amount of node resources owned by the system and the historical service processing data of the system; determines the service processing duration and node occupancy of each historical service based on the historical service processing data; establishes multiple resource pools according to the service processing durations of all historical services; determines the node allocation amount of each resource pool based on the node occupancy of all historical services and the total amount of node resources; allocates the node resources of the system to each resource pool according to the node allocation amount of each resource pool; establishes a service queue in each resource pool according to the node allocation amount of each resource pool; and manages the resources of the system according to all the resource pools and the service queues of each resource pool. It realizes the operation isolation of different services, the fixed operation time of the same service, and relative fairness for different users when using cluster resources, ensuring the service stability and resource usage fairness of the earthquake processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The scope of the present disclosure can be better understood by reading the following detailed description of exemplary embodiments in conjunction with the accompanying drawings. The accompanying drawings included are:

[0018] Figure 1 The overall flowchart of a method for managing system resources provided by an embodiment of the present application;

[0019] Figure 2 The structural block diagram of a device for managing system resources provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limitations of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0021] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0022] If similar descriptions such as "first\second\third" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" only distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0024] Embodiment 1:

[0025] In the field of seismic exploration, some complex operations require a large amount of resources for calculation, while simple operations only require a small amount of resources for calculation. However, in order to ensure the calculation efficiency, the calculation resources in this calculation platform need to at least meet the normal calculation requirements of complex operations. Of course, in practice, the number of platform users and the requirements for calculation efficiency also need to be considered, and more calculation resources will be set. However, in many cases, these calculations will not participate in the calculation, which will lead to a large waste of calculation resources and cannot well improve the overall calculation efficiency. For example, when performing simple operation calculations, perhaps only a small part of the overall resources is needed to quickly complete the simple operation calculations, and the other unparticipated resources are in an idle state, but it also does not well improve the processing efficiency of the entire platform and system. Of course, this situation can be solved by selecting multi-task parallel processing. However, in the general division of system resources, this method may lead to competition for resources between tasks or resource conflict situations, making the progress of each task affect each other and also difficult to improve the overall efficiency of the system and platform.

[0026] Regarding the problems existing in the prior art, such as Figure 1 As shown, this application provides a method for managing system resources. The method is applied to an electronic device, and the electronic device can be a server, a mobile terminal, a computer, a cloud platform, etc. The functions realized by the device data processing provided in the embodiments of this application can be implemented by a processor of the electronic device calling program code. Among them, the program code can be stored in a computer storage medium. The method for managing system resources includes:

[0027] Step S1: Obtain the total amount of node resources owned by the system and the historical service processing data of the system.

[0028] In some actual situations, since a system requires a lot of computing resources, and these computing resources are provided by CPUs or GPUs. Due to the limited computing resources that a single CPU or GPU can provide, there are many CPUs or GPUs in a system as carriers of computing resources. In this application, each CPU or GPU is regarded as a node resource. If this application wants to manage the resources of a system, it is necessary to consider the actual usage situation of the system and manage the resources of the system in combination with the actual usage situation of the system. And the historical business processing data of a system is the data that can best reflect the actual usage situation of the system. Therefore, the historical business processing data of this system is required in this application. Since managing the resources of a system will inevitably involve the reallocation of computing resources, the total amount of node resources of this system also needs to be obtained in this application.

[0029] Step S2: Determine the business processing duration and node occupancy of each historical business according to the historical business processing data.

[0030] The historical business processing data contains the business processing duration of each historical business and the node occupancy when each historical business is being processed. Therefore, we can determine the business processing duration and node occupancy of each historical business according to the historical business processing data.

[0031] Step S3: Establish multiple resource pools according to the business processing durations of all historical businesses.

[0032] In some embodiments, step S3 "Establish multiple resource pools according to the business processing durations of all historical businesses" includes:

[0033] Step S31: Obtain a preset processing duration threshold.

[0034] Step S32: Determine the business category of each historical business according to the business processing duration of each historical business and the processing duration threshold, where the business category includes: complex computing business and simple computing business.

[0035] Since there are complex computing businesses and simple computing businesses in historical businesses, and the business processing durations of complex computing businesses and simple computing businesses vary greatly, and the resources occupied by the two types of computing businesses also vary greatly. Therefore, when reallocating resources, it is necessary to separate the two types of businesses. So in this application, it is necessary to obtain a preset processing duration threshold, and determine the business category of each historical business through the size relationship between the business processing duration of each historical business and the processing duration threshold, that is, determine whether each historical business belongs to a complex computing business or a simple computing business.

[0036] Step S33: Count the first service quantity of the complex calculation services and the second service quantity of the simple calculation services in all historical services.

[0037] Step S34: Determine the resource pool quantities of various types of resource pools according to the first service quantity and the second service quantity, where the types of the resource pools include: complex service resource pools and simple service resource pools.

[0038] The purpose of this application is to reallocate the resources of the system to improve the overall processing efficiency of the system and the utilization rate of resources. Therefore, the two types of services need to be processed separately. So, two types of resource pools are designed in this application, namely complex service resource pools and simple service resource pools. The complex service resource pools are mainly used to process complex calculation services, while the simple service resource pools are used to process simple calculation services, separating the processing of the two types of services and avoiding the mutual influence caused by the different required resources and processing durations of the two services.

[0039] Although the two types of resource pools isolate the processing of the two types of services, if there are a large number of each type of service, there will still be a large amount of queuing and resource waste problems. Therefore, in this application, it is also necessary to determine the quantities of various types of resource pools according to the first service quantity and the second service quantity.

[0040] Therefore, in some embodiments, step S34 "Determine the resource pool quantities of various types of resource pools according to the first service quantity and the second service quantity" includes:

[0041] Step S341: Obtain the first preset processing capacity preset for each complex service resource pool and the second preset processing capacity preset for each simple service resource pool.

[0042] Step S342: Determine the first resource pool quantity of the complex service resource pool according to the first service quantity and the first preset processing capacity.

[0043] Step S343: Determine the second resource pool quantity of the simple service resource pool according to the second service quantity and the second preset processing capacity.

[0044] Because if a resource pool is too large, it will also cause resource waste and affect the processing efficiency of the services. Therefore, in this application, different processing capacities are preset for different types of resource pools. This processing capacity is the optimal processing capacity of a resource pool. In such a case of processing capacity, the resources of the resource pool can be well utilized, and a better efficiency improvement effect can also be achieved. Therefore, in this application, it is necessary to determine the quantities of the corresponding types of resource pools according to the quantities of various services.

[0045] Step S35: Establish multiple resource pools in the system according to the number of resource pools of various types.

[0046] After obtaining the number of various resource pools according to Step S34, the resource pools can be established in the system according to the number of various resource pools.

[0047] Step S4: Determine the node allocation amount of each resource pool according to the node occupancy and the total node resources of all historical services.

[0048] After determining the number of various resource pools, it is necessary to allocate reasonable resources to each resource pool, that is, the node allocation amount of each resource pool.

[0049] In some embodiments, Step S4 "Determine the node allocation amount of each resource pool according to the node occupancy and the total node resources of all historical services" includes:

[0050] Step S41: Obtain the first preset processing duration preset for the complex computing service.

[0051] Step S42: Determine the first node allocation amount of each complex service resource pool according to the node occupancy, the service processing duration, and the first preset processing duration of each complex computing service.

[0052] Step S43: Determine the second node allocation amount of each simple service resource pool according to the total node resources, the first node allocation amount, the number of the first resource pools, and the number of the second resource pools.

[0053] For complex computing services, unilaterally shortening the processing duration cannot improve the resource utilization rate and system processing efficiency of the system to the best. Therefore, a first preset processing duration is preset in this application, and the first preset processing duration is the appropriate processing duration of the complex computing service set according to actual needs. Since the processing duration is affected by the node occupancy, the relationship between each node and the processing duration can be obtained according to the node occupancy and the service processing duration of each historical service, and then the first node allocation amount of each complex service resource pool can be determined according to the first preset processing duration. After determining the first node allocation amount of each complex service resource pool, the total number of nodes required by the complex service resource pool can be obtained. Then, by subtracting the number of nodes allocated to the complex service resource pool from the total node resources, the remaining node resources for multiple simple service resource pools are left. Then, the remaining node resources are evenly allocated to each simple service resource pool, and the second node allocation amount of each simple service resource pool can be obtained.

[0054] Step S5: Allocate the node resources of the system to each resource pool according to the node allocation amount of each resource pool.

[0055] After determining the node allocation amounts of each resource pool, all node resources are allocated to each resource pool according to the node allocation amounts, realizing the isolation of multiple services, reducing the impact between multiple services, and improving the system's processing efficiency for services and the utilization rate of resources.

[0056] Step S6: Establish service queues in each resource pool according to the node allocation amount of each resource pool.

[0057] After steps S3 - S5, we allocate node resources to different resource pools, improving the resource utilization rate of the system and also enhancing the overall processing efficiency of the system for services. For each simple service resource pool, the number of node resources in it far exceeds the node resources required for each simple computing service. Therefore, it is necessary to re - allocate the node resources in each resource pool, that is, to re - divide the node resources by establishing service queues in each resource pool. Although the node resources in each complex service resource pool may just be enough to process complex computing services, the service queues can still be established in this way, but perhaps only one service queue can be established. And when the number of service queues in a resource pool is 1, it does not affect the overall resource pool's processing of services.

[0058] Therefore, in some embodiments, step S6, "establish service queues in each resource pool according to the node allocation amount of each resource pool", includes:

[0059] Step S61: Determine the average occupancy of each simple computing service according to the node occupancy of all simple computing services.

[0060] Step S62: Determine the average processing duration of each simple computing service according to the service processing durations of all simple computing services.

[0061] Step S63: Obtain the second preset processing duration preset for the simple computing service.

[0062] Step S64: Determine the minimum number of nodes for each service queue according to the average occupancy, the average occupancy duration, and the second preset processing duration.

[0063] The existence of the service queue enables the resources in each resource pool to be better utilized and allocated. Therefore, it is required that the resources in each service queue reach the maximum utilization rate as much as possible. Therefore, in this application, a second preset processing duration for simple computing services is also preset. Similarly, the relationship between the number of nodes and the processing duration can be determined according to the service processing duration and node occupancy of each simple computing service. Then, the appropriate number of nodes for each simple computing service, that is, the minimum number of nodes in each service queue, can be determined according to the second preset processing duration and the relationship between the node belonging and the processing duration.

[0064] Step S65: Determine the number of queues in the service queue of each simple service resource pool according to the second node allocation amount and the minimum number of nodes.

[0065] Step S66: Establish service queues in each simple service resource pool according to the number of queues.

[0066] Since the second node allocation amount of each simple service resource pool has been determined, after determining the minimum number of nodes in each service queue, the number of queues in the service queue that can be established in each simple service resource pool can be obtained. Finally, multiple service queues are established in each simple resource pool according to the number of queues.

[0067] Of course, since when determining the complex service resource pool, the first node allocation amount in the complex service resource pool is made to match the complex computing service, the number of queues in each complex service resource pool is 1, that is, each complex service resource pool has only one service queue.

[0068] In some embodiments, step S66 "Establish service queues in each simple service resource pool according to the number of queues" includes:

[0069] Step S661: Perform the following operations for each simple service resource pool:

[0070] Step S662: Obtain the resource types of the resources of each node in the simple service resource pool.

[0071] Step S663: For each service queue, allocate the node resources with the same resource type to the service queue according to the minimum number of nodes, and determine the queue type of the service queue.

[0072] Step S664: Establish service queues in the simple service resource pool according to the number of queues and the queue type of each service queue.

[0073] For each service, the required resources are not necessarily the same. Some services require CPU resources for processing, while some services require GPU resources for processing. And GPU-type resource nodes can be used to process services as CPU resources, but CPU-type resource nodes cannot be used to process GPU services. Therefore, in this application, it is also necessary to obtain the resource types of each node resource in each simple service resource pool. Then, nodes of the same resource type are assigned to the same service queue, so that two service queues can appear in the resource pool to be used for processing different types of services. Therefore, when establishing the service queue, it is also necessary to consider the queue type of each service queue.

[0074] Step S7: Manage the resources of the system according to all the resource pools and the service queues of each resource pool.

[0075] After going through steps S3 - S664, the resources of the system are re-partitioned. However, the above process only provides a management framework, and it is still necessary to manage the resources in the system according to specific situations. Therefore, in some embodiments, step S7, "Manage the resources of the system according to all the resource pools and the service queues of each resource pool", includes:

[0076] Step S71: Obtain the service to be processed uploaded by the user and the corresponding service information, where the service information includes: service type and service category.

[0077] Step S72: Determine the target resource pool category of the service to be processed according to the service category.

[0078] Step S73: Determine the target service queue type of the service to be processed according to the service type.

[0079] Step S74: Obtain the remaining task duration of each target service queue in each target resource pool, where the target resource pool is a resource pool whose type belongs to the target resource pool category, and the target service queue is a service queue whose queue type belongs to the target service queue type.

[0080] Step S75: Determine the service queue for executing the service to be processed according to each of the remaining task durations.

[0081] Step S76: Store the task to be processed in the target resource pool and continuously obtain the current remaining duration of the service queue for execution.

[0082] Step S77: When the current remaining duration is 0, submit the task to be executed to the service queue for execution, so that the task to be processed is executed by the service queue for execution.

[0083] For the business to be processed uploaded by the user, first obtain the business category of the business to be processed, that is, whether the business to be processed is a complex calculation business or a simple calculation business. At the same time, it is also necessary to obtain the business type of the business to be processed to determine whether the business to be processed belongs to the CPU business or the GPU business.

[0084] Then, determine the target resource pool category of the business to be processed according to the business category, that is, whether the agency business should be processed by the simple business resource pool or the complex business resource pool. And determine whether the business to be processed should be processed by the CPU-type queue or the GPU-type queue according to the business type.

[0085] After determining the target resource pool category and the target business queue type of the business to be processed, the remaining task duration of each target business queue in each target resource pool can be obtained according to the target resource pool category and the target business queue type. That is, if the business to be processed is a simple calculation business of the GPU type, then its target resource pool category is the simple business resource pool, and its target business queue type is the GPU-type queue. At this time, the remaining task duration of each GPU business queue in each simple business resource pool can be obtained.

[0086] After obtaining the remaining task duration of each task queue, it can be determined which resource pool and which business queue the business to be processed belongs to. Therefore, the business to be processed can be stored in the corresponding resource pool for processing. When the remaining task duration of the practice business queue is 0, the business to be processed is actively executed.

[0087] Through the above method, this application realizes the operation isolation of different services, the operation time of the same service is fixed, different users are relatively fair when using cluster resources, ensuring the job stability and resource usage fairness of the seismic processing system, and improving the utilization efficiency of system resources and the overall processing efficiency of the system platform.

[0088] Embodiment 2:

[0089] Based on the foregoing embodiments, an embodiment of this application provides a management device for system resources. Each module included in the device and each unit included in each module can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing), or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0090] As shown Figure 2 in the figure, a management device for system resources includes: a first acquisition module 1, a first determination module 2, a first execution module 3, a second determination module 4, a second execution module 5, a third determination module 6, and a third execution module 7.

[0091] The first acquisition module 1 is used to acquire the total amount of node resources owned by the system and the historical service processing data of the system. The first determination module 2 is used to determine the service processing duration and node occupancy of each historical service according to the historical service processing data. The first execution module 3 is used to establish a plurality of resource pools according to the service processing durations of all historical services. The second determination module 4 is used to determine the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources. The second execution module 5 is used to allocate the node resources of the system to each resource pool according to the node allocation amount of each resource pool. The third determination module 6 is used to establish a service queue in each resource pool according to the node allocation amount of each resource pool. The third execution module 7 is used to manage the resources of the system according to all the resource pools and the service queues of each resource pool.

[0092] In some embodiments, the first execution module 3 includes: a second acquisition module, a fourth determination module, a fourth execution module, a fifth determination module, and a fifth execution module.

[0093] The second acquisition module is used to acquire a preset processing duration threshold. The fourth determination module is used to determine the service category of each historical service according to the service processing duration of each historical service and the processing duration threshold. The fourth execution module is used to count the first service quantity of the complex calculation services and the second service quantity of the simple calculation services among all historical services. The fifth determination module is used to determine the number of resource pools of various types of resource pools according to the first service quantity and the second service quantity. The fifth execution module is used to establish a plurality of resource pools in the system according to the number of resource pools of various types of resource pools.

[0094] In some embodiments, the fifth determination module includes: a third acquisition module, a sixth determination module, and a seventh determination module.

[0095] The third acquisition module is used to acquire a first preset processing amount preset for each complex service resource pool and a second preset processing amount preset for each simple service resource pool. The sixth determination module is used to determine the first number of resource pools of the complex service resource pool according to the first service quantity and the first preset processing amount. The seventh determination module is used to determine the second number of resource pools of the simple service resource pool according to the second service quantity and the second preset processing amount.

[0096] In some embodiments, the second determination module 4 includes: a fourth acquisition module, an eighth determination module, and a ninth determination module.

[0097] The fourth acquisition module is configured to acquire a first preset processing duration preset for the complex computing service. The eighth determination module is configured to determine a first node allocation amount for each of the complex service resource pools according to the node occupancy, the service processing duration, and the first preset processing duration of each complex computing service. The ninth determination module is configured to determine a second node allocation amount for each of the simple service resource pools according to the total node resources, the first node allocation amount, the number of the first resource pools, and the number of the second resource pools.

[0098] In some embodiments, the third determination module 6 includes: a tenth determination module, an eleventh determination module, a fifth acquisition module, a twelfth determination module, a thirteenth determination module, and a sixth execution module.

[0099] The tenth determination module is configured to determine an average occupancy of each simple computing service according to the node occupancy of all simple computing services. The eleventh determination module is configured to determine an average processing duration of each simple computing service according to the service processing duration of all simple computing services. The fifth acquisition module is configured to acquire a second preset processing duration preset for the simple computing service. The twelfth determination module is configured to determine a minimum number of nodes for each service queue according to the average occupancy, the average occupancy duration, and the second preset processing duration. The thirteenth determination module is configured to determine the number of queues of the service queues in each simple service resource pool according to the second node allocation amount and the minimum number of nodes. The sixth execution module is configured to establish service queues in each simple service resource pool according to the number of queues.

[0100] In some embodiments, the sixth execution module includes: a sixth acquisition module, a seventh execution module, and an eighth execution module.

[0101] The sixth acquisition module is configured to acquire the resource types of the node resources in the simple service resource pool. The seventh execution module is configured to, for each service queue, allocate the node resources with the same resource type to the service queue according to the minimum number of nodes, and determine the queue type of the service queue. The eighth execution module is configured to establish service queues in the simple service resource pool according to the number of queues and the queue type of each service queue.

[0102] In some embodiments, the third execution module 7 includes: a seventh acquisition module, a fourteenth determination module, a fifteenth determination module, an eighth acquisition module, a sixteenth determination module, a ninth acquisition module, and a ninth execution module.

[0103] The seventh acquisition module is used to acquire the business to be processed uploaded by the user and the corresponding business information, where the business information includes: business type and business category. The fourteenth determination module is used to determine the target resource pool category of the business to be processed according to the business category. The fifteenth determination module is used to determine the target business queue type of the business to be processed according to the business type. The eighth acquisition module is used to acquire the remaining task duration of each target business queue in each target resource pool. The sixteenth determination module is used to determine the execution business queue of the business to be processed according to each of the remaining task durations. The ninth acquisition module is used to store the task to be processed in the target resource pool and acquire the current remaining duration of the execution business queue in real time. The ninth execution module is used to submit the task to be executed to the execution business queue when the current remaining duration is 0, so that the task to be processed is executed by the execution business queue.

[0104] Each module in the above device for managing system resources can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor in the device in the form of hardware or be independent of it, or can be stored in the memory in the processing device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0105] Embodiment 3:

[0106] In a third aspect, an electronic device is provided, including a storage and a processor. The storage stores a computer program, and when the processor executes the computer program, the steps of a method for managing system resources are implemented.

[0107] Embodiment 4:

[0108] In a fourth aspect, a storage medium is provided. The computer program stored in the storage medium can be executed by one or more processors, and the computer program can be used to implement the steps of any one of the methods for managing system resources in the first aspect.

[0109] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0110] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures, or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the order numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0111] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0112] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the couplings, direct couplings, or communication connections between the components shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms.

[0113] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in the embodiments of this application can be all integrated in a processing unit, or each unit can be separately regarded as a unit, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0115] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments. The foregoing storage medium includes various media that can store program codes, such as removable storage devices, read-only memories (ROMs), magnetic disks, or optical discs.

[0116] Alternatively, if the above-mentioned integrated units of this application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a controller to execute all or part of the methods described in the embodiments of this application. The foregoing storage medium includes various media that can store program codes, such as removable storage devices, ROMs, magnetic disks, or optical discs.

[0117] As described above, it is only the implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A method for managing system resources, characterized in that, comprising: Obtaining the total amount of node resources owned by the system and the historical service processing data of the system; Determining the service processing duration and node occupancy of each historical service according to the historical service processing data; Establishing multiple resource pools according to the service processing durations of all historical services; Determining the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources; Allocating the node resources of the system to each resource pool according to the node allocation amount of each resource pool; Establishing a service queue in each resource pool according to the node allocation amount of each resource pool; Managing the resources of the system according to all the resource pools and the service queues of each resource pool.

2. The method according to claim 1, characterized in that, The establishing multiple resource pools according to the service processing durations of all historical services includes: Obtaining a preset processing duration threshold; Determining the service category of each historical service according to the service processing duration of each historical service and the processing duration threshold, where the service category includes: complex calculation services and simple calculation services; Counting the first service quantity of the complex calculation services and the second service quantity of the simple calculation services among all historical services; Determining the number of resource pools of each type of resource pool according to the first service quantity and the second service quantity, where the types of resource pools include: complex service resource pools and simple service resource pools; Establishing multiple resource pools in the system according to the number of resource pools of each type of resource pool.

3. The method according to claim 2, characterized in that, The determining the number of resource pools of each type of resource pool according to the first service quantity and the second service quantity includes: Obtaining a first preset processing amount preset for each complex service resource pool and a second preset processing amount preset for each simple service resource pool; Determining the first number of resource pools of the complex service resource pool according to the first service quantity and the first preset processing amount; Determining the second number of resource pools of the simple service resource pool according to the second service quantity and the second preset processing amount.

4. The method according to claim 3, characterized in that, The determining the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources includes: Obtaining a first preset processing duration preset for the complex calculation service; Determining the first node allocation amount of each complex service resource pool according to the node occupancy, the service processing duration and the first preset processing duration of each complex calculation service; Determining the second node allocation amount of each simple service resource pool according to the total amount of node resources, the first node allocation amount, the first number of resource pools and the second number of resource pools.

5. The method according to claim 4, characterized in that, The establishing a service queue in each resource pool according to the node allocation amount of each resource pool includes: Determine the average occupancy of each simple computing service according to the node occupancy of all simple computing services; Determine the average processing duration of each simple computing service according to the service processing duration of all simple computing services; Obtain the second preset processing duration preset for the simple computing service; Determine the minimum number of nodes for each service queue according to the average occupancy, the average occupancy duration, and the second preset processing duration; Determine the number of queues of the service queues in each simple service resource pool according to the second node allocation amount and the minimum number of nodes; Establish service queues in each simple service resource pool according to the number of queues.

6. The method according to claim 5, wherein, the establishing service queues in each simple service resource pool according to the number of queues includes: perform the following operations for each simple service resource pool: Obtain the resource types of each node resource in the simple service resource pool; For each service queue, allocate the node resources with the same resource type to the service queue according to the minimum number of nodes, and determine the queue type of the service queue; Establish service queues in the simple service resource pool according to the number of queues and the queue type of each service queue.

7. The method according to claim 6, wherein, the managing the resources of the system according to all the resource pools and the service queues of each resource pool includes: Obtain the service to be processed uploaded by the user and the corresponding service information, where the service information includes: service type and service category; Determine the target resource pool category of the service to be processed according to the service category; Determine the target service queue type of the service to be processed according to the service type; Obtain the remaining task duration of each target service queue in each target resource pool, where the target resource pool is a resource pool whose type belongs to the target resource pool category, and the target service queue is a service queue whose queue type belongs to the target service queue type; Determine the execution service queue of the service to be processed according to each remaining task duration; Store the task to be processed in the target resource pool, and obtain the current remaining duration of the execution service queue in real time; When the current remaining duration is 0, submit the task to be executed to the execution service queue, so that the task to be processed is executed by the execution service queue.

8. A management device for system resources, wherein, comprises: A first acquisition module, configured to acquire the total amount of node resources owned by the system, and the historical service processing data of the system; A first determination module, configured to determine the service processing duration and node occupancy of each historical service according to the historical service processing data; A first execution module, configured to establish multiple resource pools according to the service processing duration of all historical services; A second determination module, configured to determine the node allocation amount of each resource pool according to the node occupancy of all historical services and the total amount of node resources; A second execution module, configured to allocate the node resources of the system to each resource pool according to the node allocation amount of each resource pool; A third determination module, configured to establish a service queue in each resource pool according to the node allocation amount of each resource pool; A third execution module, configured to manage the resources of the system according to all the resource pools and the service queues of each resource pool.

9. An electronic device, characterized in that, it includes: a memory and a processor, where a computer program is stored on the memory, and when the computer program is executed by the processor, the method according to any one of claims 1-7 is executed.

10. A storage medium, characterized in that, the computer program stored on the storage medium can be executed by one or more processors, and the computer program can be used to implement the method according to any one of claims 1-7.

Images