Queue resource allocation method and device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a queue resource allocation method and device, equipment, a medium and a program product. The method comprises the steps of obtaining a historical task execution log corresponding to each queue in a distributed cluster; determining historical task execution information of each queue in the historical time period based on the historical task execution logs; predicting target resource allocation proportion information corresponding to the current time period based on an information prediction model obtained by pre-training, the to-be-executed task information of each queue in the current time period, the historical task execution information and the historical resource allocation proportion information; and based on the target resource allocation proportion information, reallocating the total resource of the distributed cluster, so that each queue performs task execution in the current time period based on the reallocated current queue resource. Through the technical scheme of the embodiment of the invention, the automatic dynamic allocation of the queue resources can be realized, so that the resource utilization rate is improved, and the task execution efficiency is also ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and in particular to a queue resource allocation method, apparatus, device, medium and program product. Background Art

[0002] With the rapid development of computer technology, distributed clusters can be used to store and process large-scale data. Distributed clusters can be managed and tasks scheduled using schedulers. The scheduler allocates the total resources of the cluster to multiple queues, and each queue executes tasks based on the allocated resources.

[0003] At present, the operation and maintenance personnel usually configure the resource allocation ratio information of multiple queues based on their own experience, and then allocate the total resources of the cluster based on the manually configured resource allocation ratio information, so that each queue performs tasks based on the allocated fixed resources. However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] Due to the manual configuration method, the resources of each queue are always fixed, which may result in insufficient resources during task peak hours and excess resources during task trough hours, reducing resource utilization and task execution efficiency. Summary of the invention

[0005] The embodiments of the present invention provide a queue resource allocation method, apparatus, device, medium and program product to realize automatic dynamic allocation of queue resources, thereby improving resource utilization and ensuring task execution efficiency.

[0006] In a first aspect, an embodiment of the present invention provides a queue resource allocation method, including:

[0007] Obtaining a historical task execution log corresponding to each queue in the distributed cluster, wherein the historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it, wherein the historical queue resources are allocated based on the historical resource allocation ratio information;

[0008] Based on the historical task execution log, determine the historical task execution information of each queue within the historical time period;

[0009] Predict the target resource allocation ratio information corresponding to the current time period based on the pre-trained information prediction model, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information;

[0010] Based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated so that each queue performs tasks in the current time period based on the reallocated current queue resources.

[0011] In a second aspect, an embodiment of the present invention further provides a queue resource allocation device, including:

[0012] A historical task execution log acquisition module is used to acquire the historical task execution log corresponding to each queue in the distributed cluster. The historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it. The historical queue resources are allocated based on the historical resource allocation ratio information.

[0013] A historical task execution information determination module, used to determine the historical task execution information of each queue within a historical time period based on the historical task execution log;

[0014] A resource allocation ratio information prediction module is used to predict the target resource allocation ratio information corresponding to the current time period based on the information prediction model obtained by pre-training, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information;

[0015] The resource reallocation module is used to reallocate the total resources of the distributed cluster based on the target resource allocation ratio information, so that each queue executes tasks in the current time period based on the reallocated current queue resources.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the queue resource allocation method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a queue resource allocation method as provided in any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the queue resource allocation method provided by any embodiment of the present invention.

[0022] One embodiment of the above invention has the following advantages or beneficial effects:

[0023] Based on the historical task execution log corresponding to each queue in the distributed cluster, the historical task execution information of each queue in the historical time period is determined, and based on the pre-trained information prediction model, the task information to be executed in the current time period of each queue, the historical task execution information and the historical resource allocation ratio information, the target resource allocation ratio information that is more suitable for the current time period is automatically predicted. Based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated, so that each queue executes the tasks in the current time period based on the reallocated current queue resources, thereby realizing the automatic dynamic allocation of queue resources, improving resource utilization, and ensuring task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of a queue resource allocation method provided by an embodiment of the present invention;

[0026] Figure 2 is a flow chart of another queue resource allocation method provided by an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another queue resource allocation method provided by an embodiment of the present invention;

[0028] Figure 4 It is a structural schematic diagram of a queue resource allocation device provided by an embodiment of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Figure 1This is a flowchart of a queue resource allocation method provided by an embodiment of the present invention. This embodiment can be applied to the case where the total resources of a distributed cluster are dynamically allocated to multiple queues, especially to the application scenario where the scheduler dynamically allocates queue resources. The method can be executed by a queue resource allocation device, which can be implemented by software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically comprises the following steps:

[0032] S110. Obtain a historical task execution log corresponding to each queue in the distributed cluster. The historical task execution log is a log generated by each queue when executing tasks based on historical queue resources allocated to it. The historical queue resources are allocated based on historical resource allocation ratio information.

[0033] Among them, the distributed cluster can be a distributed system composed of multiple computer nodes. For example, the distributed cluster can be but is not limited to a Hadoop cluster. The queue is used to store tasks to be executed in the distributed cluster. There are multiple queues in the distributed cluster. Historical queue resources refer to the computing resources allocated to the queue in the historical time period, such as CPU and memory. The historical task execution log is used to record the specific circumstances of each queue using the allocated resources to process the tasks in the queue. The historical resource allocation ratio information refers to the ratio of resources allocated to each queue set in the historical time period to the total resources. For example, there are 3 queues in the distributed cluster, namely queue A, queue B and queue C, and the resource allocation ratio information can be: queue A is 50%, queue B is 30%, and queue C is 20%.

[0034] Specifically, based on the historical resource allocation ratio information, the total resources of the distributed cluster are allocated, and the historical queue resources allocated to each queue in the historical time period are determined. Each queue executes the tasks in the queue based on the allocated historical queue resources, thereby generating a historical task execution log. Through log collection, the historical task execution log corresponding to each queue in the distributed cluster can be obtained.

[0035] It should be noted that when the distributed cluster is initialized, resources can be allocated for the first time based on the manually configured resource allocation ratio information, so that each queue can execute tasks based on the queue resources allocated for the first time, and resource allocation can be dynamically adjusted based on the historical task execution logs obtained later.

[0036] S120 . Determine historical task execution information of each queue within a historical time period based on the historical task execution log.

[0037] The historical time period may refer to the time period before the current moment. The number of historical time periods may be one or more. For example, the historical time period may be at least one historical time period closest to the current moment. For example, when information prediction and allocation adjustment are performed with hours as the time granularity, the historical time period may refer to the nearest preset number of hourly time periods. For example, when the current moment is 4 o'clock, the historical time period may include 3 o'clock to 4 o'clock, 2 o'clock to 3 o'clock, etc. The historical task execution information may be task information used to characterize the parallel execution of each queue in the historical time period. Exemplarily, the historical task execution information may include: the number of historical tasks executed in parallel by each queue in the historical time period, the amount of historical execution data, and the historical task execution delay. The number of historical tasks refers to the number of tasks executed in parallel in the historical time period. The amount of historical execution data refers to the amount of library table data that needs to be read to execute tasks in the historical time period. The historical task execution delay is determined based on the execution delay time of each task in the historical time period. For example, the historical task execution delay may refer to the average execution delay time of tasks in the historical time period. The execution delay time of each task is determined based on the actual execution time and the actual submission time of each task. After a task is submitted, it may take some time for it to be executed due to insufficient queue resources, resulting in varying degrees of execution delay in task scheduling.

[0038] Specifically, by parsing and calculating the historical task execution log corresponding to each queue, the historical task execution information of each queue in each historical time period is determined.

[0039] S130, predicting the target resource allocation ratio information corresponding to the current time period based on the information prediction model obtained through pre-training, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information.

[0040] Among them, the information prediction model can be a neural network model for predicting target resource allocation ratio information that is more suitable for the current time period. For example, the information prediction model can be a time series data processing model, which can predict the deep learning model of current information based on historical time series data. The information prediction model can minimize the task execution delay through model training, so as to learn and predict the target resource allocation ratio information that is most suitable for the current time period, that is, the target resource allocation ratio information with the smallest task execution delay, thereby improving the task execution efficiency. The current time period refers to the time period in which the queue allocation resources need to be adjusted, that is, the time period at the current moment. The task information to be executed is used to characterize the task information that each queue needs to execute concurrently in the current time period. Exemplarily, the task information to be executed may include: the current number of tasks that each queue needs to execute in the current time period and the current execution data volume. Among them, the current number of tasks refers to the number of tasks that need to be executed in parallel in the current time period. The current execution data volume refers to the amount of library table data that needs to be read to execute tasks in the current time period.

[0041] Specifically, the information of tasks to be executed in the current time period, the historical task execution information in the historical time period, and the historical resource allocation ratio information of each queue are input into the information prediction model obtained by pre-training. The information prediction model adjusts the historical resource allocation ratio information based on the input task information to be executed and the historical task execution information. For example, if the queue resources are insufficient, the resource allocation ratio information of the queue is increased, and if the queue resources are excessive, the resource allocation ratio information of the queue is reduced, thereby determining the target resource allocation ratio information with the minimum task execution delay, and outputting the target resource allocation ratio information, thereby maximizing resource utilization. Based on the output of the information prediction model, the target resource allocation ratio information that is more suitable for the current time period can be automatically obtained. For example, the current number of tasks and the current execution data volume that each queue needs to execute in the current time period, the number of historical tasks executed in parallel in the historical time period, the historical execution data volume and the historical task execution delay, and the historical resource allocation ratio information of each queue can be input into the information prediction model obtained by pre-training, and the target resource allocation ratio information with the minimum task execution delay in the current time period is automatically predicted.

[0042] S140: Based on the target resource allocation ratio information, reallocate the total resources of the distributed cluster so that each queue executes tasks in the current time period based on the reallocated current queue resources.

[0043] The total resources of a distributed cluster refer to all computing resources of a distributed cluster. Specifically, the total resources of a distributed cluster are reallocated according to the target resource allocation ratio information, so that the current queue resources reallocated to each queue meet the target resource allocation ratio information, thereby realizing the automatic dynamic allocation of queue resources. For example, a scheduler (such as a Yarn scheduler) modifies the historical resource allocation ratio information in a configuration file to the target resource allocation ratio information, thereby triggering the reallocation of the total resources of a distributed cluster, so that each queue executes the tasks submitted in the current time period based on the more appropriate current queue resources reallocated, thereby avoiding the situation of insufficient resources during the peak period of tasks and excess resources during the valley period of tasks, improving resource utilization, and ensuring task execution efficiency.

[0044] It should be noted that since the resources allocated to each queue in the distributed cluster change dynamically over time, each queue is a tidal queue, which implements the most reasonable tidal configuration method and improves resource utilization. If the total resources of the distributed cluster are expanded or reduced as a whole, the queue resources can be automatically allocated according to the total resources after expansion or reduction in the next time period, reducing the manual operation and maintenance costs.

[0045] For example, based on the task execution log of each queue in the current time period, the execution delay time of each task can be determined. The total execution delay time of all tasks is divided by the total number of tasks to obtain the current average execution delay time of the distributed cluster. If the average execution delay time is greater than the preset delay time, a cluster resource expansion message is generated and displayed to remind the operation and maintenance personnel to expand the cluster resources and further ensure the efficiency of task execution.

[0046] The technical solution of this embodiment determines the historical task execution information of each queue in the historical time period based on the historical task execution log corresponding to each queue in the distributed cluster, and automatically predicts the target resource allocation ratio information that is more suitable for the current time period based on the pre-trained information prediction model, the task information to be executed by each queue in the current time period, the historical task execution information and the historical resource allocation ratio information; based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated, so that each queue executes the tasks of the current time period based on the reallocated current queue resources, thereby realizing the automatic dynamic allocation of queue resources, improving resource utilization, and ensuring task execution efficiency.

[0047] Based on the above technical solution, the information prediction model is trained based on the sample task data corresponding to each queue in the distributed cluster and the standard resource allocation ratio information corresponding to the sample task data; wherein the sample task data includes the historical task execution information and historical resource allocation ratio information of each queue in the first time period and the task information to be executed of each queue in the second time period; the standard resource allocation ratio information refers to the actual resource usage ratio information of each queue in the second time period, and the actual resource usage ratio information is determined based on the actual resources used by the queue and the total resources of the distributed cluster.

[0048] Among them, the first time period and the second time period are both historical time periods that have occurred, but the first time period is earlier than the second time period. The number of first time periods can be one or more. The actual resources used by the queue refer to the total resources actually used by the queue in the second time period. The actual resources used by the queue are the total resources required for the queue to execute tasks in the second time period. The actual resource usage ratio information refers to the resource allocation ratio information that is most suitable for the second time period, that is, the resource allocation ratio information with the smallest task execution delay in the second time period, so that the actual resource usage ratio information of each queue in the second time period can be used as the standard resource allocation ratio information.

[0049] Specifically, in the model training stage, the historical task execution log generated by the task execution in the first time period by the historical queue resources allocated to each queue based on the historical resource allocation ratio information is collected, and the historical task execution information of each queue in the first time period is determined based on the historical task execution log, such as the number of historical tasks executed in parallel by each queue in the first time period, the amount of historical execution data, and the historical task execution delay. The historical task execution information and historical resource allocation ratio information of each queue in the first time period and the task information to be executed by each queue in the second time period (such as the number of current tasks and the amount of current execution data to be executed by each queue in the second time period) are combined into sample task data. The historical task execution log generated by the task execution in the second time period by each queue based on the reallocated queue resources is collected, and the actual queue usage resources of each queue are determined based on the historical task execution log, and the result obtained by dividing the actual queue usage resources of each queue by the total resources of the distributed cluster is used as the actual resource usage ratio information. The actual resource usage ratio information is used as the model training label, that is, the actual resource usage ratio information is used as the standard resource allocation ratio information, and the model is supervised based on the sample task data corresponding to each queue to obtain the trained information prediction model, so as to ensure the accuracy of the resource allocation ratio information prediction. For example, the sample task data corresponding to each queue is input into the model to be trained, the sample resource allocation ratio information output by the model to be trained is obtained, the training error is determined based on the sample resource allocation ratio information and the standard resource allocation ratio information, and the training error is back-propagated to the model to be trained to adjust the network weights in the model to be trained until the preset convergence conditions are met, such as the training error is less than the preset error and tends to be stable, or when the preset number of iterations is reached, the model training is determined to be completed, thereby obtaining an information prediction model that can accurately predict the resource allocation ratio information, thereby improving the reasonable allocation and dynamic adjustment of resources and improving resource utilization.

[0050] Figure 2 A flowchart of another queue resource allocation method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment optimizes the step of "reallocating the total resources of the distributed cluster based on the target resource allocation ratio information so that each queue performs tasks in the current time period based on the reallocated queue resources". Explanations of terms that are the same or corresponding to the above embodiments are not repeated here.

[0051] See also Figure 2 Another queue resource allocation method provided in this embodiment specifically includes the following steps:

[0052] S210. Obtain a historical task execution log corresponding to each queue in the distributed cluster. The historical task execution log is a log generated by each queue when executing tasks based on historical queue resources allocated to it. The historical queue resources are allocated based on historical resource allocation ratio information.

[0053] S220: Based on the historical task execution log, determine the historical task execution information of each queue in the historical time period.

[0054] S230, predicting the target resource allocation ratio information corresponding to the current time period based on the information prediction model obtained through pre-training, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information.

[0055] S240: Determine the difference between the historical resource allocation ratio information and the target resource allocation ratio information.

[0056] The historical resource allocation ratio information may refer to the resource allocation ratio information set in the target historical time period adjacent to the current time period. The target historical time period is the historical time period closest to the current time period. The historical resource allocation ratio information also refers to the resource allocation ratio information currently set in the configuration file of the scheduler.

[0057] Specifically, by determining the difference between the historical resource allocation ratio information of the historical time period closest to the current time period and the target resource allocation ratio information corresponding to the current time period, such as the standard deviation, the degree of deviation between the historical resource allocation ratio information and the target resource allocation ratio information can be measured to determine whether to reallocate queue resources.

[0058] S250: If the difference is greater than or equal to the preset difference, the total resources of the distributed cluster are reallocated based on the target resource allocation ratio information, so that each queue executes the tasks in the current time period based on the reallocated current queue resources.

[0059] Specifically, when the difference between the historical resource allocation ratio information and the target resource allocation ratio information is greater than or equal to the preset difference, it indicates that there is a large deviation in the currently set historical resource allocation ratio information, and the resource allocation ratio information needs to be reset to meet the queue resource requirements in the current time period. At this time, the total resources of the distributed cluster can be reallocated so that the current queue resources reallocated to each queue meet the target resource allocation ratio information, and then each queue executes the tasks submitted in the current time period based on the reallocated more reasonable current queue resources, which ensures the task execution efficiency and improves resource utilization.

[0060] Exemplarily, if the difference is smaller than a preset difference, the reallocation of the total resources of the distributed cluster is stopped, so that each queue performs tasks in the current time period based on the historical queue resources allocated thereto.

[0061] Specifically, when the difference between the historical resource allocation ratio information and the target resource allocation ratio information is less than the preset difference, it indicates that the deviation of the currently set historical resource allocation ratio information is not large and is acceptable. There is no need to reset the resource allocation ratio information. At this time, the reallocation of the total resources of the distributed cluster can be stopped, so that each queue continues to execute the tasks submitted in the current time period based on the currently allocated historical queue resources, thereby avoiding frequent changes to the configuration file, reducing the cluster refresh frequency, and ensuring timely data output by the business.

[0062] The technical solution of this embodiment is to reallocate the total resources of the distributed cluster based on the target resource allocation ratio information when the difference between the historical resource allocation ratio information and the target resource allocation ratio information is greater than or equal to the preset difference, so that each queue executes the tasks submitted in the current time period based on the reallocated more reasonable current queue resources, thereby ensuring the task execution efficiency and improving resource utilization.

[0063] Figure 3 A flowchart of another queue resource allocation method provided by an embodiment of the present invention, based on the above embodiments, this embodiment optimizes the step of "executing tasks in the current time period based on the reallocated current queue resources". The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0064] See also Figure 3 Another queue resource allocation method provided in this embodiment specifically includes the following steps:

[0065] S310. Obtain a historical task execution log corresponding to each queue in the distributed cluster. The historical task execution log is a log generated by each queue when executing tasks based on historical queue resources allocated to it. The historical queue resources are allocated based on historical resource allocation ratio information.

[0066] S320: Based on the historical task execution log, determine the historical task execution information of each queue in the historical time period.

[0067] S330, predicting the target resource allocation ratio information corresponding to the current time period based on the information prediction model obtained through pre-training, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information.

[0068] S340: Based on the target resource allocation ratio information, reallocate the total resources of the distributed cluster so that each queue obtains the reallocated current queue resources.

[0069] S350: For each queue, obtain the target task assigned to the current queue in the current time period.

[0070] The current queue refers to any queue in the cluster. The task execution process of each queue is the same. The current queue is used as an example for explanation.

[0071] Specifically, the scheduler can assign each task submitted by the user to a queue that the user has permission to use, so that the resources of the queue can be used to execute the assigned task. For each queue, the target task assigned to the current queue in the current time period can be obtained. The number of target tasks can be one or more.

[0072] S360: Determine the task execution order corresponding to the current queue based on the job priority and submission time of the target task, and execute the target task based on the task execution order and the current queue resources reallocated to the current queue.

[0073] Among them, the job priority can be used to characterize the urgency of executing the target task. The job priority can be set when the user submits the job. The submission time refers to the time point when the target task is submitted. The task execution order is used to characterize the execution order of all tasks in the current queue. The higher the job priority, the earlier the task is executed; the earlier the submission time, the higher the task is executed. Tasks can be sorted in the order of job priority from high to low and submission time from early to late.

[0074] Specifically, the job priority and submission time of the target task can be weighted and summed to obtain the execution priority of each target task, and all tasks in the current queue can be sorted in order from high to low execution priority to obtain the task execution order. According to the task execution order, the current queue resources reallocated to the current queue are used to execute the assigned target tasks, so that the target tasks with high job priority and early submission time are executed first. By considering both the job priority and the submission time to execute tasks, it can be avoided that the task with high job priority is submitted after the task with low job priority and needs to wait for a long time before execution, which further ensures the efficiency of task execution.

[0075] The technical solution of this embodiment determines the task execution order corresponding to the current queue based on the job priority and submission time of the target task assigned to the current queue in the current time period, and executes the target task based on the task execution order and the current queue resources reallocated to the current queue, thereby executing the task while taking into account the job priority and submission time at the same time, further ensuring the task execution efficiency.

[0076] On the basis of the above technical solutions, the target task is a task submitted by the target user. Accordingly, step S360 may include the following steps S361 and S362:

[0077] S361: Determine the currently executed task submitted by the target user in the current queue, and determine the currently used total resources corresponding to the currently executed task.

[0078] The currently executed task refers to the task in the current queue that is also submitted by the target user and is being executed. The number of currently executed tasks can be one or more. Each user can have the right to use one or more queues, that is, each user can bind one or more queues. Each queue can be used by multiple users.

[0079] Specifically, after the target task submitted by the target user is assigned to the current queue, the current execution task assigned to the current queue and also submitted by the target user can be determined, and based on the execution information of the current execution task, the total resources used to execute the current execution task, that is, the total resources currently used, can be determined.

[0080] S362. If the total resources currently in use are less than the maximum resources corresponding to the target user, the task execution order corresponding to the current queue is determined based on the job priority and submission time of the target task, and the target task is executed based on the task execution order and the current queue resources reallocated to the current queue.

[0081] The maximum resource usage allowed for each user may be preconfigured to avoid the situation where the same user monopolizes resources in the queue.

[0082] Specifically, when the total resources currently in use are less than the maximum resources used corresponding to the target user, it indicates that the current queue resources used by the target user have not exceeded the maximum resources used, and the target task can be allowed to be executed in the current queue. At this time, based on the job priority and submission time of the target task, the task execution order corresponding to the current queue is determined, and according to the task execution order, the current queue resources reallocated to the current queue are used to execute the target task, thereby avoiding the situation where the target task submitted by the target user monopolizes the resources in the current queue, thereby ensuring the normal execution of the task.

[0083] Exemplarily, if the total resources currently in use are greater than or equal to the maximum resources used corresponding to the target user, then an optional queue other than the current queue and for which the target user has permission to use is determined; based on the currently used resources and currently allocated resources of each optional queue, the resource usage level corresponding to each optional queue is determined; the optional queue with the lowest resource usage level is used as the target queue, and the target task is reallocated to the target queue for execution.

[0084] Specifically, when the total resources currently in use are greater than or equal to the maximum resources used corresponding to the target user, it indicates that the resources of the current queue used by the target user have reached or exceeded the maximum resources used, and the target task cannot be performed in the current queue. At this time, the remaining queues other than the current queue can be removed from all queues that the target user has permission to use as optional queues. The number of optional queues can be one or more. The currently used resources of each optional queue can be divided by the currently allocated resources to obtain the resource usage level corresponding to each optional queue, and the optional queue with the smallest resource usage level is used as the target queue, that is, the target queue is the optional queue with the most idle resources. The target task is reallocated to the target queue to directly execute the target task in the target queue, or the current execution task submitted by the target user in the target queue is determined, and the current total resources used corresponding to the current execution task are determined, and when the current total resources used are less than the maximum resources used corresponding to the target user, the task execution order corresponding to the target queue is determined based on the job priority and submission time of the target task, and the target task is executed according to the task execution order using the current queue resources reallocated to the target queue. If the current total resources used by the target queue are still greater than or equal to the maximum resources used by the target user, queues that meet the conditions can continue to be screened until the current total resources used by the screened queues are less than the maximum resources used by the target user, and the target task is then assigned to the finally screened queue for execution.

[0085] The following is an embodiment of a queue resource allocation device provided in an embodiment of the present invention. The device and the queue resource allocation methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the queue resource allocation device, reference can be made to the embodiment of the above queue resource allocation method.

[0086] Figure 4 This is a schematic diagram of the structure of a queue resource allocation device provided by an embodiment of the present invention. This embodiment is applicable to the case where the total resources of a distributed cluster are dynamically allocated to multiple queues, especially to the application scenario where the scheduler dynamically allocates queue resources. Figure 4 As shown, the device specifically includes: a historical task execution log acquisition module 410, a historical task execution information determination module 420, a resource allocation ratio information prediction module 430 and a resource reallocation module 440.

[0087] Among them, the historical task execution log acquisition module 410 is used to obtain the historical task execution log corresponding to each queue in the distributed cluster, and the historical task execution log is a log generated by each queue for task execution based on the historical queue resources allocated, and the historical queue resources are allocated based on the historical resource allocation ratio information; the historical task execution information determination module 420 is used to determine the historical task execution information of each queue in the historical time period based on the historical task execution log; the resource allocation ratio information prediction module 430 is used to predict the target resource allocation ratio information corresponding to the current time period based on the pre-trained information prediction model, the task information to be executed by each queue in the current time period, the historical task execution information and the historical resource allocation ratio information; the resource reallocation module 440 is used to reallocate the total resources of the distributed cluster based on the target resource allocation ratio information, so that each queue performs the task of the current time period based on the reallocated current queue resources.

[0088] The technical solution of this embodiment determines the historical task execution information of each queue in the historical time period based on the historical task execution log corresponding to each queue in the distributed cluster, and automatically predicts the target resource allocation ratio information that is more suitable for the current time period based on the pre-trained information prediction model, the task information to be executed by each queue in the current time period, the historical task execution information and the historical resource allocation ratio information; based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated, so that each queue executes the tasks of the current time period based on the reallocated current queue resources, thereby realizing the automatic dynamic allocation of queue resources, improving resource utilization, and ensuring task execution efficiency.

[0089] Optionally, the historical task execution information includes: the number of historical tasks executed in parallel by each queue in the historical time period, the amount of historical execution data, and the historical task execution delay;

[0090] The to-be-executed task information includes: the current number of tasks and the current execution data volume that each queue needs to execute within the current time period.

[0091] Optionally, the information prediction model is obtained by training based on sample task data corresponding to each queue in the distributed cluster and standard resource allocation ratio information corresponding to the sample task data;

[0092] The sample task data includes historical task execution information and historical resource allocation ratio information of each queue in the first time period and to-be-executed task information of each queue in the second time period;

[0093] The standard resource allocation ratio information refers to actual resource usage ratio information of each queue in the second time period, and the actual resource usage ratio information is determined based on the actual resources used by the queue and the total resources of the distributed cluster.

[0094] Optionally, the resource reallocation module 440 includes:

[0095] a difference determination unit, configured to determine a difference between the historical resource allocation ratio information and the target resource allocation ratio information;

[0096] A resource reallocation unit is used to reallocate the total resources of the distributed cluster based on the target resource allocation ratio information if the difference is greater than or equal to a preset difference, so that each queue executes tasks in the current time period based on the reallocated current queue resources.

[0097] Optionally, the resource reallocation module 440 further includes:

[0098] The resource allocation stopping unit is used to stop reallocating the total resources of the distributed cluster if the difference is less than a preset difference, so that each queue executes tasks in the current time period based on the historical queue resources allocated to it.

[0099] Optionally, the resource reallocation module 440 includes:

[0100] A target task acquisition unit, used for acquiring, for each queue, the target task assigned to the current queue in the current time period;

[0101] The target task execution unit is used to determine the task execution order corresponding to the current queue based on the job priority and submission time of the target task, and execute the target task based on the task execution order and the current queue resources reallocated to the current queue.

[0102] Optionally, the target task is a task submitted by a target user;

[0103] Target task execution unit, including:

[0104] The currently used total resource determination subunit is used to determine the currently executed task submitted by the target user in the current queue, and determine the currently used total resources corresponding to the currently executed task;

[0105] The target task execution subunit is used to determine the task execution order corresponding to the current queue based on the job priority and submission time of the target task if the current total resources in use are less than the maximum resources in use corresponding to the target user, and to execute the target task based on the task execution order and the current queue resources reallocated to the current queue.

[0106] Optionally, the target task execution unit further includes:

[0107] an optional queue determination subunit, configured to determine, if the total resources currently in use are greater than or equal to the maximum resources used by the target user, an optional queue other than the current queue that the target user has permission to use;

[0108] A resource usage level determination subunit, configured to determine a resource usage level corresponding to each optional queue based on currently used resources and currently allocated resources of each optional queue;

[0109] The target task reallocation subunit is used to take the optional queue with the lowest resource usage as the target queue, and reallocate the target task to the target queue for execution.

[0110] The queue resource allocation device provided in the embodiment of the present invention can execute the queue resource allocation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the queue resource allocation method.

[0111] It is worth noting that in the embodiment of the above-mentioned queue resource allocation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0112] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 5 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0114] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0115] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0116] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory 30 (RAM) and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0117] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0118] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a queue resource allocation method provided by any embodiment of the present invention, the method comprising:

[0120] Obtaining a historical task execution log corresponding to each queue in the distributed cluster, wherein the historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it, wherein the historical queue resources are allocated based on the historical resource allocation ratio information;

[0121] Based on the historical task execution log, determine the historical task execution information of each queue within the historical time period;

[0122] Predict the target resource allocation ratio information corresponding to the current time period based on the pre-trained information prediction model, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information;

[0123] Based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated so that each queue performs tasks in the current time period based on the reallocated current queue resources.

[0124] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the queue resource allocation method provided by any embodiment of the present invention.

[0125] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the queue resource allocation method provided in any embodiment of the present invention are implemented. The method includes:

[0126] Obtaining a historical task execution log corresponding to each queue in the distributed cluster, wherein the historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it, wherein the historical queue resources are allocated based on the historical resource allocation ratio information;

[0127] Based on the historical task execution log, determine the historical task execution information of each queue within the historical time period;

[0128] Predict the target resource allocation ratio information corresponding to the current time period based on the pre-trained information prediction model, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information;

[0129] Based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated so that each queue performs tasks in the current time period based on the reallocated current queue resources.

[0130] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0131] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0133] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the queue resource allocation method provided by any embodiment of the present invention.

[0135] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0136] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0137] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A queue resource allocation method, characterized in that: include: Obtaining a historical task execution log corresponding to each queue in the distributed cluster, wherein the historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it, wherein the historical queue resources are allocated based on the historical resource allocation ratio information; Based on the historical task execution log, determine the historical task execution information of each queue within the historical time period; Predict the target resource allocation ratio information corresponding to the current time period based on the pre-trained information prediction model, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information; Based on the target resource allocation ratio information, the total resources of the distributed cluster are reallocated so that each queue performs tasks in the current time period based on the reallocated current queue resources.

2. The method according to claim 1, characterized in that The historical task execution information includes: the number of historical tasks executed in parallel by each queue in the historical time period, the amount of historical execution data and the historical task execution delay; The to-be-executed task information includes: the current number of tasks and the current execution data volume that each queue needs to execute within the current time period.

3. The method according to claim 1, characterized in that The information prediction model is obtained by training based on sample task data corresponding to each queue in the distributed cluster and standard resource allocation ratio information corresponding to the sample task data; The sample task data includes historical task execution information and historical resource allocation ratio information of each queue in the first time period and to-be-executed task information of each queue in the second time period; The standard resource allocation ratio information refers to actual resource usage ratio information of each queue in the second time period, and the actual resource usage ratio information is determined based on the actual resources used by the queue and the total resources of the distributed cluster.

4. The method according to claim 1, characterized in that: The reallocating of the total resources of the distributed cluster based on the target resource allocation ratio information so that each queue performs tasks in the current time period based on the reallocated current queue resources includes: Determine a difference between the historical resource allocation ratio information and the target resource allocation ratio information; If the difference is greater than or equal to the preset difference, the total resources of the distributed cluster are reallocated based on the target resource allocation ratio information, so that each queue performs tasks in the current time period based on the reallocated current queue resources.

5. The method according to claim 4, characterized in that The method further comprises: If the difference is smaller than a preset difference, the reallocation of the total resources of the distributed cluster is stopped, so that each queue performs tasks in the current time period based on the historical queue resources allocated thereto.

6. The method according to claim 1, characterized in that Execute tasks in the current time period based on the reallocated current queue resources, including: For each queue, obtain the target task assigned to the current queue in the current time period; Based on the job priority and submission time of the target task, the task execution order corresponding to the current queue is determined, and the target task is executed based on the task execution order and the current queue resources reallocated to the current queue.

7. The method according to claim 6, characterized in that The target task is a task submitted by a target user; The determining the task execution order corresponding to the current queue based on the job priority and submission time of the target task, and executing the target task based on the task execution order and the current queue resources reallocated to the current queue, includes: Determine the currently executed task submitted by the target user in the current queue, and determine the currently used total resources corresponding to the currently executed task; If the total resources currently in use are less than the maximum resources corresponding to the target user, the task execution order corresponding to the current queue is determined based on the job priority and submission time of the target task, and the target task is executed based on the task execution order and the current queue resources reallocated to the current queue.

8. The method according to claim 7, characterized in that The method further comprises: If the total resources currently in use are greater than or equal to the maximum resources used by the target user, then determining an optional queue other than the current queue and for which the target user has permission to use; Determine the resource usage level corresponding to each optional queue based on the currently used resources and currently allocated resources of each optional queue; The selectable queue with the lowest resource usage is used as the target queue, and the target task is reallocated to the target queue for execution.

9. A queue resource allocation device, characterized in that: include: A historical task execution log acquisition module is used to acquire the historical task execution log corresponding to each queue in the distributed cluster. The historical task execution log is a log generated by each queue when executing tasks based on the historical queue resources allocated to it. The historical queue resources are allocated based on the historical resource allocation ratio information. A historical task execution information determination module, used to determine the historical task execution information of each queue within a historical time period based on the historical task execution log; A resource allocation ratio information prediction module is used to predict the target resource allocation ratio information corresponding to the current time period based on the information prediction model obtained by pre-training, the task information to be executed in each queue in the current time period, the historical task execution information and the historical resource allocation ratio information; The resource reallocation module is used to reallocate the total resources of the distributed cluster based on the target resource allocation ratio information, so that each queue executes tasks in the current time period based on the reallocated current queue resources.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the queue resource allocation method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the queue resource allocation method as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the queue resource allocation method as described in any one of claims 1 to 8.

Images