Resource allocation method and device for task queue, equipment and medium

Abstract

The invention discloses a resource allocation method and device for task queues, equipment and a medium, and belongs to the field of data processing. The method comprises the steps of obtaining a time sequence resource prediction model corresponding to a resource application type of a task queue; predicting based on task demand information of the task queue according to the obtained time sequence resource prediction model, and outputting an initial predicted resource quantity of the task queue in a future time period, the task demand information comprising task description information, or the task demand information comprising the task description information and historical resource use information; optimizing the initial predicted resource quantity by taking the maximization of the utilization rate of the resources in the cluster as a target to obtain the predicted resource quantity of the task queue in the future time period; and allocating resources to the task queue in the future time period according to the predicted resource quantity. According to the embodiment of the invention, the resource allocation strategy of the task queue with higher accuracy can be obtained, and the resource allocation requirement is met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing, and particularly relates to a resource allocation method and device for a task queue, an equipment and a medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology in various fields, the complexity of computing tasks carried by big data platforms continues to rise, and the demand for computing resources by users is also rising. In some cases, there has been a demand for about 125 physical task scenarios of 5000C (C refers to logical CPU core). Such large-scale and high-complexity computing tasks pose a serious challenge to the resource supply capacity of resource clusters, and there is an urgent need for a method that can efficiently and reasonably allocate resources to a task queue. At present, samples are extracted from the running status of the task, and the resource allocation strategy for the task queue is determined by artificial experience. However, with the large-scale and high-complexity of computing tasks, the accuracy of the resource allocation strategy determined by artificial experience is insufficient, and it is difficult to meet the resource allocation demand. SUMMARY

[0003] The embodiments of the present application provide a resource allocation method, device, equipment and medium for a task queue, which can obtain a task queue resource allocation strategy with higher accuracy and meet the resource allocation demand.

[0004] In a first aspect, the embodiments of the present application provide a resource allocation method for a task queue, comprising: obtaining a time series resource prediction model corresponding to a resource application type of a task queue; predicting based on task demand information of the task queue according to the obtained time series resource prediction model, outputting an initial predicted resource amount of the task queue in a future time period, the task demand information comprising task description information, or the task demand information comprising task description information and historical resource usage information; optimizing the initial predicted resource amount to obtain a predicted resource amount of the task queue in the future time period, with the maximum utilization rate of resources in the cluster as the target; and allocating resources to the task queue in the future time period according to the predicted resource amount.

[0005] In a second aspect, an embodiment of the present application provides a resource allocation device for a task queue, comprising: a model determination module configured to obtain a time series resource prediction model corresponding to a resource application type of the task queue; a prediction module configured to predict, based on task demand information of the task queue, an initial predicted resource amount of the task queue in a future time period according to the obtained time series resource prediction model, the task demand information comprising task description information, or the task demand information comprising the task description information and historical resource usage information; an optimization module configured to optimize the initial predicted resource amount to obtain a predicted resource amount of the task queue in the future time period, with the goal of maximizing the usage rate of resources in a cluster; and an execution module configured to allocate resources for the task queue in the future time period according to the predicted resource amount.

[0006] In a third aspect, an embodiment of the present application provides a resource allocation device for a task queue, comprising: a processor and a memory storing computer program instructions; and the processor implements the resource allocation method for a task queue of the first aspect when executing the computer program instructions.

[0007] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the resource allocation method for a task queue of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the resource allocation method for a task queue of the first aspect.

[0009] The embodiments of the present application provide a resource allocation method, device, equipment and medium for a task queue. The time series resource prediction model can be selected according to the resource application type of the task queue. The selected time series resource prediction model can predict an initial predicted resource amount in a future time period based on task demand information capable of representing the task demand of the task queue. The time series resource prediction model can analyze complex tasks of the task queue, and the obtained initial predicted resource amount is more consistent with the task scenario of the task queue and more accurate. The initial predicted resource amount is optimized with the goal of maximizing the usage rate of resources in a cluster, and a predicted resource amount capable of improving the resource usage rate of the entire cluster is obtained. The resource allocation process originally relying on human experience is changed into a data-driven optimization process, and a resource allocation strategy for the task queue with higher accuracy can be obtained, thereby meeting the resource allocation demand. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. For those skilled in the art, other drawings can be obtained without creative labor on the premise of the drawings.

[0011] Figure 1 The flowchart of the resource allocation method of the task queue provided by an embodiment of the present application; Figure 2 The flowchart of an example of the data processing flow of the long-period deep learning model provided by an embodiment of the present application; Figure 3 The flowchart of an example of the selection of the time series resource prediction model in the capacity expansion queue application type scenario provided by an embodiment of the present application; Figure 4 The flowchart of an example of obtaining the initial predicted resource amount by using the large language model technology provided by an embodiment of the present application; Figure 5 The schematic diagram of an example of the resource allocation flow of the task queue provided by an embodiment of the present application Figure 6 The schematic diagram of the resource allocation device of the task queue provided by an embodiment of the present application; Figure 7 The structural schematic diagram of the resource allocation device of the task queue provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of the specific details by those skilled in the art. The description of the embodiments below is only to provide a better understanding of the present application by showing examples of the present application. It should be noted that the acquisition, storage, use, processing, etc. of information and data in the embodiments of the present application are authorized by the user or relevant institution and comply with relevant regulations of national laws and regulations. In the embodiments of the present application, some industry existing solutions, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0013] With the rapid development of artificial intelligence technology in various fields, the complexity of computing tasks carried by big data platforms continues to rise, and users' demand for computing resources is also growing. In some cases, there have been scenarios where demand reaches 5000C about 125 physical tasks. Such large-scale, high-complexity computing tasks pose a serious challenge to the resource supply capacity of resource clusters, and there is an urgent need for a method that can efficiently and reasonably allocate resources to task queues. At the present stage, samples of the running status of tasks are extracted, and the resource allocation strategy for the task queue is determined through artificial experience. However, with the large-scale and high-complexity of computing tasks, the accuracy of the resource allocation strategy determined through artificial experience is insufficient, and it is difficult to meet the resource allocation demand.

[0014] The present application provides a task queue resource allocation method, device, equipment, medium and program product, which can analyze the complexity of the tasks of the task queue, reflect the resource application type of the resource application scenario, combine the historical resource usage and running rules of the task queue, predict the amount of resources needed for queue allocation in the future through a large model, and then optimize the resource amount predicted by the large model through a small model, with the goal of maximizing the overall resource utilization of the cluster, to obtain a resource prediction result that is more in line with the actual usage, and to change the resource allocation process that originally relies on artificial experience into a data-driven optimization process, which can obtain a resource allocation strategy with higher accuracy and meet the resource allocation demand. After the small model optimizes the predicted resource amount, the resource allocation can also be continuously adjusted during the running process according to the running status of the task queue, and an accurate resource allocation strategy can be obtained through the three-stage cyclic iteration of large model prediction, small model optimization, and running dynamic adjustment, continuously improving the accuracy of queue resource prediction and improving the rationality of resource allocation. Within a certain period, it can also meet the growing resource demand of users without expanding the queue, and to a certain extent, it can also improve the running efficiency of the task queue.

[0015] For ease of understanding, the cluster and task queue involved in the embodiments of the present application are first described. The cluster refers to a collection of multiple independent computing nodes connected and integrated through a network, managed and used as a unified, single computing resource. The computing node can be implemented as a computer server or the like. The task queue can be considered as a logical resource container and scheduling unit. The cluster can allocate resources within the cluster to the task queue, so that the task queue executes corresponding tasks using the allocated resources.

[0016] The task queue resource allocation method, device, equipment, medium and program product provided by the present application are described below.

[0017] The application provides a task queue resource allocation method, which can predict the resource amount of a task queue in a future time period. The task queue resource allocation method can be executed by a task queue resource allocation device, equipment, etc., and is not limited herein. Figure 1 The task queue resource allocation method provided by an embodiment of the application is shown in a flowchart as Figure 1 The task queue resource allocation method can include steps S101-S104.

[0018] In step S101, a time series resource prediction model corresponding to a resource application type of a task queue is obtained.

[0019] The resource application type can represent a resource application scenario. For example, the resource application type can include an added queue application type and an expanded queue application type; the added queue application type represents a resource application scenario of an added task queue; and the expanded queue application type represents a resource application scenario of an existing task queue that needs to be expanded.

[0020] The time series resource prediction model is a model that can predict the resource amount of a task queue in a future time period. The time series resource prediction model can output the resource amount of a task queue arranged in a time series manner in a future time period. Different resource application types can correspond to different time series resource prediction models, so that the selected time series resource prediction model is adapted to the resource application scenario.

[0021] In step S102, based on the obtained time series resource prediction model, the initial prediction resource amount of the task queue in the future time period is output based on task demand information of the task queue.

[0022] The input of the obtained time series resource prediction model is the task demand information of the task queue, and the output is the initial prediction resource amount in the future time period. The initial prediction resource amount in the future time period output by the time series resource prediction model can include a group of initial prediction resource amounts, and the group of initial prediction resource amounts corresponds to the entire future time period; the initial prediction resource amount in the future time period output by the time series resource prediction model can include multiple groups of initial prediction resource amounts, and each group of initial prediction resource amounts in the multiple groups of initial prediction resource amounts can correspond to a time point or a time period in the future time period, that is, the time series resource prediction model can simultaneously output the initial prediction resource amounts corresponding to multiple time points or multiple time periods in the future time period.

[0023] The task demand information is used to represent the demand of the task queue. The task demand information can include task description information, or the task demand information includes the task description information and historical resource usage information. The task description information can include information used to describe the task of the task queue, for example, the task description information can include but is not limited to one or more of the following: business scenario description, business importance description, task processing data table, task processing data volume, resource application type, task type, task quantity, task complexity, task runtime requirement, structured query language (SQL) statement of the task, and the like. The historical resource usage information is used to represent the resource usage of the task queue in a historical time length. The historical resource usage information can include but is not limited to one or more of the following: resource allocation amount, resource usage rate, time characteristic, task execution time, task concurrency quantity, task type, task processing data volume, resource application type, task complexity, task waiting quantity, task delay, and the like of the task queue in the historical time length. It should be noted that the historical time length can include a time length from a certain time point before the current time point to the current time point.

[0024] Different resource application types can correspond to different time series resource prediction models. For example, in the case of a resource application type being an added queue application type, the obtained time series resource model can include a baseline prediction model, which can provide a prediction value without historical data of a specific target itself. The added queue application type represents that the task queue is an added task queue, which does not have corresponding historical resource usage information, and can be processed by the baseline prediction model in a presumed aggregation manner for this kind of scenario. For another example, in the case of a resource application type being an expansion queue application, the obtained time series resource prediction model can include a short-period machine learning model or a long-period deep learning model. Both the short-period machine learning model and the long-period deep learning model can learn the correlation of resource usage at different time points in the historical sequence, and further improve the accuracy of the initial predicted resource amount. Whether to use the short-period machine learning model or the long-period deep learning model can be determined by comparing the historical time length of the historical resource usage information with a preset time length. The preset time length is a time length threshold for judging long and short periods, which can be set according to the scene, demand, experience, and the like, and is not limited herein, for example, the preset time length can be set to half a year. The short-period machine learning model is more suitable for a task queue with a historical time length less than or equal to the preset time length. The long-period deep learning model is more suitable for a task queue with a historical time length greater than the preset time length.

[0025] In step S103, the initial predicted resource amount is optimized to maximize the usage rate of the resources in the cluster, to obtain the predicted resource amount of the task queue in the future time period.

[0026] After obtaining the initial predicted resource amount, the initial predicted resource amount can be optimized in combination with the usage rate of the resources of the cluster as a whole, and the predicted resource amount obtained after optimization can maximize the usage rate of the resources in the cluster. Specifically, the optimization of the initial predicted resource amount can be implemented by using a resource evaluation optimal solution method (REOSM), the resource evaluation problem is modeled as a mathematical optimization problem, and an efficient Lagrange multiplier method is used to obtain the predicted resource amount after optimization.

[0027] In step S104, the resources are allocated to the task queue in the future time period according to the predicted resource amount.

[0028] The resources in the embodiments of the present application can include, but are not limited to, one or more of the following: central processing unit (CPU) resources and memory resources.

[0029] In the embodiments of the present application, the time series resource prediction model can be selected according to the resource application type of the task queue. The selected time series resource prediction model can predict the initial predicted resource amount in the future time period based on the task demand information capable of representing the task demand of the task queue. The time series resource prediction model can analyze the complex tasks of the task queue, and the obtained initial predicted resource amount is more consistent with the task scenario of the task queue and is more accurate. The initial predicted resource amount is optimized to maximize the usage rate of the resources in the cluster, and the predicted resource amount capable of improving the resource usage rate of the entire cluster is obtained. The resource allocation process originally relying on human experience is changed into a data-driven optimization process, a more accurate resource allocation strategy can be obtained to meet the resource allocation demand.

[0030] In some embodiments, the obtained time series resource prediction model includes a baseline prediction model, which can include, but is not limited to, a Bottom-Up model. The Bottom-Up model can start from the task level, gradually analyze the tasks that need to be run in the task queue, and gradually aggregate the tasks to predict the initial predicted resource amount. The task description information can be analyzed by the baseline prediction model to obtain the tasks contained in the task queue, and the executor running resource information of each task is obtained. The concurrent resource amount of multiple time windows in the task timeline is obtained by the baseline prediction model based on the executor running resource information of each task. The initial predicted resource amount of the task queue in the future time period is determined and output by the baseline prediction model according to the concurrent resource amount of the multiple time windows and the safe resource threshold.

[0031] For ease of illustration, the process of predicting the initial predicted resource amount by the baseline prediction model is described below taking the Bottom-Up model as an example. The task description information can be input into the Bottom-Up model, and the task description information can include a SQL statement. The Bottom-Up model can deeply analyze the influence of the complexity of the task corresponding to the SQL statement. The Bottom-Up model can perform task decomposition on the SQL statement to obtain tasks contained in a task queue, for example, the tasks can include but are not limited to ETL tasks, Hive tasks, Spark tasks, etc. The ETL task is a task of arranging raw data into usable and valuable data through a standard process. E represents Extract, T represents Transform, and L represents Load. The Hive task and the Spark task are report generation tasks. The executor running resource information of the tasks can be obtained according to the task categories. The executor running resource information can represent the resources required for the execution of the executor, which can also be referred to as Executor. The executor running resource information can include but is not limited to one or more of the following: the number of executors, the number of CPU cores allocated to each executor (also referred to as Executor-cores), the size of the memory allocated to each executor (also referred to as Executor-memory), the task running time, the task scheduling frequency, etc. For example, the executor running resource information of the anomaly detection task is obtained, the number of executors is 50, the number of CPU cores allocated to each executor is 4C, the memory allocated to each executor is 12G, the task running time is 2 hours, and the task scheduling frequency is to run at 1 a.m. every day. Similarly, the executor running resource information of other types of tasks also needs to be collected.

[0032] The task timeline of the tasks included in the analyzable task queue can be divided into multiple time windows, and the concurrent resource amount of each time window can be determined according to the executor resource information of each task. The concurrent resource amount in a time window can represent the resource amount required for the concurrency of tasks in the corresponding time window during the running of the task queue. The calculation method of the reference resource amount required for predicting the initial predicted resource amount can be determined according to the difference between the concurrent resource amounts in different time windows, so as to determine the reference resource amount according to the concurrent resource amounts of multiple time windows and the determined calculation method. The reference resource amount can be regarded as the resource amount that can meet the concurrency requirement of the task queue. If the difference between the concurrent resource amounts in different time windows is large, it indicates that there will be a significant peak value of the concurrent resource amount during the running of the task queue, and the peak value can be determined as the reference resource amount. If the difference between the concurrent resource amounts in different time windows is small, it indicates that the concurrent resource amount is relatively average during the running of the task queue, and there is no significant peak value of the concurrent resource amount, and the average value of the concurrent resource amounts in multiple time windows can be determined as the reference resource amount. The smaller one of the reference resource amount and the safe resource threshold value can be selected as the initial predicted resource amount. The safe resource threshold value is a resource amount threshold value that can guarantee the safety of the resources of the cluster, and can be set according to the scene, demand, experience, etc., which is not limited herein. For example, the safe resource threshold value is 10% of the total amount of resources of the cluster. In some examples, if the difference between the concurrent resource amount of one time window and the concurrent resource amounts of other time windows is higher than a difference threshold value, the smaller one of the concurrent resource amount of one time window and the safe resource threshold value is determined as the initial predicted resource amount and outputted; if the difference between the concurrent resource amounts of multiple time windows is lower than or equal to the difference threshold value, the average value of the concurrent resource amounts of multiple time windows and the smaller one of the safe resource threshold value are determined as the initial predicted resource amount and outputted. The difference threshold value is a threshold value for judging whether there will be a significant peak value of the concurrent resource amount during the running of the task queue, and can be set according to the scene, demand, experience, etc., which is not limited herein. If the difference between the concurrent resource amount of one time window and the concurrent resource amounts of other time windows is higher than the difference threshold value, it is considered that there will be a significant peak value of the concurrent resource amount during the running of the task queue, and the concurrent resource amount of this time window can be determined as the reference resource amount in the above. If the difference between the concurrent resource amounts of multiple time windows is lower than or equal to the difference threshold value, it is considered that the concurrent resource amount is relatively average during the running of the task queue, and there is no significant peak value of the concurrent resource amount, and the average value of the concurrent resource amounts of multiple time windows is determined as the reference resource amount in the above. For example, the concurrent resource amount of each time window can include the CPU concurrent resource amount and the memory concurrent resource amount, and the CPU concurrent resource amount and the memory concurrent resource amount can be calculated according to the following formulas (1) and (2): CPU 总 =∑(E 1j ×CPU 1j )+∑(E 2j× CPU 2j )+…+∑(E ij × CPU ij )+…(1) MEM 总 =∑(E 1j × MEM 1j )+∑(E 2j × MEM 2j )+…+∑(E ij × MEM ij )+ …(2) wherein, CPU 总 is the CPU concurrent resource amount in the time window; MEM 总 is the memory concurrent resource amount in the time window; E ij is the number of executors of the jth task in the ith type of task in the time window; CPU ij is the number of CPU cores allocated to each executor of the jth task in the ith type of task; MEM ij is the memory size allocated to each executor of the jth task in the ith type of task. For example, if the task queue contains three types of tasks, the first type of task includes one task, the number of executors of the task is 50, the number of CPU cores allocated to each executor of the task is 4C, and the memory size allocated to each executor of the task is 16G; the second type of task includes one task, the number of executors of the task is 30, the number of CPU cores allocated to each executor of the task is 2C, and the memory size allocated to each executor of the task is 8G; the third type of task includes one task, the number of executors of the task is 20, the number of CPU cores allocated to each executor of the task is 6C, and the memory size allocated to each executor of the task is 20G, then according to the above formulas (1) and (2), the CPU concurrent resource amount of the task queue in the time window is calculated to be 380C, and the memory concurrent resource amount is calculated to be 1440G.

[0033] In some embodiments, the historical duration corresponding to the historical resource usage information is less than or equal to a preset duration, and correspondingly, the obtained event sequence resource prediction model includes a short-period machine learning model. The short-period machine learning model can include but is not limited to an XGBoost model, etc. The resource derived feature information can be calculated based on the task demand information, and the resource derived feature information includes one or more of a resource change trend feature, a resource time sequence feature, and a resource interdependence effect feature, wherein, in the case where the resource derived feature information includes the resource time sequence feature, the resource time sequence feature is periodically encoded to obtain a resource time sequence feature with cyclicity; the task demand information and the resource derived feature information are processed by the short-period machine learning model to output an initial predicted resource amount of the task queue in a future time period.

[0034] The historical resource usage information of the task queue in the historical time length can be obtained by a yet another resource negotiator (YARN) resource manager. The historical resource usage information of the task queue in the historical time length can also be obtained by JMX or a configuration file, such as the number of task concurrences, the task execution time, and the number of task waits. The historical resource usage information can also be obtained by parsing a Hive job log, such as the task type, the amount of data processed by the task, the task complexity, and the time characteristics. The task type can indicate that the task is an ETL task, a report task, or an Ad-hoc task. The task complexity can be represented by, but is not limited to, the number of JOINs, the number of GROUP BYs, and the number of window functions. The time characteristics can represent the characteristics of the business peak period in time, such as a few hours of the day, the beginning of the month, or holidays. The core features can be extracted from the obtained task demand information, such as one or more of the resource usage rate, the resource allocation amount, the number of task waits, and the task delay. The resource usage rate can include the CPU usage rate and the memory usage rate. The resource allocation amount can include the CPU allocation amount and the memory allocation amount. The task demand information is directly obtainable information, and the resource-derived feature information is information derived from the task demand information and capable of representing the resource usage characteristics of the task queue. The resource change trend feature can represent the change trend of the resource usage of the task queue, such as the average value of the resource usage rate in a certain time length, the average value of the resource allocation value in a certain time length, the standard deviation of the resource usage rate in a certain time length, the standard deviation of the resource allocation value in a certain time length, and the like. The time length for calculating the average value and the standard deviation can be, but is not limited to, 5 minutes, 1 hour, 1 day, and the like, and can be set according to the scene, the demand, and the like. The resource time sequence feature can represent the periodic time sequence characteristics of the resource usage of the task queue, such as the collection time of the resource usage rate and the resource allocation value, which is the hour of the day, the day of the week, whether it is a holiday, and the like. The resource time sequence feature has periodicity. In order to reflect the periodicity of the resource time sequence feature, the resource time sequence feature can be periodically encoded, so that the encoded resource time sequence feature has cyclicity. For example, taking the collection time of the resource usage rate and the resource allocation value as the hour as an example, the periodic encoding can be as shown in the following formulas (3) and (4): hour_sin=sin(2×π×hour / 24) (3) hour_cos=cos(2×π×hour / 24) (4) Wherein, the hour_sin and the hour_cos can constitute a cyclic resource time sequence feature, which can reflect the cycle of 24 hours in a day; the sin() and the cos() are trigonometric functions; the hour is the collection time of the resource usage and the resource allocation value. The collection time of weeks and months can also be periodically coded in a similar manner as the hours, which will not be repeated here.

[0035] The resource synergy effect feature can represent the synergy relationship between different resources. For example, the resources include CPU resources and memory resources. In the case of executing a certain task in the task queue, the CPU resource usage and the memory resource usage have a certain proportional relationship, or the CPU resource usage and the memory resource usage have a positive correlation relationship. Here, only examples are given, and the resource synergy effect feature is not limited thereto.

[0036] It should be noted that if the collection time of the task demand information is not synchronized, the collection time of the data in the task demand information can be aligned to a uniform interval, for example, it can be set to 5 minutes as a data collection point, so as to process the task demand information. If there is data loss, data anomaly, etc. after the collection time of the task demand information is aligned, the missing data can be filled by interpolation method, and the abnormal data can be replaced by smoothing processing.

[0037] For example, after a series of processing, the data input into the short-period machine learning model can be as shown in Table 1: Table 1

[0038] The data in Table 1 is only an example, but is not limited thereto. Among them, the hour_sin and the hour_cos are cyclic resource time sequence features, which can be specifically referred to in the above, and will not be repeated here.

[0039] The task demand information and the resource derived feature information can be input into the short-period machine learning model. The short-period machine learning model uses the relationship between the task demand information, the resource derived feature information and the resource quantity learned by itself to analyze the task demand information and the resource derived feature information, and then outputs the initial predicted resource quantity of the task queue in the future time period. For example, the short-period machine learning model can include an XGBoost model. The XGBoost model has strong performance, is very suitable for regression problems of tabular data, can automatically process feature interaction and nonlinear relationship, is not sensitive to missing values of data, has higher precision, and can output an initial predicted resource quantity with higher accuracy through the XGBoost model.

[0040] It should be noted that the short-period machine learning model used to predict the initial predicted resource quantity is a pre-trained short-period machine learning model. When training the short-period machine learning model, the sample training set can be divided in chronological order, and the sample data of the first 80% of time in the sample training set is used for training, and the sample data of the last 20% of time in the sample training set is used for testing. The mean absolute error (MAE) or mean squared error (MSE) can be used as a loss function to measure the difference between the predicted value and the true value, and the short-period machine learning model is trained.

[0041] In some embodiments, the historical duration corresponding to the historical resource usage information is greater than the preset duration, and correspondingly, the obtained time series resource prediction model includes a long-period deep learning model. The long-period deep learning model can include but is not limited to a Transformer model and the like. Traditional long-period deep learning models are generally used for natural language processing (NLP) and are designed for discrete sequences. For example, the input of the Transformer model is a word vector (i.e., embedding), the position information of the words in natural language uses sinusoidal positional encoding, and the attention mechanism captures the dependency between words. However, the task demand information processed by the long-period deep learning model in the embodiments of the present application is a time series, and the data type of the task demand information is a continuous numerical value, which is different from the discrete symbols processed in natural language processing. In the embodiments of the present application, the sampling of the task demand information is generally equidistant sampling or timestamped sampling, which is also different from the irregular sampling in natural language processing. In the embodiments of the present application, the task demand information input into the long-period deep learning model can include data of hundreds to thousands or even more time points, which is also different from the sentence-level input length of the long-period deep learning model in natural language processing, which includes tens of words or tokens. Moreover, the target of the embodiments of the present application can include outputting multi-dimensional values in the future time period, i.e., outputting initial predicted resource quantities at multiple time points in the future time period, which is also different from the target of predicting the next word by the long-period deep learning model in natural language processing. If the traditional long-period deep learning model is directly used, it can cause problems such as high dimension, time series noise, overfitting, and low efficiency. In view of the above differences and problems, in order to adapt the long-period deep learning model to the use scenario of the embodiments of the present application, the long-period deep learning model needs to be improved.

[0042] The long-period deep learning model in the embodiments of the present application can be improved from one or more of the following aspects: input layer, position coding, model structure, local attention, multivariate processing, model training, and model output. The improved long-period deep learning model also changes the processing process of task demand information. Specifically, the long-period deep learning model can map the task demand information at multiple time points to a multi-dimensional feature vector using a linear projection layer; the long-period deep learning model encodes the time sequence according to the task demand information to obtain a time position code, and adds the time position code to the multi-dimensional feature vector; the encoder in the long-period deep learning model processes the multi-dimensional feature vector added with the time position code, and outputs a context-aware feature vector, which can represent the dependency relationship between the task demand information at multiple time points; the long-period deep learning model aggregates the context-aware feature vector, and outputs an initial predicted resource amount through the prediction layer of the long-period deep learning model.

[0043] The input layer of the long-period deep learning model can be improved, so that the input layer of the long-period deep learning model does not perform word embedding on the input data, but uses the existing projection layer to map the task demand information of the continuous numerical type into a multi-dimensional feature vector, for example, the task demand information at each time point is mapped into a d-dimensional feature vector, so that the long-period deep learning model can process continuous numerical type input. In order to reflect the time sequence and periodicity of the task demand information, the position encoding of the long-period deep learning model can be improved, and a time-aware position encoding (i.e., Time-aware PosEnc) or a learnable position encoding can be used to obtain a time position encoding that can reflect the time sequence of the task demand information, and the time position encoding is added to the multi-dimensional feature vector to obtain an updated multi-dimensional feature vector. Since the data type of the task demand information is continuous numerical value, some structures used for natural language processing in the long-period deep learning model can be simplified or removed, that is, a lighter long-period deep learning model can be used. The number of layers, the number of heads, and the parameter quantity of the long-period deep learning model can be reduced. The traditional long-period deep learning model can include an encoder and a decoder, and the long-period deep learning model in the embodiment of the present application can remove the decoder and retain the encoder. For example, the long-period deep learning model in the embodiment of the present application includes a target Transformer model, which is a Transformer model that removes the decoder and retains the encoder. Through the lightweight long-period deep learning model, the amount of calculation can be reduced and the risk of overfitting can be reduced. The updated multi-dimensional feature vector can be processed by the encoder in the long-period deep learning model to perform multi-dimensional feature vector parallel embedding and local self-attention mechanism processing, so as to capture the complex dependency relationship between multiple time points in the updated multi-dimensional feature vector, and output a context-aware feature vector. The context-aware feature vector can contain global information of the multi-dimensional feature vector. Specifically, the multi-dimensional feature vector with time position encoding can be processed by the encoder in the long-period deep learning model to perform parallel embedding (i.e., Variable Embedding), to obtain an embedded feature vector; the embedded feature vector is processed by the encoder in the long-period deep learning model using a local attention mechanism to output a context-aware feature vector. A unique identity vector can be learned for each data in the task demand information, and the identity vector can be combined with the corresponding data (such as CPU usage, CPU allocation value, memory usage, memory allocation value, etc., but not limited thereto), so as to capture the dependency relationship between multiple data and form an embedded feature vector with richer information. The local attention mechanism can limit each time point to only focus on the data in the nearest neighbor time window, thereby improving the processing efficiency and avoiding the adverse effects of global noise. The information of the context-aware feature vector can be aggregated into a single and comprehensive representation vector through feature aggregation, and the processing of the feature aggregation can be realized as mean pooling (i.e., Mean Pool), but is not limited thereto.The representation vector obtained after the feature aggregation is input into a prediction layer. The prediction layer can be implemented as, but is not limited to, a fully connected regression layer (FC regression layer). The representation vector is mapped into the final initial predicted resource amount by the prediction layer.

[0044] The input data of the long-period deep learning model in the embodiments of the present application can include task semantic features, data volume features, task execution features, and time features. The task semantic features can include, but are not limited to, SQL statements, table names, JOIN numbers, aggregation function numbers, and User-Defined Function (UDF) numbers. The data volume features can include, but are not limited to, input table data volumes, partition numbers, and data compression rates. The task execution features can include, but are not limited to, CPU usage rates, memory usage rates, task execution times, and resource application types. The time features can include, but are not limited to, data collection times and time period patterns. The task semantic features, the data volume features, the task execution features, and the time features can all be obtained from the task requirement information. For example, the table names can be obtained from the task processing data tables in the task requirement information, the JOIN numbers, the aggregation function numbers, and the UDF numbers can be obtained from the task complexity in the task requirement information, the input table data volumes, the partition numbers, and the data compression rates can be obtained from the task processing data volumes in the task requirement information, the CPU usage rates and the memory usage rates can be obtained from the resource usage rates in the task requirement information, and the data collection times and the time period patterns can be obtained from the time features in the task requirement information. The resource estimation problem is defined as a multivariate time series prediction problem by the long-period deep learning model. For example, the input data of the long-period deep learning model can be X i =[X t-T , X t-T+1 , …, X t-1 ], X t-T is the historical resource usage information at time t-T, X t-T+1 is the historical resource usage information at time t-T+1, X t-1 is the historical resource usage information at time t-1, X t-T , X t-T+1 , …, X t-1 are all vectors; and the long-period deep learning model outputs the initial predicted resource amounts at K time points in a future time period, which can be represented as Y i =[Y t , Y t+1 , …, Y t+K-1 ], wherein Y t is the initial predicted resource amount at time point t in the future time period, Y t+1 is the initial predicted resource amount at time point t+1 in the future time period, Y t+K-1 is the initial predicted resource amount at time point t+K-1 in the future time period, and Y tY t+1 ..., Y t+K-1 Each can include CPU resources and memory resources.

[0045] For example, Figure 2 A flowchart illustrating an example of the data processing flow for a long-cycle deep learning model provided in this application embodiment is shown below. Figure 2 As shown, the data processing flow of this long-cycle deep learning model may include steps a1 to a7.

[0046] In step a1, an input sequence based on task requirement information is received.

[0047] In step a2, the input sequence is linearly embedded to obtain a multidimensional feature vector. Here, linear embedding refers to mapping the task requirement information into a multidimensional feature vector using a linear projection layer, as described in the above embodiment.

[0048] In step a3, positional encoding is added to the multidimensional feature vector. Here, the positional encoding is the time-positional encoding described in the previous embodiment.

[0049] In step a4, the encoder processes the multidimensional feature vector with added positional encoding. Specifically, it can perform parallel embedding, local attention mechanisms, etc., as described in the above embodiments.

[0050] In step a5, feature aggregation is performed on the context-aware features output by the encoder.

[0051] In step a6, the representation vector obtained after feature aggregation is input into the prediction layer for processing.

[0052] In step a7, the initial predicted resource amount is output. The initial predicted resource amount includes the initial predicted CPU resource amount and the initial predicted memory resource amount.

[0053] The specific details of steps a1 to a7 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0054] In the training process of long-cycle deep learning models, layer normalization (LayerNorm) is applied to the sample feature vectors, and dropout is used after each sub-layer of the model. A smooth loss function (Smooth Loss) is employed. LayerNorm normalizes the sample features along the feature dimension, ensuring that the data distribution remains relatively stable as it flows through each layer of the encoder, thus accelerating convergence. Dropout randomly "drops" a portion of neurons in the network during model training via forward propagation, preventing the long-cycle deep learning model from becoming overly reliant on noise and specific patterns in the training data. This allows the model to learn more robust and generalizable features. Smooth Loss avoids overfitting and model overconfidence, acting as a regularization mechanism and making the model training process smoother and more stable. LayerNorm, Dropout, and Smooth Loss techniques can stabilize the model training process, preventing issues such as oscillations and overfitting in time series forecasting. This allows the model to learn the impact of any past time point on the current prediction. For example, a long-term deep learning model can learn the importance of a CPU usage peak at the same time a week ago for the current prediction.

[0055] To facilitate understanding, the selection process of the time series resource prediction model for the expansion queue application type in the above embodiments is summarized and explained here. Figure 3 The flowchart illustrates an example of selecting a time-series resource prediction model in a scenario involving expansion queue requests, as provided in this application embodiment. Figure 3 As shown, the selection process for the time series resource prediction model in this expansion queue application scenario may include steps b1 to b5.

[0056] In step b1, historical resource usage information is obtained from the task requirement information.

[0057] In step b2, it is determined whether the period is short based on historical resource usage information. If it is short, proceed to step b3; otherwise, proceed to step b4. The shortness of the period can be determined by whether the historical duration corresponding to the historical resource usage information is less than or equal to a preset duration. If the historical duration corresponding to the historical resource usage information is less than or equal to the preset duration, it is determined to be a short period; if the historical duration corresponding to the historical resource usage information is greater than the preset duration, it is determined to be a long period.

[0058] In step b3, the XGBoost model is used for processing. That is, the XGBoost model is determined as the time series resource prediction model.

[0059] In step b4, the Transformer model is used for processing. That is, the Transformer model is determined as a time series resource prediction model.

[0060] In step b5, the initial predicted resource quantity output by the time series resource prediction model is obtained.

[0061] The specific details of steps b1 to b5 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0062] In some embodiments, step S103 can be further refined as follows: based on the initial predicted resource quantity and preset constraints, a resource utilization model is built with the goal of maximizing the utilization rate of resources in the cluster; the resource utilization model is solved to obtain the predicted resource quantity of the task queue in the future time period.

[0063] To further optimize the predicted resource quantity, the overall resource situation of the cluster, the current resource allocation, the resource usage of the task queue, and the cluster's remaining resources can be considered. A resource utilization model can be built using a resource evaluation optimization method that aims to maximize cluster resource utilization. The factors required to build the resource utilization model include decision variables, an objective function, and constraints. The decision variables can be the initial predicted resource quantity in the above embodiment. The objective function can be set based on the summation and maximum value calculation of the utility functions of the task queues; for example, the objective function can be set as Max∑f(C i ), where f() is the utility function of the task queue. To balance efficiency and fairness and ensure that all tasks in the queue can obtain resources, the utility function can be designed as a logarithmic function, f(C i )=log(C i Constraints can be set according to the actual situation of the cluster and task queues. For example, constraints may include total resource constraints, upper and lower bound constraints for each task queue, non-negativity constraints for each task queue, and upper bound constraints for each task queue; the total resource constraint can be ∑C i ≤C total C i C represents the predicted resource quantity of the i-th task queue, or the initial predicted resource quantity of the i-th task queue. total This represents the total amount of resources available to the cluster; the upper and lower bounds for each task queue can be C. min_i ≤C i ≤C max_i C min_i C represents the lower limit of the resource quantity for the i-th task queue. max_i This represents the maximum resource limit for the i-th task queue; the non-negativity constraint for each task queue can be 0 ≤ C. i The upper limit constraint for each task queue can be C. i≤10%C total The objective function can be expressed as follows (5): (5) The parameters are explained above and will not be repeated here. The objective function is defined in variable C. i Subtracting the lower limit of resource quantity and adding 1 ensures that the logarithmic function is meaningful within its domain, optimizing the fairness of resources exceeding the guaranteed amount. In this embodiment, the objective function of the resource utilization model is a concave function, and the constraints are linear. Therefore, the resource utilization model corresponds to a convex optimization problem, which can be quickly solved using the Lagrange multiplier method to obtain the global optimal solution. The global optimal solution is the optimized predicted resource quantity, specifically the predicted resource quantity for each task queue. The predicted resource quantity may include the predicted CPU resource quantity and the predicted memory resource quantity.

[0064] For example, the total resources of the cluster are about 30,000C, and there are 84 task queues. Each task queue is configured with an upper limit and a lower limit of resources. According to the operation rules of the tasks in the task queues and the elastic allocation strategy of resources, the upper limit of the allocable resources is twice the actual resources. At the same time, it is also necessary to consider the operation requirements and performance impact of the cluster components. The allocable resources generally do not exceed 80% of the total resources, that is, the total allocable resources are about 48,000C. According to actual needs, it is also necessary to reserve the needs that were reported last year but have not yet been applied for. The "strategy engine" mode can be adopted: the YARN resource manager can periodically (e.g., every minute) provide the strategy engine with the current resource utilization rate of each queue, the number of tasks to be processed, etc. The strategy engine can build a resource utilization model based on the current status, such as the total resources, task queue configuration, constraints, etc., and can return the optimal resource allocation scheme according to the above formula (5), that is, obtain the predicted resource amount of the task queue in the future time period.

[0065] In some embodiments, if the complexity of the task indicated by the task requirement information is higher than a complexity threshold, a time-series resource prediction model corresponding to the resource request type of the task queue can be obtained; if the complexity of the task indicated by the task requirement information is lower than or equal to the complexity threshold, enhanced retrieval generation (RAG) can be performed in a preset knowledge base based on the task requirement information to obtain the resource allocation example in the knowledge base that has the highest matching degree with the task requirement information; based on the resource allocation example, the initial predicted resource amount of the task queue in a future time period can be predicted.

[0066] The complexity threshold is used to determine the level of complexity and can be set according to the scenario, requirements, experience, etc., and is not limited here. If the complexity of the task indicated by the task requirement information is higher than the complexity threshold, the tasks in the task queue are considered to have high complexity, and the task requirement information needs to be processed by a time series resource prediction model to obtain the initial predicted resource amount. If the complexity of the task indicated by the task requirement information is lower than or equal to the complexity threshold, the tasks in the task queue are considered to have low complexity, and the initial predicted resource amount can be obtained using large language model technology. For details on obtaining the initial predicted resource amount through the time series resource prediction model, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0067] Using large language model technology to obtain initial predicted resource quantities, users can input task requirement information into the intelligent assistant application through interaction. This information, formed as guide words, can include task requirement information and example templates. The task requirement information in the guide words can include the task description information described in the above embodiments. For example, in a scenario for performing anomaly security protection tasks, the guide words could be: "To further implement the important directive to accelerate the research and output of anomaly security protection capabilities to banks, it is planned to include 17 national banks in the anomaly identification intelligent model calculation scope in the near future. To ensure the stability of system operation, a separate task queue is requested specifically for running the anomaly intelligent identification model. This scenario requires the use of a large data table, approximately 800GB, primarily using HiveSQL tasks, with approximately 40 computational tasks and a concurrency of no more than 5. The expectation is to complete the computation within 8 hours. What size queue resource is required?" The task requirement information in the guide words can be obtained through multiple rounds of interaction with the user. If the user's input is incomplete, prompts and multiple rounds of question-and-answer dialogue can guide the user to supplement the task requirement information. For example, prompts such as "Is the task type HiveSQL, Spark, Flink, or others? Please confirm?" or "Is the task a regular ETL data cleaning task, or does it involve multi-table joins, complex aggregation operations, sorting, machine learning training, multiple shuffles, window calculations, etc.? Please confirm?" can guide users to supplement task requirement information. The task requirement information can be parsed to obtain key dimension information, generating a multi-dimensional vector. This multi-dimensional vector is then used for enhanced retrieval in the knowledge base. The resource allocation example with the highest matching degree with the multi-dimensional vector in the knowledge base is the resource allocation example with the highest matching degree with the task requirement information. The knowledge base may include, but is not limited to, resource application operation manuals and resource application history. The resource allocation example includes the resource allocation quantity, and the initial predicted resource quantity can be obtained based on the resource allocation quantity in the resource allocation example. For example, the resource allocation quantity in the resource allocation example can be determined as the initial predicted resource quantity, or the initial predicted resource quantity can be determined based on the relationship between the task requirement information in the resource allocation example and the current task requirement information, as well as the resource allocation quantity in the resource allocation example. In some examples, resource usage information during the stable operation of the task queue can also be obtained. The task requirement information and the resource usage information during the stable operation of the task queue can be stored in a knowledge base, which can provide a reference for the subsequent prediction of the initial predicted resource amount.

[0068] For example, Figure 4 A flowchart illustrating an example of obtaining an initial predicted resource quantity using large language modeling techniques, as provided in this application embodiment, is shown below. Figure 4 As shown, the process of obtaining the initial predicted resource quantity using large language model technology may include steps c1 to c5.

[0069] In step c1, task requirement information is input into the intelligent assistant application through multiple rounds of interactive question and answer.

[0070] In step c2, the knowledge base is retrieved. The knowledge base may include knowledge of resource request operation manuals and resource request history.

[0071] In step c3, an enhanced retrieval is performed in the knowledge base based on the task requirement information to generate a resource allocation example.

[0072] In step c4, the initial predicted resource quantity is inferred based on the resource allocation example and task requirement information.

[0073] In step c5, the initial predicted resource quantity is output.

[0074] The specific details of steps c1 to c5 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0075] In some embodiments, since the execution of tasks in the task queue changes periodically and dynamically, to avoid resource idleness, resources can be allocated to the task queue according to the predicted resource amount for a future time period, and then dynamic scheduling can be performed during the task queue's operation. The resource utilization rate of the task queue can be monitored; if the resource utilization rate of the task queue exceeds the upper limit of the scheduling utilization rate, the amount of resources allocated to the task queue is gradually increased based on a dynamic adjustment algorithm. The dynamic adjustment algorithm includes a dynamic trend strategy nonlinear adjustment algorithm, or an adjustment algorithm combining the Autocorrelation Function (ACF) and the Long Short-Term Memory Network (LSTM); if the resource utilization rate of the task queue exceeds the lower limit of the scheduling utilization rate, the amount of resources allocated to the task queue is gradually reclaimed.

[0076] YARN can monitor the resource usage of the task queue in real time, specifically monitoring CPU and memory resource utilization. It enables real-time dynamic adjustment of task queue resources at the minute-level. The upper and lower limits of scheduling utilization can be set based on scenarios, requirements, and experience, and are not limited here. For example, the upper limit could be set to 80%, and the lower limit to 50%. When the resource utilization of the task queue exceeds the upper limit, the resource amount in the task queue is increased based on a dynamic adjustment algorithm gradient. This dynamic adjustment algorithm can be a combination of short-term and long-term adjustments. For short-term adjustments, such as sudden changes in resource usage, a dynamic trend-following strategy nonlinear adjustment algorithm can be used for short-term prediction to adjust the resource allocation of the task queue. For long-term adjustments, an algorithm combining autocorrelation function and Long Short-Term Memory (LSTM) networks can be used to predict the resource utilization of the next time window based on YARN's real-time monitoring of the task queue's resource utilization, thereby adjusting the amount of resources allocated to the task queue. When the resource utilization of the task queue exceeds the lower limit, resources are gradually reclaimed from the task queue. In the above embodiments, the period for dynamically adjusting the amount of task queue resources can be customized. For example, it can be set to be dynamically adjusted every 15 minutes.

[0077] To facilitate understanding, the following will be combined with... Figure 5 This document describes the resource allocation process for the task queue in the embodiments of this application. Figure 5 This is a schematic diagram illustrating an example of the resource allocation process for a task queue provided in an embodiment of this application, as shown below. Figure 5 As shown, the resource allocation process for this task queue may include steps d1 to d7.

[0078] In step d1, the task requirement information of the task queue is obtained. The task requirement information of the task queue reflects the user's resource needs.

[0079] In step d2, if the task complexity in the task queue is low, the initial predicted resource amount is obtained using the smart assistant application.

[0080] In step d3, if the task complexity of the task queue is high, then it is determined whether the resource request type is a new queue request or an expanded queue request. The determination of task complexity can be found in the relevant descriptions in the above embodiments, and will not be repeated here. If it is a new queue request, proceed to step d4; if it is an expanded queue request, proceed to step d5.

[0081] In step d4, the initial predicted resource quantity is obtained using the Bottom-Up model.

[0082] In step d5, an initial predicted resource quantity is obtained using a dual-model architecture. This dual-model architecture may include an XGBoost model and a Transformer model. Specific details regarding the use of the XGBoost and Transformer models can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0083] In step d6, the initial predicted resource quantity is optimized using the resource evaluation optimal solution method to obtain the predicted resource quantity. The predicted resource quantity is used for resource allocation of the task queue in the future time period.

[0084] In step d7, dynamic scheduling of resources is performed during the operation of the task queue.

[0085] The specific details of steps d1 to d7 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0086] Optimal resource allocation can be achieved through iterative processes involving three stages: resource prediction, evaluation and optimization, and dynamic scheduling. This maximizes the utilization of existing cluster resources and improves the overall resource utilization rate of the cluster. Experiments have shown that the task queue resource allocation method provided in this application can obtain more accurate predicted resource quantities, improving cluster resource utilization by approximately 18%. For example, Table 2 below shows a comparison between the predicted resource quantities obtained by the task queue resource allocation method according to this application and those obtained through traditional manual experience. Compared to traditional manual experience methods, the task queue resource allocation method of this application can improve cluster resource utilization by 18%.

[0087] Table 2

[0088] In Table 2, the predicted resource amount "100C / 200G" for queue B1 corresponding to human experience indicates that the predicted CPU resource amount for human experience is 100C and the predicted memory resource amount for human experience is 200G. The predicted resource amount "75C / 240G" for queue B1 corresponding to the embodiment of this application indicates that the predicted CPU resource amount for the embodiment of this application is 75C and the predicted memory resource amount for the embodiment of this application is 240G. The predicted resource amounts for other task queues are not described here.

[0089] This application also provides a resource allocation device for a task queue. Figure 6 A schematic diagram of a resource allocation device for a task queue provided in an embodiment of this application is shown below. Figure 6 As shown, the resource allocation device 200 of the task queue may include a model determination module 201, a prediction module 202, an optimization module 203, and an execution module 204.

[0090] Model determination module 201 can be used to obtain the time series resource prediction model corresponding to the resource request type of the task queue; The prediction module 202 can be used to make predictions based on the task demand information of the task queue according to the acquired time series resource prediction model, and output the initial predicted resource quantity of the task queue in the future time period. The task demand information includes task description information, or the task demand information includes task description information and historical resource usage information. The optimization module 203 can be used to optimize the initial predicted resource amount with the goal of maximizing the utilization of resources in the cluster, so as to obtain the predicted resource amount of the task queue in the future time period. The execution module 204 can be used to allocate resources to the task queue in the future time period according to the predicted resource quantity.

[0091] In some embodiments, when the resource request type is a new queue request type, the obtained time-series resource prediction model includes a baseline prediction model. When the resource request type is an expansion queue request type, the obtained time-series resource prediction model includes a short-cycle machine learning model or a long-cycle deep learning model.

[0092] In some embodiments, the acquired time-series resource prediction model includes a baseline prediction model. The prediction module 202 can be used to: parse task description information using the baseline prediction model to obtain the tasks contained in the task queue, and obtain executor runtime resource information for each task; obtain the concurrent resource quantity for multiple time windows within the task timeline based on the executor runtime resource information of each task using the baseline prediction model; and determine and output the initial predicted resource quantity of the task queue in future time periods based on the concurrent resource quantity of multiple time windows and a safety resource threshold using the baseline prediction model.

[0093] In some examples, the prediction module 202 can be used to: if the difference between the concurrent resource quantity of a time window and the concurrent resource quantity of other time windows is higher than the difference threshold, determine the smaller of the concurrent resource quantity of a time window and the safety resource threshold as the initial predicted resource quantity and output it; if the difference between the concurrent resource quantities of multiple time windows is lower than or equal to the difference threshold, determine the smaller of the average of the concurrent resource quantities of multiple time windows and the safety resource threshold as the initial predicted resource quantity and output it.

[0094] In some embodiments, when the historical duration corresponding to the historical resource usage information is less than or equal to a preset duration, the acquired time-series resource prediction model includes a short-cycle machine learning model. The prediction module 202 can be used to: calculate resource-derived feature information based on task requirement information, where the resource-derived feature information includes one or more of resource change trend features, resource time-series features, and inter-resource synergy features; wherein, when the resource-derived feature information includes resource time-series features, the resource time-series features are periodically encoded to obtain cyclical resource time-series features; and process the task requirement information and resource-derived feature information through a short-cycle machine learning model to output the initial predicted resource quantity of the task queue in the future time period.

[0095] In some embodiments, when the historical duration corresponding to the historical resource usage information is longer than a preset duration, the acquired time-series resource prediction model includes a long-term deep learning model. The prediction module 202 can be used to: map task demand information at multiple time points into multi-dimensional feature vectors using a linear projection layer via the long-term deep learning model; encode the task demand information according to time sequence using the long-term deep learning model to obtain time position codes, and add the time position codes to the multi-dimensional feature vectors; process the multi-dimensional feature vectors with added time position codes using the encoder in the long-term deep learning model to output context-aware feature vectors, which can represent the dependencies between task demand information at multiple time points; perform feature aggregation on the context-aware feature vectors using the long-term deep learning model; and output the initial predicted resource quantity through the prediction layer of the long-term deep learning model.

[0096] In some examples, long-cycle deep learning models include target Transformer models, which are Transformer models with the decoder removed and the encoder retained.

[0097] In some examples, the prediction module 202 can be specifically used to: embed multidimensional feature vectors with added temporal location encoding in parallel through the encoder in the long-period deep learning model to obtain embedded feature vectors; and process the embedded feature vectors using a local attention mechanism through the encoder in the long-period deep learning model to output context-aware feature vectors.

[0098] In some embodiments, the optimization module 203 may be specifically used to: build a resource utilization model based on the initial predicted resource amount and preset constraints, with the goal of maximizing the utilization rate of resources in the cluster; solve the resource utilization model to obtain the predicted resource amount of the task queue in the future time period.

[0099] In some embodiments, the model determination module 201 may be specifically used to: when the complexity of the task indicated by the task requirement information is higher than a complexity threshold, obtain a time series resource prediction model corresponding to the resource request type of the task queue.

[0100] The prediction module 202 can also be used to: when the complexity of the task indicated by the task requirement information is lower than or equal to the complexity threshold, perform enhanced retrieval in a preset knowledge base based on the task requirement information to generate a resource allocation example in the knowledge base that matches the task requirement information the most; and predict the initial predicted resource amount of the task queue in the future time period based on the resource allocation example.

[0101] In some embodiments, the resource allocation device 200 for the task queue may further include a dynamic adjustment module. The dynamic adjustment module can be used to: monitor the resource utilization rate of the task queue; when the resource utilization rate of the task queue exceeds the upper limit of the scheduling utilization rate, gradually increase the amount of resources allocated to the task queue based on a dynamic adjustment algorithm, which may include a dynamic trend strategy nonlinear adjustment algorithm, or an adjustment algorithm combining an autocorrelation function and a long short-term memory network; and when the resource utilization rate of the task queue exceeds the lower limit of the scheduling utilization rate, gradually reclaim the amount of resources allocated to the task queue.

[0102] It should be noted that the resource allocation device 200 for the task queue is a device corresponding to the resource allocation method for the task queue described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0103] This application also provides a resource allocation device for a task queue. Figure 7 This is a schematic diagram of the structure of a resource allocation device for a task queue provided in an embodiment of this application, as shown below. Figure 7 As shown, the resource allocation device 300 for the task queue includes a memory 301, a processor 302, and a computer program stored on the memory 301 and executable on the processor 302.

[0104] In some examples, the processor 302 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0105] Memory 301 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the resource allocation method for task queues according to embodiments of this application.

[0106] The processor 302 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 301, so as to implement the resource allocation method of the task queue in the above embodiment.

[0107] In some examples, the resource allocation device 300 for the task queue may also include a communication interface 303 and a bus 304. For example, Figure 7 As shown, the memory 301, processor 302, and communication interface 303 are connected through bus 304 and complete communication with each other.

[0108] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 303.

[0109] Bus 304 includes hardware, software, or both, that couples components of the task queue resource allocation device 300 together. For example, and not limitingly, bus 304 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 304 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0110] This application also provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are executed by a processor, they can implement the resource allocation method for the task queue in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0111] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the resource allocation method of the task queue in the above embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0112] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0113] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0114] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A resource allocation method for a task queue, characterized in that, include: Obtain the time series resource prediction model corresponding to the resource request type of the task queue; Based on the task demand information of the task queue, the obtained time series resource prediction model makes a prediction and outputs the initial predicted resource quantity of the task queue in the future time period. The task demand information includes task description information, or the task demand information includes the task description information and historical resource usage information. With the goal of maximizing the utilization of resources in the cluster, the initial predicted resource amount is optimized to obtain the predicted resource amount of the task queue in the future time period; Allocate resources to the task queue in the future time period according to the predicted resource quantity.

2. The method according to claim 1, characterized in that, When the resource request type is a new queue request type, the obtained time series resource prediction model includes the baseline prediction model; When the resource request type is an expansion queue request type, the obtained time series resource prediction model includes a short-cycle machine learning model or a long-cycle deep learning model.

3. The method according to claim 1, characterized in that, The acquired time-series resource prediction models include the baseline prediction model; The step of predicting the initial predicted resource quantity of the task queue based on the task demand information of the task queue using the acquired time-series resource prediction model, and outputting the initial predicted resource quantity of the task queue in a future time period, includes: The task description information is parsed using the baseline prediction model to obtain the tasks contained in the task queue, and the executor runtime resource information of each task is obtained. The baseline prediction model is used to obtain the concurrent resource quantity of multiple time windows within the task timeline based on the executor running resource information of each task. The baseline prediction model determines and outputs the initial predicted resource quantity of the task queue in the future time period based on the concurrent resource quantity of multiple time windows and the safety resource threshold.

4. The method according to claim 3, characterized in that, The step of determining and outputting the initial predicted resource quantity of the task queue in a future time period based on the concurrent resource quantity of multiple time windows and the safety resource threshold includes: If the difference between the concurrent resource quantity of a time window and the concurrent resource quantity of other time windows is higher than the difference threshold, the smaller of the concurrent resource quantity of the time window and the safe resource threshold is determined as the initial predicted resource quantity and output. If the difference between the concurrent resource amounts of multiple time windows is less than or equal to the difference threshold, the smaller of the average concurrent resource amounts of multiple time windows and the safe resource threshold is determined as the initial predicted resource amount and output.

5. The method according to claim 1, characterized in that, When the historical duration corresponding to the historical resource usage information is less than or equal to a preset duration, the obtained time series resource prediction model includes a short-cycle machine learning model. The step of predicting the initial predicted resource quantity of the task queue based on the task demand information of the task queue using the acquired time-series resource prediction model, and outputting the initial predicted resource quantity of the task queue in a future time period, includes: Based on the task requirement information, resource-derived feature information is calculated. The resource-derived feature information includes one or more of the following: resource change trend features, resource time series features, and inter-resource synergy effect features. In the case where the resource-derived feature information includes resource time series features, the resource time series features are periodically encoded to obtain cyclical resource time series features. The task requirement information and the resource-derived feature information are processed by a short-cycle machine learning model to output the initial predicted resource quantity of the task queue in the future time period.

6. The method according to claim 1, characterized in that, When the historical duration corresponding to the historical resource usage information is greater than the preset duration, the obtained time series resource prediction model includes a long-term deep learning model. The step of predicting the initial predicted resource quantity of the task queue based on the task demand information of the task queue using the acquired time-series resource prediction model, and outputting the initial predicted resource quantity of the task queue in a future time period, includes: The task requirement information at multiple time points is mapped into a multi-dimensional feature vector using a linear projection layer through a long-cycle deep learning model. The time sequence encoding is performed based on the task requirement information by a long-cycle deep learning model to obtain the time position encoding, and the time position encoding is added to the multi-dimensional feature vector. The encoder in the long-cycle deep learning model processes the multidimensional feature vector with added time position encoding to output a context-aware feature vector, which can represent the dependency relationship between the task requirement information at multiple time points. The context-aware feature vector is aggregated using a long-cycle deep learning model, and the initial predicted resource quantity is output through the prediction layer of the long-cycle deep learning model.

7. The method according to claim 6, characterized in that, Long-term deep learning models include target Transformer models, which are Transformer models that remove the decoder and retain the encoder.

8. The method according to claim 6, characterized in that, The process of processing the multidimensional feature vector with added temporal position encoding through the encoder in the long-period deep learning model to output a context-aware feature vector includes: The multidimensional feature vector with added temporal position encoding is embedded in parallel by the encoder in the long-period deep learning model to obtain the embedded feature vector. The encoder in a long-cycle deep learning model uses a local attention mechanism to process the embedded feature vector and output the context-aware feature vector.

9. The method according to claim 8, characterized in that, During the training of long-cycle deep learning models, the sample feature vectors are normalized layer by layer, and random discarding is performed after the sub-layers of the long-cycle deep learning model. The loss function is a smooth loss function.

10. The method according to claim 1, characterized in that, The optimization of the initial predicted resource amount, with the goal of maximizing resource utilization in the cluster, to obtain the predicted resource amount of the task queue in the future time period includes: Based on the initial predicted resource quantity and preset constraints, a resource utilization model is built with the goal of maximizing the utilization rate of resources in the cluster. Solving the resource utilization model yields the predicted resource quantity for the task queue in the future time period.

11. The method according to claim 1, characterized in that, The time-series resource prediction model corresponding to the resource request type of the task queue includes: If the complexity of the task indicated by the task requirement information is higher than the complexity threshold, obtain the time series resource prediction model corresponding to the resource request type of the task queue. The method further includes: If the complexity of the task indicated by the task requirement information is lower than or equal to the complexity threshold, an enhanced retrieval is performed in a preset knowledge base based on the task requirement information to generate a resource allocation example in the knowledge base that has the highest matching degree with the task requirement information. Based on the resource allocation example, predict the initial predicted resource amount of the task queue in a future time period.

12. The method according to claim 1, characterized in that, Also includes: Monitor the resource utilization of the task queue; When the resource utilization rate of the task queue is greater than the upper limit of the scheduling utilization rate, the amount of resources allocated to the task queue is gradually increased based on a dynamic adjustment algorithm. The dynamic adjustment algorithm includes a dynamic trend strategy nonlinear adjustment algorithm, or an adjustment algorithm combining autocorrelation function and long short-term memory network. If the resource utilization rate of the task queue is greater than the lower limit of the scheduling utilization rate, the resources allocated to the task queue will be gradually reclaimed.

13. A resource allocation device for a task queue, characterized in that, include: The model determination module is used to obtain the time series resource prediction model corresponding to the resource request type of the task queue; The prediction module is used to make predictions based on the task demand information of the task queue according to the acquired time series resource prediction model, and output the initial predicted resource quantity of the task queue in the future time period. The task demand information includes task description information, or the task demand information includes the task description information and historical resource usage information. The optimization module is used to optimize the initial predicted resource amount with the goal of maximizing the utilization of resources in the cluster, so as to obtain the predicted resource amount of the task queue in the future time period. An execution module is used to allocate resources to the task queue in a future time period according to the predicted resource quantity.

14. A resource allocation device for a task queue, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the resource allocation method for the task queue as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the resource allocation method for the task queue as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the resource allocation method for the task queue as described in any one of claims 1 to 12.

Images