Cluster resource allocation method and device, equipment and storage medium

Abstract

The invention discloses a cluster resource allocation method and device, equipment and a storage medium, and relates to the technical field of cluster data processing, and the method comprises the steps: obtaining monitoring resource information corresponding to each tenant in a target cluster; predicting the resource demand of each tenant according to the monitoring resource information to obtain a target resource demand; and allocating cluster resources corresponding to the target cluster to each tenant based on the target resource demand. According to the method and the device, the resource use condition of each tenant in the cluster is monitored, and the resource demand of the tenant in the cluster is predicted according to the monitored resource information obtained by monitoring, so that the resource required by each tenant, namely the target resource demand, is obtained; and then, the cluster resources of the target cluster are adaptively allocated to each tenant according to the target resource demand, so that on-demand allocation of the cluster resources is effectively realized, extra capacity expansion or capacity reduction operation is not needed, waste of the cluster resources is avoided, and the management and control cost of cluster resource allocation is reduced.

Description

Technical Field

[0001] The present application relates to the field of cluster data processing technology, and in particular to a cluster resource allocation method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of information technology, big data is playing an increasingly important role in key businesses across various industries. To effectively manage and control big data resources, distributed cluster systems consisting of multiple computers are now often used to store and process large amounts of data.

[0003] As business and data volumes grow, the demand for big data job resources increases. When cluster resources run low, job queues become clogged and data processing is delayed, severely impacting the normal operation of business systems. Cluster expansion is often used to address resource shortages, but the cost of expanding big data clusters is high. Furthermore, existing cluster resources are unevenly distributed, leading to wasted resources.

[0004] Therefore, how to reduce the management and control costs of cluster resource allocation is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a cluster resource allocation method, device, equipment and storage medium, aiming to solve the technical problem that existing biological product clinical trial data cannot be effectively summarized and classified.

[0006] To achieve the above objectives, the present application proposes a cluster resource allocation method, which includes:

[0007] Obtain monitoring resource information corresponding to each tenant in the target cluster;

[0008] Predicting resource requirements of each tenant based on the monitored resource information to obtain target resource requirements;

[0009] Allocate cluster resources corresponding to the target cluster to each of the tenants based on the target resource demand.

[0010] In one embodiment, the step of predicting the resource requirements of each tenant based on the monitored resource information to obtain target resource requirements includes:

[0011] Estimating a preset resource interval based on the monitored resource information to obtain estimated resource requirements;

[0012] The resource demand of each tenant is predicted based on the estimated resource demand to obtain a target resource demand.

[0013] In one embodiment, the step of predicting the resource demand of each tenant based on the estimated resource demand to obtain the target resource demand includes:

[0014] determining an initial demand forecasting model based on the estimated resource demand;

[0015] Optimizing the initial demand forecast model to obtain a target demand forecast model.

[0016] The resource demand of each tenant is predicted by the target demand prediction model to obtain the target resource demand.

[0017] In one embodiment, the step of performing preset model optimization on the initial demand forecast model to obtain a target demand forecast model includes:

[0018] Performing a preset demarcation process on the initial demand forecast model according to a preset resource critical value to obtain an optimized demand forecast model;

[0019] The optimized demand forecasting model is subjected to preset smoothing processing to obtain a target demand forecasting model.

[0020] In one embodiment, the step of allocating the cluster resources corresponding to the target cluster to the tenants based on the target resource requirements includes:

[0021] Obtain resource allocation weights corresponding to each tenant;

[0022] Allocate the cluster resources corresponding to the target cluster to the tenants according to the target resource requirements and the resource allocation weights.

[0023] In one embodiment, the step of allocating the cluster resources corresponding to the target cluster to the tenants according to the target resource requirements and the resource allocation weights includes:

[0024] Performing a numerical comparison between the target resource requirement and the total amount of cluster resources corresponding to the target cluster;

[0025] The cluster resources corresponding to the target cluster are allocated to the tenants according to the numerical comparison result, the target resource demand, and the resource allocation weight.

[0026] In one embodiment, after allocating the cluster resources corresponding to the target cluster to each tenant according to the numerical comparison result, the target resource demand, and the resource allocation weight, the method further includes:

[0027] Obtain resource usage values ​​corresponding to each tenant;

[0028] determining a current resource error based on the target resource demand and the resource usage value;

[0029] If it is detected that the current resource error does not meet the preset error index, the cluster resources are reallocated to the tenants based on a preset resource allocation strategy.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a cluster resource allocation device, which includes:

[0031] The information acquisition module is used to obtain the monitoring resource information corresponding to each tenant in the target cluster;

[0032] A demand prediction module, configured to predict the resource demand of each tenant based on the monitored resource information to obtain a target resource demand;

[0033] A resource allocation module is configured to allocate cluster resources corresponding to the target cluster to each tenant based on the target resource demand.

[0034] In addition, to achieve the above objectives, the present application also proposes a cluster resource allocation device, which includes: a memory, a processor, and a cluster resource allocation program stored in the memory and executable on the processor, wherein the cluster resource allocation program is configured to implement the steps of the cluster resource allocation method described above.

[0035] In addition, to achieve the above purpose, the present application also proposes a storage medium, which is a storage medium. The storage medium stores a cluster resource allocation program. When the cluster resource allocation program is executed by a processor, the steps of the cluster resource allocation method as described above are implemented.

[0036] The present application provides a cluster resource allocation method, apparatus, device and storage medium, the method comprising obtaining monitoring resource information corresponding to each tenant in a target cluster; predicting the resource requirements of each tenant based on the monitoring resource information to obtain target resource requirements; and allocating cluster resources corresponding to the target cluster to each tenant based on the target resource requirements. The present application monitors the resource usage of each tenant in the cluster, and predicts the resource requirements of the tenants in the cluster based on the monitoring resource information obtained, thereby obtaining the resources required by each tenant, i.e., the target resource requirements, and then adaptively allocates the cluster resources of the target cluster to each tenant based on the target resource requirements, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or reduction operations, thereby avoiding cluster resource waste and reducing the management and control costs of cluster resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a schematic diagram of the first flow chart of the first embodiment of the cluster resource allocation method of the present application;

[0040] Figure 2 This is a second flow chart of the first embodiment of the cluster resource allocation method of the present application;

[0041] Figure 3 This is a schematic diagram of the third flow chart of the first embodiment of the cluster resource allocation method of the present application;

[0042] Figure 4 This is a schematic diagram of the fourth flow chart of the first embodiment of the cluster resource allocation method of the present application;

[0043] Figure 5 This is a schematic diagram of the first flow chart of the second embodiment of the cluster resource allocation method of the present application;

[0044] Figure 6 This is a second flow chart of the second embodiment of the cluster resource allocation method of the present application;

[0045] Figure 7 This is a schematic diagram of the module structure of the cluster resource allocation device according to an embodiment of the present application;

[0046] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the cluster resource allocation method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of this application is: obtain the monitoring resource information corresponding to each tenant in the target cluster; predict the resource requirements of each tenant based on the monitoring resource information to obtain the target resource requirements; and allocate the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements.

[0051] Currently, cluster resource allocation methods have the following problems:

[0052] 1) High resource expansion costs. The cost of expanding big data cluster machines is high, but business development requires more cluster resources, creating a dilemma.

[0053] 2) Poor security for key business operations. During the early morning peak hours, a large number of jobs compete for resources, resulting in job queue congestion and increasingly poor job timeliness, impacting the development of key business operations.

[0054] 3) Cluster tenants are unevenly busy and idle, leading to resource waste. Existing cluster resources are fixed after being allocated to tenants. Even if there are no tasks executing in a queue, the allocated resources are still occupied. Idle resources cannot be provided to other busy queues, resulting in resource waste.

[0055] To solve the above problems, this application proposes a general automatic elastic scaling solution for big data resources, which realizes elastic scaling of cluster resources through full-process automation without resource expansion. Specifically, this application monitors the resource usage of each tenant in the cluster, and predicts the resource requirements of a single tenant in the cluster based on the monitoring resource information obtained by monitoring, obtains the target resource requirements corresponding to each tenant in the target cluster, and then adaptively allocates the cluster resources of the target cluster to each tenant based on the resources required by each tenant, that is, the target resource requirements, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or reduction operations, which not only avoids resource waste, but also reduces the management and control costs of cluster resource allocation.

[0056] It should be noted that the execution entity of this embodiment can be a cluster resource allocation system, a computing service device with data processing, network communication, program execution, and cluster resource control functions, such as a tablet computer, personal computer, mobile phone, etc., or a cluster resource allocation device capable of performing the above functions, etc. This embodiment does not specifically limit this. The following uses the cluster resource allocation device (referred to as the allocation device) as the execution entity to illustrate this embodiment and the following embodiments.

[0057] Based on this, the embodiment of the present application provides a cluster resource allocation method, referring to Figure 1 , Figure 1 This is a first flow chart of the first embodiment of the cluster resource allocation method of the present application.

[0058] In this embodiment, the cluster resource allocation method includes steps S10 to S30:

[0059] Step S10: Obtain monitoring resource information corresponding to each tenant in the target cluster;

[0060] It should be understood that the cluster resource allocation method proposed in this embodiment is applicable to the Hadoop big data platform, that is, the target cluster can be a Hadoop cluster. In this case, each tenant can share the resources of the target cluster, but each tenant has an independent job processing queue and resource quota to ensure fairness and isolation in resource use. The purpose of this embodiment is to allocate the cluster resources of the target cluster to each tenant on demand at a low cost.

[0061] It is easy to understand that the above-mentioned tenant can be a resource pool or resource set, which can exclusively use part of the resources in the target cluster, such as CPU, memory, storage, etc., and the above-mentioned monitoring resource information can be the usage data of CPU, memory, storage and other resources in the tenant, as well as resource indicators determined based on the usage data, so as to subsequently analyze the resource requirements of each tenant based on usage and resource indicators.

[0062] In a specific implementation, the allocation device can obtain relevant historical data on cluster resource allocation and tenant resource usage through the Yarn (Yet Another Resource Negotiator) REST API, then pre-process the usage data to obtain resource indicators and store them in the database. For example, this embodiment can use the tenant resource allocation upper limit to evaluate the tenant's resource utilization rate, and the corresponding usage can be shown as follows:

[0063]

[0064] It should be noted that the above-mentioned Memory and Vcores (Virtual Cores) are the two main indicators used to measure and allocate resources. Among them, when a task runs in a container, it requires a certain amount of memory to perform its operations. Memory is the physical memory allocated to the application container. Vcores are a way to represent CPU resources in YARN. They do not directly correspond to physical cores, but represent the share of CPU time that can be allocated to applications. For example, suppose there is a dual-core machine and 2 Vcores are specified for a task, then the task will have the right to use the full computing power of these two cores. In practice, this may mean that the task can run two threads at the same time, or a single thread can switch between the two cores to maximize CPU time.

[0065] Step S20: predicting the resource requirements of each tenant based on the monitored resource information to obtain target resource requirements;

[0066] It is understandable that once existing cluster resources are allocated to tenants (i.e., job queues), they remain fixed. Even if no tasks are executed in the queue, the allocated resource size will be fixedly occupied, and idle resources cannot be provided to other busy queues, resulting in resource waste. To avoid cluster resource waste, this embodiment can obtain monitoring resource information related to each tenant's resource usage, predict the current resource demand of each tenant based on the monitoring resource information, and dynamically allocate cluster resources in real time on demand based on the predicted resource usage of each tenant.

[0067] Preferably, this embodiment can model the resource demand forecast of a single tenant based on the monitoring resource information collected by resource monitoring, and then form a resource allocation model for the cluster by integrating the sub-models of all tenants, and call the API interface to realize the automatic elastic scaling capability of resource allocation.

[0068] In one possible embodiment, referring to Figure 2 , Figure 2 This is a second flow chart of the first embodiment of the cluster resource allocation method of the present application. In this embodiment, step S20 may include steps A1 to A2:

[0069] Step A1, estimating a preset resource interval based on the monitored resource information to obtain estimated resource requirements;

[0070] It's important to understand that the difficulty and key to predicting tenant resource demand lies in accurately predicting the location and peak value of peak demand. Because tenant resource demand varies significantly between peak and low-peak periods, and due to the sparse sampling of data, this example initially attempted to use prediction methods for time series data, such as LSTM, transformers, and ARIMA, but the results were unsatisfactory. Therefore, interval estimation, a statistical technique, was ultimately used to predict tenant resource usage demand.

[0071] Specifically, this embodiment can obtain n historical sample data X about resource r (such as the above-mentioned Memory or Vcore) for a single tenant q at each sampling time t based on the monitoring resource information. At this time, the process of estimating the preset resource interval can be:

[0072] When the sample X is large, it is assumed to follow a normal distribution; when the sample X is small, it is assumed to follow a T distribution with n-1 degrees of freedom. Then, an interval estimate is performed on X with a confidence level α and a known variance to obtain the confidence interval [xe, x+e] for a single tenant q with respect to resource r, which is the estimated resource demand mentioned above.

[0073] Step A2: predicting the resource demand of each tenant based on the estimated resource demand to obtain a target resource demand.

[0074] It is easy to understand that each tenant in the Hadoop YARN cluster can configure the minimum and maximum CPU and memory usage quotas. Based on this principle, this embodiment can combine the interval estimation results of each tenant on the resources to be used, that is, estimate the resource demand model to form a resource demand prediction model for each tenant.

[0075] In one possible implementation, refer to Figure 3 , Figure 3 This is a third flow chart of the first embodiment of the cluster resource allocation method of this application. In this embodiment, step A2 may include steps A21 to A23:

[0076] Step A21, determining an initial demand forecasting model based on the estimated resource demand;

[0077] It should be understood that the initial model for single-tenant resource demand prediction can be obtained based on the estimated resource demand [xe, x+e], that is, the above-mentioned initial demand prediction model, and is expressed as:

[0078] Amin=xe;

[0079] Amax=x+e;

[0080] Where Amin is the predicted minimum resource allocation value of the tenant, Amax is the predicted maximum resource allocation value of the tenant, x is the point estimate of the required resource, and e is the error of the point estimate of the required resource.

[0081] Step A22, performing preset model optimization on the initial demand forecast model to obtain a target demand forecast model;

[0082] Step A23: predict the resource demand of each tenant using the target demand prediction model to obtain target resource demand.

[0083] It's important to note that the initial demand forecasting model described above relies heavily on tenants' historical resource usage data and is not robust to new jobs or temporary resource usage. As historical resource usage data increases, the model error e gradually decreases, causing Amax to approach Amin, and in extreme cases, both to reach 0. Furthermore, when tenants demand additional resources, allocating resources based solely on historical usage will only allocate limited resources to them. This means that tenants won't be able to obtain the cluster resources they need for new tasks, which can easily lead to queue congestion.

[0084] In this regard, this embodiment can optimize the initial demand forecast model and perform resource demand forecasting based on the optimized target demand forecast model. Figure 4 , Figure 4This is the fourth process schematic diagram of the first embodiment of the cluster resource allocation method of the present application. In this embodiment, step A22 may include steps A221 to A222:

[0085] Step A221, perform a preset boundary processing on the initial demand prediction model according to a preset resource threshold to obtain an optimized demand prediction model;

[0086] It is easy to understand that in this embodiment, by introducing a resource guarantee mechanism, that is, setting a lower limit A of Amin for each tenant lower , to perform boundary processing on Amax and Amin. At the same time, a robustness coefficient K can be introduced to appropriately increase Amax, and set the minimum difference between Amax and Amin to d (0 < d < 1). Then, the optimized demand prediction model after the preset boundary processing can be expressed as follows:

[0087] Amin q,t = max(Amin q,t , A lower );

[0088] Amax q,t = max(K * Amax q,t , Amin q,t + d * C);

[0089] In the formula, Amin q,t is the minimum resource allocation value of the predicted tenant q at time t, Amax q,t is the maximum resource allocation value of the predicted tenant q at time t, and C is the total cluster resources of the target cluster.

[0090] Step A222, perform a preset smoothing processing on the optimized demand prediction model to obtain a target demand prediction model.

[0091] It should be understood that due to the uncertainty of big data jobs, jobs may be executed in advance or delayed due to various reasons such as environmental problems and upstream dependencies. Therefore, in this embodiment, the optimized demand prediction model can be further subjected to model smoothing processing.

[0092] In this regard, in this embodiment, a smoothing coefficient S (S ∈ N) can be introduced. For a single tenant q, for the required resource r, at each time t, take the maximum Amin and the maximum Amax within the adjacent consecutive S △t (monitoring resource information sampling interval) minutes as the processed Amin and Amax. Then, the target demand prediction model after the preset smoothing processing can be expressed as follows:

[0093]

[0094] It is easy to understand that the above target demand forecasting model relies on the resource usage information of each tenant in the cluster in the previous T-1 day. Therefore, at the end of each day, the data of each tenant in the target cluster on that day can be updated to the model to obtain a new target demand forecasting model, thereby realizing self-iterative update of the model.

[0095] Step S30: Allocate cluster resources corresponding to the target cluster to each tenant based on the target resource demand.

[0096] It is easy to understand that after predicting the tenant resource requirements currently required by each tenant in the target cluster based on the target demand prediction model, the allocation device can allocate the cluster resources corresponding to the target cluster to each tenant on demand, thereby realizing dynamic on-demand allocation of cluster resources of the target cluster.

[0097] In this embodiment, a cluster resource allocation method suitable for any multi-tenant big data cluster with Yarn as the resource scheduling framework is proposed. In this embodiment, the allocation device does not require additional resources for expansion, reduction and other operations. It can directly monitor the resource usage of each tenant in the cluster, and predict the resource requirements of a single tenant in the cluster based on the monitoring resource information obtained by monitoring, and then obtain the target resource requirements corresponding to each tenant in the target cluster. Then, according to the resources required by each tenant, that is, the target resource requirements, the cluster resources of the target cluster are adaptively allocated to each tenant, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or reduction operations. This not only avoids resource waste, but also reduces the management and control costs of cluster resource allocation.

[0098] In addition, this embodiment can automatically predict the real-time resource needs of each tenant through the pre-built target demand prediction model. lower , K, d, S, △t, etc.), the target demand prediction model can automatically evaluate the resource requirements of each tenant, and then allocate equipment to allocate cluster resources on demand based on the target resource requirements. The entire process is automated to improve the efficiency of cluster resource management and control.

[0099] This embodiment provides a cluster resource allocation method, comprising: obtaining monitoring resource information corresponding to each tenant in a target cluster; estimating a preset resource interval based on the monitoring resource information to obtain estimated resource demand; determining an initial demand forecasting model based on the estimated resource demand; performing a preset demarcation process on the initial demand forecasting model based on a preset resource threshold to obtain an optimized demand forecasting model; and performing a preset smoothing process on the optimized demand forecasting model to obtain a target demand forecasting model. The target demand forecasting model is used to predict the resource demand of each tenant to obtain a target resource demand; and cluster resources corresponding to the target cluster are allocated to each tenant based on the target resource demand. In this embodiment, the allocation device does not require additional resources for operations such as capacity expansion or contraction. Instead, it directly monitors the resource usage of each tenant in the cluster and, based on the monitored resource information, predicts the resource demand of each tenant in the cluster to obtain the target resource demand corresponding to each tenant in the target cluster. Cluster resources of the target cluster are then adaptively allocated to each tenant based on the resources required by each tenant, i.e., the target resource demand. This effectively implements on-demand allocation of cluster resources without the need for additional capacity expansion or contraction operations, thereby avoiding resource waste and reducing the management and control costs of cluster resource allocation. In addition, this embodiment can automatically predict the real-time resource requirements of each tenant through a pre-built target demand prediction model. After the model parameters of the target demand prediction model are configured, the target demand prediction model can automatically evaluate the resource requirements of each tenant, and then allocate equipment to allocate cluster resources on demand based on the target resource requirements. The entire process is automated, thereby improving the efficiency of cluster resource management and control.

[0100] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0101] It is easy to understand that the above-mentioned preset classification model needs to be iteratively trained before classifying the processed test data. Therefore, based on the first embodiment, please refer to Figure 5 , Figure 5 This is a first flow chart of the second embodiment of the cluster resource allocation method of this application. In this embodiment, step S30 includes steps B1 to B2:

[0102] Step B1, obtaining the resource allocation weight corresponding to each tenant;

[0103] Step B2: Allocate the cluster resources corresponding to the target cluster to each tenant according to the target resource demand and the resource allocation weight.

[0104] It's important to note that existing tenant resource allocation lacks guidance and evaluation methods. Adjustments to current tenant resource allocation strategies rely entirely on experience, lacking theoretical basis and guidance. Furthermore, after manually adjusting tenant resource allocation strategies, there's no quantitative evaluation method, and the only way to determine whether a strategy is optimal is to rely on intuition.

[0105] At the same time, due to the differences in importance between the businesses handled by each tenant, higher-priority businesses can be given a moderate amount of resources and receive more attention when resources are abnormal. Therefore, this embodiment supports users to pre-configure the resource allocation weights corresponding to each tenant. Then, when performing on-demand allocation of cluster resources, the target resource requirements and resource allocation weights can be combined. In this case, the greater the resource allocation weight corresponding to the tenant, the more resources are allocated. In other words, important tenants can receive special attention in cluster resource allocation by setting a high weight.

[0106] In a feasible implementation manner, in this embodiment, step B2 includes steps B21 to B22:

[0107] Step B21, performing a numerical comparison between the target resource requirement and the total amount of cluster resources corresponding to the target cluster;

[0108] Step B22: Allocate the cluster resources corresponding to the target cluster to each tenant according to the numerical comparison result, the target resource demand, and the resource allocation weight.

[0109] It should be understood that in the output of the above target demand forecasting model, Amin q , t To predict the minimum resource allocation value of tenant q at time t, Amin q , t To predict the maximum resource allocation value of tenant q at time t, when actually allocating resources, the maximum value of resources allocated to each tenant can be directly based on Amin q , t However, the minimum value of resources allocated to each tenant must ensure the normal operation of each tenant's functions. In this case, the sum of the minimum resource allocation values ​​of each tenant on the cluster should be guaranteed to be Cmin. t It should not be greater than the total cluster resource amount C of the target cluster. Therefore, it is necessary to perform a numerical comparison between the target resource demand and the total cluster resource amount corresponding to the target cluster, and allocate the cluster resources corresponding to the target cluster to each tenant based on the numerical comparison result, the target resource demand, and the resource allocation weight.

[0110] Specifically, if the resource amount of each tenant's minimum resource allocation value is determined based on the target resource demand:

[0111] ① If the total resources are less than the cluster's total resources, the remaining resources will be allocated to tenants according to their weights;

[0112] ② If it is equal to the total resources of the cluster, allocation is made according to the target resource demand predicted by the model;

[0113] ③ If it is greater than the total cluster resources, the difference in cluster resources will be allocated according to the reverse tenant weights. Based on the above ideas, the on-demand allocation strategy for cluster resources of the target cluster can be expressed as follows:

[0114]

[0115] Where Rmin q,t The minimum resource allocation value of tenant q at time t given by the allocation strategy, Rmax q,t is the maximum resource allocation value of tenant q at time t given by the allocation strategy, W q Assign weights to the resources corresponding to tenant q.

[0116] This implementation can support pre-setting tenant resource allocation weights to meet the needs of different tenant queues with different business scenario importances, thereby forming a cluster resource allocation strategy by integrating the target demand prediction model for predicting single tenant resource needs and the resource allocation weights corresponding to each tenant, thereby realizing dynamic on-demand allocation of big data resources.

[0117] Based on the first embodiment, please refer to Figure 6 , Figure 6 This is a second flow chart of the second embodiment of the cluster resource allocation method of the present application. In this embodiment, step B22 further includes steps C1 to C3:

[0118] Step C1, obtaining the resource usage value corresponding to each tenant;

[0119] Step C2, determining a current resource error according to the target resource demand and the resource usage value;

[0120] Step C3: If it is detected that the current resource error does not meet the preset error index, the cluster resources are reallocated to the tenants based on a preset resource allocation strategy.

[0121] It is understood that to ensure the reliability of dynamic resource allocation, this embodiment can identify resource allocation anomalies based on the resource usage of each tenant after dynamic resource allocation. However, for tenants, the negative impact of insufficient allocated resources is greater. Therefore, this embodiment can determine resource allocation anomalies based on the loss value of the model. When calculating the model error, the greater the resource allocation weight corresponding to the tenant, the greater the impact on the error.

[0122] Specifically, in this embodiment, the root mean square error (RMSE) can be used to evaluate the model error. Therefore, the calculation formula for the current resource error loss can be expressed as follows:

[0123]

[0124] In the formula, Real q,t is the resource usage value of tenant q at time t.

[0125] It is easy to understand that if the loss is detected to be greater than the given threshold L, the current resource error does not meet the preset error index, and the resource allocation at the current moment is considered unreasonable, which is identified as a model abnormality.

[0126] In addition, if there is a need to evaluate and control resource waste, similar abnormal judgment can be made on another indicator of the model, the resource waste value waste. The calculation formula of the resource waste value waste can be expressed as follows:

[0127]

[0128] It is understood that when a model anomaly is identified, the model's fallback mechanism is triggered. At this point, the allocation device can disable dynamic resource allocation and instead allocate resources according to a preset resource allocation policy. For example, the preset resource allocation policy can directly allocate cluster resources to each tenant based on their corresponding resource allocation weights.

[0129] In addition, after the model is withdrawn, the allocation device can continue to evaluate the calculated allocation model and perform model function testing based on the given model restart test parameter P. If the model loss is detected to be within the threshold range in multiple consecutive test cycles thereafter, the anomaly can be considered to have been eliminated, and dynamic resource allocation can be restarted based on the latest target demand forecast model.

[0130] This implementation also sets up an abnormality identification and withdrawal mechanism. By configuring the expected effect of resource allocation, after the target demand prediction model is enabled, it can automatically identify the current resource error corresponding to the resource allocation result in real time. If it is judged that the resource allocation does not meet expectations based on the current resource error, the target demand prediction model will be automatically withdrawn to protect tenant resources.

[0131] This embodiment discloses obtaining resource allocation weights corresponding to each tenant; performing a numerical comparison between the target resource demand and the total amount of cluster resources corresponding to the target cluster; and allocating the cluster resources corresponding to the target cluster to each tenant based on the numerical comparison result, the target resource demand, and the resource allocation weights. This implementation supports presetting tenant resource allocation weights to meet the need for different tenant queues with different business scenario importances. By integrating a target demand prediction model for predicting a single tenant's resource demand and the resource allocation weights corresponding to each tenant, a cluster resource allocation strategy is formed, thereby achieving dynamic, on-demand allocation of big data resources.

[0132] In addition, this embodiment obtains the resource usage value corresponding to each tenant; determines the current resource error based on the target resource demand and resource usage value; and if it is detected that the current resource error does not meet the preset error index, the cluster resources are reallocated to each tenant based on the preset resource allocation strategy. This embodiment also provides an exception identification and withdrawal mechanism. By configuring the expected effect of resource allocation, after the target demand forecast model is enabled, the current resource error corresponding to the resource allocation result is automatically identified in real time. If the resource allocation is judged to be not in line with expectations based on the current resource error, the target demand forecast model is automatically withdrawn to protect tenant resources.

[0133] This application also provides a cluster resource allocation device, please refer to Figure 7 , Figure 7 This is a schematic diagram of the module structure of the cluster resource allocation device according to an embodiment of the present application. In this embodiment, the cluster resource allocation device includes:

[0134] Information acquisition module 701, used to obtain monitoring resource information corresponding to each tenant in the target cluster;

[0135] The demand prediction module 702 is configured to predict the resource demand of each tenant based on the monitored resource information to obtain a target resource demand;

[0136] The resource allocation module 703 is configured to allocate the cluster resources corresponding to the target cluster to the tenants based on the target resource requirements.

[0137] As an implementable method, in this embodiment, the demand prediction module 702 is further configured to estimate a preset resource interval based on the monitored resource information to obtain an estimated resource demand;

[0138] The demand prediction module 702 is further configured to predict the resource demand of each tenant based on the estimated resource demand to obtain a target resource demand.

[0139] As an implementable method, in this embodiment, the demand forecasting module 702 is further configured to determine an initial demand forecasting model based on the estimated resource demand;

[0140] The demand forecasting module 702 is further configured to perform preset model optimization on the initial demand forecasting model to obtain a target demand forecasting model;

[0141] The demand prediction module 702 is further configured to predict the resource demand of each tenant using the target demand prediction model to obtain target resource demand.

[0142] As an implementable method, in this embodiment, the demand forecasting module 702 is further configured to perform a preset demarcation process on the initial demand forecasting model according to a preset resource critical value to obtain an optimized demand forecasting model;

[0143] The demand forecasting module 702 is further configured to perform preset smoothing processing on the optimized demand forecasting model to obtain a target demand forecasting model.

[0144] As an implementable method, in this embodiment, the resource allocation module 703 is further configured to obtain the resource allocation weight corresponding to each tenant;

[0145] The resource allocation module 703 is further configured to allocate the cluster resources corresponding to the target cluster to the tenants according to the target resource requirements and the resource allocation weights.

[0146] As an implementable method, in this embodiment, the resource allocation module 703 is further configured to perform a numerical comparison between the target resource requirement and the total amount of cluster resources corresponding to the target cluster;

[0147] The resource allocation module 703 is further configured to allocate the cluster resources corresponding to the target cluster to the tenants according to the numerical comparison result, the target resource demand, and the resource allocation weight.

[0148] As an implementable method, in this embodiment, the resource allocation module 703 is further configured to obtain the resource usage value corresponding to each tenant;

[0149] The resource allocation module 703 is further configured to determine a current resource error based on the target resource requirement and the resource usage value;

[0150] The resource allocation module 703 is further configured to reallocate the cluster resources to the tenants based on a preset resource allocation strategy if it is detected that the current resource error does not meet a preset error index.

[0151] The cluster resource allocation device provided in this application utilizes the cluster resource allocation method described in the aforementioned embodiments to address the technical issues surrounding cluster resource allocation. Compared to the prior art, the cluster resource allocation device provided in this application achieves the same beneficial effects as the cluster resource allocation method described in the aforementioned embodiments. Other technical features of the cluster resource allocation device are the same as those disclosed in the aforementioned embodiments and are not further detailed here.

[0152] The present application provides a cluster resource allocation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the cluster resource allocation method in the above-mentioned embodiment 1.

[0153] Reference below Figure 8, which shows a schematic diagram of the structure of a cluster resource allocation device suitable for implementing embodiments of the present application. The cluster resource allocation device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The cluster resource allocation device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0154] like Figure 8 As shown, the cluster resource allocation device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the cluster resource allocation device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the cluster resource allocation device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a cluster resource allocation device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0155] In particular, according to the embodiments disclosed herein, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed herein include a cluster resource allocation program product, which includes a cluster resource allocation program carried on a computer-readable medium, the cluster resource allocation program containing program code for executing the method shown in the flowchart. In such an embodiment, the cluster resource allocation program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the cluster resource allocation program is executed by processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed herein are performed.

[0156] The cluster resource allocation device provided in this application utilizes the cluster resource allocation method described in the aforementioned embodiment to address the technical issues surrounding cluster resource allocation. Compared to the prior art, the cluster resource allocation device provided in this application achieves the same beneficial effects as the cluster resource allocation method described in the aforementioned embodiment. Other technical features of the cluster resource allocation device are the same as those disclosed in the aforementioned embodiment and are not further detailed here.

[0157] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0159] The present application provides a storage medium having computer-readable program instructions (ie, a cluster resource allocation program) stored thereon, wherein the computer-readable program instructions are used to execute the cluster resource allocation method in the above-mentioned embodiment.

[0160] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0161] The above storage medium may be included in the cluster resource allocation device; or may exist independently without being assembled into the cluster resource allocation device.

[0162] The storage medium carries one or more programs. When the one or more programs are executed by the cluster resource allocation device, the cluster resource allocation device can perform cluster resource allocation.

[0163] The cluster resource allocation program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0164] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and cluster resource allocation program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the part of this module, program segment or code contains one or more executable instructions for realizing the prescribed logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0166] The readable storage medium provided in this application is a storage medium that stores computer-readable program instructions (i.e., a cluster resource allocation program) for executing the aforementioned cluster resource allocation method, thereby resolving the technical issues surrounding cluster resource allocation. Compared to the prior art, the beneficial effects of the storage medium provided in this application are similar to those of the cluster resource allocation method provided in the aforementioned embodiments and are not further elaborated here.

[0167] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A cluster resource allocation method, characterized in that: The method comprises: Obtain monitoring resource information corresponding to each tenant in the target cluster; Predicting resource requirements of each tenant according to the monitored resource information to obtain target resource requirements; The cluster resources corresponding to the target cluster are allocated to the tenants based on the target resource demand.

2. The cluster resource allocation method according to claim 1, characterized in that: The step of predicting the resource requirements of each tenant according to the monitored resource information to obtain the target resource requirements includes: Estimating a preset resource interval according to the monitored resource information to obtain an estimated resource demand; The resource demand of each tenant is predicted based on the estimated resource demand to obtain a target resource demand.

3. The cluster resource allocation method according to claim 2, characterized in that: The step of predicting the resource demand of each tenant based on the estimated resource demand to obtain the target resource demand includes: determining an initial demand forecasting model based on the estimated resource demand; Optimizing the initial demand forecasting model to obtain a target demand forecasting model; The resource demand of each tenant is predicted by the target demand prediction model to obtain the target resource demand.

4. The cluster resource allocation method according to claim 3, characterized in that: The step of optimizing the preset model of the initial demand forecasting model to obtain the target demand forecasting model includes: Performing preset demarcation processing on the initial demand forecasting model according to preset resource critical values ​​to obtain an optimized demand forecasting model; The optimized demand forecasting model is subjected to a preset smoothing process to obtain a target demand forecasting model.

5. The cluster resource allocation method according to claim 1, characterized in that: The step of allocating the cluster resources corresponding to the target cluster to each of the tenants based on the target resource demand includes: Obtain resource allocation weights corresponding to each tenant; The cluster resources corresponding to the target cluster are allocated to the tenants according to the target resource demand and the resource allocation weight.

6. The cluster resource allocation method according to claim 5, characterized in that: The step of allocating the cluster resources corresponding to the target cluster to each tenant according to the target resource demand and the resource allocation weight includes: Performing a numerical comparison between the target resource requirement and the total amount of cluster resources corresponding to the target cluster; The cluster resources corresponding to the target cluster are allocated to the tenants according to the numerical comparison result, the target resource demand and the resource allocation weight.

7. The cluster resource allocation method according to claim 6, characterized in that: After allocating the cluster resources corresponding to the target cluster to each tenant according to the numerical comparison result, the target resource demand and the resource allocation weight, the method further includes: Obtain resource usage values ​​corresponding to each tenant; Determining a current resource error according to the target resource requirement and the resource usage value; If it is detected that the current resource error does not meet the preset error index, the cluster resources are reallocated to the tenants based on a preset resource allocation strategy.

8. A cluster resource allocation device, characterized in that: The cluster resource allocation device comprises: The information acquisition module is used to obtain the monitoring resource information corresponding to each tenant in the target cluster; A demand prediction module, used to predict the resource demand of each tenant according to the monitored resource information to obtain a target resource demand; A resource allocation module is used to allocate cluster resources corresponding to the target cluster to each tenant based on the target resource demand.

9. A cluster resource allocation device, characterized in that: The device comprises: a memory, a processor, and a cluster resource allocation program stored in the memory and executable on the processor, wherein the cluster resource allocation program is configured to implement the steps of the cluster resource allocation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a cluster resource allocation program, and when the cluster resource allocation program is executed by the processor, the steps of the cluster resource allocation method according to any one of claims 1 to 7 are implemented.

Images