Cluster resource allocation method, device, equipment and storage medium

Abstract

This application discloses a cluster resource allocation method, apparatus, device, and storage medium. This application relates to the field of cluster data processing technology. The method includes: acquiring monitoring resource information corresponding to each tenant in a target cluster; predicting the resource needs of each tenant based on the monitoring resource information to obtain target resource needs; and allocating the cluster resources corresponding to the target cluster to each tenant based on the target resource needs. This application monitors the resource usage of each tenant in the cluster and predicts the resource needs of the tenants in the cluster based on the monitoring resource information obtained from the monitoring, thereby obtaining the resources required by each tenant, i.e., the target resource needs. Then, it adaptively allocates the cluster resources of the target cluster to each tenant based on the target resource needs, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or contraction operations, thus avoiding cluster resource waste and reducing the management cost of cluster resource allocation.

Description

Technical Field

[0001] This application relates to the field of cluster data processing technology, and in particular to a cluster resource allocation method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of information technology, big data is playing an increasingly important role in the critical business operations of various industries. In order to achieve effective management and control of big data resources, large-scale data is now often stored and processed based on distributed cluster systems composed of multiple computers.

[0003] As business volume and data volume increase, the demand for big data operation resources also rises. When resources are insufficient in the cluster, issues such as job queue congestion and data processing delays occur, severely impacting the normal operation of business systems. Currently, cluster expansion is commonly used to avoid resource shortages, but expanding big data cluster machines is costly. On the other hand, existing cluster resources are unevenly allocated, resulting in resource waste.

[0004] Therefore, how to reduce the management and control costs of cluster resource allocation is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a cluster resource allocation method, apparatus, equipment and storage medium, which aims to solve the technical problem that existing clinical trial data of biological products cannot be effectively summarized and classified.

[0006] To achieve the above objectives, this application proposes a cluster resource allocation method, which includes:

[0007] Obtain monitoring resource information for each tenant in the target cluster;

[0008] Based on the monitored resource information, the resource requirements of each tenant are predicted to obtain the target resource requirements;

[0009] Based on the target resource requirements, the cluster resources corresponding to the target cluster are allocated to each tenant.

[0010] In one embodiment, the step of predicting the resource needs of each tenant based on the monitored resource information to obtain the target resource needs includes:

[0011] Based on the monitored resource information, a preset resource range is estimated to obtain the estimated resource requirements;

[0012] Based on the estimated resource requirements, the resource requirements of each tenant are predicted to obtain the target resource requirements.

[0013] In one embodiment, the step of predicting the resource requirements of each tenant based on the estimated resource requirements to obtain the target resource requirements includes:

[0014] Based on the estimated resource demand, an initial demand forecasting model is determined;

[0015] The initial demand forecasting model is optimized using a preset model to obtain the target demand forecasting model;

[0016] The target resource requirements are obtained by predicting the resource requirements of each tenant using the target demand prediction model.

[0017] In one embodiment, the step of optimizing the initial demand forecasting model to obtain the target demand forecasting model includes:

[0018] The initial demand forecasting model is subjected to a preset boundary process based on a preset resource threshold to obtain an optimized demand forecasting model.

[0019] The optimized demand prediction model is subjected to a preset smoothing process to obtain the target demand prediction model.

[0020] In one embodiment, the step of allocating the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements includes:

[0021] Obtain the resource allocation weight corresponding to each tenant;

[0022] Based on the target resource requirements and the resource allocation weights, the cluster resources corresponding to the target cluster are allocated to each tenant.

[0023] In one embodiment, the step of allocating the cluster resources corresponding to the target cluster to each tenant according to the target resource requirements and the resource allocation weight includes:

[0024] A numerical comparison is made between the target resource requirement and the total cluster resources corresponding to the target cluster;

[0025] Based on the numerical comparison results, the target resource requirements, and the resource allocation weights, the cluster resources corresponding to the target cluster are allocated to each tenant.

[0026] In one embodiment, after allocating the cluster resources corresponding to the target cluster to each tenant based on the numerical comparison results, the target resource requirements, and the resource allocation weights, the method further includes:

[0027] Obtain the resource usage values ​​corresponding to each tenant;

[0028] The current resource error is determined based on the target resource requirement and the resource usage value.

[0029] If the current resource error is detected to be inconsistent with the preset error index, the cluster resources will be redistributed to each tenant based on the preset resource allocation strategy.

[0030] Furthermore, to achieve the above objectives, this application also proposes a cluster resource allocation device, which includes:

[0031] The information acquisition module is used to acquire monitoring resource information corresponding to each tenant in the target cluster;

[0032] The demand forecasting module is used to forecast the resource demand of each tenant based on the monitored resource information to obtain the target resource demand.

[0033] The resource allocation module is used to allocate the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements.

[0034] In addition, to achieve the above objectives, this application also proposes a cluster resource allocation device, which includes: a memory, a processor, and a cluster resource allocation program stored in the memory and executable on the processor, wherein the cluster resource allocation program is configured to implement the steps of the cluster resource allocation method described above.

[0035] In addition, to achieve the above objectives, this application also proposes a storage medium, which stores a cluster resource allocation program. When the cluster resource allocation program is executed by a processor, it implements the steps of the cluster resource allocation method described above.

[0036] This application provides a cluster resource allocation method, apparatus, device, and storage medium. The method includes acquiring monitoring resource information corresponding to each tenant in a target cluster; predicting the resource needs of each tenant based on the monitoring resource information to obtain target resource needs; and allocating the cluster resources corresponding to the target cluster to each tenant based on the target resource needs. This application monitors the resource usage of each tenant in the cluster and predicts the resource needs of the tenants in the cluster based on the monitoring resource information obtained from the monitoring, thereby obtaining the resources required by each tenant, i.e., the target resource needs. Then, the cluster resources of the target cluster are adaptively allocated to each tenant according to the target resource needs, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or contraction operations, thus avoiding cluster resource waste and reducing the management and control costs of cluster resource allocation. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a first flowchart illustrating the first embodiment of the cluster resource allocation method of this application;

[0040] Figure 2 This is a second flowchart illustrating the first embodiment of the cluster resource allocation method of this application;

[0041] Figure 3 This is a schematic diagram of the third process of the first embodiment of the cluster resource allocation method of this application;

[0042] Figure 4 This is a schematic diagram of the fourth process of the first embodiment of the cluster resource allocation method of this application;

[0043] Figure 5 This is a schematic diagram of the first process of the second embodiment of the cluster resource allocation method of this application;

[0044] Figure 6 This is a second flowchart illustrating the second embodiment of the cluster resource allocation method of this application;

[0045] Figure 7 This is a schematic diagram of the module structure of the cluster resource allocation device according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the cluster resource allocation method in this application embodiment.

[0047] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of this application is: to obtain the monitoring resource information corresponding to each tenant in the target cluster; to predict the resource demand of each tenant based on the monitoring resource information and obtain the target resource demand; and to allocate the cluster resources corresponding to the target cluster to each tenant based on the target resource demand.

[0051] Currently, the cluster resource allocation method has the following problems:

[0052] 1) High cost of resource expansion. The cost of expanding big data cluster machines is high, but due to business development, more cluster resources are needed, leading to a dilemma.

[0053] 2) Poor support for critical business operations. During the early morning peak hours, numerous jobs compete for resources, causing queue congestion and increasingly poor timeliness, thus impacting the operation of critical business.

[0054] 3) Uneven workload among cluster tenants leads to resource waste. This means that existing cluster resources are allocated to tenants in a fixed manner. Even if there are no tasks executing in the queue, the allocated resource size remains unchanged, preventing idle resources from being made available to other busy queues, resulting in resource waste.

[0055] To address the aforementioned issues, this application proposes a general automated elastic scaling solution for big data resources. This solution automates the entire process of elastic scaling of cluster resources without requiring resource expansion. Specifically, this application monitors the resource usage of each tenant in the cluster and predicts the resource needs of individual tenants based on the monitored resource information. This yields the target resource needs corresponding to each tenant in the target cluster. Then, based on the resources required by each tenant—the target resource needs—the cluster resources of the target cluster are adaptively allocated to each tenant. This effectively achieves on-demand allocation of cluster resources without requiring additional expansion or contraction operations. This not only avoids resource waste but also reduces the management costs of cluster resource allocation.

[0056] It should be noted that the executing entity in this embodiment can be a cluster resource allocation system, or a computing service device with data processing, network communication, program execution, and cluster resource regulation functions, such as a tablet computer, personal computer, or mobile phone, or a cluster resource allocation device capable of performing the above functions. This embodiment does not specifically limit the specific implementation. The following uses a cluster resource allocation device (hereinafter referred to as the allocation device) as the executing entity to describe this embodiment and the following embodiments.

[0057] Based on this, embodiments of this application provide a cluster resource allocation method, referring to... Figure 1 , Figure 1 This is a first flowchart illustrating the first embodiment of the cluster resource allocation method of this application.

[0058] In this embodiment, the cluster resource allocation method includes steps S10 to S30:

[0059] Step S10: Obtain the monitoring resource information corresponding to each tenant in the target cluster;

[0060] It should be understood that the cluster resource allocation method proposed in this embodiment is applicable to Hadoop big data platforms, that is, the target cluster can be a Hadoop cluster. In this case, each tenant can share the resources of the target cluster, but each tenant has an independent job processing queue and resource quota to ensure the fairness and isolation of resource use. The purpose of this embodiment is to allocate the cluster resources of the target cluster to each tenant on demand at low cost.

[0061] It is easy to understand that the aforementioned tenants can be a resource pool or resource set, which can exclusively use a portion of the resources in the target cluster, such as CPU, memory, and storage. The aforementioned monitoring resource information can be the usage data of resources such as CPU, memory, and storage in the tenants, as well as the resource indicators determined based on the usage data, so as to conduct subsequent analysis of the resource needs of each tenant based on the usage and resource indicators.

[0062] In practical implementation, the allocation device can obtain historical data related to cluster resource allocation and tenant resource usage through the Yarn (Yet Another Resource Negotiator) REST API. This data is then preprocessed to obtain resource metrics, which are then stored in the database. For example, this embodiment can use the tenant resource allocation cap to evaluate the tenant's resource utilization rate, and the corresponding usage scenario is as follows:

[0063]

[0064] It's important to note that Memory and Vcores (Virtual Cores) are two primary metrics used to measure and allocate resources. When a task runs within a container, it requires a certain amount of memory to perform its operations; Memory refers to the physical memory allocated to the application container. Vcores, on the other hand, are a way of representing CPU resources in YARN. They don't directly correspond to physical cores but rather represent the share of CPU time available to an application. For example, assuming a dual-core machine and assigning 2 Vcores to a task, that task will have access to the full computing power of both cores. In practice, this could mean the task can run two threads simultaneously, or a single thread can switch between two cores to maximize CPU time.

[0065] Step S20: Based on the monitored resource information, predict the resource requirements of each tenant to obtain the target resource requirements;

[0066] It is understandable that existing cluster resources, once allocated to tenants (i.e., job queues), remain fixed. Even if no tasks are executing in the queue, the allocated resource size will still be occupied, preventing idle resources from being made available to other busy queues, resulting in resource waste. To avoid cluster resource waste, this embodiment can, after obtaining monitoring resource information related to the resource usage of each tenant, predict the current resource needs of each tenant based on the monitoring resource information, and dynamically allocate cluster resources in real time according to the predicted resource usage of each tenant.

[0067] Preferably, this embodiment can model the resource demand prediction of a single tenant based on the monitoring resource information collected by resource monitoring, and then form a cluster resource allocation model by integrating the sub-models of all tenants, and call the API interface to realize the automatic elastic scaling capability of resource allocation.

[0068] In one feasible implementation, refer to Figure 2 , Figure 2 This is a second flowchart illustrating the first embodiment of the cluster resource allocation method of this application. In this embodiment, step S20 may include steps A1 to A2:

[0069] Step A1: Estimate the estimated resource requirements by performing a preset resource range estimation based on the monitored resource information;

[0070] It is important to understand that the difficulty and key point of tenant resource demand forecasting lies in accurately predicting the location and peak value of the peak. Because tenant resource demand varies significantly between peak and low periods, and due to the sparse sampling of data, this embodiment initially attempted to use forecasting methods for time-series data such as LSTM, transformer, and ARIMA, but the results were unsatisfactory. Therefore, interval estimation from statistics was ultimately chosen to forecast tenant resource usage demand.

[0071] Specifically, this embodiment can obtain n historical sample data X for a single tenant q at each sampling time t regarding resource r (such as Memory or Vcore mentioned above) based on monitoring resource information. In this case, the process of estimating the aforementioned preset resource interval can be:

[0072] When the sample X is large, it is assumed to follow a normal distribution; when the sample X is small, it is assumed to follow a T-distribution with n-1 degrees of freedom. Then, interval estimation with confidence level α and known variance is performed on X to obtain the confidence interval [xe, x+e] for a single tenant q with respect to resource r, which is the estimated resource demand mentioned above.

[0073] Step A2: Based on the estimated resource demand, predict the resource demand of each tenant to obtain the target resource demand.

[0074] It is easy to understand that each tenant in a Hadoop Yarn cluster can configure minimum and maximum CPU and memory usage limits. Based on this principle, this embodiment can combine the interval estimation results of each tenant regarding the resources they need to use, that is, estimate resource requirements and model the resource requirement prediction model for each tenant.

[0075] In one feasible implementation, refer to Figure 3 , Figure 3 This is a third flowchart illustrating the first embodiment of the cluster resource allocation method of this application. In this embodiment, step A2 may include steps A21 to A23:

[0076] Step A21: Determine the initial demand forecasting model based on the estimated resource demand;

[0077] It is important to understand that, based on the estimated resource demand [xe, x+e], an initial model for single-tenant resource demand forecasting can be obtained, namely the initial demand forecasting model mentioned above, which can be expressed as:

[0078] Amin = xe;

[0079] Amax = x + e;

[0080] In the formula, Amin is the predicted minimum resource allocation value for the tenant, Amax is the predicted maximum resource allocation value for the tenant, x is the estimated demand resource point, and e is the error of the demand resource point estimation.

[0081] Step A22: Optimize the initial demand forecasting model using a preset model to obtain the target demand forecasting model;

[0082] Step A23: Predict the resource requirements of each tenant using the target demand prediction model to obtain the target resource requirements.

[0083] It should be noted that the initial demand forecasting model described above heavily relies on tenants' historical resource usage data and is not robust to new jobs or temporary resource usage. As historical resource usage data increases, the model error e will gradually decrease, causing Amax to approach Amin, and in extreme cases, both may be zero. Furthermore, when tenants have additional resource usage needs, if resource allocation is based solely on historical usage, only limited resources will be allocated to tenants. Tenants may not be able to obtain the necessary cluster resources for new tasks, potentially leading to queue congestion.

[0084] To address this, this embodiment can optimize the initial demand forecasting model and perform resource demand forecasting based on the optimized target demand forecasting model. In one feasible implementation, refer to... Figure 4 , Figure 4This is the fourth process schematic diagram of the first embodiment of the cluster resource allocation method of this application. In this embodiment, step A22 may include steps A221 to A222:

[0085] Step A221, perform a preset boundary processing on the initial demand prediction model according to a preset resource threshold to obtain an optimized demand prediction model;

[0086] It is easy to understand that in this embodiment, by introducing a resource guarantee mechanism, that is, setting a lower limit A of Amin for each tenant lower , to perform boundary processing on Amax and Amin. At the same time, a robustness coefficient K can be introduced to moderately increase Amax, and the minimum difference between Amax and Amin is set to d (0 < d < 1). Then, the optimized demand prediction model after the preset boundary processing can be expressed as follows:

[0087] Amin q,t = max(Amin q,t , A lower );

[0088] Amax q,t = max(K * Amax q,t , Amin q,t + d * C);

[0089] In the formula, Amin q,t is the minimum resource allocation value of predicted tenant q at time t, Amax q,t is the maximum resource allocation value of predicted tenant q at time t, and C is the total cluster resources of the target cluster.

[0090] Step A222, perform a preset smoothing process on the optimized demand prediction model to obtain a target demand prediction model.

[0091] It should be understood that due to the uncertainty of big data jobs, jobs may be executed in advance or delayed due to various reasons such as environmental problems and upstream dependencies. Therefore, in this embodiment, the optimized demand prediction model can be further subjected to model smoothing processing.

[0092] In this regard, in this embodiment, a smoothing coefficient S (S ∈ N) can be introduced. For a single tenant q, for the required resource r, at each time t, take the maximum Amin and the maximum Amax within adjacent consecutive S △t (monitoring resource information sampling interval) minutes as the processed Amin and Amax. Then, the target demand prediction model after the preset smoothing process can be expressed as follows:

[0093]

[0094] It is easy to understand that the above target demand prediction model relies on the resource usage information of each tenant in the cluster on the previous T-1 day. Therefore, at the end of each day, the data of each tenant in the target cluster can be updated to the model to obtain a new target demand prediction model, thus realizing the self-iterative update of the model.

[0095] Step S30: Based on the target resource requirements, allocate the cluster resources corresponding to the target cluster to each tenant.

[0096] It is easy to understand that after predicting the current tenant resource requirements of each tenant in the target cluster based on the target demand prediction model, the allocation device can allocate the cluster resources corresponding to the target cluster to each tenant on demand, thereby realizing the dynamic on-demand allocation of the cluster resources of the target cluster.

[0097] This embodiment proposes a cluster resource allocation method suitable for any multi-tenant big data cluster using Yarn as its resource scheduling framework. In this embodiment, the allocation device does not require additional resources for expansion or contraction. It can directly monitor the resource usage of each tenant in the cluster and predict the resource demand of each individual tenant based on the monitored resource information. This yields the target resource demand corresponding to each tenant in the target cluster. Then, based on the resources required by each tenant, i.e., the target resource demand, the cluster resources of the target cluster are adaptively allocated to each tenant. This effectively achieves on-demand allocation of cluster resources without the need for additional expansion or contraction operations. This not only avoids resource waste but also reduces the management and control costs of cluster resource allocation.

[0098] Furthermore, this embodiment can automatically predict the real-time resource demand of each tenant through a pre-built target demand prediction model. Once the model parameters of the target demand prediction model (i.e., the aforementioned A) are configured... lower After considering parameters such as K, d, S, and Δt, the target demand prediction model can automatically assess the resource needs of each tenant and then allocate equipment to the cluster resources on demand based on the target resource needs. The entire process is automated, improving the efficiency of cluster resource management.

[0099] This embodiment provides a cluster resource allocation method, which includes: acquiring monitoring resource information corresponding to each tenant in the target cluster; estimating the estimated resource demand by performing a preset resource range estimation based on the monitoring resource information; determining an initial demand prediction model based on the estimated resource demand; performing preset boundary processing on the initial demand prediction model according to a preset resource threshold to obtain an optimized demand prediction model; and performing preset smoothing processing on the optimized demand prediction model to obtain a target demand prediction model. The target demand prediction model is used to predict the resource demand of each tenant to obtain the target resource demand; and the cluster resources corresponding to the target cluster are allocated to each tenant based on the target resource demand. In this embodiment, the allocation device does not require additional resources for expansion or contraction. It can directly monitor the resource usage of each tenant in the cluster and predict the resource demand of a single tenant in the cluster based on the monitoring resource information obtained from the monitoring, thereby obtaining the target resource demand corresponding to each tenant in the target cluster. Then, based on the resources required by each tenant, i.e., the target resource demand, the cluster resources of the target cluster are adaptively allocated to each tenant, thereby effectively realizing on-demand allocation of cluster resources without the need for additional expansion or contraction operations. This not only avoids resource waste but also reduces the management cost of cluster resource allocation. In addition, this embodiment can automatically predict the real-time resource demand of each tenant through a pre-built target demand prediction model. After the model parameters of the target demand prediction model are configured, the target demand prediction model can automatically evaluate the resource demand of each tenant, and then allocate the cluster resources on demand based on the target resource demand. The whole process is automated, which improves the efficiency of cluster resource management.

[0100] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0101] It is easy to understand that the above-mentioned preset classification model needs to be iteratively trained before classifying the processed experimental data. Therefore, based on the first embodiment, please refer to... Figure 5 , Figure 5 This is a first flowchart illustrating the second embodiment of the cluster resource allocation method of this application. In this embodiment, step S30 includes steps B1 to B2:

[0102] Step B1: Obtain the resource allocation weights corresponding to each tenant;

[0103] Step B2: Allocate the cluster resources corresponding to the target cluster to each tenant according to the target resource requirements and the resource allocation weight.

[0104] It should be noted that the current tenant resource allocation lacks guidance and evaluation methods. In other words, adjustments to the current tenant resource allocation strategy rely entirely on experience, without theoretical basis or guidance. Furthermore, after manually adjusting the tenant resource allocation strategy, there is a lack of quantitative evaluation methods; judgments are based solely on intuition to determine whether it is better.

[0105] Meanwhile, given the varying importance of the services handled by different tenants, higher-priority services can receive appropriate resource allocation and greater attention during resource anomalies. Therefore, this embodiment supports users pre-configuring resource allocation weights for each tenant. Then, when allocating cluster resources on demand, the target resource requirements and allocation weights can be combined. In this case, the higher the resource allocation weight of a tenant, the more resources it receives; that is, important tenants can receive priority in cluster resource allocation by setting a high weight.

[0106] In one feasible implementation, step B2 includes steps B21 to B22:

[0107] Step B21: Compare the target resource requirements with the total cluster resources corresponding to the target cluster.

[0108] Step B22: Based on the numerical comparison results, the target resource requirements, and the resource allocation weights, allocate the cluster resources corresponding to the target cluster to each tenant.

[0109] It is important to understand that in the results output by the above target demand prediction model, Amin q , t To predict the minimum resource allocation value for tenant q at time t, Amin q , t To predict the maximum resource allocation value for tenant q at time t, when the allocation actually takes place, the maximum resource allocation value to each tenant can be directly based on Amin. q , t This process can proceed, but the minimum resources allocated to each tenant must ensure that each tenant's functions operate normally. Therefore, the sum of the minimum resource allocation values ​​for each tenant in the cluster, Cmin, should be guaranteed. t The target resource requirement should not exceed the total cluster resource C of the target cluster. Therefore, it is necessary to compare the target resource requirement with the total cluster resource corresponding to the target cluster, and allocate the cluster resource corresponding to the target cluster to each tenant based on the comparison result, the target resource requirement and the resource allocation weight.

[0110] Specifically, if the minimum resource allocation values ​​for each tenant are determined based on the target resource requirements, then the total resource quantity is calculated by summing these values:

[0111] ①If the resource weight is less than the total cluster resources, the remaining resources will be allocated to tenants according to their weights.

[0112] ②If the total cluster resources are equal, then the resources will be allocated according to the target resource requirements predicted by the model;

[0113] ③ If the value is greater than the total cluster resources, the difference in cluster resources will be allocated according to the reverse tenant weight. Based on the above idea, the on-demand allocation strategy for the target cluster resources can be expressed as follows:

[0114]

[0115] In the formula, Rmin q,t Rmax is the minimum resource allocation value for tenant q at time t given by the allocation strategy. q,t The maximum resource allocation value for tenant q at time t is given by the allocation strategy, W. q Assign weights to the resources corresponding to tenant q.

[0116] This implementation method can support pre-setting tenant resource allocation weights to meet the needs of different tenant queues with different business scenario importance. By integrating the target demand prediction model for predicting single tenant resource demand and the resource allocation weights corresponding to each tenant, a cluster resource allocation strategy is formed, thereby realizing dynamic on-demand allocation of big data resources.

[0117] Based on the first embodiment, please refer to Figure 6 , Figure 6 This is a second flowchart illustrating the second embodiment of the cluster resource allocation method of this application. In this embodiment, steps C1 to C3 are included after step B22:

[0118] Step C1: Obtain the resource usage value corresponding to each tenant;

[0119] Step C2: Determine the current resource error based on the target resource requirement and the resource usage value;

[0120] Step C3: If the current resource error is detected to be inconsistent with the preset error index, the cluster resources are redistributed to each tenant based on the preset resource allocation strategy.

[0121] Understandably, to ensure the reliability of dynamic resource allocation, this embodiment can identify resource allocation anomalies based on the resource usage of each tenant after dynamic resource allocation. For tenants, insufficient resource allocation has a greater negative impact. Therefore, this embodiment can determine resource allocation anomalies based on the model's loss value. When calculating model error, the larger the resource allocation weight of a tenant, the greater its impact on the error.

[0122] Specifically, this embodiment can use the root mean square error (RMSE) to evaluate the model error. Therefore, the formula for calculating the current resource error loss can be expressed as follows:

[0123]

[0124] In the formula, Real q,t This represents the resource usage value of tenant q at time t.

[0125] It is easy to understand that if the detected loss is greater than the given threshold L, the current resource error does not meet the preset error index, and the resource allocation at the current moment is considered unreasonable and identified as a model anomaly.

[0126] In addition, if there is a need to assess and control resource waste, a similar anomaly detection can be performed on another indicator of the model, the resource waste value (waste). The formula for calculating the resource waste value (waste) can be expressed as follows:

[0127]

[0128] Understandably, when a model anomaly is detected, a model rollback mechanism will be triggered. At this time, the allocation device can disable dynamic resource allocation and instead allocate resources according to a preset resource allocation strategy. For example, this preset resource allocation strategy could be to directly allocate cluster resources to each tenant according to the resource allocation weight corresponding to each tenant.

[0129] In addition, after the model is rolled back, the allocation device can continue to evaluate the calculated allocation model and restart the model function test based on the given model test parameter P. If the model loss is detected to be within the threshold range in several consecutive detection cycles, the anomaly can be considered eliminated, and the dynamic allocation of resources can be restarted based on the latest target demand prediction model.

[0130] This implementation also includes an anomaly identification and rollback mechanism. By configuring the expected effect of resource allocation, after the target demand prediction model is enabled, the current resource error corresponding to the resource allocation result can be automatically identified in real time. If the resource allocation does not meet expectations based on the current resource error, the target demand prediction model can be automatically rolled back to protect tenant resources.

[0131] This embodiment discloses the acquisition of resource allocation weights for each tenant; numerical comparison of the target resource demand and the total cluster resources corresponding to the target cluster; and allocation of cluster resources corresponding to the target cluster to each tenant based on the numerical comparison results, the target resource demand, and the resource allocation weights. This embodiment supports pre-setting tenant resource allocation weights to meet the needs of different tenant queues with varying business scenario importance. By integrating the target demand prediction model for predicting single-tenant resource demand with the resource allocation weights corresponding to each tenant, a cluster resource allocation strategy is formed, thereby achieving dynamic on-demand allocation of big data resources.

[0132] Furthermore, this embodiment also obtains the resource usage values ​​corresponding to each tenant; determines the current resource error based on the target resource demand and resource usage values; if the current resource error is detected to be inconsistent with the preset error index, the cluster resources are reallocated to each tenant based on the preset resource allocation strategy. This embodiment also sets up an anomaly identification and rollback mechanism. By configuring the expected effect of resource allocation, after the target demand prediction model is enabled, the current resource error corresponding to the resource allocation result is automatically identified in real time. If it is determined based on the current resource error that the resource allocation does not meet expectations, the target demand prediction model is automatically rolled back to protect tenant resources.

[0133] This application also provides a cluster resource allocation device, please refer to... Figure 7 , Figure 7 This is a schematic diagram of the module structure of the cluster resource allocation device according to an embodiment of this application. In this embodiment, the cluster resource allocation device includes:

[0134] The information acquisition module 701 is used to acquire monitoring resource information corresponding to each tenant in the target cluster.

[0135] The demand forecasting module 702 is used to forecast the resource demand of each tenant based on the monitored resource information to obtain the target resource demand.

[0136] The resource allocation module 703 is used to allocate the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements.

[0137] As one possible implementation, in this embodiment, the demand prediction module 702 is further used to estimate a preset resource range based on the monitored resource information to obtain the estimated resource demand;

[0138] The demand forecasting module 702 is also used to forecast the resource demand of each tenant based on the estimated resource demand, and obtain the target resource demand.

[0139] As one possible implementation, in this embodiment, the demand forecasting module 702 is further used to determine an initial demand forecasting model based on the estimated resource demand;

[0140] The demand forecasting module 702 is also used to perform preset model optimization on the initial demand forecasting model to obtain a target demand forecasting model;

[0141] The demand forecasting module 702 is also used to forecast the resource demand of each tenant through the target demand forecasting model to obtain the target resource demand.

[0142] As one possible implementation, in this embodiment, the demand forecasting module 702 is further used to perform preset boundary processing on the initial demand forecasting model according to a preset resource threshold value to obtain an optimized demand forecasting model.

[0143] The demand forecasting module 702 is also used to perform a preset smoothing process on the optimized demand forecasting model to obtain the target demand forecasting model.

[0144] As one possible implementation, in this embodiment, the resource allocation module 703 is also used to obtain the resource allocation weight corresponding to each tenant;

[0145] The resource allocation module 703 is further configured to allocate the cluster resources corresponding to the target cluster to each tenant according to the target resource requirements and the resource allocation weight.

[0146] As one possible implementation, in this embodiment, the resource allocation module 703 is also used to perform a numerical comparison between the target resource requirement and the total amount of cluster resources corresponding to the target cluster;

[0147] The resource allocation module 703 is further configured to allocate the cluster resources corresponding to the target cluster to each tenant based on the numerical comparison results, the target resource requirements, and the resource allocation weight.

[0148] As one possible implementation, in this embodiment, the resource allocation module 703 is also used to obtain the resource usage value corresponding to each tenant;

[0149] The resource allocation module 703 is further configured to determine the current resource error based on the target resource requirement and the resource usage value;

[0150] The resource allocation module 703 is further configured to reallocate the cluster resources to each tenant based on a preset resource allocation strategy if the current resource error is detected to be inconsistent with the preset error index.

[0151] The cluster resource allocation device provided in this application, employing the cluster resource allocation method in the above embodiments, can solve the technical problem of cluster resource allocation. Compared with the prior art, the beneficial effects of the cluster resource allocation device provided in this application are the same as those of the cluster resource allocation method provided in the above embodiments, and other technical features in the cluster resource allocation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0152] This application provides a cluster resource allocation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the cluster resource allocation method in Embodiment 1 above.

[0153] The following is for reference. Figure 8This document illustrates a structural diagram of a cluster resource allocation device suitable for implementing embodiments of this application. The cluster resource allocation device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The cluster resource allocation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0154] like Figure 8 As shown, the cluster resource allocation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the cluster resource allocation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the cluster resource allocation device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows cluster resource allocation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0155] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a cluster resource allocation program product, which includes a cluster resource allocation program carried on a computer-readable medium, the cluster resource allocation program containing program code for performing the methods shown in the flowcharts. In such embodiments, the cluster resource allocation program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the cluster resource allocation program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0156] The cluster resource allocation device provided in this application, employing the cluster resource allocation method in the above embodiments, can solve the technical problem of cluster resource allocation. Compared with the prior art, the beneficial effects of the cluster resource allocation device provided in this application are the same as those of the cluster resource allocation method provided in the above embodiments, and other technical features in this cluster resource allocation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0159] This application provides a storage medium having computer-readable program instructions (i.e., a cluster resource allocation program) stored thereon, which are used to execute the cluster resource allocation method in the above embodiments.

[0160] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0161] The aforementioned storage medium may be included in the cluster resource allocation device; or it may exist independently and not be assembled into the cluster resource allocation device.

[0162] The aforementioned storage medium carries one or more programs. When the aforementioned one or more programs are executed by the cluster resource allocation device, the cluster resource allocation device becomes: cluster resource allocation.

[0163] Cluster resource allocation program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and cluster resource allocation program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0166] The readable storage medium provided in this application is a storage medium that stores computer-readable program instructions (i.e., a cluster resource allocation program) for executing the above-described cluster resource allocation method, and is capable of solving the technical problem of cluster resource allocation. Compared with the prior art, the beneficial effects of the storage medium provided in this application are the same as the beneficial effects of the cluster resource allocation method provided in the above embodiments, and will not be repeated here.

[0167] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A cluster resource allocation method, characterized in that, The method includes: Obtain monitoring resource information for each tenant in the target cluster; Based on the monitored resource information, the resource requirements of each tenant are predicted to obtain the target resource requirements; Based on the target resource requirements, allocate the cluster resources corresponding to the target cluster to each tenant; The step of predicting the resource needs of each tenant based on the monitored resource information to obtain the target resource needs includes: Based on the monitored resource information, a preset resource range is estimated to obtain the estimated resource requirements; Based on the estimated resource demand, the resource demand of each tenant is predicted to obtain the target resource demand; The step of predicting the resource requirements of each tenant based on the estimated resource requirements to obtain the target resource requirements includes: Based on the estimated resource demand, an initial demand forecasting model is determined; The initial demand forecasting model is optimized using a preset model to obtain the target demand forecasting model; The target resource demand is obtained by predicting the resource demand of each tenant using the target demand prediction model. The step of optimizing the initial demand forecasting model to obtain the target demand forecasting model includes: The initial demand forecasting model is subjected to a preset boundary process based on a preset resource threshold to obtain an optimized demand forecasting model. The optimized demand prediction model is subjected to a preset smoothing process to obtain the target demand prediction model.

2. The cluster resource allocation method as described in claim 1, characterized in that, The step of allocating the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements includes: Obtain the resource allocation weight corresponding to each tenant; Based on the target resource requirements and the resource allocation weights, the cluster resources corresponding to the target cluster are allocated to each tenant.

3. The cluster resource allocation method as described in claim 2, characterized in that, The step of allocating the cluster resources corresponding to the target cluster to each tenant according to the target resource requirements and the resource allocation weight includes: A numerical comparison is made between the target resource requirement and the total cluster resources corresponding to the target cluster; Based on the numerical comparison results, the target resource requirements, and the resource allocation weights, the cluster resources corresponding to the target cluster are allocated to each tenant.

4. The cluster resource allocation method as described in claim 3, characterized in that, After allocating the cluster resources corresponding to the target cluster to each tenant based on the numerical comparison results, the target resource requirements, and the resource allocation weights, the process further includes: Obtain the resource usage values ​​corresponding to each tenant; The current resource error is determined based on the target resource requirement and the resource usage value. If the current resource error is detected to be inconsistent with the preset error index, the cluster resources will be redistributed to each tenant based on the preset resource allocation strategy.

5. A cluster resource allocation device, characterized in that, The cluster resource allocation device includes: The information acquisition module is used to acquire monitoring resource information corresponding to each tenant in the target cluster; The demand forecasting module is used to forecast the resource demand of each tenant based on the monitored resource information to obtain the target resource demand. The resource allocation module is used to allocate the cluster resources corresponding to the target cluster to each tenant based on the target resource requirements. The demand forecasting module is further configured to estimate the estimated resource demand by performing a preset resource range estimation based on the monitored resource information; and to predict the resource demand of each tenant based on the estimated resource demand to obtain the target resource demand. The demand forecasting module is further configured to determine an initial demand forecasting model based on the estimated resource demand; optimize the initial demand forecasting model using a preset model to obtain a target demand forecasting model; and forecast the resource demand of each tenant using the target demand forecasting model to obtain the target resource demand. The demand forecasting module is further configured to perform a preset boundary processing on the initial demand forecasting model based on a preset resource threshold to obtain an optimized demand forecasting model; and to perform a preset smoothing processing on the optimized demand forecasting model to obtain a target demand forecasting model.

6. A cluster resource allocation device, characterized in that, The device includes: a memory, a processor, and a cluster resource allocation program stored on the memory and executable on the processor, the cluster resource allocation program being configured to implement the steps of the cluster resource allocation method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a cluster resource allocation program, which, when executed by a processor, implements the steps of the cluster resource allocation method as described in any one of claims 1 to 4.

Images