Resource allocation method and device, storage medium, program product and electronic equipment

Abstract

The invention discloses a resource allocation method and device, a storage medium, a program product and electronic equipment. The method relates to the field of big data, and comprises the following steps: acquiring a resource allocation request of a target application for a distributed cluster, and determining a target resource quantity requested by the resource allocation request; under the condition that the target application is the first type of application, determining target allocation information according to the target resource quantity and historical resource use information of the target application on the distributed cluster; when the target application is a second type of application, determining target allocation information according to the target resource quantity and historical resource use information of a plurality of first type of applications associated with the distributed cluster; and determining a resource allocation operation on the target application based on the target allocation information. Through the resource allocation method and device, the problem that the resource allocation accuracy is low due to the fact that the resources in the distributed cluster are allocated to the application according to the resource quantity requested to be allocated by the application in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a resource allocation method, device, storage medium, program product, and electronic device. Background Art

[0002] As the digital transformation of financial institutions continues to deepen, the demand for big data resources from applications within the institutions is increasing. Distributed server clusters need to meet the resource demands of each application as much as possible, which poses a great challenge to existing cluster resources.

[0003] At present, in the relevant technology, the resource allocation strategy is usually based on the resource request of the application, that is, the user submits a resource application to the cluster management platform based on the resource demand estimated by the application, and the platform allocates resources according to these requests. However, when users request resources through applications, there are many subjective factors, and the actual resources required by the application may be far lower than the amount requested. In this case, allocating resources according to the amount of resources requested by the application may not only lead to a waste of resources (that is, the allocated resources far exceed the actual demand), but also may cause resource competition (that is, high-demand applications cannot run normally due to insufficient resources). Therefore, there is a problem of low accuracy in resource allocation.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The main purpose of the present application is to provide a resource allocation method, device, storage medium, program product and electronic device to solve the problem in the related art that resources in a distributed cluster are allocated to applications according to the amount of resources requested by the application, resulting in low accuracy of resource allocation.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a resource allocation method is provided. The method comprises: obtaining a resource allocation request of a target application for a distributed cluster, and determining a target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs; when the target application is a first-class application, determining target allocation information according to the target resource amount and historical resource usage information of the target application for the distributed cluster, wherein the first-class application refers to an application that has been allocated resources of the distributed cluster, and the target allocation information indicates whether the target resource amount is allocated to the target application; when the target application is a second-class application, determining the target allocation information according to the target resource amount and historical resource usage information of multiple first-class applications associated with the distributed cluster; and determining a resource allocation operation for the target application based on the target allocation information.

[0007] Furthermore, the historical resource usage information includes the resource usage of each target component by the target application at multiple time points, and the target component is a component in a distributed cluster, wherein the resource allocation method also includes: determining the resource usage saturation information of the target application in a first historical time period based on the resource usage of each target component by the target application at multiple time points and the allocated resources of the target application; determining the total resource usage of the target application in a second historical time period based on the resource usage of each target component by the target application at multiple time points; determining the resource usage deficiency of the target application based on the total resource usage and the allocated resources of the target application; and determining the target allocation information based on the target resource amount, the resource usage saturation information, and the resource usage deficiency.

[0008] Furthermore, the resource allocation method also includes: when the resource usage saturation information indicates that the resource usage of the target application is saturated, determining the target allocation information based on the resource usage deficiency amount and the target resource amount; when the resource usage saturation information indicates that the resource usage of the target application is unsaturated, determining that the target allocation information indicates that it is prohibited to allocate the target resource amount to the target application.

[0009] Furthermore, the resource allocation method also includes: obtaining the growth amount of the target application's job in the first time period relative to the job in the second historical time period to obtain the job growth amount, wherein the first time period is associated with the second historical time period; calculating the additional resource demand amount of the target application in the first time period based on the job growth amount; and determining the target allocation information based on the resource usage deficit amount, the additional resource demand amount and the target resource amount.

[0010] Furthermore, the resource allocation method also includes: determining a reference application from multiple first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; predicting a predicted resource demand of the target application in a second time period based on historical resource usage information of the reference application for the distributed cluster to obtain a first resource demand; and determining target allocation information based on the first resource demand and the target resource quantity.

[0011] Furthermore, the resource allocation method also includes: obtaining multiple target jobs to be executed by the target application within a second time period; predicting the predicted resource demand of the target application within the second time period based on the resource usage when the distributed cluster executes the reference job, and obtaining a second resource demand, wherein the reference job and the target job have the same job type; determining the target allocation information based on the first resource demand, the second resource demand and the target resource demand.

[0012] Furthermore, the resource allocation method also includes: for each first-category application in the distributed cluster, calculating the average resource utilization rate of the first-category application in each third historical time period in multiple third historical time periods; determining resource recovery information based on the average resource utilization rate of each third historical time period, wherein the resource recovery information is at least used to characterize whether the resources allocated to the first-category application are to be recovered.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a resource allocation device is provided. The device includes: an acquisition module, which is used to acquire a resource allocation request of a target application for a distributed cluster, and determine the target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs; a first determination module, which is used to determine the target allocation information according to the target resource amount and the historical resource usage information of the target application for the distributed cluster when the target application is a first-class application, wherein the first-class application refers to an application that has been allocated resources of the distributed cluster, and the target allocation information indicates whether the target resource amount is allocated to the target application; a second determination module, which is used to determine the target allocation information according to the target resource amount and the historical resource usage information of multiple first-class applications associated with the distributed cluster when the target application is a second-class application; and a third determination module, which is used to determine the resource allocation operation for the target application based on the target allocation information.

[0014] Furthermore, the historical resource usage information includes the resource usage of each target component by the target application at multiple time points, and the target component is a component in a distributed cluster, wherein the first determination module also includes: a first determination submodule, used to determine the resource usage saturation information of the target application in the first historical time period based on the resource usage of each target component by the target application at multiple time points and the allocated resources of the target application; a second determination submodule, used to determine the total resource usage of the target application in the second historical time period based on the resource usage of each target component by the target application at multiple time points; a third determination submodule, used to determine the resource usage deficiency of the target application based on the total resource usage and the allocated resources of the target application; and a fourth determination submodule, used to determine the target allocation information based on the target resource amount, the resource usage saturation information and the resource usage deficiency.

[0015] Furthermore, the fourth determination submodule also includes: a first determination unit, used to determine the target allocation information based on the resource usage deficiency amount and the target resource amount when the resource usage saturation information indicates that the resource usage of the target application is saturated; and a second determination unit, used to determine that the target allocation information indicates that it is prohibited to allocate the target resource amount to the target application when the resource usage saturation information indicates that the resource usage of the target application is unsaturated.

[0016] Furthermore, the first determination unit also includes: an acquisition subunit, used to obtain the growth amount of the job of the target application in the first time period relative to the job in the second historical time period, and obtain the job growth amount, wherein the first time period is associated with the second historical time period; a calculation subunit, used to calculate the additional resource demand amount of the target application in the first time period based on the job growth amount; a determination subunit, used to determine the target allocation information based on the resource usage deficit amount, the additional resource demand amount and the target resource amount.

[0017] Furthermore, the second determination module also includes: a fifth determination sub-module, used to determine a reference application from multiple first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; a prediction sub-module, used to predict the predicted resource demand of the target application in the second time period based on the historical resource usage information of the reference application for the distributed cluster, and obtain the first resource demand; and a sixth determination sub-module, used to determine the target allocation information based on the first resource demand and the target resource amount.

[0018] Furthermore, the sixth determination submodule also includes: an acquisition unit, used to acquire multiple target jobs to be executed by the target application within the second time period; a prediction unit, used to predict the predicted resource demand of the target application within the second time period based on the resource usage when the distributed cluster executes the reference job, and obtain the second resource demand, wherein the reference job and the target job have the same job type; a third determination unit, used to determine the target allocation information based on the first resource demand, the second resource demand and the target resource demand.

[0019] Furthermore, the resource allocation device also includes: a calculation module, which is used to calculate the average resource utilization rate of the first category application in each third historical time period in multiple third historical time periods for each first category application in the distributed cluster; a fourth determination module, which is used to determine resource recovery information based on the average resource utilization rate of each third historical time period, wherein the resource recovery information is at least used to characterize whether the resources allocated to the first category application are to be recovered.

[0020] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above resource allocation method.

[0021] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, the electronic device comprising a memory storing an executable program; and a processor for running the program, wherein the above-mentioned resource allocation method is executed when the program is running.

[0022] In order to achieve the above objective, according to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of the above resource allocation method when executed by a processor.

[0023] In an embodiment of the present application, when the target application is a first-category application, the target allocation information of the resource allocation request for the target application is determined based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, so that for a known application, based on the relationship between the application's own historical resource usage on the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met. When the target application is a second-category application, the target allocation information is determined based on the target resource amount and the historical resource usage information of multiple first-category applications associated with the distributed cluster, so that for a newly added application, based on the relationship between the historical resource usage of existing applications in the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met, thereby dynamically determining whether to allocate resources to the application based on actual conditions, thereby improving the accuracy of resource allocation.

[0024] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining whether to allocate resources to an application, realizes the technical effect of improving the accuracy of resource allocation, and solves the technical problem in the related technology of allocating resources in a distributed cluster to an application according to the amount of resources requested by the application, resulting in low accuracy of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of a computer terminal provided according to an embodiment of the present application;

[0027] Figure 2 The process of the resource allocation method provided in the embodiment of the present application is Figure 1 ;

[0028] Figure 3 The process of the resource allocation method provided in the embodiment of the present application is Figure 2 ;

[0029] Figure 4 is a schematic diagram of a resource allocation device provided according to an embodiment of the present application;

[0030] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0034] Example 1

[0035] According to an embodiment of the present application, a method embodiment of a resource allocation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a resource allocation method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the resource allocation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned resource allocation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] Under the above operating environment, this application provides Figure 2 The resource allocation method shown. Figure 2 The process of the resource allocation method provided in the embodiment of the present application is Figure 1 .

[0042] Step S201 : obtaining a resource allocation request of a target application for a distributed cluster, and determining a target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs.

[0043] Optionally, electronic devices, application systems, servers and other devices may be used as the execution subject of the present application. In this embodiment, the target processing system is used as the execution subject to execute the above-mentioned resource allocation method.

[0044] Optionally, the target application may be an application in the target organization. Depending on the actual application scenario, the type of the target application may also vary. For example, in a financial scenario, the target application may be a payment application. Therefore, the application type of the target application is not specifically limited in this application.

[0045] Optionally, the distributed cluster may be a cluster in the target organization, and the distributed cluster is used to provide resources for processing application jobs for the target application. In some embodiments, the aforementioned resources include at least computing resources, and the computing resources may be virtual cores (vCores). Virtual cores are virtualization units used to measure and allocate CPU (Central Processing Unit) resources. In a distributed cluster, it allows the resource manager to control and schedule CPU resources more finely.

[0046] Optionally, the resource allocation request of the target application includes the target resource amount requested by the target application. The resource allocation request of the target application may be generated by the developer or operation and maintenance personnel of the target application according to the operation requirements of the application. The resource allocation request is sent by the developer or operation and maintenance personnel to the target processing system through the target application. The target processing system may be used to manage a distributed cluster. The target processing system may determine the target resource amount from the resource allocation request.

[0047] Step S202, when the target application is a first-category application, determine the target allocation information based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, wherein the first-category application refers to the application to which the resources of the distributed cluster have been allocated, and the target allocation information indicates whether the target resource amount is allocated to the target application.

[0048] Optionally, the first type of application can be understood as the stock application in the target organization, which is an application that has obtained resource allocation. For this type of application, it can be determined whether to meet its current resource request based on its past resource usage records on the distributed cluster. Specifically, factors such as the resource usage peak and average usage of the application at different time points can be analyzed to determine whether the resource amount in its resource allocation request is reasonable and whether the target resource amount needs to be allocated to the application. This analysis based on historical data helps to improve the accuracy and efficiency of resource allocation.

[0049] For example, the resource usage deficit of the target application can be calculated based on the historical resource usage information of the target application, and the target allocation information can be determined based on the size relationship between the resource usage deficit and the target resource amount. For example, if the resource usage deficit exceeds the target resource amount, the target allocation information is determined to indicate that the target resource amount is allocated to the target application, otherwise, the target allocation information is determined to indicate that the target resource amount is not allocated to the target application.

[0050] In some embodiments, the resource usage of each component by the application at multiple time points is determined as the historical resource usage information of the application, and the unit of the resource usage of each component by the application at multiple time points is "vCore*Seconds". Optionally, the target processing system can collect the historical resource usage information of each first-category application in the distributed cluster in the following way:

[0051] For each first-class application, the resource usage of each component in the distributed cluster is collected. According to the collection cycle corresponding to different components, the collected data will be written to files generated every 30s, 60s, and 300s respectively:

[0052] 30s cycle: The default collection cycle is 30s for real-time indicators;

[0053] 60s cycle: The default collection cycle is 60s for real-time indicators;

[0054] 300s cycle: all indicators collected in periods other than 30s or 60s;

[0055] Monitor the format of the file written:

[0056] "Cluster ID|Cluster Name|Display Name|Service Name|Indicator ID|Collection Time|Collection Host|Unit|Indicator Value", where each field is separated by "|", for example:

[0057] 1|xx1|Host|Host|10000413|2019 / 06 / 1810:05:00|189-66-254-146|vCore*Seconds

[0058] |900,00;

[0059] 1|xx1|Host|Host|10000413|2019 / 06 / 1810:05:00|189-66-254-152|vCore*Seconds

[0060] |300,000;

[0061] 2|xx2|Host|Host|10000413|2019 / 06 / 1810:05:00|189-66-254-163|vCore*Seconds

[0062] |600,000;

[0063] Among them, the aforementioned display name can be understood as the component name, the service name refers to the service in the application, and the service name is used to determine the application corresponding to the component. The indicator value can be understood as the resource usage. The indicator value counted every 30s (or 60s, 300s) can be understood as the total amount of resources used by the application in the previous 30s (or 60s, 300s) window. Specifically, assuming that within the time interval of 30s before a certain point in time, the target application uses N vCores for a certain component and lasts for the entire 30s, then the resource usage of the target application for the component at this time point is 30NvCore*Seconds. This unit measures the product of the virtual core and the usage time, which can simultaneously reflect the intensity of resource usage and the duration of resource usage.

[0064] Optional. The unit of the target resource amount is vCore.

[0065] Step S203: when the target application is a second-category application, target allocation information is determined according to the target resource amount and historical resource usage information of a plurality of first-category applications associated with the distributed cluster.

[0066] Optionally, the second type of application can be understood as a new application in the target organization, which is an application that requests resource allocation for the first time. For this type of application, you can refer to the historical resource usage information of multiple first-type applications in the distributed cluster, and find the resource usage pattern of similar applications, which can be used as a reference for resource allocation. For example, if the job type and data processing requirements of a new application are similar to those of a known application, then the resource requirements of the new application can be predicted based on the resource usage of the known application. Optionally, you can also determine the jobs that the second type of application needs to process, and then predict the resource requirements of the new application based on the resource usage of multiple first-type applications when processing the same type of jobs. Therefore, based on the resource requirements of the new application, it is judged whether the amount of resources in its resource allocation request is reasonable and whether the target amount of resources needs to be allocated to the application.

[0067] For example, the resource demand of the new application is counted according to the above method, and the target allocation information is determined according to the size relationship between the resource demand of the new application and the target resource amount. For example, if the resource demand of the new application exceeds the target resource amount, the target allocation information is determined to indicate that the target resource amount is allocated to the target application, otherwise, the target allocation information is determined to indicate that the target resource amount is not allocated to the target application.

[0068] Step S204: determining a resource allocation operation for the target application based on the target allocation information.

[0069] Optionally, the resource allocation operation includes at least a first allocation operation and a second allocation operation, the first allocation operation refers to an operation of allocating a target amount of resources to a target application, and the second allocation operation refers to an operation of prohibiting allocation of resources to the target application.

[0070] Optionally, the target processing system may determine that the resource allocation operation is a first allocation operation and execute the first allocation operation when the target allocation information indicates that the target resource amount is allocated to the target application.

[0071] Optionally, the target processing system may determine that the resource allocation operation is a second allocation operation and execute the second allocation operation when the target allocation information indicates that the target resource amount is not allocated to the target application.

[0072] In an embodiment of the present application, when the target application is a first-category application, the target allocation information of the resource allocation request for the target application is determined based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, so that for a known application, based on the relationship between the application's own historical resource usage on the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met. When the target application is a second-category application, the target allocation information is determined based on the target resource amount and the historical resource usage information of multiple first-category applications associated with the distributed cluster, so that for a newly added application, based on the relationship between the historical resource usage of existing applications in the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met, thereby dynamically determining whether to allocate resources to the application based on actual conditions, thereby improving the accuracy of resource allocation.

[0073] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining whether to allocate resources to an application, realizes the technical effect of improving the accuracy of resource allocation, and solves the technical problem in the related technology of allocating resources in a distributed cluster to an application according to the amount of resources requested by the application, resulting in low accuracy of resource allocation.

[0074] Optionally, in the resource allocation method provided in the embodiment of the present application, the historical resource usage information includes the resource usage of each target component by the target application at multiple time points, and the target component is a component in a distributed cluster, wherein the target allocation information is determined based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, including: determining the resource usage saturation information of the target application in a first historical time period based on the resource usage of each target component by the target application at multiple time points and the allocated resources of the target application; determining the total resource usage of the target application in a second historical time period based on the resource usage of each target component by the target application at multiple time points; determining the resource usage deficiency of the target application based on the total resource usage and the allocated resources of the target application; determining the target allocation information based on the target resource amount, the resource usage saturation information, and the resource usage deficiency.

[0075] Optionally, the target components refer to individual computing and storage units in a distributed cluster.

[0076] Optionally, the first historical time period may be a month, a week, 15 days or a day in the past, and the multiple time points include at least some time points belonging to the first historical time period. In the process of determining the resource usage saturation information of the target application in the first historical time period according to the resource usage of each target component of the target application at multiple time points and the allocated resources of the target application, for each time point, the target processing system may calculate the resource usage of each target component according to the target application at the time point to obtain the total resource usage at the time point, and determine that the time point is a busy time point when the total resource usage exceeds a certain proportion (e.g., 80%) of the allocated resources, otherwise, determine that the time point is not a busy point. For example, the unit of resource usage of the target application for the components mentioned above is "vCore*Seconds", and the unit of allocated resources is "vCore". The target processing system can divide the former by the corresponding collection time, convert the resource usage into "vCore" units, and then sum up the converted resource usage of each component to obtain the total resource usage in "vCore" units, which can be compared with the allocated resources.

[0077] Optionally, after determining whether each time point is a busy point, for each hour in the first historical time period, if more than a certain proportion (e.g., 80%) of the time points in the hour are busy time points, the hour is determined to be a busy period, otherwise, the hour is determined not to be a busy period. Further, for each day in the first time period, if there are more than N (e.g., N=4) hours of busy periods in the day, the day is determined to be a busy day, otherwise, the day is determined not to be a busy day. Further, if the number of busy days in the first historical time period exceeds a certain proportion (e.g., 50%) of the first historical time period, it is determined that the resource usage saturation information of the target application in the first historical time period represents the resource usage saturation of the target application, and if the number of busy days in the first time period does not exceed a certain proportion (e.g., 50%) of the first historical time period, it is determined that the resource usage saturation information of the target application in the first historical time period represents the resource usage saturation of the target application.

[0078] Optionally, the second historical time period is different from the first historical time period, and the second historical time period may be a peak business period, such as 4 hours on a certain day (e.g., 20:00-00:00). In the process of determining the total resource usage of the target application in the second historical time period based on the resource usage of each target component by the target application at multiple time points, the target processing system may filter out time points belonging to the second historical time period from multiple time points to obtain multiple target time points, and then for each target time point, calculate the total resource usage at the target time point (in "vCore*Seconds") based on the resource usage of each target component by the target application at the target time point, and then sum the total resource usage at all target time points to obtain the total resource usage in the second historical time period (in "vCore*Seconds") as a unit.

[0079] In some embodiments, batch jobs in an application are usually executed periodically on a daily basis, so the first historical time period may be selected as a certain day in history, and the second historical time period may be selected as a peak business period in a day.

[0080] After determining the total amount of resource usage in the second historical time period, the target processing system may convert the total amount of resource usage in the second historical time period into "vCore" units, for example, by dividing the total amount of resource usage by the length of the second historical time period to obtain the total amount of resource usage in "vCore" units. Thereafter, the difference between the total amount of resource usage of the target application in the second historical time period and the amount of allocated resources (in "vCore" units) is calculated to obtain the resource usage deficit amount.

[0081] Optionally, after calculating and obtaining the resource usage saturation information and the resource usage deficiency amount, the target processing system may determine the target allocation information according to the target resource amount, the resource usage saturation information and the resource usage deficiency amount. For example, when the resource usage is saturated and the resource usage deficiency amount is greater than or equal to the target resource amount, the target allocation information is determined to indicate that the target resource amount is allocated to the target application; conversely, when the resource usage is not saturated, or the resource usage deficiency amount is less than the target resource amount, the target allocation information is determined to indicate that the target resource amount is not allocated to the target application.

[0082] In some embodiments, the target processing system can also calculate the resource usage deviation, for example, according to the following formula: (actual allocated resources for each job - mean of resource usage) / standard deviation of resource usage. If the skewness exceeds 50 in the positive direction, it is determined that the consumption of tenant resources by individual jobs is unreasonable, and an early warning message is generated to alert relevant staff.

[0083] In some embodiments, the target processing system may generate reports based on the aforementioned calculated resource usage saturation, information resource usage deficiency, and resource usage deviation for staff to review.

[0084] It should be noted that, through the above process, it is possible to determine the target allocation information based on the information of the resource saturation dimension and the resource deficiency dimension, thereby improving the accuracy of the determined target allocation information and further improving the accuracy of resource allocation.

[0085] Optionally, in the resource allocation method provided in an embodiment of the present application, target allocation information is determined based on the target resource amount, resource usage saturation information and resource usage deficiency amount, including: when the resource usage saturation information indicates that the resource usage of the target application is saturated, the target allocation information is determined based on the resource usage deficiency amount and the target resource amount; when the resource usage saturation information indicates that the resource usage of the target application is unsaturated, determining that the target allocation information indicates that it is prohibited to allocate the target resource amount to the target application.

[0086] In some embodiments, when the resource usage saturation information indicates that the resource usage of the target application is saturated, the target processing system may determine that the target allocation information indicates that the target resource amount is allocated to the target application when the resource usage deficit amount is greater than or equal to the target resource amount; conversely, when the resource usage deficit amount is less than the target resource amount, the target allocation information indicates that the target resource amount is not allocated to the target application.

[0087] In some embodiments, when the resource usage saturation information represents the resource usage saturation of the target application, the target processing system may also determine the target allocation information based on the resource usage deficit amount, the numerical relationship between the target resource amounts, and the job growth amount of the target application in the future time period.

[0088] Optionally, when the resource usage saturation information indicates that resource usage of the target application is not saturated, determining that the target allocation information indicates that allocation of the target resource amount to the target application is prohibited.

[0089] It should be noted that, through the above method, it is possible to determine the target allocation information based on the information of the resource saturation dimension and the resource deficiency dimension, thereby achieving accurate determination of the target allocation information, thereby improving the accuracy of resource allocation.

[0090] Optionally, in the resource allocation method provided in an embodiment of the present application, target allocation information is determined based on the resource usage deficit and the target resource amount, including: obtaining the growth amount of the target application's job in a first time period relative to the job in a second historical time period to obtain the job growth amount, wherein the first time period is associated with the second historical time period; calculating the additional resource demand amount of the target application in the first time period based on the job growth amount; and determining the target allocation information based on the resource usage deficit, the additional resource demand amount, and the target resource amount.

[0091] Optionally, since the workload of the target application at different stages is not constant but may change, the target processing system may determine the target allocation information in combination with the workload growth of the target application in the future time period.

[0092] Optionally, the above-mentioned first time period is a future time period, and it is associated with the second historical time period. For example, if the second historical time period is a peak business period in a historical date, such as 20:00-00:00 on a historical day (such as the 3rd of last month), then the first time period is a peak business period in a future date, such as 20:00-00:00 on a future day (such as the 1st of next month).

[0093] Optionally, the growth amount of the target application's jobs in the first time period relative to the jobs in the second historical time period can be recorded in the resource allocation request, and the target processing system can determine the job growth amount from the resource allocation request, wherein the job growth amount includes the type and corresponding quantity of the increased jobs.

[0094] Furthermore, the target processing system can calculate the additional resource demand corresponding to the job growth in the first time period based on the reference resource usage corresponding to each type of job (in units of vCore*Seconds). For example, the reference resource usage corresponding to each job in the job growth is added to obtain the additional resource demand corresponding to the job growth in the first time period. The reference resource usage corresponding to the job can be determined by the target processing system based on the historical resource usage of the same type of job. For example, the historical resource usage corresponding to the completion of one execution of the same type of job is determined as the reference resource usage. The historical resource usage corresponding to one execution of the job can be obtained by accumulating the resource usage at each time point between the start execution time and the end execution time of the job.

[0095] Optionally, after determining the additional resource requirement, the target processing system may determine the target allocation information according to the resource usage deficit, the additional resource requirement, and the target resource amount. For example, the additional resource requirement is first divided by the length of the first time period to obtain the additional resource requirement in units of vCores, and then the sum of the resource usage deficit and the additional resource requirement is calculated to obtain the total resource requirement, so that when the total resource requirement is greater than or equal to the target resource amount, the target allocation information is determined to indicate that the target resource amount is allocated to the target application, and when the total resource requirement is less than the target resource amount, the target allocation information is determined to indicate that the target resource amount is not allocated to the target application.

[0096] It should be noted that, through the above process, it is possible to determine the target allocation information in combination with the job growth amount of the target application, thereby improving the accuracy of determining the target allocation information, and further improving the accuracy of resource allocation.

[0097] Optionally, in the resource allocation method provided in an embodiment of the present application, target allocation information is determined based on the target resource amount and historical resource usage information of multiple first-category applications associated with the distributed cluster, including: determining a reference application from multiple first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; predicting the predicted resource demand of the target application in a second time period based on the historical resource usage information of the reference application for the distributed cluster to obtain the first resource demand; and determining the target allocation information based on the first resource demand and the target resource amount.

[0098] Optionally, the target processing system can obtain application information of the target application and application information of the first category of applications, where the application information includes but is not limited to the data processing logic of the application (such as business rules, data processing flow, algorithms used), business scenarios, peak resource usage time, etc. The target processing system can evaluate the similarity between the target application and the first category of applications based on preset similarity evaluation rules and application information. The target processing system can also input the application information of the target application and the application information of the first category of applications into the target neural network model, and output the similarity between the target application and the first category of applications through the target neural network model, wherein the target neural network model can be trained based on a training sample set, where the training samples in the training sample set include application information of two sample applications, and the true labels of the training samples represent the similarity between the two sample applications.

[0099] Optionally, after determining the similarity between the target application and each first-category application, the target processing system may determine the first-category application with a similarity greater than a preset similarity and the highest similarity as a reference application.

[0100] Optionally, after determining the reference application, the target processing system can predict the predicted resource demand of the target application in the second time period based on the historical resource usage information of the reference application for the distributed cluster, and obtain the first resource demand. In some embodiments, batch jobs in the application are usually executed periodically on a daily basis. Therefore, the target processing system can determine the total resource usage (in units of "vCore*Seconds") of the reference application in the fourth historical time period (e.g., a certain historical day) based on the historical resource usage information of the reference application for the distributed cluster, and then determine the total resource usage of the reference application in the fourth historical time period as the predicted resource demand of the target application in the second time period (e.g., a certain future day), and obtain the first resource demand. Optionally, the second time period is associated with the fourth historical time period. Among them, determining the total resource usage of the reference application in the fourth historical time period is the same as the above-mentioned process of determining the total resource usage of the target application in the second historical time period, so it will not be repeated here.

[0101] Optionally, after determining the first resource requirement, the target processing system may determine the target allocation information based on the numerical relationship between the first resource requirement and the target resource. For example, the first resource requirement is first converted into "vCore" units, such as dividing the first resource requirement by the length of the second time period to obtain the first resource requirement in "vCore" units. Thereafter, when the first resource requirement is greater than or equal to the target resource, the target allocation information is determined to indicate that the target resource is allocated to the target application, and when the first resource requirement is less than the target resource, the target allocation information is determined to indicate that the target resource is not allocated to the target application.

[0102] It should be noted that by determining a reference application similar to the target application from multiple first-category applications and evaluating whether the requested resource amount of the target application is reasonable based on the resource usage of the reference application, the accuracy of the determined target allocation information can be effectively improved.

[0103] Optionally, in the resource allocation method provided in an embodiment of the present application, target allocation information is determined based on the first resource demand and the target resource amount, including: obtaining multiple target jobs to be executed by the target application within a second time period; predicting the predicted resource demand of the target application within the second time period based on the resource usage when the distributed cluster executes the reference job, and obtaining a second resource demand, wherein the reference job and the target job have the same job type; determining the target allocation information based on the first resource demand, the second resource demand and the target resource amount.

[0104] In some embodiments, the target processing system may also predict the resource demand of the target application according to the target job to be executed by the target application, wherein the target job may be a batch job.

[0105] In some embodiments, the second time period is a future time period. Since batch jobs in applications are usually executed periodically on a daily basis, the second time period may be a certain day in the future.

[0106] For example, assuming that the new application (target application) has 10 high-priority jobs to complete between 6 and 8 a.m. every day, the computing resources required for the distributed cluster to execute the same type of job (i.e., the reference job) within these 2 hours are 900,000 vCore*Seconds, so at least 125 vCores (900,000 / 3600 / 2=125 vCores) must be applied for.

[0107] At the same time, assuming that the new application needs to complete 100 low-priority jobs between 8:00 and 20:00, the computing resource usage required for the distributed cluster to execute the same type of job (i.e., the reference job) within these 12 hours is 9,000,000 vCore*Seconds. If calculated based on the 12-hour period, at least 208 vCores (9,000,000 / 3600 / 12=208.33 vCores) must be applied for.

[0108] In addition, suppose that the newly added application has 3 multi-batch jobs every day, and a total of 12 batches need to be run from 8:00 to 20:00 every day. Each batch is required to be completed within 15 minutes. When the distributed cluster executes the same type of job (i.e., the reference job), the computing resource usage of each job is 90,000, so at least 300 vCores (90,000*3 / 15 / 60=300vCores) must be applied for. Therefore, the total computing resource amount of the newly added resources is: max (result of step 1, result of step 2) + step 3, that is, max (125,208) + 300 = 508vCores, that is, 508vCores is determined as the second resource requirement.

[0109] Optionally, after obtaining the second resource requirement, the target processing system may determine the target allocation information according to the first resource requirement, the second resource requirement and the target resource requirement. For example, the maximum resource requirement is determined from the first resource requirement and the second resource requirement, and then the maximum resource requirement is compared with the target resource requirement to determine the target allocation information according to the comparison result.

[0110] For example, when the maximum resource demand is greater than or equal to the target resource amount, the target allocation information indicates that the target resource amount is allocated to the target application; when the maximum resource demand is less than the target resource amount, the target allocation information indicates that the target resource amount is not allocated to the target application.

[0111] It should be noted that, through the above-mentioned method, it is achieved to evaluate whether the requested resource amount of the target application is reasonable in combination with the work content of the target application, thereby further improving the accuracy of the determined target allocation information.

[0112] Optionally, in the resource allocation method provided in an embodiment of the present application, for each first-category application in a distributed cluster, the average resource usage rate of the first-category application in each third historical time period in multiple third historical time periods is calculated; based on the average resource usage rate of each third historical time period, resource recovery information is determined, wherein the resource recovery information is at least used to characterize whether the resources allocated to the first-category application are to be recovered.

[0113] In some embodiments, the third historical time period may be a certain month in history. The plurality of third historical time periods may include the first month in the past, the second month in the past, the third month in the past, and the like.

[0114] For each first-category application in each third historical time period, the target processing system may calculate the resource usage rate of the first-category application on each day in the third historical time period, and then average the resource usage rate on each day in the third historical time period to obtain the average resource usage rate of the first-category application in the third historical time period. For example, the total resource usage of the first-category application on a certain day may be divided by the allocated resource amount corresponding to the application to obtain the resource usage rate of the day.

[0115] In some embodiments, if the monthly average resource usage rate of a first-category application is 0% for three consecutive months, all its resources and the resource pool itself will be recycled; if the monthly average resource usage rate of a first-category application is lower than 50% for three consecutive months, half of the unused resources will be recycled to the resource pool; if the monthly average resource usage rate of a first-category application is lower than 70% for six consecutive months, half of its unused resources will be recycled; if the above situation does not occur for a first-category application, the resources allocated to the first-category application will not be recycled.

[0116] It should be noted that, through the above process, the idle resources allocated to each first-category application are effectively recovered, thereby avoiding resource waste and further improving the accuracy of resource allocation.

[0117] In an optional embodiment, Figure 3 The process of the resource allocation method provided in the embodiment of the present application is Figure 2 ,according to Figure 3 An optional application process of the resource allocation method is described. Figure 3As shown, the target processing system periodically collects resource usage information of the first type of application in the distributed cluster based on a preset monitoring cycle and stores it in a preset storage area. After obtaining the resource allocation request of the target application for the distributed cluster, the application type (first type of application, second type of application) is identified, and corresponding analysis is performed based on the identification result. For example, if the target application is a first type of application, resource saturation analysis, resource usage defect analysis, resource usage deviation analysis, etc. are performed to determine the target matching information. If the target application is a second type of application, the resource usage of the known first type of application and the job content of the target application are analyzed based on similarity to determine the target matching information.

[0118] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining whether to allocate resources to an application, realizes the technical effect of improving the accuracy of resource allocation, and solves the technical problem in the related technology of allocating resources in a distributed cluster to an application according to the amount of resources requested by the application, resulting in low accuracy of resource allocation.

[0119] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0120] Example 2

[0121] The embodiment of the present application also provides a resource allocation device. It should be noted that the resource allocation device of the embodiment of the present application can be used to execute the resource allocation method provided by the embodiment of the present application. The resource allocation device provided by the embodiment of the present application is introduced below.

[0122] According to an embodiment of the present application, a device for implementing the above resource allocation method is also provided, such as Figure 4 As shown, the device comprises:

[0123] The acquisition module 401 is used to acquire a resource allocation request of a target application for a distributed cluster and determine a target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs;

[0124] A first determination module 402 is used to determine target allocation information according to the target resource amount and historical resource usage information of the target application on the distributed cluster when the target application is a first type of application, wherein the first type of application refers to an application to which resources of the distributed cluster have been allocated, and the target allocation information indicates whether to allocate the target resource amount to the target application;

[0125] The second determination module 403 is used to determine the target allocation information according to the target resource amount and the historical resource usage information of the plurality of first-category applications associated with the distributed cluster when the target application is a second-category application;

[0126] The third determining module 404 is configured to determine a resource allocation operation for a target application based on the target allocation information.

[0127] In an embodiment of the present application, when the target application is a first-category application, the target allocation information of the resource allocation request for the target application is determined based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, so that for a known application, based on the relationship between the application's own historical resource usage on the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met. When the target application is a second-category application, the target allocation information is determined based on the target resource amount and the historical resource usage information of multiple first-category applications associated with the distributed cluster, so that for a newly added application, based on the relationship between the historical resource usage of existing applications in the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met, thereby dynamically determining whether to allocate resources to the application based on actual conditions, thereby improving the accuracy of resource allocation.

[0128] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining whether to allocate resources to an application, realizes the technical effect of improving the accuracy of resource allocation, and solves the technical problem in the related technology of allocating resources in a distributed cluster to an application according to the amount of resources requested by the application, resulting in low accuracy of resource allocation.

[0129] Optionally, in the resource allocation device provided in the embodiment of the present application, the historical resource usage information includes the resource usage of each target component by the target application at multiple time points, and the target component is a component in a distributed cluster, wherein the first determination module also includes: a first determination submodule, used to determine the resource usage saturation information of the target application in the first historical time period according to the resource usage of each target component by the target application at multiple time points and the allocated resources of the target application; a second determination submodule, used to determine the total resource usage of the target application in the second historical time period according to the resource usage of each target component by the target application at multiple time points; a third determination submodule, used to determine the resource usage deficiency of the target application according to the total resource usage and the allocated resources of the target application; and a fourth determination submodule, used to determine the target allocation information according to the target resource amount, the resource usage saturation information, and the resource usage deficiency.

[0130] Optionally, in the resource allocation device provided in the embodiment of the present application, the fourth determination submodule also includes: a first determination unit, used to determine the target allocation information based on the resource usage deficiency amount and the target resource amount when the resource usage saturation information indicates that the resource usage of the target application is saturated; and a second determination unit, used to determine that the target allocation information indicates that it is prohibited to allocate the target resource amount to the target application when the resource usage saturation information indicates that the resource usage of the target application is unsaturated.

[0131] Optionally, in the resource allocation device provided in the embodiment of the present application, the first determination unit also includes: an acquisition subunit, used to acquire the growth amount of the job of the target application in the first time period relative to the job in the second historical time period, and obtain the job growth amount, wherein the first time period is associated with the second historical time period; a calculation subunit, used to calculate the additional resource demand amount of the target application in the first time period based on the job growth amount; and a determination subunit, used to determine the target allocation information based on the resource usage deficit amount, the additional resource demand amount and the target resource amount.

[0132] Optionally, in the resource allocation device provided in the embodiment of the present application, the second determination module also includes: a fifth determination sub-module, used to determine a reference application from multiple first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; a prediction sub-module, used to predict the predicted resource demand of the target application in the second time period based on the historical resource usage information of the reference application for the distributed cluster, and obtain the first resource demand; and a sixth determination sub-module, used to determine the target allocation information based on the first resource demand and the target resource amount.

[0133] Optionally, in the resource allocation device provided in the embodiment of the present application, the sixth determination submodule also includes: an acquisition unit, used to acquire multiple target jobs to be executed by the target application within the second time period; a prediction unit, used to predict the predicted resource demand of the target application within the second time period based on the resource usage when the distributed cluster executes the reference job, and obtain the second resource demand, wherein the reference job and the target job have the same job type; a third determination unit, used to determine the target allocation information based on the first resource demand, the second resource demand and the target resource demand.

[0134] Optionally, in the resource allocation device provided in the embodiment of the present application, the resource allocation device also includes: a calculation module, used to calculate, for each first-category application in the distributed cluster, the average resource usage rate of the first-category application in each third historical time period in multiple third historical time periods; a fourth determination module, used to determine resource recovery information based on the average resource usage rate of each third historical time period, wherein the resource recovery information is at least used to characterize whether the resources allocated to the first-category application are to be recovered.

[0135] It should be noted that the acquisition module 401, the first determination module 402, the second determination module 403 and the third determination module 404 correspond to steps S201 to S204 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0136] Example 3

[0137] An embodiment of the present application may provide an electronic device, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0138] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain a resource allocation request of the target application for the distributed cluster, and determine the target resource quantity requested by the resource allocation request, wherein the distributed cluster provides the target application with resources for processing application jobs; when the target application is a first-category application, determine the target allocation information based on the target resource quantity and the historical resource usage information of the target application for the distributed cluster, wherein the first-category application refers to an application that has been allocated resources of the distributed cluster, and the target allocation information indicates whether the target resource quantity is allocated to the target application; when the target application is a second-category application, determine the target allocation information based on the target resource quantity and the historical resource usage information of multiple first-category applications associated with the distributed cluster; and determine the resource allocation operation for the target application based on the target allocation information.

[0140] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the resource usage saturation information of the target application in the first historical time period based on the resource usage of each target component and the allocated resources of the target application at multiple time points; determine the total resource usage of the target application in the second historical time period based on the resource usage of each target component by the target application at multiple time points; determine the resource usage deficiency of the target application based on the total resource usage and the allocated resources of the target application; determine the target allocation information based on the target resource amount, the resource usage saturation information and the resource usage deficiency.

[0141] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: when the resource usage saturation information indicates that the resource usage of the target application is saturated, determine the target allocation information based on the resource usage deficiency amount and the target resource amount; when the resource usage saturation information indicates that the resource usage of the target application is unsaturated, determine that the target allocation information indicates that it is prohibited to allocate the target resource amount to the target application.

[0142] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the growth amount of the target application's job in the first time period relative to the job in the second historical time period to obtain the job growth amount, wherein the first time period is associated with the second historical time period; calculate the additional resource demand amount of the target application in the first time period based on the job growth amount; determine the target allocation information based on the resource usage deficit amount, the additional resource demand amount and the target resource amount.

[0143] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine a reference application from multiple first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; predict the predicted resource demand of the target application in a second time period based on the historical resource usage information of the reference application for the distributed cluster to obtain the first resource demand; determine the target allocation information based on the first resource demand and the target resource amount.

[0144] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain multiple target jobs to be executed by the target application in the second time period; predict the predicted resource demand of the target application in the second time period based on the resource usage when the distributed cluster executes the reference job, and obtain the second resource demand, wherein the reference job and the target job have the same job type; determine the target allocation information based on the first resource demand, the second resource demand and the target resource demand.

[0145] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: for each first-category application in the distributed cluster, calculate the average resource usage rate of the first-category application in each third historical time period in multiple third historical time periods; determine resource recovery information based on the average resource usage rate of each third historical time period, wherein the resource recovery information is at least used to characterize whether the resources allocated to the first-category application are to be recovered.

[0146] In an embodiment of the present application, when the target application is a first-category application, the target allocation information of the resource allocation request for the target application is determined based on the target resource amount and the historical resource usage information of the target application for the distributed cluster, so that for a known application, based on the relationship between the application's own historical resource usage on the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met. When the target application is a second-category application, the target allocation information is determined based on the target resource amount and the historical resource usage information of multiple first-category applications associated with the distributed cluster, so that for a newly added application, based on the relationship between the historical resource usage of existing applications in the distributed cluster and the requested target resource amount, it is evaluated whether the resource allocation request of the application needs to be met, thereby dynamically determining whether to allocate resources to the application based on actual conditions, thereby improving the accuracy of resource allocation.

[0147] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining whether to allocate resources to an application, realizes the technical effect of improving the accuracy of resource allocation, and solves the technical problem in the related technology of allocating resources in a distributed cluster to an application according to the amount of resources requested by the application, resulting in low accuracy of resource allocation.

[0148] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations shown.

[0149] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] Example 4

[0151] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the resource allocation method provided in the first embodiment.

[0152] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0153] The present application also provides a computer program product, which is suitable for executing the steps of the resource allocation method when executed on a data processing device.

[0154] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0160] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A resource allocation method, characterized in that: include: Obtaining a resource allocation request of a target application for a distributed cluster, and determining a target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs; In the case where the target application is a first type of application, determining target allocation information according to the target resource amount and historical resource usage information of the target application on the distributed cluster, wherein the first type of application refers to an application that has been allocated resources of the distributed cluster, and the target allocation information indicates whether the target resource amount is allocated to the target application; In the case where the target application is a second-category application, determining target allocation information according to the target resource amount and historical resource usage information of a plurality of first-category applications associated with the distributed cluster; A resource allocation operation for the target application is determined based on the target allocation information.

2. The method according to claim 1, characterized in that The historical resource usage information includes resource usage of each target component by the target application at multiple time points, and the target component is a component in the distributed cluster, wherein the target allocation information is determined according to the target resource amount and the historical resource usage information of the target application on the distributed cluster, including: Determine resource usage saturation information of the target application in a first historical time period according to resource usage of each target component by the target application at multiple time points and allocated resources of the target application; Determine the total resource usage of the target application in a second historical time period according to the resource usage of each target component by the target application at multiple time points; Determining a resource usage deficit of the target application according to the total resource usage and the allocated resource amount of the target application; The target allocation information is determined according to the target resource amount, the resource usage saturation information and the resource usage deficit amount.

3. The method according to claim 2, characterized in that Determining the target allocation information according to the target resource amount, the resource usage saturation information, and the resource usage deficiency amount includes: In a case where the resource usage saturation information indicates that the resource usage of the target application is saturated, determining the target allocation information according to the resource usage deficit amount and the target resource amount; In a case where the resource usage saturation information indicates that resource usage of the target application is not saturated, determining that the target allocation information indicates that allocation of the target resource amount to the target application is prohibited.

4. The method according to claim 3, characterized in that Determining the target allocation information according to the resource usage deficit amount and the target resource amount includes: Obtaining a growth amount of jobs of the target application in a first time period relative to jobs in the second historical time period to obtain a job growth amount, wherein the first time period is associated with the second historical time period; Calculating the additional resource demand of the target application in the first time period according to the job growth amount; The target allocation information is determined according to the resource usage deficit, the additional resource requirement, and the target resource amount.

5. The method according to claim 1, characterized in that Determining target allocation information according to the target resource amount and historical resource usage information of multiple first-category applications associated with the distributed cluster includes: Determining a reference application from a plurality of first-category applications associated with the distributed cluster, wherein the application similarity between the reference application and the target application is greater than a preset similarity; Predicting a predicted resource demand of the target application in a second time period according to historical resource usage information of the reference application on the distributed cluster, to obtain a first resource demand; The target allocation information is determined according to the first resource requirement and the target resource amount.

6. The method according to claim 5, characterized in that Determining the target allocation information according to the first resource demand and the target resource amount includes: Acquire a plurality of target jobs to be executed by the target application within the second time period; Predicting the predicted resource demand of the target application in the second time period according to the resource usage of the distributed cluster when executing the reference job, to obtain a second resource demand, wherein the reference job is of the same job type as the target job; The target allocation information is determined according to the first resource requirement, the second resource requirement, and the target resource amount.

7. The method according to claim 1, characterized in that The method further comprises: For each first-category application in the distributed cluster, calculating an average resource usage rate of the first-category application in each third historical time period in a plurality of third historical time periods; Resource recovery information is determined according to the average resource usage rate in each third historical time period, wherein the resource recovery information is at least used to indicate whether to recover the resources allocated to the first category of applications.

8. A resource allocation device, characterized in that: include: an acquisition module, configured to acquire a resource allocation request of a target application for a distributed cluster, and determine a target resource amount requested by the resource allocation request, wherein the distributed cluster provides resources for the target application to process application jobs; A first determination module is used to determine target allocation information according to the target resource amount and historical resource usage information of the target application on the distributed cluster when the target application is a first type of application, wherein the first type of application refers to an application to which resources of the distributed cluster have been allocated, and the target allocation information indicates whether the target resource amount is allocated to the target application; A second determination module is used to determine target allocation information according to the target resource amount and historical resource usage information of multiple first-category applications associated with the distributed cluster when the target application is a second-category application; The third determining module is used to determine a resource allocation operation for the target application based on the target allocation information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the resource allocation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the resource allocation method described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the resource allocation method according to any one of claims 1 to 7 are implemented.

Images