Resource allocation method and device, equipment, medium and program product

Abstract

The invention provides a resource allocation method and device, equipment, a medium and a program product, which can be applied to the field of artificial intelligence and the field of financial science and technology. The method comprises the steps that based on a to-be-executed job task, the resource demand quantity of a to-be-evaluated service platform is evaluated, an evaluation result is obtained, and the evaluation result represents the resource quantity occupied by the initial resource execution job task of the to-be-evaluated service platform; performing resource recovery on at least one initial resource of the to-be-evaluated service platform according to the evaluation result to obtain an updated service platform; performing resource gap detection on the service platform based on the resource execution data of the service platform to obtain a detection result; and based on the detection result, carrying out resource allocation on the service platform to obtain a resource allocation result.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and financial technology, and more specifically to a resource allocation method, device, equipment, medium and program product. Background Art

[0002] With the continuous innovation of business scenarios and the massive increase in business volume, the resource consumption of the big data platform used to perform business tasks is also increasing. Therefore, for business service platforms such as big data platforms, the response time of business services is low, and there is an imbalance in resource allocation on the business service platform, resulting in inefficient resource use. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a resource allocation method, apparatus, device, medium and program product.

[0004] According to a first aspect of the present disclosure, a resource allocation method is provided, comprising: evaluating the resource demand of a service platform to be evaluated based on a job task to be executed, and obtaining an evaluation result, wherein the evaluation result represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task; recovering at least one initial resource of the service platform to be evaluated according to the evaluation result, and obtaining an updated service platform; performing resource gap detection on the service platform based on resource execution data of the service platform, and obtaining a detection result; and allocating resources to the service platform based on the detection result, and obtaining a resource allocation result.

[0005] According to an embodiment of the present disclosure, the resource execution data of the service platform includes data point utilization rate and resource queue data, the resource queue data includes resource waiting time and idle resource amount of idle resources in the idle resource queue, and resource occupation time and occupied resource amount of occupied resources in the occupied resource queue of the service platform, and the occupied resources are occupied by job tasks; based on the resource execution data of the service platform, the service platform is detected for resource gaps to obtain detection results, including: determining the resource saturation index of the service platform according to the data point utilization rate of the service platform, wherein the data point utilization rate includes the central processing unit utilization rate, the memory utilization rate and the network traffic utilization rate; determining the queue waiting time index according to the resource waiting time of idle resources in the idle resource queue of the service platform; when both the resource saturation index and the queue waiting time index meet the preset index conditions, determining the queue resource allocation result of the service platform based on the cumulative sum between the occupied resource amount of the occupied resource queue and the idle resource amount of the idle resource queue, and the product between the cumulative sum and the resource waiting time of the idle resources; determining the detection result according to the ratio between the queue resource allocation result of the service platform and the resource occupation time of the occupied resource queue.

[0006] According to an embodiment of the present disclosure, a data point usage rate corresponds to a preset operating time period of a service platform; based on the data point usage rate of the service platform, a resource saturation index of the service platform is determined, including: comparing the data point usage rate of the data points of the service platform in the preset operating time period with a preset usage rate threshold to obtain a usage rate comparison result; when the usage rate comparison result indicates that the data point usage rate of the data point exceeds the preset usage rate threshold, the data points in the preset operating time period are determined as saturated data points; when the number of saturated data points exceeds the preset number threshold, the preset operating time period is determined as a resource saturation period; based on the number of resource saturation periods, the resource saturation index of the service platform is determined.

[0007] According to the embodiments of the present disclosure, resource requirements include storage resource requirements and computing resource requirements; based on the job tasks to be executed, the resource requirements of the service platform to be evaluated are evaluated to obtain evaluation results, including: based on a preset compression ratio, compressing the business data volume of the job tasks to determine the storage resource requirements; based on the resource value that the job tasks need to occupy, determining the computing resource requirements; determining the evaluation results based on the storage resource requirements and the computing resource requirements.

[0008] According to an embodiment of the present disclosure, the computing resource demand is determined based on the resource value that the job task needs to occupy, including: determining the job duration assigned to the job task according to the job priority of the job task; and determining the first computing resource demand of the service platform to be evaluated based on the ratio of the resource value that the job task needs to occupy and the job duration.

[0009] According to an embodiment of the present disclosure, the computing resource requirement is determined based on the resource value that needs to be consumed by the job task, and the method also includes: determining the second computing resource requirement of the service platform to be evaluated according to the ratio between the resource value that needs to be consumed by the job task and the task execution time of the subtask of the job task, wherein the job task is divided into N batches of subtasks, and N is an integer greater than 1; determining the computing model resources used to execute the job task in the service platform to be evaluated, and the computing model resources include operator computing memory, operator parallel number and computing model parallel number; determining the third computing resource requirement of the service platform to be evaluated according to the product of operator computing memory, operator parallel number and computing model parallel number; determining the computing resource requirement according to the first computing resource requirement, the second computing resource requirement and the third computing resource requirement.

[0010] According to an embodiment of the present disclosure, at least part of the initial resources of the service platform to be evaluated are recycled according to the evaluation results to obtain an updated service platform, including: determining a resource pool associated with the service platform to be evaluated, the resource pool being used to allocate initial resources to the service platform to be evaluated; determining a resource utilization rate according to a ratio between the amount of resources occupied by executing job tasks represented by the evaluation results and the amount of initial resources allocated to the platform to be evaluated; obtaining a utilization rate comparison result based on a comparison between the resource utilization rate and a preset resource utilization rate threshold; executing a preset resource recycling strategy based on the utilization rate comparison result to obtain a resource recovery rate for the initial resources; and recycling the initial resources of the service platform to be evaluated based on the resource recovery rate to obtain an updated service platform.

[0011] A second aspect of the present disclosure provides a resource allocation device, including: an evaluation module, which is used to evaluate the resource demand of a service platform to be evaluated based on a job task to be executed, and obtain an evaluation result, wherein the evaluation result represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task; a recovery module, which is used to recover at least one initial resource of the service platform to be evaluated according to the evaluation result, and obtain an updated service platform; a detection module, which is used to perform resource gap detection on the service platform based on the resource execution data of the service platform, and obtain a detection result; and an allocation module, which is used to allocate resources to the service platform based on the detection result, and obtain a resource allocation result.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0014] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] According to the embodiments of the present disclosure, based on the job tasks to be executed, the resource demand of the service platform to be evaluated is evaluated to obtain an evaluation result; based on the evaluation result, at least one initial resource of the service platform to be evaluated is recycled to obtain an updated service platform; based on the resource execution data of the service platform, the service platform is detected for resource gaps to obtain a detection result; based on the detection result, resources are allocated to the service platform to obtain a resource allocation result. By evaluating the resource demand based on the job tasks to be executed, the problem of poor evaluation accuracy caused by manual evaluation in the prior art is avoided, so as to achieve efficient and accurate evaluation of resource demand, thereby enabling reasonable resource recycling of initial resources. At the same time, by detecting resource gaps on the service platform based on the resource execution data of the service platform, the problem of inefficient resources still existing in the service platform in actual applications is avoided, thereby achieving the effect of improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the resource allocation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart of a resource allocation method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 3 A schematic diagram schematically shows a resource allocation method according to an embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram showing a structural block diagram of a resource allocation device according to an embodiment of the present disclosure; and

[0021] Figure 5 A block diagram of an electronic device suitable for implementing a resource allocation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0028] The embodiments of the present disclosure provide a resource allocation method, which evaluates the resource demand of a service platform to be evaluated based on a job task to be executed, and obtains an evaluation result, wherein the evaluation result represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task; according to the evaluation result, at least one initial resource of the service platform to be evaluated is recycled to obtain an updated service platform; based on the resource execution data of the service platform, a resource gap detection is performed on the service platform to obtain a detection result; based on the detection result, resources are allocated to the service platform to obtain a resource allocation result.

[0029] Figure 1 The application scenario diagram of the resource allocation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the resource allocation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the resource allocation device provided in the embodiment of the present disclosure can generally be set in the server 105. The resource allocation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the resource allocation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] The following will be based on Figure 1 The scenario described is used to describe the resource allocation method of the disclosed embodiment in detail.

[0037] Figure 2 The flowchart of the resource allocation method according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the resource allocation method of this embodiment includes operations S210 to S240.

[0039] In operation S210, based on the job tasks to be executed, the resource demand of the service platform to be evaluated is evaluated to obtain an evaluation result, which represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job tasks.

[0040] According to an embodiment of the present disclosure, the job task to be executed may be a data processing task, a log management task, etc. executed on the service platform to be evaluated.

[0041] According to an embodiment of the present disclosure, the service platform to be evaluated may be a service platform in a big data service cluster.

[0042] According to an embodiment of the present disclosure, the initial resources of the service platform to be evaluated may be resources allocated to the service platform to be evaluated by a resource pool.

[0043] According to an embodiment of the present disclosure, by evaluating the resource demand of the service platform to be evaluated, the amount of resources required for the service platform to be evaluated to execute the task, ie, the evaluation result, is determined.

[0044] In operation S220, at least one initial resource of the service platform to be evaluated is recycled according to the evaluation result to obtain an updated service platform.

[0045] According to the embodiments of the present disclosure, by recycling the initial resources, the problem of too many idle resources on the service platform can be avoided, so as to maximize the use of resources and avoid waste of resources.

[0046] In operation S230, a resource gap detection is performed on the service platform based on the resource execution data of the service platform to obtain a detection result.

[0047] According to an embodiment of the present disclosure, resource execution data may be data generated by the service platform when executing a job task, including data such as the amount of resources occupied by executing the job task and the amount of idle resources.

[0048] According to an embodiment of the present disclosure, gap detection can be to determine whether there is a resource gap when the current service platform executes a job task, that is, whether there is a difference between the amount of resources required to execute the job task and the amount of resources of the current service platform.

[0049] In operation S240, resources are allocated to the service platform based on the detection result to obtain a resource allocation result.

[0050] According to an embodiment of the present disclosure, resources are allocated to the service platform based on the detection results, which may be resource recovery operations and resource expansion operations, so as to reduce inefficient resources and avoid resource gaps.

[0051] According to the embodiments of the present disclosure, based on the job tasks to be executed, the resource demand of the service platform to be evaluated is evaluated to obtain an evaluation result; based on the evaluation result, at least one initial resource of the service platform to be evaluated is recycled to obtain an updated service platform; based on the resource execution data of the service platform, the service platform is detected for resource gaps to obtain a detection result; based on the detection result, resources are allocated to the service platform to obtain a resource allocation result. By evaluating the resource demand based on the job tasks to be executed, the problem of poor evaluation accuracy caused by manual evaluation in the prior art is avoided, so as to achieve efficient and accurate evaluation of resource demand, thereby enabling reasonable resource recycling of initial resources. At the same time, by detecting resource gaps on the service platform based on the resource execution data of the service platform, the problem of inefficient resources still existing in the service platform in actual applications is avoided, thereby achieving the effect of improving resource utilization.

[0052] According to an embodiment of the present disclosure, the resource execution data of the service platform includes data point utilization rate and resource queue data, the resource queue data includes resource waiting time and idle resource amount of idle resources in the idle resource queue, and resource occupation time and occupied resource amount of occupied resources in the occupied resource queue of the service platform, and the occupied resources are occupied by job tasks; based on the resource execution data of the service platform, the service platform is detected for resource gaps to obtain detection results, including: determining the resource saturation index of the service platform according to the data point utilization rate of the service platform, wherein the data point utilization rate includes the central processing unit utilization rate, the memory utilization rate and the network traffic utilization rate; determining the queue waiting time index according to the resource waiting time of idle resources in the idle resource queue of the service platform; when both the resource saturation index and the queue waiting time index meet the preset index conditions, determining the queue resource allocation result of the service platform based on the cumulative sum between the occupied resource amount of the occupied resource queue and the idle resource amount of the idle resource queue, and the product between the cumulative sum and the resource waiting time of the idle resources; determining the detection result according to the ratio between the queue resource allocation result of the service platform and the resource occupation time of the occupied resource queue.

[0053] According to an embodiment of the present disclosure, after the service platform starts to execute the job task, the data point usage rate of the service platform is collected once a minute to determine the resource saturation index.

[0054] According to an embodiment of the present disclosure, after the service platform starts to execute the job task, the resource waiting time of the idle resources in the idle resource queue of the service platform is collected once a minute. The queue waiting time indicator can be the average resource waiting time of the idle resources in all idle resource queues.

[0055] According to an embodiment of the present disclosure, when the resource saturation index and the queue waiting time index both meet the preset index conditions, for example, when the resource saturation index is greater than 2 and the queue waiting time index is greater than 5 minutes, it is judged that there is a resource gap problem in the service platform, and the resource gap needs to be detected to obtain the detection result.

[0056] According to an embodiment of the present disclosure, resource gap=(the cumulative sum of the occupied resource amount of the occupied resource queue and the idle resource amount of the idle resource queue The average resource waiting time of idle resources of all idle queues) / the average resource occupation time of occupied resource queues of all occupied queues.

[0057] According to the embodiments of the present disclosure, based on the amount of occupied resources and idle resources of the service platform, it is evaluated whether there is still a resource gap in the allocated resources of the current service platform, and the service platform is managed and recycled or expanded according to the calculated resource gap, so as to improve the resource utilization rate of the service platform and reduce inefficient resources.

[0058] According to an embodiment of the present disclosure, a data point usage rate corresponds to a preset operating time period of a service platform; based on the data point usage rate of the service platform, a resource saturation index of the service platform is determined, including: comparing the data point usage rate of the data points of the service platform in the preset operating time period with a preset usage rate threshold to obtain a usage rate comparison result; when the usage rate comparison result indicates that the data point usage rate of the data point exceeds the preset usage rate threshold, the data points in the preset operating time period are determined as saturated data points; when the number of saturated data points exceeds the preset number threshold, the preset operating time period is determined as a resource saturation period; based on the number of resource saturation periods, the resource saturation index of the service platform is determined.

[0059] According to an embodiment of the present disclosure, the preset operating time period of the service platform may be that each hour when the service platform executes a job task is regarded as a preset operating time period.

[0060] According to an embodiment of the present disclosure, the data point usage rate of the service platform in the preset operating time period is compared with the preset usage rate threshold. For example, it can be counted whether the usage rate of each data point exceeds 80%, wherein the preset usage rate threshold can be set according to actual needs and is not limited here.

[0061] According to an embodiment of the present disclosure, when the number of saturated data points exceeds a preset number threshold, it may be, for example, when the utilization rate of each data point mentioned above exceeds 80%, and the number of saturated data points in the i-th preset operating time period exceeds 80%, the i-th preset operating time period is determined as a resource saturation period, wherein the preset number threshold can be set according to actual needs and is not limited here.

[0062] According to an embodiment of the present disclosure, the number of all resource saturation time periods is counted and the number value is used as the resource saturation index of the service platform. For example, among all preset operating time periods, the i-th preset operating time period and the j-th preset operating time period are resource saturation time periods, and the resource saturation index is 2.

[0063] According to the embodiments of the present disclosure, resource requirements include storage resource requirements and computing resource requirements; based on the job tasks to be executed, the resource requirements of the service platform to be evaluated are evaluated to obtain evaluation results, including: based on a preset compression ratio, compressing the business data volume of the job tasks to determine the storage resource requirements; based on the resource value that the job tasks need to occupy, determining the computing resource requirements; determining the evaluation results based on the storage resource requirements and the computing resource requirements.

[0064] According to an embodiment of the present disclosure, the business data volume of the operation task is compressed based on a preset compression ratio, which can be, for example, evaluated at a compression ratio of 6 to 1 for the business data volume. For example, if 1.5G of data needs to be retained for 1 year every day, 1.5G of data needs to be retained for 1 year. 365 / 6=91G storage resource requirement, where the preset compression ratio can be set according to actual needs and is not limited here.

[0065] According to an embodiment of the present disclosure, the computing resource demand may be the average computing resource consumption of a single job task. For example, the resource value required for the job task provided by the service platform to be evaluated may be calculated based on the average resource consumption of all jobs on all service platforms of the big data service cluster (90,000 vCore Seconds) to calculate the required computing resources.

[0066] According to an embodiment of the present disclosure, the storage resource requirement and the computing resource requirement are taken together as the resource amount occupied by the initial resource execution job task of the service platform to be evaluated, that is, the evaluation result.

[0067] According to an embodiment of the present disclosure, the computing resource demand is determined based on the resource value that the job task needs to occupy, including: determining the job duration assigned to the job task according to the job priority of the job task; and determining the first computing resource demand of the service platform to be evaluated based on the ratio of the resource value that the job task needs to occupy and the job duration.

[0068] According to an embodiment of the present disclosure, different job tasks correspond to different job priorities, and different job priorities correspond to different job durations.

[0069] For example, a high-priority job to be completed between 6:00 and 8:00 every day requires 900,000 vCores of computing resources to be used during these 2 hours. Seconds, so at least 125 vCores of the first computing resource requirement must be applied for (900,000 / 3600 / 2 = 125 vCores).

[0070] The low priority job to be completed between 8:00 and 20:00 requires 9,000,000 vCores of computing resources to be used during these 12 hours. Seconds, at least 208 vCores of the first computing resource requirement must be applied for (9,000,000 / 3600 / 12 = 208.33 vCores).

[0071] According to an embodiment of the present disclosure, the computing resource requirement is determined based on the resource value that needs to be consumed by the job task, and the method also includes: determining the second computing resource requirement of the service platform to be evaluated according to the ratio between the resource value that needs to be consumed by the job task and the task execution time of the subtask of the job task, wherein the job task is divided into N batches of subtasks, and N is an integer greater than 1; determining the computing model resources used to execute the job task in the service platform to be evaluated, and the computing model resources include operator computing memory, operator parallel number and computing model parallel number; determining the third computing resource requirement of the service platform to be evaluated according to the product of operator computing memory, operator parallel number and computing model parallel number; determining the computing resource requirement according to the first computing resource requirement, the second computing resource requirement and the third computing resource requirement.

[0072] According to an embodiment of the present disclosure, based on the ratio between the resource value consumed by the job task and the task execution time of the subtasks of the job task, for example, there may be 3 job tasks that are multi-batch jobs, each job task needs to run a total of 12 batches of subtasks from 8:00 to 20:00 every day, and each batch of subtasks is required to be completed within 15 minutes (i.e., the task execution time of the subtask), then at least 300 second computing resource requirements vCore (90,000 3 / 15 / 60=300 vCores).

[0073] According to an embodiment of the present disclosure, the computing model resource may be a Turing computing resource.

[0074] According to the embodiment of the present disclosure, the third computing resource requirement = operator computing memory (maximum 1024G, set according to actual needs) Operator Parallelism Calculate the number of parallel models 70 / 400, e.g. 50 4 5 70 / 400=175vCore.

[0075] According to an embodiment of the present disclosure, the computing resource requirement=max(first computing resource requirement)+second computing resource requirement+third computing resource requirement, that is, max(125, 208)+300+175=683 vCores.

[0076] According to an embodiment of the present disclosure, at least part of the initial resources of the service platform to be evaluated are recycled according to the evaluation results to obtain an updated service platform, including: determining a resource pool associated with the service platform to be evaluated, the resource pool being used to allocate initial resources to the service platform to be evaluated; determining a resource utilization rate according to a ratio between the amount of resources occupied by executing job tasks represented by the evaluation results and the amount of initial resources allocated to the platform to be evaluated; obtaining a utilization rate comparison result based on a comparison between the resource utilization rate and a preset resource utilization rate threshold; executing a preset resource recycling strategy based on the utilization rate comparison result to obtain a resource recovery rate for the initial resources; and recycling the initial resources of the service platform to be evaluated based on the resource recovery rate to obtain an updated service platform.

[0077] According to an embodiment of the present disclosure, the resource utilization rate may be determined based on the ratio between the amount of resources occupied by executing the job task represented by the evaluation result and the amount of initial resources allocated to the platform to be evaluated, for example, based on formula (1).

[0078] (1);

[0079] Among them, Ai is the resource amount of the i-th initial resource, B is the resource amount of the resource pool, and ci is the resource amount occupied by executing the i-th job task i.

[0080] According to an embodiment of the present disclosure, a usage comparison result is obtained based on comparing the resource usage rate with a preset resource usage rate threshold. The resource usage rate can be compared with the preset resource usage rate threshold within a preset comparison period to determine whether the resource usage rate reaches the preset resource usage rate threshold.

[0081] According to an embodiment of the present disclosure, a preset resource recovery strategy is executed based on the usage comparison result to obtain a resource recovery rate for the initial resources, that is, after judging that the resource usage rate reaches a preset resource usage rate threshold, a resource recovery strategy corresponding to the preset resource usage rate threshold is determined to determine the amount of resources that need to be recovered.

[0082] For example, storage resources and computing resources are compared separately.

[0083] For computing resources: if the resource utilization rate is 0% for 3 consecutive months, all resources should be recycled to the resource pool; if the resource utilization rate is lower than 50% for 3 consecutive months, half of the unused resources should be recycled to the resource pool; if the resource utilization rate is lower than 70% for 6 consecutive months, half of the unused resources should be recycled to the resource pool.

[0084] For storage resources: If the resource utilization rate is 0% for 3 consecutive months, all storage resources should be recycled; if the resource utilization rate is lower than 50% for 3 consecutive months, half of the remaining storage resources should be recycled to the resource pool; if the resource utilization rate is lower than 70% for 6 consecutive months, half of the unused resources should be recycled to the resource pool.

[0085] Figure 3 The following is a schematic diagram of a resource allocation method according to an embodiment of the present disclosure.

[0086] The resource allocation method disclosed in the present invention may be as follows: Figure 3 As shown, in operation S310, based on a preset compression ratio, the task data volume of the job task is compressed to determine the storage resource requirement; in operation S320, based on the resource value that the job task needs to occupy, the computing resource requirement is determined. In operation S330, the evaluation result is determined based on the above storage resource requirement and computing resource requirement. In operation S340, at least one initial resource of the service platform to be evaluated is recycled according to the evaluation result to obtain an updated service platform. In operation S350, the resource saturation index of the service platform is determined according to the data point utilization rate of the service platform; in operation S360, the queue waiting time index is determined according to the resource waiting time of the idle resources in the idle resource queue of the service platform. In operation S370, based on the above resource saturation index and queue waiting time index, the service platform is detected for a gap to obtain an inspection result. In operation S380, based on the detection result, resources are allocated to the service platform to obtain a resource allocation result.

[0087] Based on the above resource allocation method, the present disclosure also provides a resource allocation device. Figure 4 The device is described in detail.

[0088] Figure 4 The structure block diagram of the resource allocation device according to the embodiment of the present disclosure is schematically shown.

[0089] like Figure 4 As shown, the resource allocation device 400 of this embodiment includes an evaluation module 410 , a recovery module 420 , a detection module 430 and an allocation module 440 .

[0090] The evaluation module 410 is used to evaluate the resource demand of the service platform to be evaluated based on the job task to be executed, and obtain an evaluation result, which represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task. In one embodiment, the evaluation module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0091] The recycling module 420 is used to recycle at least one initial resource of the service platform to be evaluated according to the evaluation result to obtain an updated service platform. In one embodiment, the recycling module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0092] The detection module 430 is used to perform resource gap detection on the service platform based on the resource execution data of the service platform to obtain a detection result. In one embodiment, the detection module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0093] The allocation module 440 is used to allocate resources to the service platform based on the detection result to obtain a resource allocation result. In one embodiment, the allocation module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0094] According to the embodiments of the present disclosure, based on the job tasks to be executed, the resource demand of the service platform to be evaluated is evaluated to obtain an evaluation result; based on the evaluation result, at least one initial resource of the service platform to be evaluated is recycled to obtain an updated service platform; based on the resource execution data of the service platform, the service platform is detected for resource gaps to obtain a detection result; based on the detection result, resources are allocated to the service platform to obtain a resource allocation result. By evaluating the resource demand based on the job tasks to be executed, the problem of poor evaluation accuracy caused by manual evaluation in the prior art is avoided, so as to achieve efficient and accurate evaluation of resource demand, thereby enabling reasonable resource recycling of initial resources. At the same time, by detecting resource gaps on the service platform based on the resource execution data of the service platform, the problem of inefficient resources still existing in the service platform in actual applications is avoided, thereby achieving the effect of improving resource utilization.

[0095] According to an embodiment of the present disclosure, any multiple modules of the evaluation module 410, the recycling module 420, the detection module 430 and the allocation module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the evaluation module 410, the recycling module 420, the detection module 430 and the allocation module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the evaluation module 410, the recycling module 420, the detection module 430 and the allocation module 440 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0096] According to an embodiment of the present disclosure, the detection module 430 includes a first determination submodule, a second determination submodule, a third determination submodule, and a fourth determination submodule.

[0097] The first determination submodule is used to determine the resource saturation index of the service platform according to the data point utilization rate of the service platform, wherein the data point utilization rate includes the central processing unit utilization rate, the memory utilization rate and the network traffic utilization rate.

[0098] The second determining submodule is used to determine a queue waiting time indicator according to the resource waiting time of the idle resources in the idle resource queue in the service platform.

[0099] The third determination submodule is used to determine the queue resource allocation result of the service platform based on the cumulative sum of the occupied resources of the occupied resource queue and the idle resources of the idle resource queue, and the product of the cumulative sum and the resource waiting time of the idle resources, when both the resource saturation index and the queue waiting time index meet the preset index conditions.

[0100] The fourth determination submodule is used to determine the detection result according to the ratio between the queue resource allocation result of the service platform and the resource occupation time of the occupied resource queue.

[0101] According to an embodiment of the present disclosure, the first determining submodule includes a first comparing unit, a first determining unit, a second determining unit, and a third determining unit.

[0102] The first comparison unit is used to compare the data point usage rate of the data point of the service platform in the preset operation period with the preset usage rate threshold to obtain a usage rate comparison result.

[0103] The first determining unit is configured to determine the data point in the preset running time period as a saturated data point when the usage rate comparison result indicates that the usage rate of the data point exceeds a preset usage rate threshold.

[0104] The second determining unit is configured to determine a preset operating time period as a resource saturation time period when the number of saturated data points exceeds a preset number threshold.

[0105] The third determining unit is used to determine the resource saturation index of the service platform according to the number of resource saturation time periods.

[0106] According to an embodiment of the present disclosure, the evaluation module 410 includes a fifth determination submodule, a sixth determination submodule, and a seventh determination submodule.

[0107] The fifth determination submodule is used to compress the business data volume of the operation task based on a preset compression ratio to determine the storage resource requirement.

[0108] The sixth determination submodule is used to determine the computing resource requirement based on the resource value that the job task needs to occupy.

[0109] The seventh determination submodule is used to determine the evaluation result according to the storage resource requirement and the computing resource requirement.

[0110] According to an embodiment of the present disclosure, the sixth determining submodule includes a fourth determining unit and a fifth determining unit.

[0111] The fourth determining unit is used to determine the job duration allocated to the job task according to the job priority of the job task.

[0112] The fifth determining unit is used to determine the first computing resource requirement of the service platform to be evaluated according to the ratio of the resource value required to be occupied by the job task and the job duration.

[0113] According to an embodiment of the present disclosure, the sixth determining submodule further includes a sixth determining unit, a seventh determining unit, an eighth determining unit and a ninth determining unit.

[0114] The sixth determination unit is used to determine the second computing resource requirement of the service platform to be evaluated based on the ratio between the resource value required to be consumed by the job task and the task execution time of the subtask of the job task, wherein the job task is divided into N batches of subtasks, and N is an integer greater than 1.

[0115] The seventh determination unit is used to determine the computing model resources used to execute the job task in the service platform to be evaluated, and the computing model resources include operator computing memory, operator parallel number and computing model parallel number.

[0116] The eighth determination unit is used to determine the third computing resource requirement of the service platform to be evaluated according to the product of the operator computing memory, the operator parallel number and the computing model parallel number.

[0117] The ninth determining unit is used to determine the computing resource requirement according to the first computing resource requirement, the second computing resource requirement and the third computing resource requirement.

[0118] According to an embodiment of the present disclosure, the recovery module 420 includes an eighth determining submodule, a ninth determining submodule, a first obtaining submodule, a second obtaining submodule, and a third obtaining submodule.

[0119] The eighth determination submodule is used to determine a resource pool associated with the service platform to be evaluated, where the resource pool is used to allocate initial resources to the service platform to be evaluated.

[0120] The ninth determination submodule is used to determine the resource utilization rate according to the ratio between the amount of resources occupied by executing the job task represented by the evaluation result and the amount of initial resources allocated to the platform to be evaluated.

[0121] The first obtaining submodule is used to obtain a usage comparison result based on the comparison between the resource usage rate and a preset resource usage rate threshold.

[0122] The second obtaining submodule is used to execute a preset resource recovery strategy based on the usage rate comparison result to obtain a resource recovery rate for the initial resources.

[0123] The third submodule is used to recycle the initial resources of the service platform to be evaluated based on the resource recovery rate to obtain an updated service platform.

[0124] Figure 5 A block diagram of an electronic device suitable for implementing a resource allocation method according to an embodiment of the present disclosure is schematically shown.

[0125] like Figure 5As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0126] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0127] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0128] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0129] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0130] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the resource allocation method provided by the embodiment of the present disclosure.

[0131] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0132] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0134] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0135] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0137] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A resource allocation method, characterized in that: The method comprises: Based on the job task to be executed, the resource demand of the service platform to be evaluated is evaluated to obtain an evaluation result, wherein the evaluation result represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task; Reclaim at least one initial resource of the service platform to be evaluated according to the evaluation result to obtain an updated service platform; Performing resource gap detection on the service platform based on the resource execution data of the service platform to obtain a detection result; Based on the detection result, resources are allocated to the service platform to obtain a resource allocation result.

2. The method according to claim 1, characterized in that The resource execution data of the service platform includes data point usage rate and resource queue data, wherein the resource queue data includes resource waiting time and idle resource amount of idle resources in an idle resource queue, and resource occupation time and occupied resource amount of occupied resources in an occupied resource queue of the service platform, wherein the occupied resources are occupied by the job task; The performing resource gap detection on the service platform based on the resource execution data of the service platform to obtain the detection result includes: Determining a resource saturation index of the service platform according to a data point utilization rate of the service platform, wherein the data point utilization rate includes a CPU utilization rate, a memory utilization rate, and a network traffic utilization rate; Determine a queue waiting time indicator according to the resource waiting time of the idle resources in the idle resource queue in the service platform; In the case where both the resource saturation index and the queue waiting time index meet the preset index conditions, based on the cumulative sum of the occupied resource amount of the occupied resource queue and the idle resource amount of the idle resource queue, and the product of the cumulative sum and the resource waiting time of the idle resources, determine the queue resource allocation result of the service platform; The detection result is determined according to the ratio between the queue resource allocation result of the service platform and the resource occupation time of the occupied resource queue.

3. The method according to claim 2, characterized in that The data point usage rate corresponds to a preset operation time period of the service platform; Determining the resource saturation index of the service platform according to the data point utilization rate of the service platform includes: Comparing the usage rate of the data points of the service platform in the preset operation period with a preset usage rate threshold to obtain a usage rate comparison result; If the usage rate comparison result indicates that the data point usage rate of the data point exceeds the preset usage rate threshold, determining the data point in the preset operating time period as a saturated data point; When the number of the saturated data points exceeds a preset number threshold, determining the preset operating time period as a resource saturation period; A resource saturation index of the service platform is determined according to the number of the resource saturation time periods.

4. The method according to claim 1, characterized in that: The resource requirements include storage resource requirements and computing resource requirements; The resource demand of the service platform to be evaluated is evaluated based on the job task to be executed, and the evaluation result is obtained, including: Based on a preset compression ratio, compress the business data volume of the job task to determine the storage resource requirement; Determine the computing resource requirement based on the resource value that the job task needs to occupy; The evaluation result is determined according to the storage resource requirement and the computing resource requirement.

5. The method according to claim 4, characterized in that The determining the computing resource requirement based on the resource value required to be occupied by the job task includes: Determine the duration of the operation assigned to the operation task according to the operation priority of the operation task; The first computing resource requirement of the service platform to be evaluated is determined according to the ratio of the resource value required to be occupied by the job task to the job duration.

6. The method according to claim 5, characterized in that The determining the computing resource requirement based on the resource value to be consumed by the job task further includes: Determine the second computing resource requirement of the service platform to be evaluated according to the ratio between the resource value required to be consumed by the job task and the task execution time of the subtask of the job task, wherein the job task is divided into N batches of subtasks, where N is an integer greater than 1; Determine the computing model resources for executing the job task in the service platform to be evaluated, wherein the computing model resources include operator computing memory, operator parallelism, and computing model parallelism; Determine the third computing resource requirement of the service platform to be evaluated according to the product of the operator computing memory, the operator parallel number and the computing model parallel number; The computing resource requirement is determined according to the first computing resource requirement, the second computing resource requirement, and the third computing resource requirement.

7. The method according to claim 1, characterized in that The step of recovering at least part of the initial resources of the service platform to be evaluated according to the evaluation result to obtain an updated service platform includes: Determining a resource pool associated with the service platform to be evaluated, wherein the resource pool is used to allocate the initial resources to the service platform to be evaluated; Determine the resource utilization rate according to the ratio between the amount of resources occupied by executing the job task represented by the evaluation result and the amount of initial resources allocated to the platform to be evaluated; Comparing the resource usage rate with a preset resource usage rate threshold, obtaining a usage rate comparison result; Executing a preset resource recovery strategy based on the usage rate comparison result to obtain a resource recovery rate for the initial resources; and The initial resources of the service platform to be evaluated are recycled based on the resource recycling rate to obtain the updated service platform.

8. A resource allocation device, characterized in that: The device comprises: An evaluation module, used to evaluate the resource demand of the service platform to be evaluated based on the job task to be executed, and obtain an evaluation result, wherein the evaluation result represents the amount of resources occupied by the initial resources of the service platform to be evaluated to execute the job task; A recycling module, configured to recycle at least one initial resource of the service platform to be evaluated according to the evaluation result to obtain an updated service platform; a detection module, configured to perform resource gap detection on the service platform based on resource execution data of the service platform to obtain a detection result; and The allocation module is used to allocate resources to the service platform based on the detection result to obtain a resource allocation result.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images