Method for adjusting computing resources, electronic equipment and storage medium

Abstract

The invention provides a method for adjusting computing resources, which comprises the following steps of: obtaining a plurality of initial task requests, obtaining a plurality of candidate task requests based on the initial task requests, obtaining a plurality of intermediate task requests after all allocable computing resources of a resource allocation component YARN are applied, and sending the intermediate task requests to a resource allocation component YARN; according to the plurality of candidate task requests and the plurality of intermediate task requests, determining candidate computing resources corresponding to each intermediate task request, and calling the candidate computing resources to execute the intermediate task requests so as to realize scheduling of the computing resources; according to the method and the device, when the server does not apply for the computing resources, the requested computing resources of the server do not need to be released, so that the task execution efficiency is improved, meanwhile, the situation that the computing resources are called but not used is reduced, and the task execution efficiency is improved. And the utilization rate of resources is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computing resource processing, and in particular to a method for adjusting computing resources, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of science and technology and the widespread application of artificial intelligence technology, various application scenarios of artificial intelligence, whether it is complex model training, large-scale data processing, or task execution, require powerful computing resources as support. In modern computer systems, processing various types of user requests or data analysis tasks through server clusters is a common operation mode. When a server needs to perform multiple tasks simultaneously, the demand for computing resources is increasing. Therefore, dynamically allocating computing resources according to the actual load of the cluster to improve the computing resource utilization of the machine is a current hot research direction.

[0003] The method of adjusting computing resources in the existing technology is that when there are no remaining available resources after a server's computing resources are applied for, when a new task requires computing resources to execute, this task needs to wait for the server's computing resources to be released before applying for computing resources. It can be seen that this may easily cause the waiting time to be too long, reducing the efficiency of task execution. At the same time, there is a situation where computing resources are called but not used, which may easily cause waste of resources. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention adopts a technical solution: a method for adjusting computing resources, comprising the following steps:

[0005] S100: Acquire several initial task requests, wherein the initial task requests are Internet-related task requests uploaded by users.

[0006] S200 , based on the initial task request, obtaining a plurality of candidate task requests, wherein the candidate task requests are initial task requests that are passed by submitting a computing resource application request to a resource allocation component YARN.

[0007] S300, when all the allocatable computing resources of the resource allocation component YARN have been applied for, obtain several intermediate task requests, wherein the intermediate task requests are initial task requests that are rejected when submitting computing resource application requests to the resource allocation component YARN because all the allocatable computing resources of the resource allocation component YARN have been applied for.

[0008] S400 , according to a plurality of candidate task requests and a plurality of intermediate task requests, determining a candidate computing resource corresponding to each intermediate task request and calling the candidate computing resource to execute the intermediate task request to implement scheduling of computing resources.

[0009] S500 , based on the candidate task request status corresponding to the candidate computing resources, releasing the candidate computing resources used for the intermediate task request.

[0010] The present invention protects an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for adjusting computing resources when executing the computer program.

[0011] The present invention protects a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for adjusting computing resources is implemented.

[0012] The present invention has at least the following beneficial effects: the present invention is a method for adjusting computing resources, comprising the following steps: obtaining a number of initial task requests, obtaining a number of candidate task requests based on the initial task requests, obtaining a number of intermediate task requests when all allocatable computing resources of the resource allocation component YARN have been applied for, determining the candidate computing resources corresponding to each intermediate task request based on the number of candidate task requests and the number of intermediate task requests, and calling the candidate computing resources to execute the intermediate task requests to implement the scheduling of computing resources, releasing the candidate computing resources for use by the intermediate task requests based on the candidate task requests corresponding to the candidate computing resources. When the server has no computing resources to apply for, the present invention does not need to wait for the computing resources applied for by the server to be released, thereby improving the efficiency of task execution. At the same time, it reduces the situation where computing resources are called but not used, thereby improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of a method for adjusting computing resources provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] Example

[0017] An embodiment of the present invention provides a method for adjusting computing resources, the method comprising the following steps: Figure 1 As shown:

[0018] S100: Acquire several initial task requests, wherein the initial task requests are Internet-related task requests uploaded by users.

[0019] S200 , based on the initial task request, obtaining a plurality of candidate task requests, wherein the candidate task requests are initial task requests that are passed by submitting a computing resource application request to a resource allocation component YARN.

[0020] Specifically, when the initial computing resource requirement corresponding to the initial task request is G, computing resources of G are requested from the resource allocation component YARN, where G is the initial computing resource requirement corresponding to the initial task request.

[0021] Furthermore, the initial computing resource requirement is the size of the required initial computing resources, wherein the initial computing resources are the computing resources required to execute the initial task request.

[0022] S300, when all the allocatable computing resources of the resource allocation component YARN have been applied for, obtain several intermediate task requests, wherein the intermediate task requests are initial task requests that are rejected when submitting computing resource application requests to the resource allocation component YARN because all the allocatable computing resources of the resource allocation component YARN have been applied for.

[0023] S400 , according to a plurality of candidate task requests and a plurality of intermediate task requests, determining a candidate computing resource corresponding to each intermediate task request and calling the candidate computing resource to execute the intermediate task request to implement scheduling of computing resources.

[0024] Specifically, S400 also includes the following steps:

[0025] S401, obtain candidate task request list B = {B1, ..., B i ,……,B n}, B i is the i-th candidate task request, i=1...n, and n is the number of candidate task requests in the candidate task request list.

[0026] S402, according to B, determine the designated task request list D = {D1, ..., D r ,……,D s}, D ris the rth specified task request, r=1…s, s is the number of specified task requests in the specified task request list.

[0027] Specifically, S402 also includes the following steps:

[0028] S4021, obtain the candidate computing resource utilization list Y corresponding to B = {Y1, ..., Y i ,……,Y n}, Y i B i The corresponding candidate computing resource utilization rate is the ratio of the computing resource occupied by the candidate task request from the start of execution to the tth moment to the initial computing resource demand.

[0029] Specifically, t is a preset time span threshold, wherein those skilled in the art know that t can be selected according to actual needs, all of which fall within the protection scope of the present invention and will not be described in detail here.

[0030] S4022, determine D based on Y, where Y i ≤Y 0 When B i For a specified task request, Y 0 The default utilization threshold.

[0031] Specifically, Y 0 The value range of Y is 0.5 to 0.8, wherein those skilled in the art know that Y can be adjusted according to actual needs. 0 The selection of , all fall within the protection scope of the present invention, and will not be described in detail here.

[0032] S403, obtain the intermediate computing resource demand list U corresponding to the intermediate task request list = {U1, ..., U j ,……,U m}, U j The intermediate computing resource demand corresponding to the j-th intermediate task request, j=1...m, m is the number of intermediate task requests in the intermediate task request list, wherein the intermediate computing resource demand is the size of the computing resources required from the start of executing the intermediate task request to the t-th moment.

[0033] S404, obtain the specified computing resource utilization list X corresponding to D = {X1, ..., X r ,……,X s}, X r D r The corresponding specified computing resource utilization, wherein the specified computing resource utilization is the candidate computing resource utilization corresponding to the specified task request obtained from Y.

[0034] S405: According to U and X, determine the candidate computing resources corresponding to each intermediate task request and call the candidate computing resources to execute the intermediate task request to realize the scheduling of computing resources. j ≤1-X r When D r The corresponding initial computing resource is the candidate computing resource corresponding to the jth intermediate task request and calls D r The corresponding computing resource specifies the jth intermediate task request.

[0035] From the above content, we can see that based on the occupancy of computing resources in the server, when the server has no computing resources to apply for, there is no need to wait for the computing resources applied for by the server to be released. The computing resource space that has been applied for can be called to perform related tasks, which improves the efficiency of task execution and is less likely to cause waste of resources.

[0036] S500 , based on the candidate task request status corresponding to the candidate computing resources, releasing the candidate computing resources used for the intermediate task request.

[0037] Specifically, S500 also includes the following steps:

[0038] S501, obtain any candidate computing resource E 0 And real-time E 0 The corresponding candidate task request execution process is detected.

[0039] S502, when QE T >QE 0 T When E 0 The corresponding target task request is released at time T to execute E 0 The corresponding candidate task request, where QE T For E 0 The corresponding candidate task request's demand for computing resources at time T, QE 0 T is the candidate computing resource E at time T 0 The idle computing resources in , T is a moment in the process that the candidate task requests to execute.

[0040] Specifically, S502 also includes the following steps:

[0041] S1, get E 0 The corresponding target occupancy state is a state in which the candidate computing resource is occupied by the candidate task request and the intermediate task request.

[0042] S2, according to the target occupancy status, obtain the target quantity, wherein the target quantity is E at time T0 The number of intermediate task requests executed in .

[0043] S3, when the target number is 1, the computing resources that execute the intermediate task request in the candidate computing resources are released to execute E 0 The corresponding candidate task request.

[0044] S4, when the target quantity is not 1, obtain the candidate computing resource E 0 The corresponding final task request list EA={EA1,……,EA e ,……,EA f} and the candidate priority list FA corresponding to EA = {FA1, ..., FA e ,……,FA f}, EA e is the e-th final task request, e=1……f, f is the number of final task requests in the final task request list, FA e For EA e The corresponding priority to be selected, wherein the final task request is the candidate computing resource E 0 The intermediate task request executed at time T, the priority to be selected is the completion rate of the final task request at time T.

[0045] S5, when FA e >0 and FA e When it is the smallest candidate priority in FA, get FA e Corresponding EA e The key computing resource occupancy P e , where the key computing resource occupancy is the size of the candidate computing resources occupied at time T.

[0046] S6, when P e ≥QE T At time T, EA e The candidate computing resources occupied by the corresponding final task request are released to execute E 0 The corresponding candidate task request.

[0047] S7, when P e <QE T When FA is selected, FA is excluded. e The smallest candidate priority among the two other candidates is used to obtain the sum of the key computing resource occupancy of the final task requests corresponding to these two candidate priorities.

[0048] S8, repeat steps S6 to S8 until the total amount of key computing resources acquired is not less than QE T , and release the candidate computing resources occupied by the final task request to execute E0 The corresponding candidate task request.

[0049] From the above, we can see that when the server has no computing resources to apply for, there is no need to wait for the server's applied computing resources to be released, which improves the efficiency of task execution. At the same time, it reduces the situation where computing resources are called but not used, thereby improving resource utilization.

[0050] A method for adjusting computing resources provided by an embodiment of the present invention includes the following steps: obtaining a number of initial task requests, obtaining a number of candidate task requests based on the initial task requests, obtaining a number of intermediate task requests when all allocatable computing resources of the resource allocation component YARN have been applied for, determining the candidate computing resources corresponding to each intermediate task request based on the number of candidate task requests and the number of intermediate task requests, and calling the candidate computing resources to execute the intermediate task requests to realize the scheduling of computing resources, and releasing the candidate computing resources for use by the intermediate task requests based on the candidate task requests corresponding to the candidate computing resources. When the server has no computing resources to apply for, the present invention does not need to wait for the computing resources applied for by the server to be released, thereby improving the efficiency of task execution. At the same time, it reduces the situation where computing resources are called but not used, thereby improving resource utilization.

[0051] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which can be provided in an electronic device to store at least one readable instruction or at least one program for implementing one of the methods in the method embodiment. The at least one readable instruction or the at least one program is loaded and executed by the processor to implement:

[0052] S100: Acquire several initial task requests, wherein the initial task requests are Internet-related task requests uploaded by users.

[0053] S200 , based on the initial task request, obtaining a plurality of candidate task requests, wherein the candidate task requests are initial task requests that are passed by submitting a computing resource application request to a resource allocation component YARN.

[0054] S300, when all the allocatable computing resources of the resource allocation component YARN have been applied for, obtain several intermediate task requests, wherein the intermediate task requests are initial task requests that are rejected when submitting computing resource application requests to the resource allocation component YARN because all the allocatable computing resources of the resource allocation component YARN have been applied for.

[0055] S400 , according to a plurality of candidate task requests and a plurality of intermediate task requests, determining a candidate computing resource corresponding to each intermediate task request and calling the candidate computing resource to execute the intermediate task request to implement scheduling of computing resources.

[0056] S500 , based on the candidate task request status corresponding to the candidate computing resources, releasing the candidate computing resources used for the intermediate task request.

[0057] An embodiment of the present invention further provides an electronic device, including a processor and the non-transitory computer-readable storage medium. The storage medium may be provided in the electronic device to store at least one readable instruction or at least one program for implementing one of the methods in the method embodiment. The at least one readable instruction or the at least one program is loaded and executed by the processor to implement:

[0058] S100: Acquire several initial task requests, wherein the initial task requests are Internet-related task requests uploaded by users.

[0059] S200 , based on the initial task request, obtaining a plurality of candidate task requests, wherein the candidate task requests are initial task requests that are passed by submitting a computing resource application request to a resource allocation component YARN.

[0060] S300, when all the allocatable computing resources of the resource allocation component YARN have been applied for, obtain several intermediate task requests, wherein the intermediate task requests are initial task requests that are rejected when submitting computing resource application requests to the resource allocation component YARN because all the allocatable computing resources of the resource allocation component YARN have been applied for.

[0061] S400 , according to a plurality of candidate task requests and a plurality of intermediate task requests, determining a candidate computing resource corresponding to each intermediate task request and calling the candidate computing resource to execute the intermediate task request to implement scheduling of computing resources.

[0062] S500 , based on the candidate task request status corresponding to the candidate computing resources, releasing the candidate computing resources used for the intermediate task request.

[0063] Although some specific embodiments of the present invention have been described in detail by way of example, it will be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for adjusting computing resources, characterized in that: The method comprises the following steps: S100, obtaining a plurality of initial task requests, wherein the initial task requests are Internet-related task requests uploaded by users; S200, based on the initial task request, obtaining a plurality of candidate task requests, wherein the candidate task requests are initial task requests that have been submitted to the resource allocation component YARN and passed the computing resource application request; S300, after all allocatable computing resources of the resource allocation component YARN have been applied for, obtaining several intermediate task requests, wherein the intermediate task requests are initial task requests that were rejected when submitting computing resource application requests to the resource allocation component YARN because all allocatable computing resources of the resource allocation component YARN have been applied for; S400, based on a plurality of candidate task requests and a plurality of intermediate task requests, determining a candidate computing resource corresponding to each intermediate task request and calling the candidate computing resource to execute the intermediate task request to implement scheduling of the computing resource; S500 , based on the candidate task request status corresponding to the candidate computing resources, releasing the candidate computing resources used for the intermediate task request.

2. The method for adjusting computing resources according to claim 1, characterized in that: When the initial computing resource requirement corresponding to the initial task request is G, apply for computing resources of G from the resource allocation component YARN, where G is the initial computing resource requirement corresponding to the initial task request.

3. The method for adjusting computing resources according to claim 2, characterized in that: The initial computing resource requirement is the size of the required initial computing resources, wherein the initial computing resources are the computing resources required when executing the initial task request.

4. The method for adjusting computing resources according to claim 1, wherein: S400 also includes the following steps: S401, obtain candidate task request list B = {B1, ..., B i ,……,B n }, B i is the i-th candidate task request, i=1…n, n is the number of candidate task requests in the candidate task request list; S402, according to B, determine the designated task request list D = {D1, ..., D r ,……,D s }, D r is the rth specified task request, r=1…s, s is the number of specified task requests in the specified task request list; S403, obtain the intermediate computing resource demand list U corresponding to the intermediate task request list = {U1, ..., U j ,……,U m },U j is the intermediate computing resource requirement corresponding to the jth intermediate task request, j=1…m, m is the number of intermediate task requests in the intermediate task request list, wherein the intermediate computing resource requirement is the amount of computing resources required from the start of executing the intermediate task request to the tth time; S404, obtain the specified computing resource utilization list X corresponding to D = {X1, ..., X r ,……,X s }, X r D r The corresponding specified computing resource utilization, wherein the specified computing resource utilization is the candidate computing resource utilization corresponding to the specified task request obtained from Y; S405: According to U and X, determine the candidate computing resources corresponding to each intermediate task request and call the candidate computing resources to execute the intermediate task request to realize the scheduling of computing resources. j ≤1-X r When D r The corresponding initial computing resource is the candidate computing resource corresponding to the jth intermediate task request and calls D r The corresponding computing resource specifies the jth intermediate task request.

5. The method for adjusting computing resources according to claim 4, characterized in that: S402 also includes the following steps: S4021, obtain the candidate computing resource utilization list Y corresponding to B = {Y1, ..., Y i ,……,Y n }, Y i B i The corresponding candidate computing resource utilization rate, where the candidate resource utilization rate is the ratio of the computing resources occupied by the candidate task request from the start of execution to the tth time to the initial computing resource demand; S4022, determine D based on Y, where Y i ≤Y 0 When B i For a specified task request, Y 0 The default utilization threshold.

6. The method for adjusting computing resources according to claim 5, characterized in that: Y 0 The value range is 0.5 to 0.

8.

7. The method for adjusting computing resources according to claim 1, wherein: S500 also includes the following steps: S501, obtain any candidate computing resource E 0 And real-time E 0 The corresponding candidate task request execution process is detected. S502, when QE T >QE 0 T When E 0 The corresponding target task request is released at time T to execute E 0 The corresponding candidate task request, where QE T For E 0 The corresponding candidate task request's demand for computing resources at time T, QE 0 T is the candidate computing resource E at time T 0 The idle computing resources in , T is a moment in the process that the candidate task requests to execute.

8. A non-transitory computer-readable storage medium, characterized in that The storage medium stores at least one readable instruction or at least one program, and the at least one readable instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: The device comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images