Method, device, computer equipment and storage medium for allocating YARN computing resources

Abstract

The present invention discloses a method, apparatus, computer equipment, and storage medium for allocating Yarn computing resources. The method includes periodically obtaining the current number of tasks submitted by a data aggregation queue and the current number of tasks submitted by a data scheduling queue, and calculating a current load factor; setting a corresponding current priority factor based on the task priorities in the data aggregation queue and the data scheduling queue; and updating the current Yarn computing resource allocation ratio of the data aggregation queue and the data scheduling queue based on the current load factor and the current priority factor. The present invention proposes the concepts of load factor and priority factor. Under the premise of satisfying the scheduler principle, the maximum proportion of adaptive allocation of different queues can be calculated according to this method, which has the advantage of more reasonable resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of emerging information technology, and in particular to a method, apparatus, computer equipment, and storage medium for allocating YARN computing resources. Background Art

[0002] Traditional capacity schedulers allocate a fixed YARN computing resource ratio (Configured Capacity) to the data aggregation queue (data_merge) and data scheduling queue (data_dev) based on prior experience. This fixed ratio fails to fully account for the serial nature of data aggregation and data scheduling, as well as the isolated timing characteristics of some data aggregation operations. Consequently, it fails to maximize resource utilization between the two queues at the appropriate time. Summary of the Invention

[0003] The purpose of the present invention is to provide a method, apparatus, computer equipment and storage medium for allocating YARN computing resources, aiming to solve the problem that existing schedulers cannot achieve the maximum resource utilization ratio for each queue at the appropriate time point.

[0004] To solve the above technical problems, the present invention aims to achieve the following technical solutions: providing a method for allocating YARN computing resources, comprising:

[0005] Regularly obtain the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculate the current load factor;

[0006] According to the task priorities in the data aggregation queue and the data scheduling queue, setting corresponding current priority factors;

[0007] According to the current load factor and the current priority factor, the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue is updated.

[0008] In addition, the technical problem to be solved by the present invention is to provide a device for allocating YARN computing resources, comprising:

[0009] The load factor calculation unit is used to periodically obtain the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculate the current load factor;

[0010] a priority factor setting unit, configured to set a corresponding current priority factor according to the task priorities in the data aggregation queue and the data scheduling queue;

[0011] The resource allocation updating unit is configured to update the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor.

[0012] In addition, an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the method for allocating Yarn computing resources described in the first aspect above.

[0013] In addition, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method for allocating Yarn computing resources described in the first aspect above.

[0014] The embodiment of the present invention discloses a method, apparatus, computer equipment and storage medium for allocating Yarn computing resources. The method includes periodically obtaining the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculating the current load factor; setting the corresponding current priority factor according to the task priorities in the data aggregation queue and the data scheduling queue; and updating the current Yarn computing resource ratio of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor. The embodiment of the present invention proposes the concepts of load factor and priority factor. Under the premise of satisfying the scheduler principle, the maximum proportion of adaptive allocation of different queues can be calculated according to this method, which has the advantage of more reasonable resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a process for allocating YARN computing resources provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a sub-flow diagram of step S101 provided in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a sub-flow diagram of step S202 provided in an embodiment of the present invention;

[0019] Figure 4 A schematic block diagram of a device for YARN computing resources provided by an embodiment of the present invention;

[0020] Figure 5 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0025] See also Figure 1 , Figure 1 A schematic diagram of a process for allocating YARN computing resources provided by an embodiment of the present invention;

[0026] like Figure 1 As shown, the method includes steps S101 to S103.

[0027] S101, regularly obtaining the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculating the current load factor;

[0028] S102, setting corresponding current priority factors according to the task priorities in the data aggregation queue and the data scheduling queue;

[0029] S103: Update the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor.

[0030] To facilitate understanding of the invention, we first introduce the data aggregation queue and the data scheduling queue. The data aggregation queue and the data scheduling queue are two major links in the ETL process, belonging to two different tenant queues, and the tasks meet the serial order of data aggregation first and then data scheduling. That is to say, the daily scheduling task will automatically trigger the corresponding data scheduling queue task only after the task of the data aggregation queue is completed; when these two queues execute the corresponding tasks, the Capacity scheduler allocates Yarn resources. The allocation ratio of Yarn resources is the key to the efficiency of the queue execution task. Ideally, the application's request for Yarn resources should be met immediately, but in reality, resources are often limited, especially in a very busy cluster. A request for application resources often needs to wait for a period of time before the corresponding resources are obtained; based on this, the present invention proposes a method for dynamically allocating Yarn resources.

[0031] Specifically, the present invention proposes the concepts of load factor and priority factor. On the premise of meeting the Capacity scheduler principle, the maximum proportion of adaptive allocation of data aggregation queues and data scheduling queues can be calculated according to this method, and the Yarn computing resources can be refreshed regularly using the Yarn interface without causing any business failures. It is suitable for dynamic allocation of resource proportions of data aggregation queues and data scheduling queues, and is an optimization strategy for adaptive allocation of Yarn resources in serialized scenarios. It realizes the allocation of reasonable resources according to actual business needs, and has the effect of shortening scheduling time and improving production efficiency.

[0032] Specifically, the current YARN computing resource ratio between the data aggregation queue and the data scheduling queue is calculated using the following formula:

[0033] x+y=A(x∈[1%,A%), y∈[1%,A%));

[0034] x / y=α*β(α=3.0);

[0035]

[0036] Among them, x represents the current Yarn computing resource ratio of the data aggregation queue, y represents the current Yarn computing resource ratio of the data scheduling queue, A represents the total resource ratio that can be allocated to x and y in the entire cluster environment after excluding the Yarn computing resources reserved by other queues. The value range of A is not greater than 100%, β represents the current load factor, α represents the current priority factor, and β represents the current load factor. x Indicates the current number of tasks in the data aggregation queue, β y Indicates the current number of tasks in the data scheduling queue.

[0037] In this embodiment, the total of the entire Yarn computing resources is 100%. After deducting the resources pre-occupied by other queues, 90% can be set for the data aggregation queue and the data scheduling queue, that is, A is preferably 90%, which can be selected according to actual conditions; the current priority factor α is a preset value set according to the importance of the business, which can be set according to the preset business level standard or manually set. The current priority factor α represents the priority of data scheduling using Yarn computing resources, which can be tested and set according to actual conditions. Since the data aggregation queue has a higher business priority than the data scheduling queue, the initial preferred value here is 3. The larger the value, the more the calculated allocation ratio of Yarn computing resources is biased towards the data aggregation queue. Based on this, when the number of current tasks submitted by the data aggregation queue is larger, the current priority factor α can be increased.

[0038] Specifically, A, β, α, β x and β y Substituting the value of into the above formula, we can calculate the current YARN computing resource ratio x of the data aggregation queue and the current YARN computing resource ratio y of the data scheduling queue. Specifically, when obtaining the current number of tasks of the data aggregation queue and the current number of tasks of the data scheduling queue, if the current number of tasks of the data aggregation queue is 0, the current YARN computing resource ratio of the data aggregation queue is set to the preset ratio. Similarly, if the current number of tasks of the data scheduling queue is 0, the current YARN computing resource ratio of the data scheduling queue is set to the preset ratio. The preset ratio here is preferably 1%, which is used to maintain the minimum computing standby requirement.

[0039] In one embodiment, if Figure 2 As shown, step S101 includes:

[0040] S201, according to the start time of different tasks, the timing script starts the corresponding task according to the start time and moves the task to the data aggregation queue;

[0041] This step is used to periodically count the current number of tasks submitted to Yarn by the data aggregation queue. Each task has a scheduled start time. When the corresponding task start time arrives, the scheduled script starts the task and then moves the task to the data aggregation queue. For example, the data aggregation queue of the day has two tasks (a and b) in time series. Task a is configured to start at 0:30 every day, and task b is configured to start at 1:30 every day. Then, when the Yarn interface command is used between 0:30 and 1:30, the number of tasks currently running submitted to Yarn by the data aggregation queue is 1. After 1:30, the number of tasks currently running submitted to Yarn is 2. Based on this, after starting the corresponding tasks according to the start time configuration of different tasks, the current number of tasks serially submitted to Yarn in the data aggregation queue can be obtained at different times.

[0042] S202: Determine the merge end time of each task, schedule the corresponding task according to the merge end time by the timing script, and move the task to the data scheduling queue;

[0043] This step is used to periodically count the current number of tasks submitted to Yarn by the data scheduling queue. The execution time of each task in the data aggregation queue is uncertain. Only after the execution in the data aggregation queue is completed, the execution end time is also the merge end time. Then the timing script starts the corresponding task according to the merge end time; then the data scheduling program is automatically triggered and the task is moved to the data scheduling queue. At this time, the data scheduling queue submits the task to Yarn and updates the current number of tasks submitted to Yarn by the data scheduling queue. For example, for task a and task b in the above example, assuming that the merge end time of task a is 1:00 and the merge end time of task b is 2:00, then the number of tasks currently running submitted to Yarn by the data scheduling queue obtained using the Yarn interface command between 1:00 and 2:00 is 1, and the number of tasks currently running submitted to Yarn after 2:00 is 2. The specific process of determining the merge end time of each task is explained in the following steps S301-S303.

[0044] S203. Count the current number of tasks in the data aggregation queue and the current number of tasks in the data scheduling queue at preset time intervals, and update the current load factor according to the ratio of the current number of tasks in the data aggregation queue to the current number of tasks in the data scheduling queue.

[0045] In this step, statistics are performed at a preset time interval. Specifically, the current number of tasks in the data aggregation queue and the current number of tasks in the data scheduling queue can be counted according to the statistical method of steps S201-202. Then, the current number of tasks in the data aggregation queue currently counted is divided by the current number of tasks in the data scheduling queue, that is, β x / β y The ratio of is taken as the current load factor.

[0046] In one embodiment, if Figure 3 As shown, the process of determining the merging end time of each task is specifically introduced, and step S202 includes:

[0047] S301, in the data aggregation queue, regularly scan each task to see if there is a corresponding storage mark record in the MySQL table, if yes, go to step S202, if not, jump to step S203;

[0048] S302: Determine the end time of the corresponding task, schedule the corresponding task by the timing script, and trigger the data scheduling program to move the task to the data scheduling queue;

[0049] S303: Wait for the next scan.

[0050] In this embodiment, in the data aggregation queue, each task will generate a new hive partition table during execution. The number of records in the hive partition table generated by each task is scanned periodically, or the corresponding partition HDFS directory connected to the hive partition table is scanned to see whether it is generated. If the number of records reaches a preset record value or the partition HDFS directory has been generated, a record of the corresponding task entry mark is inserted into the MySQL table. That is to say, during the periodic scanning process, it is determined whether each task has a corresponding entry mark record in the MySQL table to determine whether the task is completed. If there is a corresponding entry mark, the task is determined to be completed.

[0051] When it is determined that the corresponding task is completed, the data scheduling program can be triggered to move the task to the data scheduling queue.

[0052] The embodiment of the present invention further provides a device for allocating Yarn computing resources, which is used to execute any embodiment of the method for allocating Yarn computing resources. Figure 4 , Figure 4 This is a schematic block diagram of an apparatus for allocating YARN computing resources provided by an embodiment of the present invention.

[0053] like Figure 4As shown, the device 400 for allocating Yarn computing resources includes: a load factor calculation unit 401, a priority factor setting unit 402, and a resource allocation updating unit 403.

[0054] The load factor calculation unit 401 is used to periodically obtain the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculate the current load factor;

[0055] The priority factor setting unit 402 is used to set the corresponding current priority factor according to the task priorities in the data aggregation queue and the data scheduling queue;

[0056] The resource allocation updating unit 403 is configured to update the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor.

[0057] The device fully considers the serial characteristics of services between queues and the isolated characteristics of queue tasks. By dynamically refreshing the YRAN computing resources of the data aggregation queue and the data scheduling queue, it ensures that when the sum of the YRAN computing resources of the data aggregation queue and the data scheduling queue is constant, the current YARN computing resource ratio of each queue is dynamically allocated to achieve the goal of maximizing the overall scheduling resource utilization.

[0058] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0059] The above-mentioned device for allocating Yarn computing resources can be implemented in the form of a computer program, which can be used in Figure 5 Runs on the computer equipment shown.

[0060] See also Figure 5 , Figure 5 Schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 500 is a server, which can be an independent server or a server cluster composed of multiple servers.

[0061] See Figure 5 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0062] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a method for allocating Yarn computing resources.

[0063] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0064] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the method for allocating Yarn computing resources.

[0065] The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0066] Those skilled in the art will understand that Figure 5 The embodiment of the computer device shown in the figure does not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and processor are the same as those in the figure. Figure 5 The embodiments shown are consistent and will not be described again here.

[0067] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0068] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for allocating YARN computing resources according to an embodiment of the present invention.

[0069] The storage medium is a physical, non-transient storage medium, for example, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0070] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for allocating YARN computing resources, characterized in that: include: Regularly obtain the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculate the current load factor; wherein the data aggregation queue and the data scheduling queue belong to two different tenant queues, and the tasks meet the serial order of entering the data aggregation first and then reaching the data scheduling; According to the task priorities in the data aggregation queue and the data scheduling queue, setting corresponding current priority factors; Update the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor; The current YARN computing resource ratio between the data aggregation queue and the data scheduling queue is calculated using the following formula: ； ； ； Among them, x represents the current Yarn computing resource ratio of the data aggregation queue, x∈[1%,A%), y∈[1%,A%), y represents the current Yarn computing resource ratio of the data scheduling queue, A represents the total resource ratio that can be allocated to x and y in the entire cluster environment after excluding the Yarn computing resources occupied by other queues, and the value range of A is not greater than 100%. β represents the current load factor, and α represents the current priority factor. The value of α is a preset value set according to the importance of the business. The initial value of α is 3. α represents the priority of data scheduling using Yarn computing resources. x Indicates the current number of tasks in the data aggregation queue, β y Indicates the current number of tasks in the data scheduling queue, β x ,β y ≠0.

2. The method for allocating YARN computing resources according to claim 1, characterized in that: The method of periodically obtaining the current number of tasks in the data aggregation queue and the current number of tasks in the data scheduling queue, and calculating the current load factor, includes: According to the start time of different tasks, the scheduled script starts the corresponding task according to the start time and moves the task to the data aggregation queue; Determine the merge end time of each task, schedule the corresponding task according to the merge end time by the timing script, and move the task to the data scheduling queue; At preset time intervals, the current number of tasks of the data aggregation queue and the current number of tasks of the data scheduling queue are counted, and the current load factor is updated according to the ratio of the current number of tasks of the data aggregation queue to the current number of tasks of the data scheduling queue.

3. The method for allocating YARN computing resources according to claim 2, characterized in that: The determining of the merge end time of each task, scheduling the corresponding task according to the merge end time by the timing script, and moving the task to the data scheduling queue includes: In the data aggregation queue, each task is scanned regularly to see if there is a corresponding storage mark record in the MySQL table. If so, the corresponding task is determined to be completed; When it is determined that the corresponding task is completed, the timing script schedules the corresponding task and triggers the data scheduling program to move the task to the data scheduling queue.

4. The method for allocating YARN computing resources according to claim 3, characterized in that: In the data aggregation queue, each task is periodically scanned to see if there is a corresponding storage mark record in the MySQL table. If so, the corresponding task is determined to be completed, including: In the data aggregation queue, the number of records in the Hive partition table generated by each task during operation is scanned periodically, or the corresponding partition HDFS directory connected to the Hive partition table is scanned to see whether it has been generated. If the number of records reaches a preset record value or the partition HDFS directory has been generated, a record with the corresponding task entry mark is inserted into the MySQL table, and the corresponding task is determined to be completed.

5. The method for allocating YARN computing resources according to claim 1, characterized in that: After the current number of tasks in the data aggregation queue and the current number of tasks in the data scheduling queue are obtained at regular intervals, the method further includes: When the current number of tasks in the data aggregation queue is 0, the current Yarn computing resource ratio of the data aggregation queue is set to a preset ratio.

6. The method for allocating YARN computing resources according to claim 1, characterized in that: After the current number of tasks in the data aggregation queue and the current number of tasks in the data scheduling queue are obtained at regular intervals, the method further includes: When the current number of tasks in the data scheduling queue is 0, the current Yarn computing resource ratio of the data scheduling queue is set to a preset ratio.

7. A device for allocating YARN computing resources, characterized in that: include: A load factor calculation unit, configured to periodically obtain the current number of tasks submitted by the data aggregation queue and the current number of tasks submitted by the data scheduling queue, and calculate the current load factor; wherein the data aggregation queue and the data scheduling queue belong to two different tenant queues, and tasks satisfy the serial order of first entering the data aggregation queue and then reaching the data scheduling queue; a priority factor setting unit, configured to set a corresponding current priority factor according to the task priorities in the data aggregation queue and the data scheduling queue; A resource allocation updating unit, configured to update the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the current load factor and the current priority factor; The resource allocation updating unit is specifically configured to calculate the current YARN computing resource allocation of the data aggregation queue and the data scheduling queue according to the following formula: ； ； ； Among them, x represents the current Yarn computing resource ratio of the data aggregation queue, x∈[1%,A%), y∈[1%,A%), y represents the current Yarn computing resource ratio of the data scheduling queue, A represents the total resource ratio that can be allocated to x and y in the entire cluster environment after excluding the Yarn computing resources occupied by other queues, and the value range of A is not greater than 100%. β represents the current load factor, and α represents the current priority factor. The value of α is a preset value set according to the importance of the business. The initial value of α is 3. α represents the priority of data scheduling using Yarn computing resources. x Indicates the current number of tasks in the data aggregation queue, β y Indicates the current number of tasks in the data scheduling queue, β x ,β y ≠0.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for allocating YARN computing resources is implemented as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method for allocating YARN computing resources according to any one of claims 1 to 6.

Images