Job execution method of YARN queue and related equipment thereof

Abstract

The invention provides a job execution method of a YARN queue and related equipment, and the method comprises the steps: obtaining a stacking type result of the YARN queue according to a preset period, the YARN queue comprises a plurality of jobs, and the stacking type result comprises a first stacking type result which cannot create a resource application operation for the jobs of the YARN queue, and / or a second stacking type result which cannot create a resource application operation for the jobs of the YARN queue, and / or a second stacking type result which cannot create a resource application operation for the jobs of the YARN queue; the job of the YARN queue cannot execute a second accumulation type result of a job execution operation after creating a resource application operation; the execution operation of the YARN queue is correspondingly adjusted according to the accumulation type result, and the execution operation comprises the adjustment of the number of the resource application operations and / or the adjustment of the number of the job execution operations.

Description

Technical Field

[0001] The present application relates to the field of YARN queue management, and in particular to a YARN queue job execution method and related equipment. Background Art

[0002] YARN is an important component of Hadoop, mainly used for resource management and job scheduling. When YARN obtains a job submitted by a user, YARN assigns it to the YARN queue. Then, YARN allocates corresponding resources to the job according to the total amount of YARN queue resources to complete the execution of the job.

[0003] Currently, there is a backlog of jobs in the YARN queue, which causes the jobs in the YARN queue to fail to execute normally. Summary of the invention

[0004] The present application provides a method for executing a job in a YARN queue and related devices thereof, which are used to adjust the job execution operation in the YARN queue to reduce the situation where the job is piled up in the YARN queue.

[0005] A first aspect of the present application provides a method for executing a job of a YARN queue, comprising: obtaining a backlog type result of the YARN queue according to a preset period, wherein the YARN queue includes multiple jobs, and the backlog type result includes a first backlog type result that a resource application operation cannot be created for the job of the YARN queue, and / or a second backlog type result that the job of the YARN queue cannot perform a job execution operation after creating a resource application operation; adjusting the execution operation of the YARN queue accordingly according to the backlog type result, the execution operation including adjusting the number of the resource application operations, and / or adjusting the number of the job execution operations.

[0006] In some implementations, when the accumulation type result is the first accumulation type result, the cause of the first accumulation type result is obtained, and then the execution operation of the YARN queue is adjusted according to the cause of the first accumulation type result.

[0007] In some implementations, it is determined that the cause of the first accumulation-type result is that the total amount of queue resources of the YARN queue has no idle resources, and then the total amount of queue resources of the YARN queue is increased.

[0008] In some embodiments, it is determined that the cause of the first accumulation-type result is that there are idle resources in the total amount of queue resources of the YARN queue, then at least part of the idle resources of the total amount of queue resources are used to execute resource application operations of the YARN queue to increase the number of resource application operations of the YARN queue.

[0009] In some embodiments, the idle percentage of idle resources of the total queue resources is obtained, and the application percentage of job creation resource application operations for the YARN queue of the total queue resources is obtained; and the idle percentage is reduced by at least part of the idle percentage and the application percentage is increased according to a first preset step size to increase the number of resource application operations of the YARN queue.

[0010] In some embodiments, when the accumulation type result is a second accumulation type result, the number of resource creation application operations performed on the jobs of the YARN queue in the total queue resources of the YARN queue is reduced to increase the number of job execution operations of the jobs of the YARN queue after the resource creation application operation.

[0011] In some implementations, at least some of the job-corresponding resources of the plurality of jobs are released according to the processing priorities of the jobs in the YARN queue.

[0012] In some embodiments, the application ratio of the total queue resources for executing the create resource application operation for the jobs of the YARN queue is obtained, and the execution ratio of the total queue resources for executing the job execution operation after the create resource application operation is obtained; at least part of the application ratio is reduced according to a second preset step size to reduce the number of create resource application operations executed for the jobs of the YARN queue in the total queue resources of the YARN queue, and the execution ratio is increased to increase the number of job execution operations for the jobs of the YARN queue after the create resource application operation.

[0013] A second aspect of the present application provides an electronic device, comprising: a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the job execution method of the YARN queue of the first aspect.

[0014] A third aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores program data, and the program data can be executed to implement the job execution method of the YARN queue of the first aspect.

[0015] Compared with the prior art, the technical solution of the present application obtains the accumulation type result of the YARN queue including multiple jobs according to a preset period, wherein the accumulation type result includes a first accumulation type result that a resource application operation cannot be created for the job of the YARN queue, and / or a second accumulation type result that the job of the YARN queue cannot perform the job execution operation after creating the resource application operation. Then, the execution operation of the YARN queue is adjusted accordingly according to the accumulation type result. Since the execution operation includes adjusting the number of resource application operations and / or adjusting the number of job execution operations, refined execution adjustment of the YARN queue job is achieved, thereby reducing the situation of job accumulation in the YARN queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 It is a flowchart of the first embodiment of the method for executing a job in a YARN queue of the present application; Figure 2 It is a flowchart of the second embodiment of the method for executing a job in a YARN queue of the present application; Figure 3 It is a flowchart of the third embodiment of the method for executing a job in a YARN queue of the present application; Figure 4 It is a flowchart of the fourth embodiment of the method for executing a job in a YARN queue of the present application; Figure 5 It is a flowchart of the fifth embodiment of the method for executing a job in a YARN queue of the present application; Figure 6 This is a full flow diagram of the job execution method of the YARN queue in this application; Figure 7 It is a schematic diagram of the job execution device of the YARN queue of this application; Figure 8 This is a schematic diagram of the YARN system architecture for this application; Fig. 9 This is a schematic diagram of the YARN electronic equipment structure of this application. DETAILED DESCRIPTION

[0018] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0019] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0022] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0023] After YARN receives the job submitted by the user, it assigns the job to the YARN queue to form multiple YARN queues including multiple jobs. In order to execute the jobs in the YARN queue, YARN will allocate corresponding resources to the jobs in the YARN queue according to the resource situation.

[0024] In the process of fully executing the jobs in the YARN queue, YARN first creates a resource application operation for the jobs in the YARN queue, that is, through ApplicationMaster (AM), it applies to the jobs in the YARN queue to use containers to execute the jobs in the YARN queue, which will consume some of YARN's resources. Furthermore, after creating the resource application operation, YARN performs a job execution operation on the jobs in the YARN queue, that is, it executes the jobs in the YARN queue through containers, which also consumes some of YARN's resources. Therefore, the job backlog in the YARN queue comes from the job backlog caused by the inability to create a resource application operation for the job, the job backlog caused by the inability to execute the job execution operation after the resource application operation is created, and the job backlog caused by insufficient total queue resources in the YARN queue.

[0025] Based on this, see Figure 1 , Figure 1 The following shows the job execution method of the YARN queue, including: 101. Obtaining an accumulation type result of a YARN queue according to a preset period, wherein the YARN queue includes multiple jobs, and the accumulation type result includes a first accumulation type result that a resource application operation cannot be created for the job of the YARN queue, and / or a second accumulation type result that a job execution operation cannot be performed after the resource application operation is created for the job of the YARN queue; YARN monitors the running status of the YARN queue to obtain the running status information of the YARN queue, and then determines whether there is a job backlog in the YARN queue based on the running status information. When there is a job backlog in the YARN queue, YARN analyzes the cause of the job backlog and obtains the backlog type result of the YARN queue.

[0026] The accumulation type results include a first accumulation type result that a resource application operation cannot be created for a job in a YARN queue, and a second accumulation type result that a job in a YARN queue cannot perform a job execution operation after a resource application operation is created.

[0027] In some implementations, YARN obtains the running status information according to a preset period, and simultaneously obtains the job accumulation type result and the corresponding execution operation.

[0028] Furthermore, the preset period is adjustable. In some implementations, the preset period may be an adjustable fixed period or an adjustable variable period.

[0029] 102. Adjust the execution operation of the YARN queue accordingly according to the accumulation type result, wherein the execution operation includes adjusting the number of resource application operations and / or adjusting the number of job execution operations.

[0030] YARN adjusts the execution operation of the YARN queue accordingly according to the specific situation of the accumulation type result and then executes the job of the YARN queue, wherein the execution operation includes adjusting the number of resource application operations and / or adjusting the number of job execution operations.

[0031] By obtaining the accumulation type result of the job of the YARN queue and adjusting the execution operation of the YARN queue accordingly according to the accumulation type result, the autonomous adjustment execution of the job of the YARN queue is realized, thereby reducing the probability of the occurrence of the accumulation of the job of the YARN queue and enhancing the probability of the normal execution of the job of the YARN queue. Furthermore, since the execution operation includes adjusting the number of resource application operations and / or adjusting the number of job execution operations, the refined execution adjustment of the YARN queue job is realized, thereby reducing the accumulation of the job of the YARN queue.

[0032] In some implementations, the total amount of queue resources of the YARN queue is constant. By adjusting the execution operation accordingly based on the above-mentioned accumulation type result, the job accumulation of the YARN queue that is not caused by insufficient total amount of queue resources of the YARN queue can be reduced, that is, the problem of job accumulation of the YARN queue caused by the reasonable total amount of queue resources of the YARN queue but the rigid operation configuration of the YARN queue is solved, thereby maximizing the utilization of the constant total amount of queue resources of the YARN queue.

[0033] Furthermore, in the case where the accumulation type result is that the resource request operation cannot be created for the job in the YARN queue, YARN analyzes the cause more precisely and branches the processing accordingly. Figure 2 , which shows a schematic diagram of the corresponding execution operation adjustment when it is impossible to create a resource application operation for a job in a YARN queue because there are no idle resources in the total amount of queue resources of the YARN queue, including: 201. It is determined that the cause of the first accumulation type result is that the total amount of queue resources of the YARN queue has no idle resources, and the total amount of queue resources for executing operations of the YARN queue is increased.

[0034] When YARN further determines that the cause of the first accumulation type result is that the total amount of queue resources of the YARN queue has no idle resources, the total amount of queue resources for executing operations of the YARN queue is increased, so that the jobs of the YARN queue are executed with sufficient resources.

[0035] See also Figure 3 , which shows a schematic diagram of the corresponding execution operation adjustment when it is impossible to create a resource application operation for a job in a YARN queue because there are idle resources in the total amount of queue resources of the YARN queue, including: 301. Determine that the cause of the first accumulation type result is that there are idle resources in the total amount of queue resources of the YARN queue, then use at least part of the idle resources in the total amount of queue resources to perform resource application operations for executing operations of the YARN queue, so as to increase the number of resource application operations for executing operations of the YARN queue.

[0036] YARN further determines that the cause of the first accumulation type result is that there are idle resources in the total amount of queue resources of the YARN queue, and then uses the idle resources in the total amount of queue resources to perform resource application operations for executing operations of the YARN queue to increase the number of resource application operations for executing operations of the YARN queue.

[0037] In some implementations, when YARN determines that the cause of the first accumulation type result is that there are idle resources in the total amount of queue resources of the YARN queue, YARN uses all idle resources in the total amount of queue resources for resource application operations to execute execution operations of the YARN queue.

[0038] In some implementations, when YARN determines that the first accumulation type result is caused by idle resources in the total amount of queue resources of the YARN queue, YARN uses part of the idle resources in the total amount of queue resources for resource application operations to execute execution operations of the YARN queue.

[0039] In some embodiments, YARN obtains the idle ratio of the total amount of queue resources, and obtains the application ratio of the total amount of queue resources for the resource creation application operation of the YARN queue job, and then reduces the idle ratio to increase the application ratio, so that the resource creation application operation is executed for the job of the YARN queue.

[0040] In some implementations, YARN may increase the application ratio by obtaining part or all of the idle ratio, thereby causing the job in the YARN queue to perform a resource application creation operation.

[0041] In some embodiments, YARN can reduce some or all of the idle percentage by a first preset step size and increase the application percentage by a first preset step size, so that the job of the YARN queue performs a resource application operation. In some embodiments, the first preset step size is set to 2%-7%, such as 2%, 5%, 7%. Further, the first preset step size is adjustable.

[0042] See also Figure 4 , Figure 4 The flowchart for processing the second accumulation type result in which the accumulation type result is that the job of the YARN queue cannot perform the job execution operation after the resource application operation is created is shown, specifically including: 401. When the accumulation type result is the second accumulation type result, the resource creation application operation for the job of the YARN queue is reduced in the total amount of queue resources of the YARN queue to increase the number of job execution operations performed by the job of the YARN queue after the resource creation application operation.

[0043] YARN determines that the accumulation type result is the second accumulation type result, which indicates that the accumulation of jobs in the YARN queue is caused by too many job execution operations and insufficient resource application operations. Therefore, the resource creation application operation for the job execution of the YARN queue in the total queue resources of the YARN queue is replaced with the job execution operation for the job of the YARN queue after the resource creation application operation. That is, the resource creation application operation for the job execution of the YARN queue in the total queue resources of the YARN queue is reduced to increase the job execution operation of the job of the YARN queue after the resource creation application operation, thereby alleviating the accumulation of jobs in the YARN queue.

[0044] In some embodiments, YARN obtains the application ratio of the total queue resources for executing the create resource application operation for jobs in the YARN queue, and obtains the execution ratio of the total queue resources for executing the job execution operation after the create resource application operation, and then reduces the application ratio and increases the execution ratio in the total queue resources of the YARN queue, thereby reducing the create resource application operation for jobs in the YARN queue and increasing the job execution operations after the create resource application operation for jobs in the YARN queue.

[0045] In some embodiments, after obtaining the application ratio and the execution ratio, YARN adjusts the application ratio and the execution ratio according to the second preset step size, that is, the application ratio is reduced by the second preset step size and the execution ratio is increased by the second preset step size, so as to reduce the total amount of queue resources of the YARN queue to execute the creation resource application operation for the job of the YARN queue, so as to increase the job execution operation of the job of the YARN queue after the creation resource application operation. In some embodiments, the second preset step size is set to 2%-7%, for example, 2%, 5%, 7%. Further, the second preset step size is adjustable.

[0046] See also Figure 5 , Figure 5 It shows that after the second accumulation type result that the accumulation type result is that the job of the YARN queue cannot perform the job execution operation after the resource application operation is created, YARN first releases the job of the YARN queue, instead of directly reducing the total amount of queue resources of the YARN queue to perform the resource application operation on the job of the YARN queue, so as to increase the job execution operation of the job of the YARN queue after the resource application operation is created, including: 501. Release at least some of the job corresponding resources of the multiple jobs according to the processing priorities of the jobs in the YARN queue.

[0047] YARN obtains the processing priorities of multiple jobs in the YARN queue, and then releases at least some of the job corresponding resources of the multiple jobs in the YARN queue according to the processing priorities.

[0048] It is understandable that the present invention refines the job accumulation situation of the YARN queue by setting the associated accumulation situation and executing the operation, thereby reducing the probability of job accumulation of the YARN queue. Further, when the total amount of queue resources of the YARN queue is constant, the job accumulation of the YARN queue caused by the rigid imbalance of the operation configuration of the YARN queue is reduced by the associated accumulation situation and executing the operation, the total amount of queue resources of the constant YARN queue is utilized to the maximum extent, and the utilization rate of the total amount of queue resources of the constant YARN queue is improved.

[0049] See also Figure 6 , Figure 6 The whole process diagram of a method for executing a job in a YARN queue is shown. After starting, YARN reads the running status information of YARN, that is, the running status information of each YARN queue, and then detects whether there is a job accumulation in each YARN queue. After YARN determines that there is a job accumulation in the YARN queue, it analyzes the job accumulation type to obtain the accumulation type result, and then branches the processing according to the specific accumulation type result.

[0050] When the result of the pileup type is determined to be unable to create an AM, that is, unable to create a resource application operation for the YARN queue job, it is further determined whether the total amount of queue resources of the YARN queue has free resources, and then branch processing is performed again based on whether there are free resources. If there are no free resources, the total amount of queue resources is increased; if there are free resources, the proportion of free resources is reduced and the proportion of AM is increased, that is, the aforementioned Figure 3 The corresponding implementation method of .

[0051] If the stacking type is determined to be unable to apply for a container after AM is created, the corresponding resources of at least some jobs in the YARN queue are released first according to the processing priority, that is, the aforementioned Figure 5 The corresponding implementation method reduces the proportion of AM and increases the proportion of container, that is, Figure 4 The corresponding implementation method of .

[0052] It should be noted that, except for the case where there is no free resource in the total amount of queue resources, other processing methods are to modify the queue configuration when the total amount of queue resources of the YARN queue is constant. For details, please refer to the previous Figure 3 , Figure 4 and Figure 5 The specific implementation methods are not described here. It can be seen that in some implementation methods, the present invention is based on the situation that the total amount of YARN queue resources is constant, and creates a more refined adjustment strategy under the situation that the total amount of YARN queue resources is constant, that is, based on obtaining specific accumulation type results and adjusting the quantitative relationship between the proportions of the total amount of YARN queue resources components accordingly, that is, the quantitative relationship between the idle proportion, the application proportion and the execution proportion, which solves the problem in the prior art that YARN job accumulation is generally attributed to the insufficient total amount of YARN queue resources and the total amount of YARN queue resources is simply and directly expanded or the resources in the execution process are simply released to form a sufficient total amount of YARN queue resources, thereby achieving maximum utilization of the total amount of YARN constant resource queues and improving the utilization rate of the total amount of YARN constant resource queues.

[0053] See also Figure 7 , Figure 7 A schematic diagram showing a job execution apparatus of a YARN queue includes: The acquisition module 701 is used to acquire the accumulation type result of the YARN queue according to a preset period, wherein the YARN queue includes multiple jobs, and the accumulation type result includes a first accumulation type result that a resource application operation cannot be created for the job of the YARN queue, and / or a second accumulation type result that the job of the YARN queue cannot perform a job execution operation after the resource application operation is created; The adjustment module 702 is used to adjust the execution operation of the YARN queue according to the accumulation type result, and the execution operation includes adjusting the number of resource application operations and / or adjusting the number of job execution operations.

[0054] See also Figure 8 , Figure 8 A schematic diagram of a YARN system architecture is shown, including a control module, a data analysis module, a monitoring module and a YARN cluster module.

[0055] The monitoring module is used to periodically collect the operating status information of each queue in the YARN cluster, such as the total usage of the queue resources of the YARN queue and the job status information of the YARN queue, and save the collected operating status information to the database for subsequent use.

[0056] The data analysis module is used to read the running status information collected by the queue monitoring module to analyze the queue status to obtain the accumulation type result, and then output the corresponding execution operation based on the accumulation type result.

[0057] The control module is used to provide a configuration entry for the user, and the user can adjust the preset period to achieve the job execution frequency of the YARN queue, and can also adjust the first preset step size and the second preset step size.

[0058] The YARN cluster module is used to generate multiple YARN queues including multiple jobs, and execute the jobs of the YARN queues according to the execution operations of the data analysis module.

[0059] This application also includes a schematic diagram of the YARN electronic equipment structure, see Fig. 9 The method comprises: a memory 901 and a processor 902; wherein the memory is used to store programs; and the processor is used to execute the programs in the memory, including executing the methods in the aforementioned embodiments; In some implementations, the electronic device further includes a bus system that electrically connects the memory 901 and the processor 902. The bus system is used to connect the memory and the processor so that the memory and the processor communicate with each other.

[0060] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0061] In some embodiments, the memory 901 may be an internal storage unit of the cloud device, for example, a hard disk or memory of the cloud device. In other embodiments, the memory 901 may also be an external storage device of the cloud device, for example, a plug-in hard disk equipped on the cloud device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory may also include both an internal storage unit of the cloud device and an external storage device. The memory is used to store an operating system, an application, a boot loader (BootLoader), data, and other programs, such as program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0062] The present application also includes a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute any one of the methods in the aforementioned embodiments.

[0063] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0064] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer readable medium may at least include: any entity or device capable of carrying the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0065] The above are only implementation methods of the present application, and are not intended to limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for executing a job in a YARN queue, characterized in that: include: Acquire an accumulation type result of the YARN queue according to a preset period, wherein the YARN queue includes a plurality of jobs, and the accumulation type result includes a first accumulation type result that a resource application operation cannot be created for the job of the YARN queue, and / or a second accumulation type result that a job execution operation cannot be performed after the resource application operation is created for the job of the YARN queue; The execution operation of the YARN queue is adjusted accordingly according to the accumulation type result, and the execution operation includes adjusting the number of the resource application operations and / or adjusting the number of the job execution operations.

2. The method for executing a job in a YARN queue according to claim 1, characterized in that: The step of adjusting the execution operation of the YARN queue according to the stacking type result includes: When the accumulation type result is the first accumulation type result, the cause of the first accumulation type result is obtained, and then the execution operation of the YARN queue is adjusted according to the cause of the first accumulation type result.

3. The method for executing a job in a YARN queue according to claim 2, characterized in that: The execution operation further includes increasing the total amount of queue resources of the YARN queue, and the obtaining of the cause of the first accumulation type result and then adjusting the execution operation of the YARN queue according to the cause of the first accumulation type result include: If it is determined that the cause of the first accumulation type result is that the total amount of queue resources of the YARN queue has no idle resources, the total amount of queue resources of the YARN queue is increased.

4. The method for executing a job in a YARN queue according to claim 2, characterized in that: The obtaining the cause of the first accumulation type result, and then adjusting the execution operation of the YARN queue according to the cause of the first accumulation type result, includes: It is determined that the cause of the first accumulation type result is that there are idle resources in the total amount of queue resources of the YARN queue, then at least part of the idle resources of the total amount of queue resources are used to execute resource application operations of the YARN queue to increase the number of the resource application operations of the YARN queue.

5. The method for executing a job in a YARN queue according to claim 4, characterized in that: The using at least part of the idle resources of the total amount of the queue resources to execute the resource application operation of the YARN queue to increase the number of the resource application operations of the YARN queue includes: Obtaining the idle ratio of the idle resources of the total resources of the queue, and obtaining the application ratio of the job creation resource application operation of the YARN queue of the total resources of the queue; The idle ratio is reduced at least in part and the application ratio is increased according to a first preset step size, so as to increase the number of the resource application operations of the YARN queue.

6. The method for executing a job in a YARN queue according to claim 1, characterized in that: The step of adjusting the execution operation of the YARN queue according to the stacking type result includes: When the accumulation type result is the second accumulation type result, the number of resource creation application operations performed on the jobs of the YARN queue in the total queue resources of the YARN queue is reduced to increase the number of job execution operations of the jobs of the YARN queue after the resource creation application operation.

7. The method for executing a job in a YARN queue according to claim 6, characterized in that: After the accumulation type result is the second accumulation type result, in the reducing the total amount of queue resources of the YARN queue, performing a resource application creation operation on the job of the YARN queue to increase the number of jobs in the YARN queue before the job execution operation after the resource application creation operation, further comprising: At least part of the job corresponding resources of the multiple jobs are released according to the processing priority of the jobs in the YARN queue.

8. The method for executing a job in a YARN queue according to claim 6, characterized in that: The reducing the total amount of queue resources of the YARN queue by performing a resource application creation operation on the job of the YARN queue to increase the job execution operation of the job of the YARN queue after the resource application creation operation includes: Obtaining the application ratio of the total resources of the queue for executing the resource application creation operation for the jobs of the YARN queue, and obtaining the execution ratio of the total resources of the queue for executing the job execution operation after the resource application creation operation; According to a second preset step size, at least part of the application ratio is reduced to reduce the number of resource application creation operations performed on the jobs of the YARN queue in the total queue resources of the YARN queue, and the execution ratio is increased to increase the number of job execution operations of the jobs of the YARN queue after the resource application creation operation.

9. The method for executing a job in a YARN queue according to any one of claims 1, 2, 4 to 8, characterized in that: The execution operation is an execution operation under the condition that the total amount of queue resources of the YARN queue is constant.

10. An electronic device, characterized in that: The electronic device comprises: a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the job execution method of the YARN queue as claimed in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and the program data can be executed to implement the job execution method of the YARN queue according to any one of claims 1 to 9.

Images