Resource scheduling method, device, electronic device and storage medium

By monitoring the resource usage information of Spark computing cluster in real time, dynamically adjusting the number of executors and memory/CPU allocation, the problem of slow computing and waste caused by fixed resource configuration is solved, and the computing performance and resource utilization are improved.

CN111813548BActive Publication Date: 2024-11-26BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010619438.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-30
Publication Date
2024-11-26
Estimated Expiration
2040-06-30

AI Technical Summary

Technical Problem

In Spark computing cluster, resource configuration remains fixed and cannot be automatically adjusted according to changes in computing jobs, resulting in slow calculation of computing jobs or waste of resources.

Method used

By monitoring the resource usage information of the computing cluster in real time, determining resource usage exceptions, and adjusting the number of executors and memory/CPU allocation amount according to the exception type, and dynamically adjusting resource configuration.

Benefits of technology

Automatic adjustment of resource configuration of computing jobs is realized, ensuring computing performance and speed, reducing resource waste, and improving resource utilization rate of computing clusters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111813548B_ABST
    Figure CN111813548B_ABST
Patent Text Reader

Abstract

The present application relates to a resource scheduling method, device, electronic device and storage medium, the method comprising: real-time monitoring of resource usage information of computing jobs in a computing cluster; when it is determined that the computing job has abnormal resource usage according to the resource usage information, determining the scheduling strategy of the executor corresponding to the computing job; and executing the scheduling operation corresponding to the scheduling strategy on the executor. This technical solution dynamically adjusts the memory and / or CPU usage of each executor during the execution of the computing job, allocates the resources required for normal execution to the computing job, ensures its performance and execution speed, reduces resource idleness, avoids resource waste, and improves the resource utilization rate of the entire computing cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a resource scheduling method, device, electronic device and storage medium. Background Art

[0002] Spark is a fast and general computing cluster designed for large-scale data processing. It adopts a distributed computing framework and is of great significance in the processing and calculation of massive data, machine learning and data mining.

[0003] Resource allocation is a very important aspect when executing computing jobs in Spark. Currently, whether it is static resource allocation or dynamic resource allocation, resources are allocated to each computing job at the beginning of task execution based on set values ​​or based on indicators such as the amount of computing data, task completion time, total resources, and number of tasks. Before the computing job is completed, its corresponding resources will not change.

[0004] However, since resource configuration is fixed and cannot be automatically adjusted according to changes in computing jobs, computing jobs may be slow due to insufficient resources, or there may be idle resources in computing jobs, resulting in resource waste. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present application provide a resource scheduling method, device, electronic device and storage medium.

[0006] According to one aspect of an embodiment of the present application, a resource scheduling method is provided, including:

[0007] Real-time monitoring of resource usage information of computing jobs in computing clusters;

[0008] When it is determined according to the resource usage information that the computing job has resource usage anomalies, determining a scheduling strategy for an executor corresponding to the computing job;

[0009] The scheduling operation corresponding to the scheduling strategy is executed on the executor.

[0010] Optionally, the real-time monitoring of resource usage information of computing jobs in the computing cluster includes:

[0011] Collecting at least one of the following resource usage data of computing jobs in the computing cluster in real time: CPU utilization data, memory utilization data, and garbage collection data;

[0012] The resource usage data within a preset time period is aggregated into resource usage information of the computing job.

[0013] Optionally, determining a scheduling strategy of an executor corresponding to the computing job includes:

[0014] Determine the exception type corresponding to the resource usage exception;

[0015] When the exception type is a memory exception, determining the scheduling strategy corresponding to the exception type is: adjusting the number of executors corresponding to the computing job and / or adjusting the memory allocation of the executor;

[0016] When the exception type is a CPU exception, the scheduling strategy corresponding to the exception type is determined as: adjusting the number of executors corresponding to the computing job and / or adjusting the number of CPU cores corresponding to the executors.

[0017] Optionally, the resource usage exception includes insufficient memory or free memory, and determining, according to the resource usage information, that the computing job has a resource usage exception includes:

[0018] Determine the memory utilization rate within a preset time period according to the memory utilization data, and determine the average garbage collection time within the preset time period according to the garbage collection data;

[0019] When the memory utilization rate is greater than or equal to the first memory threshold within the preset time period, and the average garbage collection time is greater than or equal to the first time period threshold, determining that the resource usage anomaly is insufficient memory;

[0020] When the memory utilization is less than or equal to the second memory threshold and the average garbage collection duration is less than or equal to the second duration threshold, it is determined that the resource usage anomaly is memory idle.

[0021] Optionally, when it is determined that the resource usage anomaly is insufficient memory, determining a scheduling strategy for an executor corresponding to the computing job includes:

[0022] Obtain the memory allocation corresponding to the executor and the maximum allocatable memory of a single computing node in the computing cluster;

[0023] When the difference between the memory allocation amount and the maximum allocatable memory amount meets a first preset condition, determining that the scheduling strategy is to increase the memory allocation amount of the executor;

[0024] When the difference between the memory allocation amount and the maximum allocatable memory amount does not satisfy the first preset condition, determining the scheduling strategy is to increase the number of executors corresponding to the computing job.

[0025] Optionally, when it is determined that there is free memory in the computing job, determining a scheduling strategy for an executor corresponding to the computing job includes:

[0026] Determine the CPU utilization within the preset time period according to the CPU utilization data;

[0027] When the CPU utilization is greater than or equal to a first CPU threshold, determining that the scheduling strategy is to reduce the memory allocation of the executor;

[0028] When the CPU utilization is less than the first CPU threshold, determining the scheduling strategy is to reduce the number of executors corresponding to the computing job.

[0029] Optionally, the scheduling strategy also includes: scheduling duration;

[0030] The executing the scheduling operation corresponding to the scheduling strategy on the executor includes:

[0031] During the scheduling duration, the memory allocation amount, the number of executors and / or the number of CPU cores are adjusted.

[0032] Optionally, determining an average garbage collection duration within a preset duration according to the garbage collection data includes:

[0033] Counting the total garbage collection duration corresponding to the computing job within the preset duration according to the garbage collection data;

[0034] Obtain the number of executors corresponding to the computing job;

[0035] The total garbage collection duration is divided by the number of executors to obtain the average garbage collection duration corresponding to each executor.

[0036] According to another aspect of an embodiment of the present application, a resource scheduling device is provided, including:

[0037] The monitoring module is used to monitor the resource usage information of computing jobs in the computing cluster in real time;

[0038] An analysis module, configured to determine a scheduling strategy for an executor corresponding to the computing job when it is determined according to the resource usage information that the computing job has resource usage anomalies;

[0039] The scheduling module is used to execute the scheduling operation corresponding to the scheduling strategy on the executor.

[0040] According to another aspect of the embodiment of the present application, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0041] The memory is used to store computer programs;

[0042] The processor is used to implement the above method steps when executing a computer program.

[0043] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0044] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0045] In real time, the resource usage of each computing job is monitored to determine whether there is an abnormality in resource usage for each computing job. When an abnormality occurs, the resources of each executor of the computing job can be adjusted in real time. In this way, the memory and / or CPU usage of each executor is dynamically adjusted during the execution of the computing job, and the resources required for normal execution are allocated to the computing job to ensure its performance and execution speed, reduce idle resources, avoid resource waste, and improve the resource utilization rate of the entire computing cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flowchart of a resource scheduling method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a resource scheduling method provided in another embodiment of the present application;

[0050] Figure 3 A flowchart of a resource scheduling method provided in another embodiment of the present application;

[0051] Figure 4 A flowchart of a resource scheduling method provided in another embodiment of the present application;

[0052] Figure 5 A flowchart of a resource scheduling method provided in another embodiment of the present application;

[0053] Figure 6 A block diagram of a resource scheduling device provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] In the computing cluster Spark, there are multiple computing nodes deployed in a distributed manner, and multiple executors can run on each computing node. Spark converts the submitted computing jobs into tasks, schedules the tasks to each executor, and the executors execute the assigned tasks.

[0057] Executor is a worker process responsible for running Tasks in Spark jobs. Each Task is independent of each other. When the Spark application is started, the Executor is started at the same time and always exists with the entire life cycle of the Spark application. If an Executor fails or crashes, the Spark application can continue to execute and the tasks on the failed Task will be scheduled to other Executors to continue running.

[0058] Spark allocates resources to a job based on the data volume, completion time, computational complexity, etc. The resources Spark allocates to each job include: the number of executors, the memory allocation of each executor, the number of CPU cores allocated to each executor, etc. Therefore, in this embodiment, resource scheduling mainly adjusts the number of executors, memory allocation, and CPU core allocation.

[0059] The number of executors and the number of CPU cores corresponding to each executor determine the number of tasks that can be executed in parallel. For example, if there are 6 executors and each executor has 2 CPU cores, 12 tasks can be executed in parallel. After the 12 tasks are executed, the next batch of 12 tasks will be replaced. If the number of executors is increased to 10 and the number of CPU cores corresponding to each executor remains unchanged, the number of tasks executed in parallel will be 20. If the number of executors remains unchanged and the number of CPU cores corresponding to each executor is increased to 5, the number of tasks executed in parallel will be 30. It can be seen that by increasing the number of executors or the number of CPU cores corresponding to each executor, the task parallelism capability can be increased, and the computing performance, that is, the computing speed, will also be improved accordingly.

[0060] The amount of memory allocated to each Executor determines how much data can be cached during the calculation process, how often the disk is written, and the frequency and time of garbage collection (GC). If the amount of memory allocated to each Executor is small, when executing tasks, it may cause the Java Virtual Machine (JVM) heap memory to be frequently full, and garbage collection to be performed frequently, which slows down the calculation speed. In addition, if the memory is insufficient, more data needs to be written to the disk, and the disk input and output volume (number of reads and writes) is high. If the amount of memory for each Executor is increased, the GC frequency and time can be reduced, and the calculation speed can be increased; more data can be cached in memory, the amount of data written to disk can be reduced, the disk input and output volume can be reduced, and the calculation performance can be improved.

[0061] Generally speaking, the node memory of each computing node is fixed, and a portion of the memory needs to be reserved for the operating system and Spark daemon. Therefore, the product of the number of executors on each computing node and the memory allocation of each executor is less than the maximum allocatable memory of the computing node. For example, if the node memory of a computing node is 64GB, of which 1GB is used for the operating system and Spark daemon, then its maximum allocatable memory is 63GB.

[0062] In the embodiment of the present application, the resource usage of each computing job in the computing cluster (Spark) is monitored in real time to determine whether the computing job has resource usage anomalies such as insufficient resources or resource waste. When resource usage anomalies occur, the scheduling strategy of the Executor that executes the computing job is determined, and the Executor is adjusted according to the scheduling strategy.

[0063] First, a resource scheduling method provided by an embodiment of the present invention is introduced below.

[0064] Figure 1 A flow chart of a resource scheduling method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0065] Step S11, real-time monitoring of resource usage information of computing jobs in the computing cluster.

[0066] Optionally, step S11 includes: collecting at least one of the following resource usage data of computing jobs in the computing cluster in real time: CPU utilization data, memory utilization data and garbage collection data; and aggregating the resource usage data within a preset time period into resource usage information of the computing job.

[0067] Among them, GC mainly cleans up objects that are no longer used in the memory, freeing up memory space for users to create other new objects. Whenever GC occurs, a message will be printed in the work log. Therefore, through log analysis, the total GC duration corresponding to each computing job can be statistically obtained. Optionally, this embodiment determines whether there is a resource anomaly based on the total GC duration. In addition to the total GC duration, the GC data can also include GC frequency, etc.

[0068] Step S12: when it is determined according to the resource usage information that a computing job has resource usage anomalies, a scheduling strategy for an executor corresponding to the computing job is determined.

[0069] Abnormal resource usage includes insufficient resources or idle resources. The resources in this embodiment mainly include memory and / or CPU, that is, insufficient memory or idle memory, insufficient CPU or idle CPU.

[0070] When a job has insufficient memory, the scheduling strategy can be to increase the number of executors or increase the memory allocation for each executor; conversely, when a job has free memory, the scheduling strategy can be to reduce the number of executors or reduce the memory allocation for each executor.

[0071] When the job has insufficient CPU, the scheduling strategy can be to increase the number of CPU cores allocated to each Executor; conversely, when the job has idle CPU, the scheduling strategy is to reduce the number of CPU cores allocated to each Executor.

[0072] Step S13, executing a scheduling operation corresponding to the scheduling strategy on the executor.

[0073] In Spark, you can adjust the parameters in the spark-submit shell script to implement resource scheduling operations on the Executor.

[0074] For example, the specific parameters in the spark-submit shell script are as follows:

[0075] (1) num-executors 3, configure the number of executors to 3;

[0076] (2) executor-memory 100m, configure the memory allocation for each Executor to 100 MB;

[0077] (3)executor-cores 3: configure the number of CPU cores corresponding to each Executor to 3.

[0078] In this embodiment, whether each computing job has abnormal resource usage is determined in real time based on the resource usage of the monitored computing job. When an abnormality occurs, the resources of each executor of the computing job can be adjusted in real time. In this way, the memory and / or CPU usage of each executor is dynamically adjusted during the execution of the computing job, and the resources required for normal execution are allocated to the computing job to ensure its performance and execution speed, reduce resource idleness, avoid resource waste, and improve the resource utilization rate of the entire computing cluster.

[0079] In an optional embodiment, the total CPU utilization of the computing job can be calculated based on the CPU utilization data. If the total CPU utilization is less than or equal to a preset CPU threshold, such as less than 30%, the CPU core corresponding to each Executor can be reduced; conversely, if the CPU utilization of each Executor within a preset time period is greater than or equal to another preset CPU threshold, such as less than 90%, the CPU core corresponding to each Executor can be increased.

[0080] Spark is a memory-based distributed computing engine, and most of the jobs it executes are memory-intensive computing tasks. The resource utilization and computing efficiency of the cluster are strongly dependent on memory usage. Therefore, the scheduling of memory resources is crucial. In an optional embodiment, it can be determined whether the computing job has insufficient memory or free memory based on memory usage data and GC data.

[0081] Optionally, in step S12, determining a scheduling strategy for an executor corresponding to the computing job includes:

[0082] Determine the exception type corresponding to the resource usage exception;

[0083] When the exception type is a memory exception, determining the scheduling strategy corresponding to the exception type is: adjusting the number of executors corresponding to the computing job and / or adjusting the memory allocation of the executor;

[0084] When the exception type is a CPU exception, the scheduling strategy corresponding to the exception type is determined as: adjusting the number of executors corresponding to the computing job and / or adjusting the number of CPU cores corresponding to the executors.

[0085] Optionally, the scheduling strategy further includes: scheduling duration. The above step S13 includes: adjusting the memory allocation, the number of executors and / or the number of CPU cores within the scheduling duration.

[0086] For example, the scheduling duration is 30 minutes. During the 30 minutes, the corresponding scheduling operation is performed. When 30 minutes are reached, the resource usage of the Job before can be restored, or a new resource scheduling can be performed again based on the resource usage information within these 30 minutes.

[0087] Figure 2 This is a flow chart of a resource scheduling method provided by another embodiment of the present application. Figure 2 As shown, resource usage anomalies include insufficient memory or free memory. In step S12, determining that a computing job has resource usage anomalies according to resource usage information includes:

[0088] Step S21, determining the memory utilization rate within a preset time period according to the memory utilization data, and determining the average garbage collection time within the preset time period according to the garbage collection data;

[0089] Step S22, when the memory utilization rate is greater than or equal to the first memory threshold within the preset time length, and the average garbage collection time length is greater than or equal to the first time length threshold, determining that the resource usage abnormality is insufficient memory;

[0090] Step S23: When the memory utilization is less than or equal to the second memory threshold and the average garbage collection duration is less than or equal to the second duration threshold, it is determined that the resource usage anomaly is memory idle.

[0091] Optionally, the first memory threshold and the second memory threshold may be set to be the same or different. The first duration threshold and the second duration threshold may be set to be the same or different.

[0092] Optionally, the preset duration may be equal to the scheduled duration, such as 30 minutes, or the preset duration may be equal to or less than the scheduled duration.

[0093] For example, the preset duration may be 30 minutes. If the memory utilization of the job within 30 minutes is greater than 50% and the average GC duration is greater than 1 minute, it is determined that the memory is insufficient. For another example, if the memory utilization of the job within 30 minutes is greater than or equal to 30% and the average GC duration is less than 30 seconds, it is determined that the memory is free.

[0094] The average GC duration in step S21 can be calculated by the following steps:

[0095] Step A1, calculating the total garbage collection duration corresponding to the job within a preset duration according to the garbage collection data statistics;

[0096] Step A2, obtaining the number of executors corresponding to the computing job;

[0097] Step A3: Divide the total garbage collection time by the number of executors to obtain the average garbage collection time corresponding to each executor.

[0098] For example, if the total GC duration of a job in 30 minutes is T and the number of Executors is n, then the average garbage collection duration of each Executor is

[0099] In an optional embodiment, when it is determined that the resource usage anomaly is insufficient memory, the amount of memory used by the computing job can be increased by increasing the memory allocation of the Executor or increasing the number of Executors. Figure 3 This is a flow chart of a resource scheduling method provided by another embodiment of the present application. Figure 3 As shown, the scheduling strategy of the executor corresponding to the computing job is determined in the above step S12, including:

[0100] Step S31, obtaining the memory allocation corresponding to the executor and the maximum allocatable memory of a single computing node in the computing cluster.

[0101] Step S32: When the difference between the memory allocation amount and the maximum allocatable memory amount meets a first preset condition, determining the scheduling strategy to increase the memory allocation amount of the executor.

[0102] For example, the first preset condition is that the product of the memory allocation amount and the preset multiple is less than the maximum allocatable memory amount, or the first preset condition is that the difference between the memory allocation amount and the maximum allocatable memory amount is greater than or equal to a preset value.

[0103] Step S33: when the difference between the memory allocation amount and the maximum allocatable memory amount does not satisfy the first preset condition, determine the scheduling strategy to increase the number of executors corresponding to the computing job.

[0104] For example, the product of the memory allocation amount and the preset multiple is greater than or equal to the maximum allocatable memory amount, or the difference between the memory allocation amount and the maximum allocatable memory amount is less than or equal to the preset value.

[0105] For example, the memory allocation amount of each Executor is m, and the maximum allocatable memory amount is M. For example, the first preset condition is that the product of the memory allocation amount and the preset multiple is less than the maximum allocatable memory amount, and the preset multiple can be set to 1.5.

[0106] When 1.5m<M, that is, there is still a certain amount of allocatable memory space on the computing node, the memory allocation of the Executor can be increased. The amount of memory increased / decreased for the Executor each time can be a certain multiple of the memory allocation, such as increasing the memory by 0.5m each time; or the amount of memory increased / decreased each time can be determined based on the maximum allocatable memory, such as increasing by 0.2M each time; or a fixed amount of memory can be set each time to increase / decrease, such as increasing by 200MB; and so on.

[0107] If you increase the memory allocation of the Executor, you can keep the number of Executors unchanged.

[0108] When 1.5m≥M, the memory space that can be allocated to the computing node is insufficient, and the memory of each executor cannot be increased. The number of executors can be increased. For example, the number of executors can be increased / decreased by a fixed amount each time, such as increasing 2 executors each time; or the number of executors increased / decreased each time can be a certain multiple of the number of executors n, such as when n=1, 1 executor is increased each time, and when n≥2, INT(0.5n) executors are increased each time; and so on.

[0109] If you increase the number of Executors, you can keep the memory allocation of the Executors unchanged.

[0110] Optionally, when the product of the memory allocation and the preset multiple is less than the maximum allocatable memory, the memory allocation of each executor and the number of executors can be increased at the same time. For example, when 1.5m<M, each executor is given an additional 200MB of memory and two executors are added.

[0111] In this embodiment, whether there is insufficient memory for the Job is analyzed based on memory usage data and GC data. When there is insufficient memory, memory scheduling is performed by increasing the memory allocation of each Executor and / or increasing the number of Executors. When the memory allocation of the Executor is increased, the GC frequency and time can be reduced, the calculation speed can be increased, the amount of data written to the disk during the calculation process can be reduced, the disk input and output volume can be reduced, and the computing performance can be improved. When the number of Executors is increased, the number of Tasks processed in parallel can be increased, and the increase in computing speed reduces the completion time of the Job. Therefore, this embodiment can improve the computing performance of the entire Job by improving the computing performance of a single Task and / or increasing the number of Tasks processed in batches.

[0112] In an optional embodiment, when it is determined that the resource usage anomaly is idle memory, it is possible to select how to release the idle memory based on the CPU usage corresponding to the computing job.

[0113] Figure 4 This is a flow chart of a resource scheduling method provided by another embodiment of the present application. Figure 4 As shown, when it is determined that there is free memory for the computing job, the scheduling strategy of the executor corresponding to the computing job is determined in the above step S12, including:

[0114] Step S41, determining the CPU utilization within a preset time period according to the CPU utilization data;

[0115] Step S42, when the CPU utilization is greater than or equal to the first CPU threshold, determining the scheduling strategy to reduce the memory allocation of the executor;

[0116] Step S43: When the CPU utilization is less than the first CPU threshold, determine the scheduling strategy to reduce the number of executors corresponding to the computing job.

[0117] For example, the first CPU threshold is 50%. When the CPU utilization is greater than or equal to 50%, that is, although the memory is idle, when the CPU utilization is high, if the number of Executors is reduced, the total number of CPU cores corresponding to the Job will also be reduced, and the task parallel processing capability will be reduced, the calculation speed will decrease, and the CPU utilization will increase. In this way, the calculation performance of the Job will also decrease. Therefore, at this time, the memory allocation of the Executor can be reduced to release the idle memory while ensuring that the task parallel processing capability remains unchanged.

[0118] When the CPU utilization is less than 50%, that is, the memory is idle and the CPU utilization is not high, you can reduce the number of executors to release not only the memory allocated to the executors, but also the CPU cores allocated to the executors. Optionally, you can also reduce the number of executors and reduce the memory allocation of the executors at the same time.

[0119] In this embodiment, when memory is idle, the idle memory is released based on the CPU usage to improve memory resource utilization and avoid wasting memory resources. In addition, when the CPU usage is different, different memory release strategies are adopted to improve the accuracy of memory scheduling while ensuring computing performance, avoiding affecting the normal processing of tasks.

[0120] The method of this embodiment is described in detail below by taking the scheduling of memory resources as an example. Figure 5 This is a flow chart of a resource scheduling method provided by another embodiment of the present application. Figure 5 As shown, the resource scheduling method includes the following steps:

[0121] Step S501, monitoring the CPU utilization data, memory utilization data and GC data of the Job;

[0122] Step S502, determine whether it is 30 minutes since the last resource scheduling, if yes, execute step S503, if no, return to step S501;

[0123] Step S503, calculate the memory utilization, average GC duration and CPU utilization, obtain the number of Executors n corresponding to the Job, the memory allocation m corresponding to each Executor and the maximum allocatable memory M of a single computing node;

[0124] Step S504, determine whether the memory utilization is greater than or equal to 50%, and whether the average GC duration is greater than or equal to 1 minute. If yes, that is, the memory corresponding to the Job is insufficient, and execute step S508; if no, the memory corresponding to the Job is free, and execute step S505;

[0125] Step S505, determining whether the CPU utilization is greater than or equal to 50%, if yes, executing step S506, if no, executing step S507;

[0126] Step S506, reduce the memory allocation of Executor to 0.5m;

[0127] Step S507, reducing the number of Executors allocated to the Job to 0.5n;

[0128] Step S508, determine whether 1.5m is greater than or equal to M, if yes, execute step S510, if no, execute step S509;

[0129] Step S509, increase the memory allocation of Executor to 1.5m;

[0130] Step S510, increase the number of Executors to 1.5n;

[0131] Step S511, determine whether the job is completed, if yes, execute step S512, if not, return to step S501;

[0132] Step S512: Release the Executor allocated to the Job.

[0133] In this embodiment, it is determined in real time whether the Job has insufficient memory or free memory according to the resource usage of the Job monitored. When insufficient memory or free memory occurs, the number of executors or the memory allocation of the executors can be adjusted in real time according to the memory utilization, average GC duration and CPU utilization. When there is insufficient memory, increasing the memory allocation of the executor can reduce the GC frequency and time, improve the calculation speed, reduce the amount of data written to the disk during the calculation process, reduce the disk input and output, and improve the calculation performance. Increasing the number of executors can increase the number of tasks processed in parallel, and the improvement in calculation speed reduces the completion time of the Job. Thus, the computing performance of the entire Job is improved by improving the computing performance of a single Task and / or increasing the number of batch processing Tasks. In addition, when free memory occurs, the free memory is released based on the CPU usage to improve the utilization of memory resources and avoid wasting memory resources. In addition, when the CPU usage is different, different memory release strategies are adopted to improve the accuracy of memory scheduling while ensuring computing performance, so as to avoid affecting the normal processing of tasks.

[0134] The following are device embodiments of the present application, which can be used to execute method embodiments of the present application.

[0135] Figure 6 This is a block diagram of a resource scheduling device provided in an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 6 As shown, the resource scheduling device includes:

[0136] Monitoring module 61, used for real-time monitoring of resource usage information of computing jobs in the computing cluster;

[0137] An analysis module 62 is used to determine a scheduling strategy for an executor corresponding to a computing job when it is determined based on the resource usage information that a computing job has resource usage anomalies;

[0138] The scheduling module 63 is used to execute the scheduling operation corresponding to the scheduling strategy on the executor.

[0139] The present application also provides an electronic device, such as Figure 7 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0140] Memory 1503, used for storing computer programs;

[0141] The processor 1501 is used to implement the steps of the above method embodiment when executing the computer program stored in the memory 1503.

[0142] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0143] The communication interface is used for communication between the above electronic device and other devices.

[0144] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0145] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0146] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method embodiment described below are implemented.

[0147] It should be noted that, for the above-mentioned device, electronic device and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0148] It should be further noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0149] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A resource scheduling method, characterized in that: include: Real-time monitoring of resource usage information of computing jobs in computing clusters; When it is determined according to the resource usage information that the computing job has resource usage anomalies, determining a scheduling strategy for an executor corresponding to the computing job; Executing a scheduling operation corresponding to the scheduling strategy on the executor; The real-time monitoring of resource usage information of computing jobs in the computing cluster includes: Collecting at least one of the following resource usage data of computing jobs in the computing cluster in real time: CPU utilization data, memory utilization data, and garbage collection data; Aggregating the resource usage data within a preset time period into resource usage information of the computing job; The resource usage anomaly includes insufficient memory or free memory, and determining that the computing job has a resource usage anomaly according to the resource usage information includes: Determine the memory utilization rate within a preset time period according to the memory utilization data, and determine the average garbage collection time within the preset time period according to the garbage collection data; When the memory utilization rate is greater than or equal to the first memory threshold within the preset time period, and the average garbage collection time is greater than or equal to the first time period threshold, determining that the resource usage anomaly is insufficient memory; When the memory utilization is less than or equal to the second memory threshold and the average garbage collection duration is less than or equal to the second duration threshold, it is determined that the resource usage anomaly is memory idle.

2. The method according to claim 1, characterized in that: The determining of the scheduling strategy of the executor corresponding to the computing job includes: Determine the exception type corresponding to the resource usage exception; When the exception type is a memory exception, determining the scheduling strategy corresponding to the exception type is: adjusting the number of executors corresponding to the computing job and / or adjusting the memory allocation of the executor; When the exception type is a CPU exception, the scheduling strategy corresponding to the exception type is determined as: adjusting the number of executors corresponding to the computing job and / or adjusting the number of CPU cores corresponding to the executors.

3. The method according to claim 1, characterized in that When it is determined that the resource usage anomaly is insufficient memory, determining a scheduling strategy for the executor corresponding to the computing job includes: Obtain the memory allocation corresponding to the executor and the maximum allocatable memory of a single computing node in the computing cluster; When the difference between the memory allocation amount and the maximum allocatable memory amount meets a first preset condition, determining that the scheduling strategy is to increase the memory allocation amount of the executor; When the difference between the memory allocation amount and the maximum allocatable memory amount does not satisfy the first preset condition, determining the scheduling strategy is to increase the number of executors corresponding to the computing job.

4. The method according to claim 1, characterized in that: When it is determined that there is free memory in the computing job, determining a scheduling strategy for an executor corresponding to the computing job includes: Determine the CPU utilization within the preset time period according to the CPU utilization data; When the CPU utilization is greater than or equal to a first CPU threshold, determining that the scheduling strategy is to reduce the memory allocation of the executor; When the CPU utilization is less than the first CPU threshold, determining the scheduling strategy is to reduce the number of executors corresponding to the computing job.

5. The method according to claim 2, characterized in that: The scheduling strategy also includes: scheduling duration; The executing the scheduling operation corresponding to the scheduling strategy on the executor includes: During the scheduling duration, the memory allocation amount, the number of executors and / or the number of CPU cores are adjusted.

6. The method according to claim 1, characterized in that The determining, according to the garbage collection data, an average garbage collection duration within a preset duration includes: Counting the total garbage collection duration corresponding to the computing job within the preset duration according to the garbage collection data; Obtain the number of executors corresponding to the computing job; The total garbage collection duration is divided by the number of executors to obtain the average garbage collection duration corresponding to each executor.

7. A resource scheduling device, characterized in that: include: The monitoring module is used to monitor the resource usage information of computing jobs in the computing cluster in real time; An analysis module, configured to determine a scheduling strategy for an executor corresponding to the computing job when it is determined according to the resource usage information that the computing job has resource usage anomalies; A scheduling module, used for executing a scheduling operation corresponding to the scheduling strategy on the executor; The monitoring module is used to collect at least one of the following resource usage data of the computing job in the computing cluster in real time: CPU utilization data, memory utilization data and garbage collection data; and summarize the resource usage data within a preset time period into resource usage information of the computing job; The analysis module is used to determine the memory utilization rate within a preset time period according to the memory utilization data, and to determine the average garbage collection time within the preset time period according to the garbage collection data; When the memory utilization rate is greater than or equal to the first memory threshold within the preset time period, and the average garbage collection time is greater than or equal to the first time period threshold, determining that the resource usage anomaly is insufficient memory; When the memory utilization is less than or equal to the second memory threshold and the average garbage collection duration is less than or equal to the second duration threshold, it is determined that the resource usage anomaly is memory idle.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.