A job scheduling method, a job scheduling system and a storage medium

Abstract

The application provides a job scheduling method, a job scheduling system and a storage medium, wherein the method comprises the following steps: in the process that the job scheduling system polls and traverses a job instance list at a preset cycle frequency, obtaining executable time and configuration information of a job j in the job instance list; in the case that the job j meets a scheduling condition based on the configuration information, obtaining historical execution record information of the job j in a last scheduling cycle N; wherein the historical execution record information comprises an actual execution duration T, a planned application resource number and an actual used resource number; determining a job resource idle factor of the job j in the last scheduling cycle N based on the planned application resource number and the actual used resource number; determining a response ratio of the job j based on the configuration information, the executable time, the job resource idle factor and the actual execution duration; and sending a resource application of the job j to a resource manager of a big data computing platform according to the response ratio of the job j.

Description

Technical Field

[0001] This application relates to the field of data processing technology in financial technology (Fintech), and includes, but is not limited to, a job scheduling method, a job scheduling system, and a storage medium. Background Technology

[0002] With the development of computing, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, Fintech also places higher demands on technology. In the Fintech field, with the continuous enrichment and improvement of the banking and financial system, and the rapid growth of data volume within financial systems, financial systems typically use Hadoop (a distributed computing platform) for data processing. YARN (Yet Another Resource Negotiator), a core component of the Hadoop platform, is a new type of resource manager. Here, YARN is a general-purpose resource management system that provides unified resource management and scheduling for Hadoop computing tasks.

[0003] In related technologies, batch job scheduling methods involve a job scheduling system calculating the response ratio of each job based on its pre-execution waiting time and its average execution time within the previous period N. The job with the highest response ratio is then selected from the batch jobs for scheduling. This involves submitting the resource request of the job with the highest response ratio to the Hadoop big data computing platform, which then allocates resources to the job based on the stored data and the resource request, thus scheduling and executing the job. However, during the historical execution of jobs, due to actual resource allocation issues in YARN, the obtained average execution time is inaccurate, leading to inaccurate calculated response ratios. Summary of the Invention

[0004] This application provides a job scheduling method, a job scheduling system, and a storage medium to solve the problem in related technologies where the average execution time obtained is inaccurate due to the actual resource allocation problem of YARN, which in turn leads to inaccurate response ratio calculation.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a job scheduling method, including:

[0007] During the process of polling and traversing the job instance list at a preset periodic frequency, the executable time and configuration information of job j in the job instance list are obtained.

[0008] Based on the configuration information, if it is determined that job j meets the scheduling conditions, the historical execution record information of job j in the previous scheduling period N is obtained; wherein, the historical execution record information includes the actual execution duration T, the planned number of resources requested, and the actual number of resources used;

[0009] Based on the planned number of resources requested and the actual number of resources used, the job resource idle factor of job j in the previous scheduling period N is determined;

[0010] Based on the configuration information, the executable time, the job resource idle factor, and the actual execution duration, the response ratio of job j is determined;

[0011] Based on the response ratio of job j, a resource request for job j is sent to the resource manager of the big data computing platform.

[0012] This application provides a job scheduling device, including:

[0013] The module is used to obtain the executable time and configuration information of job j in the job instance list during the process of polling and traversing the job instance list at a preset periodic frequency in the job scheduling system.

[0014] The obtaining module is further configured to, based on the configuration information and determining that the job j meets the scheduling conditions, obtain historical execution record information of the job j in the previous scheduling period N; wherein, the historical execution record information includes the actual execution duration T, the planned number of resources requested, and the actual number of resources used;

[0015] The determination module is used to determine the job resource idle factor of job j in the previous scheduling period N based on the planned number of resources requested and the actual number of resources used;

[0016] The determining module is further configured to determine the response ratio of job j based on the configuration information, the executable time, the job resource idle factor, and the actual execution duration;

[0017] The sending module is used to send the resource request of job j to the resource manager of the big data computing platform according to the response ratio of job j.

[0018] This application provides a job scheduling system, including:

[0019] Memory, used to store executable instructions;

[0020] The processor implements the above method when executing executable instructions stored in memory.

[0021] This application provides a storage medium storing executable instructions for inducing a processor to execute the above-described method.

[0022] The embodiments of this application have the following beneficial effects:

[0023] This application's embodiments address the problem in related technologies where inaccurate response ratios, caused by resource idleness, lead to errors in job scheduling order. Here, the job scheduling system calculates job resource idle factors under different resource states and, based on the job execution time under different resource states and the corresponding job resource idle factors, calculates the job's execution time relative to a resource-sufficient state. This improves the accuracy of the average job execution time, thereby increasing the accuracy of the job's response ratio at the current moment and enabling the job scheduling system to schedule job execution more rationally. Simultaneously, it reduces the deviation rate of job response values ​​under different resource states, optimizing the job scheduling order. Attached Figure Description

[0024] Figure 1A This is the intent of the execution timeline of job K during resource-scarce periods, as provided in related technologies;

[0025] Figure 1B This is the intent of the execution timeline of job K provided in related technologies when resources are sufficient;

[0026] Figure 2A This is a schematic diagram illustrating the execution of job K during periods of resource scarcity, as provided in related technologies.

[0027] Figure 2B This is a schematic diagram illustrating the execution of job K under conditions of sufficient resources, as provided in related technologies.

[0028] Figure 3 This is a schematic diagram of an optional architecture of the terminal provided in an embodiment of this application;

[0029] Figure 4 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram illustrating the execution order of multiple jobs provided in the embodiments of this application;

[0031] Figure 6 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0032] Figure 7 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram illustrating four scenarios for classifying tasks based on importance and urgency, as provided in the embodiments of this application.

[0034] Figure 9 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0035] Figure 10 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0036] Figure 11 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0037] Figure 12 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0038] Figure 13 This is an optional flowchart illustrating the job scheduling method provided in an embodiment of this application;

[0039] Figure 14 This is an optional flowchart of the job scheduling method provided in the embodiments of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0042] To better understand the job scheduling method provided in the embodiments of this application, the job scheduling methods in related technologies will first be described:

[0043] The job scheduling system, acting as a workflow-based batch job scheduler, is responsible for scheduling big data jobs according to dependencies and predefined scheduling strategies. The job scheduling system submits computation jobs to the Hadoop big data computing platform, which then calculates the corresponding data based on the stored data and the computational logic submitted by the scheduling system.

[0044] In related technologies, the batch job scheduling method uses the Highest Response Ratio Next (HRN) algorithm. The HRN scheduling algorithm considers the waiting time of each job and the average execution time in the previous period N to obtain the response ratio of each job, and selects the job with the highest response ratio from all jobs for execution.

[0045] Here, the response ratio R can be calculated as follows (Formula 1).

[0046]

[0047] Where T is the average execution time within the previous period N, and W is the waiting time of the job in the backup queue. When the job scheduling system needs to schedule jobs, it calculates the response ratio of each job and assigns the job with the highest response ratio R to execution. In this way, even for long jobs, as their waiting time increases, W / T also increases, giving them a chance to be prioritized for execution.

[0048] However, the average execution time T within the previous period N refers to the estimated execution time for a Hive on Spark job. Using the calculation method described above (Formula 1), there is a significant deviation due to actual YARN resource allocation issues. In other words, for the same job with the same amount of data, the average execution time can vary considerably depending on whether YARN resources are sufficient or insufficient. This inaccuracy in the average execution time T causes the highest response ratio first algorithm to fail to deliver as expected; that is, a short job may be estimated as a long job. Under the same waiting time W, the highest response ratio may be incorrectly calculated as lower than other jobs, resulting in later delivery than other jobs.

[0049] For example, the batch processing job is identified as ads_rpt.blc_scale_trade_1d, and this job is referred to as job K. Job K is of type Hive on Spark, and its resource requirement is 40 executors. The job scheduling system needs to submit job K at 05:20 (when resources are scarce) and 21:28 (when resources are plentiful) on the same day, and analyze the execution time of job K through the Spark monitoring (User Interface, UI) interface, referring to... Figure 1A , Figure 1B , Figure 2A and Figure 2B As shown, Figure 1A This diagram illustrates the execution timeline of job K during periods of resource scarcity. Figure 1B This diagram illustrates the execution timeline of job K when resources are plentiful. Figure 2A This diagram illustrates the execution of job K during periods of resource scarcity. Figure 2B The diagram illustrates the execution of job K when resources are plentiful.

[0050] See Figure 1A and Figure 2A When job K in the job scheduling system starts batch processing at 5:20, due to resource constraints in the YARN cluster, only the driver module starts before job K (job0) begins. During job0's execution, only 9 executors start successively. Since job K requires 40 executors, the remaining executors are added gradually after 5:45. Furthermore, job K's execution time during this resource-constrained period is 16 minutes. It should be noted that due to the characteristics of Spark tasks, as soon as resources are allocated, the cluster will start batch processing based on the existing resources, rather than waiting for all resources to be allocated before starting batch processing.

[0051] See Figure 1B and Figure 2B When job K in the job scheduling system runs at 21:28, due to sufficient (or idle) YARN set resources, all 40 executors required for job 0 to execute have been started before job 0 starts, and the execution time of job K when resources are sufficient is 6.6 minutes. Clearly, when YARN set resources are sufficient, the execution time of job 0 is significantly shortened.

[0052] Here, during resource-scarce periods, job K actually uses 9 executors, with an actual execution time of 16 minutes = 960 seconds. The waiting time W is assumed to be 300 seconds. Using the above (Formula 1), the response ratio R of job K during resource-scarce periods is calculated. u The value is 1.3125. When resources are plentiful, job K actually uses 40 executors, with an actual execution time of 6.6 min = 396 s. The waiting time W is also assumed to be 300 s. Using the above (Formula 1), the response ratio R of job K when resources are plentiful is calculated. e The value is 1.7576; further, the deviation rate RD of the response ratio of task K under resource scarcity and resource abundance conditions is determined as:

[0053] As can be seen from the above, under the same job and the same amount of data (computation), the execution time of job K varies greatly under different resource allocation conditions. This leads to an inaccurate average execution time of job K, which in turn leads to an inaccurate response ratio R value of job K, ultimately resulting in reduced competitiveness with other jobs.

[0054] The following describes an exemplary application of the job scheduling system provided in this application. This job scheduling system can be implemented as any terminal with a screen display function, such as a laptop, tablet, desktop computer, or intelligent robot, or as a server. The following will describe an exemplary application of the job scheduling system when implemented as a terminal.

[0055] See Figure 3 , Figure 3 This is a schematic diagram of the structure of the terminal 100 provided in the embodiments of this application. Figure 3 The terminal 100 shown includes at least one processor 110, at least one network interface 120, a user interface 130, and a memory 150. The various components in the terminal 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 140.

[0056] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] User interface 130 includes one or more output devices 131 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0058] Memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 150 may optionally include one or more storage devices physically located remote from processor 110. Memory 150 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0059] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0060] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0061] The input processing module 153 is used to detect and translate one or more user inputs or interactions from one or more input devices 132.

[0062] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 3 A job scheduling device 154 stored in memory 150 is shown. This job scheduling device 154 can be a job scheduling device in terminal 100, and can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 1541, a determination module 1542, a sending module 1543, and a processing module 1544. These modules are logically linked and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0063] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the job scheduling method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0064] The job scheduling method provided in this application will be described below with reference to the exemplary application and implementation of the terminal 100 provided in the embodiments of this application. See also Figure 4 , Figure 4 This is an optional flowchart illustrating the job scheduling method provided in this application embodiment, which will be combined with... Figure 4 The steps shown are explained below.

[0065] Step 201: During the process of polling and traversing the job instance list at a preset periodic frequency in the job scheduling system, obtain the executable time and configuration information of job j in the job instance list.

[0066] In this embodiment of the application, the preset period frequency is a fixed period frequency that is set in advance. For example, the preset period frequency can be 5 seconds.

[0067] In this embodiment of the application, the executable time can be understood as the start time when the job is currently executable.

[0068] In this embodiment of the application, a job can be understood as an operation performed on data, and job j is at least one of the job instance lists. Jobs include short jobs and long jobs. A short job can be understood as a job with a short execution time, and a long job can be understood as a job with a long execution time.

[0069] In this embodiment of the application, a job instance can be understood as a job executed within each scheduling cycle; for example, if a job is scheduled to be executed once a day, then the job executed each day is an instance of that job. A job instance list can be understood as a list consisting of multiple jobs.

[0070] In this embodiment, the configuration information of the job includes, but is not limited to, job identifier, job description, job type, job dependent jobs, start time within the scheduling period, timeout start time, resource requirements, estimated execution duration, job fixed priority, and number of retries.

[0071] In a feasible offline big data computing scenario, refer to Figure 5 As shown, the job scheduling system contains a large offline computing job M, which can be divided into four smaller jobs: H, A, B, and C. The execution of jobs A and B depends on the execution result of job H, while there is no dependency between jobs A and B. The execution of job C depends on the execution results of jobs A and B. The execution process among jobs H, A, B, and C forms a Directed Acyclic Graph (DAG). Table 1 illustrates the configuration information for different types of jobs.

[0072]

[0073] Table 1

[0074] Step 202: Based on the configuration information, if it is determined that job j meets the scheduling conditions, obtain the historical execution record information of job j in the previous scheduling period N.

[0075] The historical execution record information includes the actual execution duration, the number of resources planned to be requested, and the number of resources actually used. Of course, the historical execution record information may also include the job execution date, job name, actual number of retries, and resource status at the time of job execution.

[0076] In this embodiment, the planned resource request number is the number of resources that need to be requested determined from the job's configuration information, while the actual resource usage number is the actual number of resources allocated to job A during the actual execution of the job.

[0077] It should be noted that the relationship between the actual number of resources E_Used used during the actual execution of the job and the actual execution time T is as follows:

[0078] In a feasible scenario, taking assignment A and assignment B as examples, refer to... Figure 5 As shown in Table 1, assuming the scheduling period N is 5 days and the current date is 20220506, and jobs A and B are executed once a day, the historical execution records of jobs A and B in the previous scheduling period N (20220501-20220505) are shown in Table 2.

[0079]

[0080] Table 2

[0081] In this embodiment, the scheduling period N is a preset duration; for example, N can be a duration set in weeks, N can also be a duration set in months, and of course, N can be other durations. This application does not impose specific limitations on this. The execution process of the job can be understood as the process in which the job is repeatedly executed in the same order at fixed time intervals within the preset duration.

[0082] In this embodiment of the application, the actual execution time can be understood as the execution time of job j when it is executed for the nth time in the previous scheduling period N, where n is an integer greater than or equal to 1 and less than or equal to N, N is the total number of days in a scheduling period, and N is a positive integer.

[0083] In some embodiments, the configuration information of job j includes the job type of job j, the dependent jobs of job j, and the start time of job j within the scheduling period. The process of determining that job j meets the scheduling conditions based on the configuration information in step 202 can be implemented as follows: the job type is a preset job type, the dependent jobs of job j have been completed, and the executable time of job j reaches the start time within the scheduling period.

[0084] In this embodiment, the job types include, but are not limited to, Spark ON Hive and Java. Here, Spark ON Hive can be understood as data being stored in Hive, with users using HQL for data processing and analysis. However, when a user submits an execution request, the underlying layer undergoes Hive parsing, optimization, and compilation, ultimately running as a Spark job. It should be noted that, due to the characteristics of Java, the cluster will only perform batch job execution based on available resources once all resources have been allocated.

[0085] In this embodiment of the application, the preset job type can be Spark ON Hive.

[0086] In this embodiment of the application, the fact that the dependent job of job j has been successfully executed can be understood as the upstream job of job j having been completed.

[0087] In this embodiment of the application, the executable time of job j reaching the start time within the scheduling period can be understood as the executable time of job j being after the start time within the scheduling period. For example, if the start time within the scheduling period of job j is set to 05:00:00 every day, and the current executable time of job j is 05:30:05, the current executable time of job j is after the start time within the scheduling period of job j, therefore the executable time of job j reaches the start time within the scheduling period.

[0088] It should be noted that Spark is a fast and general-purpose computing engine designed specifically for large-scale data processing. Spark uses in-memory computing technology, which can perform analysis and computation in memory before the data is written to disk.

[0089] The HIVE mentioned above is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The HIVE data warehouse tool can map structured data files to a database table and provide SQL query functionality.

[0090] The Hive SQL described above can transform SQL statements into MapReduce tasks for execution, using its own SQL to query and analyze data stored in the Hadoop Distributed File System. Hive uses SQL-like statements to achieve fast MapReduce statistics, allowing users unfamiliar with MapReduce to easily query, summarize, and analyze data using SQL.

[0091] In some feasible scenarios, if the dependent jobs of job j have been completed and the executable time of job j reaches the start time within the scheduling period, but the job type of job j is not the preset job type, the job scheduling system can calculate the response ratio R of job j according to the job scheduling method in related technologies.

[0092] Step 203: Based on the planned number of resources requested and the actual number of resources used, determine the job resource idle factor of job j in the previous scheduling period N.

[0093] In this embodiment, the job resource idle factor is determined by the planned number of resources requested by job j and the actual number of resources used.

[0094] Step 204: Determine the response ratio of job j based on configuration information, executable time, job resource idle factor, and actual execution time.

[0095] Step 205: Based on the response ratio of job j, send the resource request for job j to the resource manager of the big data computing platform.

[0096] In this embodiment, the big data computing platform can be Hadoop, and the resource manager can be Yarn. In Yarn, the resource allocation unit is the Executor. An Executor is a Java Virtual Machine (JVM) process in a Spark program, responsible for executing the specific tasks of a Spark job. A Spark task sets the resources required for the entire Spark job by setting the number of central processing unit (CPU) cores used by a single Executor and the amount of memory multiplied by the number of Executors.

[0097] Here, resources include, but are not limited to, the operating system's memory (mem), CPU, input / output (IO) transfer speed, disk, and network bandwidth.

[0098] In this embodiment, firstly, during the process of the job scheduling system polling and traversing the job instance list at a preset periodic frequency, the executable time and configuration information of job j in the job instance list are obtained; then, based on the configuration information, if it is determined that job j meets the scheduling conditions, the historical execution record information of job j in the previous scheduling period N is obtained, such as the actual execution time, the number of resources planned to be requested, and the number of resources actually used; then, based on the number of resources planned to be requested and the number of resources actually used, the job resource idle factor of job j in the previous scheduling period N is determined; further, based on the configuration information, executable time, job resource idle factor Z, and actual execution time T, the response ratio of job j is determined; finally, if there are multiple jobs j that meet the conditions, the priority of the jobs is determined according to the size of the response ratio, and the resource request of the higher priority job is sent to the resource manager of the big data computing platform first.

[0099] This application provides a job scheduling method. During the process of polling and traversing a job instance list at a preset periodic frequency in the job scheduling system, the executable time and configuration information of job j in the job instance list are obtained. Based on the configuration information, if it is determined that job j meets the scheduling conditions, the historical execution record information of job j in the previous scheduling period N is obtained. The historical execution record information includes the actual execution duration T, the planned number of resources requested, and the actual number of resources used. Based on the planned number of resources requested and the actual number of resources used, the job resource idle factor of job j in the previous scheduling period N is determined. Based on the configuration information, executable time, job resource idle factor, and actual execution duration, the response ratio of job j is determined. According to the response ratio of job j, a resource request for job j is sent to the resource manager of the big data computing platform. Thus, by introducing a job resource idle factor, the problem of inaccurate response ratio calculations due to resource idleness issues in related technologies is solved, which in turn leads to errors in job scheduling order. Here, the job scheduling system calculates the job resource idle factor under different resource states, and based on the execution time of the job under different resource states and the corresponding job resource idle factor, it calculates the execution time of the job relative to the execution time under a resource-sufficient state. This improves the accuracy of the average execution time of the job, thereby improving the accuracy of the job's response ratio at the current moment, enabling the job scheduling system to schedule job execution more rationally. At the same time, it reduces the deviation rate of job response values ​​when the job is under different resource states, optimizing the job scheduling order.

[0100] See Figure 6 , Figure 6 This is an optional flowchart illustrating the job scheduling method provided in this application embodiment, which will be combined with... Figure 5 The steps shown are explained below.

[0101] Step 301: During the process of polling and traversing the job instance list at a preset periodic frequency in the job scheduling system, obtain the executable time and configuration information of job j in the job instance list.

[0102] Step 302: Based on the configuration information, if it is determined that job j meets the scheduling conditions, obtain the historical execution record information of job j in the previous scheduling period N.

[0103] The historical execution record information includes the actual execution duration, the number of resources planned for application, and the number of resources actually used.

[0104] Step 303: Based on the planned number of resources requested and the actual number of resources used, determine the job resource idle factor of job j in the previous scheduling period N.

[0105] In some embodiments, step 303, which determines the job resource idle factor of job j in the previous scheduling period N based on the planned resource request number and the actual resource usage number, is combined with... Figure 7 To elaborate further,

[0106] Step 401: For the nth execution process of job j in the previous scheduling period N, calculate the first sum of the planned resource request number and the actual resource usage number of job j.

[0107] Step 402: Calculate the first product of the actual number of resources used and the first preset value.

[0108] In this embodiment of the application, the first preset value can be 2.

[0109] Step 403: Calculate the ratio of the first sum and the first product to obtain the job resource idle factor of job j in the nth time, until the N job resource idle factors of job j in the previous scheduling period N are obtained.

[0110] In this embodiment, after the job scheduling system obtains the historical execution record information of job j in the previous scheduling period N, it calculates the planned resource request number E_Need for job j in the nth execution process within the previous scheduling period N. n Compared with the actual number of resources used, E_Used n First, calculate the sum of the first values; then, calculate the actual resource usage E_Used. n The first product of the first sum and the first product is used as the job resource idle factor Zn of job j in the nth time. Finally, for the N execution processes of job j in the previous scheduling period N, the N job resource idle factors Zn of job j in the previous scheduling period N are obtained.

[0111] In this embodiment of the application, the job resource idle factor Zn of job j in the nth time can be obtained by the following (Formula 2).

[0112]

[0113] Among them, Z n E_Need is the job resource idle factor for job j during the nth scheduling cycle N; n The number of resources to be requested by job j before its nth execution within a scheduling period N; E_Used n Let be the number of resources actually used by job j during the nth actual execution within a scheduling period N; here, both the planned resource request number and the actual resource usage number can be obtained from the historical execution record information of the nth execution.

[0114] In a feasible scenario, taking jobs A and B as examples, referring to Tables 1 and 2, job A plans to request 40 executors. Within one scheduling period N, job A actually uses 40 executors, 33 executors, 4 executors, 10 executors, and 6 executors respectively. Job B plans to request 4 executors, and within one scheduling period N, job B actually uses 4 executors. The job scheduling system uses the above (Formula 2) to calculate the N job resource idle factors Z for job A within one scheduling period N. A The results are shown in Table 3, which includes the job execution date, job name, planned resource request quantity for job A, and actual resource usage quantity for job A. The job scheduling system uses the above (Formula 2) to calculate the N job resource idle factors Z for job B within a scheduling period N. B n, the results are shown in Table 4, which includes the job execution date, job name, the planned resource request number for job B, and the actual resource usage number for job B.

[0115] Job execution date Assignment Name Planned number of resources requested Actual number of resources used Job resource idle factor 20220501 Homework A 40 executors 40 executors 1 20220502 Homework A 40 executors 33 executors 1.1061 20220503 Homework A 40 executors 4 executors 5.5 20220504 Homework A 40 executors 10 executors 2.5 20220505 Homework A 40 executors 6 executors 3.8333

[0116] Table 3

[0117] Job execution date Assignment Name Planned number of resources requested Actual number of resources used Job resource idle factor 20220501 Homework B 4 executors 4 executors 1 20220502 Homework B 4 executors 4 executors 1 20220503 Homework B 4 executors 4 executors 1 20220504 Homework B 4 executors 4 executors 1 20220505 Homework B 4 executors 4 executors 1

[0118] Table 4

[0119] Step 304: Obtain the execution completion time of the dependent jobs of job j, the number of times job j failed to execute during this execution, and the maximum fixed priority set for the jobs in the job instance list.

[0120] In this embodiment, the maximum fixed priority is the highest level determined based on the task's business importance and output time, i.e., the maximum value that can be set for the task's fixed priority; for example, the maximum fixed priority L... max The fixed priority of a job can be 10, and the fixed priority of a job can be a positive integer between 1 and 10. It should be noted that the higher the fixed priority of a job, the higher the priority. That is to say, under at least two identical conditions, the job with the higher priority will be scheduled first.

[0121] In this embodiment of the application, the execution failure count is the number of times job j fails to execute in each actual execution process. The execution failure count is also called the actual retries count. The number of times the job fails to execute in this execution process is also called the actual retries count of the job in this execution process.

[0122] In a feasible application scenario, refer to Figure 8As shown, tasks are categorized based on their business importance (level of importance) and output time (level of urgency). The horizontal axis represents increasing urgency from left to right, and the vertical axis represents increasing importance from bottom to top. This allows for the classification of tasks into four categories: important and urgent, unimportant but urgent, important but not urgent, and unimportant and not urgent. For important and urgent tasks, the higher the fixed priority value, the lower the fixed priority value for unimportant and not urgent tasks.

[0123] Step 305: Based on the executable time and the execution completion time of dependent jobs, determine the waiting time of job j in the backup status queue.

[0124] In this embodiment, after the dependent jobs of job j have completed execution, the job scheduling system obtains the execution completion time of the dependent jobs and polls the job instance list at a preset periodic frequency to determine the initial start time when job j should have initiated scheduling; wherein, the initial start time is the time obtained by varying the execution completion time of the dependent jobs of job j at a preset periodic frequency. Further, the job scheduling system calculates the executable time of job j and subtracts the initial start time of job j to obtain the waiting time W of job j in the backup status queue.

[0125] In a feasible scenario, referring to Table 1, taking jobs A, B, and H as examples, assuming the current date is 20220506 and the current time is 05:30:05 AM; since jobs A and B depend on the execution result of job H, and job H completes execution at 05:10:00, the executable time for both jobs A and B is 05:30:05, reaching the start time of their respective scheduling cycles at 05:00:00; furthermore, both jobs A and B are of the preset job type Spark ONHive, thus both jobs A and B meet the scheduling conditions. It should be noted that since job H completes execution at 05:10:00, and the preset cycle frequency is 5 seconds, jobs A and B should have initiated scheduling at 05:10:05; however, due to insufficient YARN resources, the execution of jobs A and B failed until 05:30:05. At this point, the waiting time for both job A and job B is: W = Date(05:30:05) - Date(05:10:05) = 1200s. Job A failed 1 time between 05:10:05 and 05:30:05, and job B failed 2 times between 05:10:05 and 05:30:05. It should be noted that the number of times the jobs failed between 05:10:05 and 05:30:05 is also called the actual number of retries between 05:10:05 and 05:30:05.

[0126] Step 306: Determine the dynamic priority of job j based on the executable time, timeout start time, waiting time, number of retries, number of execution failures, and maximum fixed priority.

[0127] In this embodiment, the dynamic priority of a job is determined by the number of times the job has failed in its historical execution process, and the time difference between the executable time and the timeout start time during the current execution process. Here, the greater the time difference between the executable time and the timeout start time, the higher the dynamic priority of the job; the more times the job fails, the lower the dynamic priority of the job.

[0128] In this embodiment of the application, the configuration information may further include: the fixed priority of job j, the timeout start time, and the number of retries. The number of retries is the number of times the job is allowed to fail during actual execution.

[0129] In this embodiment of the application, the job scheduling system determines the dynamic priority of job j based on the executable time, timeout start time, waiting time, number of retries, number of execution failures, and maximum fixed priority of job j.

[0130] In some embodiments, the process of determining the dynamic priority of job j based on executable time, timeout start time, waiting time, number of retries, number of execution failures, and maximum fixed priority in step 306 is combined with Figure 9 To elaborate further,

[0131] Step 501: Calculate the difference between the executable time and the timeout start time to obtain the timeout duration of job j.

[0132] In this embodiment of the application, the configuration information of job j includes the timeout start time, and the timeout duration CT is equal to the difference between the executable time of job j and the timeout start time.

[0133] In one feasible scenario, referring to Table 1, taking assignments A, B, and H as examples, assuming the current date is 20220506 and the current time is 05:30:05 AM; the timeout start time for both assignments A and B is set to 05:20:00 AM every day, so the timeout duration CT for assignment A is Date(05:30:05) - Date(05:20:00) = 605 seconds, and the timeout duration CT for assignment B is Date(05:30:05) - Date(05:20:00) = 605 seconds.

[0134] Step 502: Calculate the second sum of the waiting time and the timeout time.

[0135] Step 503: Calculate the ratio of the second sum to the first waiting time.

[0136] Step 504: Calculate the first difference between the number of retry attempts and the number of execution failures.

[0137] Step 505: Calculate the ratio of the first difference to the second number of retry attempts.

[0138] Step 506: Calculate the product of the maximum fixed priority, the first ratio, and the second ratio to obtain the dynamic priority of the job.

[0139] In this embodiment, the job scheduling system subtracts the timeout start time from the executable time, and the difference is used as the timeout duration CT of job j. The system calculates the second sum (W+CT) of the waiting time W and the timeout duration CT, and then calculates the first ratio (W+CT) / W of the second sum (W+CT) to the waiting time W. Further, it calculates the first difference (ET-ST) between the number of retry attempts ET and the number of execution failures ST; it then calculates the second ratio (ET-ST) / ET of the first difference (ET-ST) to the number of retry attempts ET. Finally, it sets the maximum fixed priority L... max The product of the first ratio (W+CT) / W and the second ratio (ET-ST) / ET is used as the dynamic priority L of job j. adj .

[0140] In this embodiment of the application, the dynamic priority of job j can be obtained as follows (Formula 3).

[0141]

[0142] Among them, L adj L represents the dynamic priority of task j. max The maximum fixed priority is given by W, where W is the waiting time of job j in the backup status queue, CT is the timeout duration of job j, ET is the number of retries set for job j, and ST is the number of execution failures of job j in this execution process.

[0143] In a feasible scenario, referring to Table 1, taking tasks A, B, and H as examples, assuming the current date is 20220506 and the current time is 05:30:05 AM; the timeout start time for both tasks A and B is set to 05:20:00 AM daily, the timeout duration for task A is CT = 605s, the waiting time for task A is W = 1200s, the number of retries for task A is ET = 5, and the number of execution failures for task A is ST = 1; the timeout duration for task B is CT = 605s, the waiting time for task B is W = 1200s, the number of retries for task B is ET = 5, and the number of execution failures for task B is ST = 2; and the maximum fixed priority L... max =10, and using the above (Formula 3), calculate the dynamic priority L of task A.adj Given a value of 12.03, calculate the dynamic priority L of task B. adj It is 9.03.

[0144] Step 307: Based on the actual execution time of job j during the nth execution process in the previous scheduling period N and the corresponding job resource idle factor, determine the average execution time of job j in the previous scheduling period N.

[0145] In this embodiment of the application, the job scheduling system obtains the actual execution time Tn and job resource idle factor Zn corresponding to the nth execution of job j in the previous scheduling period N from the historical execution record information of job j. For the previous scheduling period N, based on the obtained N actual execution times Tn and N job resource idle factors Zn, the average execution time T of job j in the previous scheduling period N is determined.

[0146] In some embodiments, step 307, which determines the average execution time T of job j within the previous scheduling period N based on the actual execution time Tn of job j during its nth execution process in the previous scheduling period N, and the job resource idle factor Zn, is combined with... Figure 10 To provide further clarification,

[0147] Step 601: Obtain the maximum and minimum actual execution times among the N actual execution times.

[0148] Step 602: Calculate the third ratio of the actual execution time to the corresponding job resource idle factor during the nth execution process.

[0149] Step 603: Subtract the third ratio corresponding to the longest actual execution time and the third ratio corresponding to the shortest actual execution time from the N third ratios to obtain N-2 third ratios.

[0150] Step 604: Calculate the average of the N-2 third ratios to obtain the average execution time.

[0151] In this embodiment of the application, the average execution time T of job j can be obtained as follows (Formula 4).

[0152]

[0153] Where T is the average execution time of job j. Let T be the third ratio of the actual execution time Tn to the job resource idle factor Zn during the nth execution. max Z represents the maximum actual execution time of job j within the previous scheduling period N. max T represents the maximum job resource idle factor corresponding to the maximum actual execution time of job j. minZ represents the minimum actual execution time of job j within the previous scheduling period N. min This represents the minimum job resource idle factor corresponding to the minimum actual execution time of job j. Indicates the maximum actual execution time T max The corresponding third ratio, Represents the minimum actual execution time T min The corresponding third ratio.

[0154] In a feasible scenario, referring to Tables 2, 3, and 4, taking jobs A and B as examples, the scheduling period N = 5. The actual execution times of job A in the previous scheduling period N are 1000s, 1200s, 10000s, 4000s, and 7000s, respectively. The job resource idle factors of job A in the previous scheduling period are 1, 1.1061, 5.5, 2.5, and 3.8333, respectively. The maximum actual execution time T of job A in the previous scheduling period N is... max =10000, the minimum actual execution time T of job A in the previous scheduling cycle N. min =1000; Substituting the above parameters into Formula 4, we obtain the average execution time of task A.

[0155] Similarly, the actual execution times of job B in the previous scheduling period N were 1000s, 1100s, 1200s, 1300s, and 1100s, respectively. The job resource idle factor of job B in the previous scheduling period was 1. The maximum actual execution time T of job B in the previous scheduling period N was... max =10000, the minimum actual execution time T of job B in the previous scheduling period N. min =1000; Substituting the above parameters into Formula 4, we obtain the average execution time of task B.

[0156] Using data from related technologies as an example, job K actually uses 9 executors during resource-scarce periods, plans to request 40 executors, and has an actual execution time of 16 minutes (960 seconds). The waiting time W is assumed to be 300 seconds. Using the above (Formula 2), the job resource idle factor Z for job K during resource-scarce periods is calculated to be 3. If only the job resource idle factor is considered, the response ratio R for job K during resource-scarce periods is calculated using the following (Formula 5). u It is 1.8507.

[0157]

[0158] Where R represents the job response ratio, and W represents the job's waiting time in the ready queue. This represents the average execution time of the job within the previous scheduling period N.

[0159] When resources are plentiful, job K actually uses 40 executors. The planned resource request is for 40 executors, and the actual execution time is 6.6 min = 396 s. The waiting time W is also assumed to be 300 s. Using the above (Formula 2), the job resource idle factor Z for job K during resource-scarce periods is calculated to be 1. If only the job resource idle factor is considered, the response ratio R of job K during resource-sufficient periods is calculated using the above (Formula 5). e The value is 1.7576; furthermore, considering the idle factor of job resources, the deviation rate RD of the response ratio of job K under resource shortage and resource abundance conditions is determined as follows: In related technologies, without considering the job resource idle factor, the deviation rate RD of the response ratio of job K under resource shortage and resource abundance conditions is -25.32%. Clearly, compared with the deviation rate obtained by considering the job resource idle factor, the deviation rate obtained without considering the job resource idle factor is optimized from -25.32% to 6.31%, and the positive deviation rate is more conducive to the normal delivery of Hive on Spark tasks.

[0160] As can be seen from the above, the less resources actually allocated to a job, the longer the job execution time. This embodiment adds a resource idle factor Z, and uses the value of dividing the actual job execution time T by the resource idle factor Z as the new job execution time to participate in the calculation of the job response ratio. That is, the idle factor Z of the job resources is determined by the arithmetic mean of the sum of the number of resources requested by the job plan (such as the total number of executors) and the number of resources actually used by the job (such as the number of executors), with the actual number of resources used as the denominator. By comparing the job execution time T with the actual job idle resource factor Z, it can be closer to the execution time of the job when resources are sufficient, so the average execution time of the job is more accurate, and the calculated response ratio of the job at the current moment is also more accurate, which is more conducive to the normal delivery of Hive on Spark tasks.

[0161] Step 308: Determine the response ratio of job j based on waiting time, fixed job priority, dynamic job priority, and average execution time.

[0162] In this embodiment, the response ratio of job j is based on the waiting time W of job j and the fixed priority L of job j. fix The dynamic priority L of task j adj It is obtained by combining the average execution time T of job j in the previous scheduling period N.

[0163] In some embodiments, the process of determining the response ratio of job j based on waiting time, fixed job priority, dynamic job priority, and average execution time in step 308 is combined with Figure 11 To elaborate further,

[0164] Step 701: Determine the priority coefficient of job j based on the fixed priority, dynamic priority, and maximum fixed priority of the job.

[0165] In this embodiment, the priority coefficient CM of job j is based on the fixed priority L of job j. fix The dynamic priority L of task j adj It is obtained from the maximum fixed priority.

[0166] In some embodiments, the process of determining the priority coefficient of job j based on job fixed priority, job dynamic priority, and maximum fixed priority in step 701 is combined with Figure 12 To elaborate further,

[0167] Step 801: Calculate the third sum of the fixed priority and dynamic priority of the job.

[0168] Step 802: Calculate the ratio of the third sum to the fourth value of the maximum fixed priority.

[0169] Step 803: Calculate the sum of the fourth ratio and the second preset value to obtain the priority coefficient.

[0170] In this embodiment of the application, the priority coefficient CM of job j can be obtained as follows (Formula 6).

[0171]

[0172] Where CM is the priority coefficient of task j, L fix L represents the fixed priority of task j. adj L represents the dynamic priority of task j. max This indicates the highest fixed priority, and the second preset value is 1.

[0173] In a feasible scenario, referencing Figure 5 As shown in Table 1, taking task A and task B as examples, the maximum fixed priority L max The priority of task A is 10, and the task priority is fixed at L. fix The dynamic priority L of task A is 10. adj The value is 12.03. Substituting the above parameters into the above (Formula 6), we get the priority coefficient CM of task A as 3.20.

[0174] Similarly, the maximum fixed priority Lmax The priority of task B is 10, and the task priority is fixed at L. fix The dynamic priority L of task B is 5. adj The value is 9.03. Substituting the above parameters into the above (Formula 6), we get the priority coefficient CM of task B as 2.40.

[0175] As can be seen from the above, in this embodiment of the application, by introducing a priority coefficient for the job, different priorities are configured for different jobs under the premise of the same waiting time, and the response ratio of different jobs is calculated based on the differentiated priorities, so that the job with the larger response ratio gets priority delivery, which meets the business usage expectations; the scheduling order of the jobs is optimized, and the different scheduling of different jobs is also realized.

[0176] Step 702: Determine the response ratio of job j based on the waiting time, priority coefficient, and average execution time.

[0177] In this embodiment of the application, the response ratio R of job j is obtained based on the waiting time W of job j, the priority coefficient CM of job j, and the average execution time T of job j in the previous scheduling period N.

[0178] In some embodiments, the process of determining the response ratio of job j based on waiting time, priority coefficient, and average execution time in step 702 is combined with Figure 13 To elaborate further,

[0179] Step 901: Calculate the second product of the waiting time and the priority coefficient.

[0180] Step 902: Calculate the fifth ratio of the second product to the average execution time.

[0181] Step 903: Calculate the sum of the fifth ratio and the third preset value to obtain the response ratio of task j.

[0182] In this embodiment of the application, the response ratio of job j can be obtained as follows (Formula 7).

[0183]

[0184] Where R is the response ratio of job j, W is the waiting time of job j in the backup state queue, CM is the priority coefficient of job j, T is the average execution time of job j in the previous scheduling period N, and the third preset value is 1.

[0185] In a feasible scenario, referencing Figure 5As shown, taking job A and job B as examples, the waiting time W of job A is 1200s, the priority coefficient CM of job A is 3.20, and the average execution time T of job A in the previous scheduling period N is 1503.67s. Substituting the above parameters into the above (Formula 7), the response ratio R of job A is 3.55.

[0186] Similarly, the waiting time W of job B is 1200s, the priority coefficient CM of job B is 2.40, and the average execution time T of job B in the previous scheduling period N is 1133.33s. Substituting the above parameters into the above (Formula 7), we get the response ratio R of job B as 3.54.

[0187] Step 309: Based on the response ratio of job j, send the resource request for job j to the resource manager of the big data computing platform.

[0188] In this embodiment, after the job scheduling system obtains the response ratio of job j, it sorts all the response ratios of job j in descending order, and sends the resource request of the job corresponding to the response ratio to the resource manager of the big data computing platform according to the sorted response ratio.

[0189] In a feasible application scenario, taking jobs A and B as examples, job A has a response ratio (R) of 3.55, and job B has a response ratio of 3.54. The job scheduling system determines that job A's response ratio is greater than job B's, meaning job A receives priority for delivery and execution. In other words, the job scheduling system first sends the resource request for job A to the YARN resource manager of the big data computing platform, and then sends the resource request for job B to the YARN resource manager. The YARN resource manager will allocate resources according to the delivery order of jobs A and B, satisfying job A's resource request first, and then job B's resource request; that is, the YARN resource manager allocates resources based on its existing resources, prioritizing job A for execution, and then allocating resources based on its existing resources for job B. Furthermore, jobs A and B in the job scheduling system execute the Spark program sequentially. Then, after jobs A and B have completed execution, the execution record information of jobs A and B during this execution process is synchronized through the YARN resource synchronization module in the job scheduling system. Here, the execution record information for Task A and Task B during this execution process is shown in Table 5.

[0190] Homework Category Homework A Homework B Job label 1648799282759_508981 1648799282759_508985 The number of resources requested in this implementation plan 40 executors 4 executors Actual resource usage during this execution 33 executors 4 executors Actual execution time 1200 seconds 1000 seconds

[0191] Table 5

[0192] Here, the execution record information of job A and job B during this execution process is synchronized through the YARN resource synchronization module in the job scheduling system. The process is as follows:

[0193] First, the log parsing module is started through the YARN resource synchronization module in the job scheduling system. Second, the log parsing module calls the Spark REST Application Programming Interface (API) to query the log information of the current execution of jobs A and B from the YARN resource manager, based on the job identifiers of jobs A and B, such as spark jobId. Then, the log parsing module obtains the log information of jobs A and B and parses the execution record information of jobs A and B from the log information, such as the number of resources requested in the execution plan, the number of resources actually used, and the actual execution time. Finally, the execution record information of jobs A and B is recorded in the job execution pipeline record table in the job scheduling system to facilitate the query of the execution status of each job in the job execution pipeline record table later.

[0194] In a feasible application scenario, refer to Figure 14 As shown, the job scheduling method provided in the embodiments of this application will be further explained.

[0195] Step 1001: The job scheduling system initiates a polling traversal of the job instance list at a preset periodic frequency (e.g., 5 seconds). At 05:30:05, the traversal of job instances is initiated.

[0196] Step 1002: Based on the configuration information of each job obtained, the job scheduling system determines that the job meets the basic scheduling conditions.

[0197] In this embodiment, the configuration information of the job includes, but is not limited to, job identifier, job description, job type, job dependent jobs, start time within the scheduling period, timeout start time, resource requirements, estimated execution duration, job fixed priority, and number of retries.

[0198] In this embodiment of the application, the scheduling conditions include basic scheduling conditions. Based on the configuration information of the job, determining that the job meets the basic scheduling conditions includes: the job's dependent (upstream) jobs have been completed, and the job's executable time has reached the start time within the scheduling cycle.

[0199] Step 1003: The job scheduling system determines whether the job type is Spark ON Hive.

[0200] In this embodiment of the application, if it is determined that the job type is Spark ON Hive, then step 1004 is executed; if it is determined that the job type is not Spark ON Hive, then step 1005 is executed.

[0201] Step 1004: The job scheduling system selects the first strategy corresponding to the Spark ON Hive type.

[0202] Here, the first strategy is the strategy corresponding to the response ratio of the computational job provided in this application.

[0203] Step 1005: The job scheduling system selects the second strategy.

[0204] Here, the first strategy is the strategy corresponding to the response ratio of the computational job provided by the relevant technology.

[0205] Step 1006: The job scheduling system calculates the response ratio of each job and sorts the jobs in descending order of response ratio to obtain the sorting result.

[0206] Step 1007: The job scheduling system schedules job delivery according to the sorting results.

[0207] Step 1008: The job scheduling system sends the resource requests for each job to the resource manager YARN of the big data computing platform in the order of submission.

[0208] Step 1009: YARN resource manager allocates resources to jobs according to the order in which the jobs are submitted.

[0209] Step 1010: Each job in the job scheduling system is executed according to the allocated resources.

[0210] Step 1011: After each job is completed, trigger the YARN resource synchronization module in the job scheduling system.

[0211] Step 1012: The job scheduling system uses the log parsing module in the YARN resource synchronization module to call the Spark REST API based on the job identifier of each job to query the log information of each job in the YARN resource manager for the current execution.

[0212] Step 1013: Parse the execution record information of each job from the log information and record the execution record information of each job into the job execution flow record table in the job scheduling system.

[0213] As described above, in this embodiment, by introducing a job resource idle factor, the execution time of a job under resource-sufficient conditions can be calculated, thereby improving the accuracy of the average execution time of the job and the accuracy of the response ratio of the job at the current moment, enabling the job scheduling system to schedule job execution more reasonably. Simultaneously, it reduces the deviation rate of job response values ​​when jobs are in different resource states, optimizing the job scheduling order. By introducing a job priority coefficient, different priorities are configured for different jobs under the same waiting time condition, and the response ratio of different jobs is calculated based on the differentiated priorities, allowing jobs with higher response ratios to receive priority delivery, meeting business usage expectations. This achieves optimization of the job scheduling order and differentiated scheduling for different jobs.

[0214] The following continues to describe the exemplary structure of the job scheduling device 154 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software module stored in the job scheduling device 154 of the memory 150 can be the job scheduling device in the terminal 100, including:

[0215] The module 1541 is used to obtain the executable time and configuration information of job j in the job instance list during the process of polling and traversing the job instance list at a preset periodic frequency in the job scheduling system.

[0216] The module 1541 is further configured to obtain historical execution record information of job j in the previous scheduling period N, based on configuration information and after determining that job j meets the scheduling conditions; wherein, the historical execution record information includes the actual execution duration T, the number of resources requested in the plan, and the number of resources actually used.

[0217] Module 1542 is used to determine the job resource idle factor of job j in the previous scheduling period N based on the planned number of resources requested and the actual number of resources used.

[0218] The determination module 1542 is also used to determine the response ratio of job j based on configuration information, executable time, job resource idle factor and actual execution time;

[0219] The sending module 1543 is used to send the resource request of job j to the resource manager of the big data computing platform according to the response ratio of job j.

[0220] In some embodiments, the configuration information includes the job type of job j, the dependent jobs of job j, and the start time within the scheduling period of job j. Based on the configuration information, it is determined that job j meets the scheduling conditions, including: the job type is a preset job type, the dependent jobs of job j have been completed, and the executable time of job j has reached the start time within the scheduling period.

[0221] In some embodiments, the processing module 1544 is further configured to, for the nth execution process of job j in the previous scheduling period N, calculate the first sum of the planned resource request number and the actual resource usage number of job j; calculate the first product of the actual resource usage number and the first preset value; calculate the ratio of the first sum and the first product to obtain the job resource idle factor of job j in the nth execution, until N job resource idle factors of job j in the previous scheduling period N are obtained.

[0222] In some embodiments, the obtaining module 1541 is further configured to obtain the execution completion time of the dependent jobs of job j, the number of execution failures of job j during this execution, and the maximum fixed priority set for jobs in the job instance list; the determining module 1542 is further configured to determine the waiting time of job j in the backup status queue based on the executable time and the execution completion time of dependent jobs; determine the job dynamic priority of job j based on the executable time, timeout start time, waiting time, number of retries, number of execution failures, and maximum fixed priority; determine the average execution time of job j in the previous scheduling period N based on the actual execution time of job j in the nth execution process in the previous scheduling period N and the corresponding job resource idle factor; and determine the response ratio of job j based on the waiting time, job fixed priority, job dynamic priority, and average execution time.

[0223] In some embodiments, the processing module 1544 is further configured to calculate the difference between the executable time and the timeout start time to obtain the timeout duration of job j;

[0224] Calculate the second sum of the waiting time and the timeout time;

[0225] Calculate the ratio of the second sum to the first waiting time;

[0226] Calculate the first difference between the number of retry attempts and the number of execution failures;

[0227] Calculate the ratio of the first difference to the second number of retry attempts;

[0228] The dynamic priority of the task is obtained by multiplying the maximum fixed priority, the first ratio, and the second ratio.

[0229] In some embodiments, the obtaining module 1541 is further configured to obtain the maximum actual execution time and the minimum actual execution time among N actual execution times; the processing module 1544 is further configured to calculate the third ratio of the actual execution time to the corresponding job resource idle factor during the nth execution process; subtract the third ratio corresponding to the maximum actual execution time and the third ratio corresponding to the minimum actual execution time from the N third ratios respectively to obtain N-2 third ratios; calculate the average of the N-2 third ratios to obtain the average execution time.

[0230] In some embodiments, the determining module 1542 is further configured to determine the priority coefficient of job j based on job fixed priority, job dynamic priority and maximum fixed priority; and to determine the response ratio of job j based on waiting time, priority coefficient and average execution time.

[0231] In some embodiments, the processing module 1544 is further configured to calculate a third sum of the fixed priority and the dynamic priority of the job; calculate a fourth ratio of the third sum to the maximum fixed priority; and calculate the sum of the fourth ratio and a second preset value to obtain a priority coefficient.

[0232] In some embodiments, the processing module 1544 is further configured to calculate a second product of the waiting time and the priority coefficient; calculate a fifth ratio of the second product to the average execution time; and calculate the sum of the fifth ratio and a third preset value to obtain the response ratio of job j.

[0233] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0234] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 4 , Figures 6-7 as well as Figures 9-14 The method shown.

[0235] The storage medium provided in this application obtains the executable time and configuration information of job j in the job instance list by polling and traversing the job instance list at a preset periodic frequency in the job scheduling system; based on the configuration information, if it is determined that job j meets the scheduling conditions, it obtains the historical execution record information of job j in the previous scheduling period N; wherein, the historical execution record information includes the actual execution duration T, the planned number of resources requested, and the actual number of resources used; based on the planned number of resources requested and the actual number of resources used, it determines the job resource idle factor of job j in the previous scheduling period N; based on the configuration information, executable time, job resource idle factor, and actual execution duration, it determines the response ratio of job j; according to the response ratio of job j, it sends the resource request of job j to the resource manager of the big data computing platform. Thus, by introducing a job resource idle factor, the problem of inaccurate response ratio calculations due to resource idleness issues in related technologies is solved, which in turn leads to errors in job scheduling order. Here, the job scheduling system calculates the job resource idle factor under different resource states, and based on the execution time of the job under different resource states and the corresponding job resource idle factor, it calculates the execution time of the job relative to the execution time under a resource-sufficient state. This improves the accuracy of the average execution time of the job, thereby improving the accuracy of the job's response ratio at the current moment, enabling the job scheduling system to schedule job execution more rationally. At the same time, it reduces the deviation rate of job response values ​​when the job is under different resource states, optimizing the job scheduling order.

[0236] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0237] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0238] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTL) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0239] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A job scheduling method, characterized in that, include: During the process of polling and traversing the job instance list at a preset periodic frequency, the executable time and configuration information of job j in the job instance list are obtained. Based on the configuration information, if it is determined that job j meets the scheduling conditions, the historical execution record information of job j in the previous scheduling period N is obtained; wherein, the historical execution record information includes the actual execution duration T, the planned number of resources requested, and the actual number of resources used; Based on the planned number of resources requested and the actual number of resources used, the job resource idle factor of job j in the previous scheduling period N is determined; Based on the configuration information, the executable time, the job resource idle factor, and the actual execution duration, the response ratio of job j is determined; Based on the response ratio of job j, send the resource request of job j to the resource manager of the big data computing platform; The step of determining the job resource idle factor of job j in the previous scheduling period N based on the planned resource request number and the actual resource usage number includes: For the nth execution process of job j within the previous scheduling period N, calculate the first sum of the planned resource request number and the actual resource usage number of job j; Calculate the first product of the actual number of resources used and the first preset value; Calculate the ratio of the first sum to the first product to obtain the job resource idle factor of job j in the nth time, until N job resource idle factors of job j in the previous scheduling period N are obtained; The configuration information includes: the job type of job j, the dependent jobs of job j, the start time of job j within the scheduling period, the fixed priority of job j, the timeout start time, and the number of retries; determining the response ratio of job j based on the configuration information, the executable time, the job resource idle factor, and the actual execution duration includes: Obtain the execution completion time of the dependent jobs of job j, the number of times job j failed to execute during this execution, and the maximum fixed priority set for the jobs in the job instance list; Based on the executable time and the execution completion time of the dependent jobs, determine the waiting time of job j in the backup status queue; The dynamic priority of job j is determined based on the executable time, the timeout start time, the waiting time, the number of retries, the number of execution failures, and the maximum fixed priority. Based on the actual execution time of job j during the nth execution process in the previous scheduling period N and the corresponding job resource idle factor, the average execution time of job j in the previous scheduling period N is determined; The response ratio of job j is determined based on the waiting time, the fixed priority of the job, the dynamic priority of the job, and the average execution time.

2. The method according to claim 1, characterized in that, The step of determining that job j meets the scheduling conditions based on the configuration information includes: the job type is a preset job type, the dependent jobs of job j have been completed, and the executable time of job j reaches the start time within the scheduling period.

3. The method according to claim 1, characterized in that, The process of determining the dynamic priority of job j based on the executable time, the timeout start time, the waiting time, the number of retry attempts, the number of execution failures, and the maximum fixed priority includes: The timeout duration of job j is obtained by calculating the difference between the executable time and the timeout start time. Calculate a second sum of the waiting time and the timeout time; Calculate the first ratio of the second sum to the waiting time; Calculate the first difference between the number of retry attempts and the number of execution failures; Calculate the second ratio of the first difference to the number of retry attempts; The dynamic priority of the task is obtained by multiplying the maximum fixed priority, the first ratio, and the second ratio.

4. The method according to claim 1, characterized in that, The step of determining the average execution time of job j within the previous scheduling period N based on the actual execution time of job j during its nth execution process in the previous scheduling period N and the corresponding job resource idle factor includes: Obtain the maximum and minimum actual execution times among N actual execution times; Calculate the third ratio of the actual execution time to the corresponding job resource idle factor during the nth execution process; From the N third ratios, subtract the third ratio corresponding to the maximum actual execution time and the third ratio corresponding to the minimum actual execution time respectively to obtain N-2 third ratios; The average of the N-2 third ratios is calculated to obtain the average execution time.

5. The method according to claim 1, characterized in that, Determining the response ratio of job j based on the waiting time, the fixed priority of the job, the dynamic priority of the job, and the average execution time includes: The priority coefficient of job j is determined based on the fixed priority of the job, the dynamic priority of the job, and the maximum fixed priority. The response ratio of job j is determined based on the waiting time, the priority coefficient, and the average execution time.

6. The method according to claim 5, characterized in that, Determining the priority coefficient of job j based on the job's fixed priority, the job's dynamic priority, and the maximum fixed priority includes: Calculate the third sum of the fixed priority and the dynamic priority of the task; Calculate the ratio of the third sum to the fourth value of the maximum fixed priority; The priority coefficient is obtained by calculating the sum of the fourth ratio and the second preset value.

7. The method according to claim 5, characterized in that, Determining the response ratio of job j based on the waiting time, the priority coefficient, and the average execution time includes: Calculate the second product of the waiting time and the priority coefficient; Calculate the fifth ratio of the second product to the average execution time; The sum of the fifth ratio and the third preset value is calculated to obtain the response ratio of the task j.

8. A job scheduling system, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 7.

9. A storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the method of any one of claims 1 to 7.

Images