Job task scheduling method and device, storage medium and electronic equipment

Abstract

The invention discloses a job task scheduling method and device, a storage medium and electronic equipment, and relates to the technical field of big data. The method comprises the steps that the arrival rate and the average workload of job tasks of the big data platform are determined; according to the arrival rate and the average workload of the job tasks, determining reserved processing time and reserved resource quantity allocated by the big data platform for each job task corresponding to different priorities; and performing resource scheduling on each job task received by the big data platform in a preset time period according to the reserved processing time and the reserved resource quantity allocated by the big data platform for each job task corresponding to different priorities. The technical problem that in the prior art, due to the fact that big data job scheduling has the advantages of being large in data size, large in task number and high in timeliness requirement, resource distribution is uneven in an actual job scheduling scene, and consequently job task processing time is long is solved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, storage medium and electronic device for scheduling a job task. Background Art

[0002] With the rise of big data technology, the amount of data is growing exponentially, and large-scale clusters need to be built to process large amounts of data. For cluster resource allocation and job scheduling, it is necessary to consider the performance parameters of the cluster (such as the number of nodes, storage capacity, and computing capacity) and the performance parameters of the job (such as job priority, data volume, and computing capacity). The strategy and algorithm of job scheduling are crucial to the performance of the cluster. How to optimize the job scheduling strategy, improve resource utilization, and reduce user waiting time while ensuring cluster performance has become a core issue in big data cluster research.

[0003] However, due to the substantial increase in the scale of big data assets and usage, big data job scheduling has a large amount of data, a large number of tasks, and high timeliness requirements. Therefore, in actual job scheduling scenarios, there is an uneven allocation of resources, which leads to a long processing time for job tasks, and in turn makes it difficult to meet the timeliness requirements of jobs of different priorities.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device, storage medium and electronic device for scheduling job tasks, so as to at least solve the technical problem in the prior art that, due to the characteristics of big data job scheduling such as large data volume, large number of tasks and high timeliness requirements, there is uneven resource allocation in actual job scheduling scenarios, resulting in long job task processing time.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of an embodiment of the present application, a method for scheduling job tasks is provided, including: determining the arrival rate and average workload of job tasks of a big data platform, wherein the arrival rate represents the arrival rate of job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the big data platform processing the received job tasks within the preset time period; determining the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities based on the arrival rate and average workload of the job tasks; and performing resource scheduling for each job task received by the big data platform within the preset time period based on the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities.

[0007] Optionally, determining the arrival rate and average workload of job tasks on the big data platform includes: obtaining the initial rate at which job tasks arrive at the big data platform; determining three probability events of the job tasks after leaving the first type of service station, wherein the first probability event represents the probability of the job task entering the second type of service station, the second probability event represents the probability of the job task leaving the big data platform, and the third probability represents the probability of the job task entering a new first type of service station, the first type of service station is used to convert the job task into an intermediate data set in the form of key-value pairs, and the second type of service station is used to merge and process intermediate data with the same primary key; based on the initial rate at which job tasks arrive at the big data platform and the three probability events after the job tasks leave the first type of service station, determine the arrival rate and average workload of job tasks on the big data platform.

[0008] Optionally, the arrival rate and average workload of job tasks on the big data platform are determined based on the initial rate at which job tasks arrive at the big data platform and three probability events after the job tasks leave the first type of service station, including: the rate at which job tasks exiting the first type of service station enter the task queue of the first type of service station as the first rate; the rate at which job tasks exiting the second type of service station enter the task queue of the first type of service station as the second rate; the rate at which job tasks exiting the first type of service station enter the task queue of the second type of service station as the third rate; a queuing network model is determined based on the initial rate, first rate, second rate, third rate at which job tasks arrive at the big data platform and the three probability events after the job tasks leave the first type of service station, wherein the queuing network model is used to simulate, in a functional form, the process in which job tasks arrive at the big data platform according to a Poisson process at the initial rate, and the service time of the big data platform for job tasks is an independently distributed random variable; the arrival rate and average workload of job tasks on the big data platform are determined based on the queuing network model.

[0009] Optionally, the arrival rate and average workload of the job tasks of the big data platform are determined according to the queuing network model, including: determining the average processing time of the job tasks at the ith service station according to the average number of job tasks in the ith service station, the average rate at which the job tasks arrive at the ith service station, and the processing speed of the ith service station for the job tasks, wherein the ith service station is any first-category service station or second-category service station; obtaining the transfer probability of the job task from the ith service station to the jth service station, wherein the jth service station is a service station different from the ith service station; obtaining the transfer time of the job task from the ith service station to the jth service station; determining the average time taken for the job task to leave the big data platform from arriving at the ith service station according to the transfer time, the transfer probability and the average processing time of the job task at the ith service station; determining the arrival rate and average workload of the job tasks of the big data platform according to the queuing network model and the average time taken for the job task to leave the big data platform from arriving at the ith service station.

[0010] Optionally, based on the arrival rate and average workload of the job tasks, the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities are determined, including: obtaining the arrival rate of the job task of the kth priority when entering the first type of service station, wherein the kth priority is any priority; obtaining the arrival rate of the job task of the kth priority when entering the second type of service station; obtaining a first function and a second function, wherein the first function is used to characterize the correlation between the processing rate of the job task of the kth priority at the first type of service station and the resource amount allocated to the job task by the first type of service station; the second function is used to characterize the correlation between the processing rate of the job task of the kth priority at the second type of service station and the resource amount allocated to the job task by the second type of service station; determining the task processing time of the job task of the kth priority according to the first function, the second function, the arrival rate of the job task of the kth priority when entering the first type of service station and the arrival rate when entering the second type of service station; determining the reserved processing time and reserved resource amount allocated by the big data platform to the job task of the kth priority according to the task processing time of the job task of the kth priority.

[0011] Optionally, based on the task processing time of the k-th priority job task, the reserved processing time and reserved resource amount allocated by the big data platform for the k-th priority job task are determined, including: obtaining the total amount of allocated resources of the big data platform and the maximum processing time set for the job tasks of each priority; based on the total amount of allocated resources, the maximum processing time set for the job tasks of each priority and the task processing time of the k-th priority job task, the reserved processing time and reserved resource amount allocated by the big data platform for the k-th priority job task.

[0012] Optionally, according to the reserved processing time and reserved resource amount allocated by the big data platform to job tasks of different priorities, resource scheduling is performed for each job task received by the big data platform within a preset time period, including: obtaining the amount of resources already occupied by the task queue where each job task is located, wherein the task queue where each job task is located includes multiple job tasks of the same priority level as the job task; determining the resource hunger of the task queue based on the reserved resource amount and the occupied resource amount of the task queue where each job task is located, wherein the resource hunger is positively correlated with the resource scheduling priority; and performing resource scheduling for each job task based on the resource hunger of the task queue where each job task is located and the reserved processing time allocated for each job task.

[0013] Optionally, the resource hunger of the task queue is determined based on the reserved resources and the occupied resources of the task queue where each job task is located, including: determining the resource hunger of the task queue in the first category of service stations based on the reserved resources of the task queue where each job task is located in the first category of service stations and the occupied resources of the task queue in the first category of service stations; and / or; determining the resource hunger of the task queue in the second category of service stations based on the reserved resources of the task queue where each job task is located in the second category of service stations and the occupied resources of the task queue in the second category of service stations.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the embodiment of the present application, a scheduling device for job tasks is also provided, including: a first determination unit, determining the arrival rate and average workload of the job tasks of the big data platform, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the big data platform processing the received job tasks within the preset time period; a second determination unit, determining the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities according to the arrival rate and average workload of the job tasks; a processing unit, performing resource scheduling for each job task received by the big data platform within the preset time period according to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities.

[0015] According to another aspect of an embodiment of the present application, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned scheduling method for job tasks.

[0016] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the scheduling method of the above-mentioned job task.

[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which implement the steps of the above-mentioned method for scheduling job tasks when executed by a processor.

[0018] In an embodiment of the present application, the arrival rate and average workload of the job tasks of the big data platform are first determined, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the job tasks received by the big data platform during the preset time period. Then, based on the arrival rate and the average workload of the job tasks, the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities are determined. Then, based on the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed for each job task received by the big data platform during the preset time period.

[0019] From the above content, it can be seen that this application takes into account the arrival characteristics of job tasks, and through real-time monitoring and historical data analysis, accurately predicts the arrival rate and average workload of job tasks received by the big data platform within a preset time period.

[0020] Secondly, according to the arrival rate and average workload of the job tasks, combined with the job priority, reasonable processing time and resource volume are pre-allocated for the jobs of different priorities. This step ensures the rapid response of high-priority jobs and fair processing of low-priority jobs by building a queuing network model and using a first-order autoregressive model to predict resource requirements.

[0021] In addition, it is equipped with a resource scheduling algorithm, which can dynamically adjust resource allocation according to the real-time resource hunger (i.e. the difference ratio between the required resources and the currently allocated resources) and the reserved processing time of the job tasks, giving priority to satisfying the needs of jobs with high resource hunger and urgent processing time. This not only improves the utilization of resources, but also enhances the responsiveness and stability of the system.

[0022] It can be seen that through the technical solution of the present application, the purpose of improving scheduling efficiency, enhancing timeliness and fairness, and optimizing resource allocation has been achieved, thereby achieving the technical effect of being able to meet the timeliness requirements of jobs of different priority levels for big data job scheduling, and thus solving the technical problem in the prior art that due to the characteristics of big data job scheduling such as large data volume, large number of tasks, and high timeliness requirements, there is uneven resource allocation in actual job scheduling scenarios, resulting in long job task processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for scheduling job tasks is shown;

[0025] Figure 2 is a flowchart of an optional method for scheduling job tasks according to an embodiment of the present application;

[0026] Figure 3 It is a flow chart of the steps of queuing network modeling in an optional method for job task scheduling according to an embodiment of the present application;

[0027] Figure 4 It is a queuing network model diagram in an optional method for job task scheduling according to an embodiment of the present application;

[0028] Figure 5 It is a flow chart of the overall technical solution of a model of an optional method for scheduling job tasks according to an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of a job task scheduling device provided according to an embodiment of the present application;

[0030] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should also be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0034] According to an embodiment of the present application, a method embodiment of a method for scheduling a job task is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] It should be noted that a job task scheduling system can be used as the execution subject of the job task scheduling method of the embodiment of the present application. It is understandable that the job task scheduling method provided in the embodiment of the present application can also be executed by other systems or devices, and the embodiment of the present application does not specifically limit this.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for scheduling a job task is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuit may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be fully or partially integrated into any of the other components in the computer terminal 10 (or mobile device). As in the method for scheduling job tasks involved in the embodiments of the present application, the data processing circuit serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for scheduling the job tasks in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the scheduling method of the job tasks described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] Under the above operating environment, this application provides Figure 2 The scheduling method of the job tasks shown. Figure 2 FIG. 1 is a flowchart of an optional method for scheduling job tasks according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0042] Step S201, determining the arrival rate and average workload of the job tasks of the big data platform.

[0043] In step S201, the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the big data platform processing the received job tasks within the preset time period.

[0044] Optionally, the job task scheduling system accurately measures the arrival rate of job tasks and their inherent average workload, providing solid data support for subsequent resource allocation and job scheduling.

[0045] Optionally, the arrival rate of job tasks quantitatively describes the rate at which the big data platform accepts job tasks within a given time, reflects the spatiotemporal variability of the job flow, and also reveals the demand for the platform's carrying capacity.

[0046] Optionally, the average workload is a statistical quantification of the resources consumed by the job task during the processing of the big data platform, covering the comprehensive demand for computing, storage and network resources during the execution of the job task.

[0047] Step S202, according to the arrival rate and average workload of the job tasks, determine the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities.

[0048] Optionally, based on the arrival rate and average workload of the job tasks, the job task scheduling system determines the reserved processing time and reserved resource amount allocated by the big data platform for each job task corresponding to different priorities, aiming to provide a resource for each job task of different priority levels on the big data platform.

[0049] Optionally, the job task scheduling system combines the real-time demand of job tasks with historical behavior patterns and uses optimization algorithms to quantify the potential demand of job tasks for platform resources, thereby achieving forward-looking and accurate allocation of limited resources.

[0050] Optionally, there is an optional flow chart of the steps of queuing network modeling in the method for job task scheduling, such as Figure 3 As shown, the modeling process of the task scheduling model using the queuing network model can be more clearly understood: first, the queuing network model is defined (ie, step S301), then the arrival rate is calculated according to the model (ie, step S302), and finally the model is solved (ie, step S303).

[0051] Step S203, according to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed on each job task received by the big data platform within a preset time period.

[0052] Optionally, based on a deep understanding of the arrival rate and average workload of job tasks, as well as the reserved processing time and reserved resource amounts planned for job tasks of different priority levels, the job task scheduling system performs resource scheduling for each job task received by the big data platform within a preset time period.

[0053] Optionally, when performing resource scheduling, the job task scheduling system first evaluates the current resource status of the platform and the priority distribution in the job queue, and then, based on this, reasonably allocates the platform's reserved resources according to the priority and processing requirements of the job tasks.

[0054] It can be seen from the contents of steps S201 to S203 that in an embodiment of the present application, the arrival rate and average workload of the job tasks of the big data platform are first determined, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the job tasks received by the big data platform during the preset time period. Then, based on the arrival rate and the average workload of the job tasks, the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities are determined. Then, based on the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed for each job task received by the big data platform during the preset time period.

[0055] From the above content, it can be seen that this application takes into account the arrival characteristics of job tasks, and through real-time monitoring and historical data analysis, accurately predicts the arrival rate and average workload of job tasks received by the big data platform within a preset time period.

[0056] Secondly, according to the arrival rate and average workload of the job tasks, combined with the job priority, reasonable processing time and resource volume are pre-allocated for the jobs of different priorities. This step ensures the rapid response of high-priority jobs and fair processing of low-priority jobs by building a queuing network model and using a first-order autoregressive model to predict resource requirements.

[0057] In addition, it is equipped with a resource scheduling algorithm, which can dynamically adjust resource allocation according to the real-time resource hunger (i.e. the difference ratio between the required resources and the currently allocated resources) and the reserved processing time of the job tasks, giving priority to satisfying the needs of jobs with high resource hunger and urgent processing time. This not only improves the utilization of resources, but also enhances the responsiveness and stability of the system.

[0058] It can be seen that through the technical solution of the present application, the purpose of improving scheduling efficiency, enhancing timeliness and fairness, and optimizing resource allocation has been achieved, thereby achieving the technical effect of being able to meet the timeliness requirements of jobs of different priority levels for big data job scheduling, and thus solving the technical problem in the prior art that due to the characteristics of big data job scheduling such as large data volume, large number of tasks, and high timeliness requirements, there is uneven resource allocation in actual job scheduling scenarios, resulting in long job task processing time.

[0059] In an optional embodiment, the job task scheduling system first obtains the initial rate at which the job tasks arrive at the big data platform, and then determines three probability events of the job tasks after leaving the first type of service station, wherein the first probability event represents the probability of the job tasks entering the second type of service station, the second probability event represents the probability of the job tasks leaving the big data platform, and the third probability represents the probability of the job tasks entering a new first type of service station. The first type of service station is used to convert the job tasks into intermediate data sets in the form of key-value pairs, and the second type of service station is used to merge and process intermediate data with the same primary key. Then, based on the initial rate at which the job tasks arrive at the big data platform and the three probability events after the job tasks leave the first type of service station, the arrival rate and average workload of the job tasks on the big data platform are determined.

[0060] Optionally, the job task scheduling system records the initial rate at which job tasks arrive at the big data platform as λ.

[0061] Optionally, the first type of service station is a map service station; the second type of service station is a reduce service station.

[0062] Optionally, there are three probability events after the job task leaves the first type of service station, that is, there are the following three situations after the job leaves the map service station.

[0063] Optionally, the first probability event represents the probability of the job task entering the second type of service station, that is, the probability of entering the reduce service station (i.e., the first probability event) is recorded as q 12 .

[0064] Optionally, the second probability event represents the probability of the job leaving the big data platform, that is, the probability of completing the processing and leaving the system (i.e., the second probability event) is recorded as q 10 .

[0065] Optionally, the third probability represents the probability of the job task entering a new first-class service station, that is, the probability of entering the map service station again and performing the next round of map tasks (i.e., the third probability) is recorded as q 11 .

[0066] Optionally, in addition to the three probability events, the job may directly leave the system and end the service after leaving the reduce service station. The probability of this situation is q 20 ; It is also possible to return to the map service station to continue the process, and the probability of this situation is q 21 .

[0067] Optionally, the first type of service station is used to convert the job task into an intermediate data set in the form of key-value pairs, and the second type of service station is used to merge and process the intermediate data with the same primary key, that is, during the execution of the MapReduce task, all data is organized into the form of <key, value>. A data set consisting of <key, value> pairs is first processed in the Map stage to generate a new intermediate data set consisting of <key, value> pairs. All <key, value> pairs with the same key value in this intermediate data set will be sent to the same Reduce task. This process is called shuffle. The data sent to the Reduce task exists in the form of key-value pairs. The Reduce task will merge and process these values ​​with the same key value and output the calculation results.

[0068] From the above content, it can be seen that the job task scheduling system obtains the initial rate of job tasks arriving at the big data platform, determines the three probability events of the job tasks after leaving the first type of service station, and then determines the arrival rate and average workload of the job tasks on the big data platform. It scientifically predicts the arrival rate and average workload of the job tasks on the big data platform, and provides a solid data science foundation for the formulation of subsequent resource reservation and dynamic scheduling strategies.

[0069] In an optional embodiment, the job task scheduling system first takes the rate of job tasks leaving the first type of service station when entering the task queue of the first type of service station as the first rate, then takes the rate of job tasks leaving the second type of service station when entering the task queue of the first type of service station as the second rate, and then takes the rate of job tasks leaving the first type of service station when entering the task queue of the second type of service station as the third rate, and then determines the queuing network model based on the initial rate of job tasks arriving at the big data platform, the first rate, the second rate, the third rate, and the three probability events of the job tasks after leaving the first type of service station, wherein the queuing network model is used to simulate the process in which job tasks arrive at the big data platform according to the Poisson process of the initial rate in the form of a function, and the service time of the big data platform for the job tasks is an independently distributed random variable, and then determines the job task arrival rate and average workload of the big data platform according to the queuing network model.

[0070] Optionally, the first rate is the rate at which the job tasks leaving the first type of service station enter the task queue of the first type of service station, that is, the rate at which the jobs leaving the map service station enter the map queue again, denoted as λ 2 .

[0071] Optionally, the second rate is the rate at which the job tasks leaving the second type of service station enter the task queue of the first type of service station, that is, the rate at which the job tasks leaving the reduce service station enter the map queue, denoted as λ 3 .

[0072] Optionally, the third rate is the rate at which the job tasks leaving the first type of service station enter the task queue of the second type of service station, that is, the rate at which only some of the jobs leaving the map service station enter the reduce queue, denoted by λ 4 .

[0073] Optionally, the queuing network model is used to simulate the process in which the job tasks arrive at the big data platform according to the Poisson process of the initial rate through the functional form, and the service time of the big data platform for the job tasks is an independently distributed random variable. Among them, the queuing network has M (M>1) nodes, each node has a mutually independent negative exponentially distributed service time, and the input from the outside to any node in the system is an independent Poisson arrival process. The M / M / 1 model is a queuing system in which user job tasks arrive according to a Poisson process with a rate of λ and the service time is an independently distributed random variable.

[0074] Optionally, since in a steady state, the rate at which user jobs leave the map and reduce servers should be equal to the rate at which they arrive, the arrival rate can be calculated. 1To represent the rate at which user jobs enter the map service station, the rate at which they leave the map service station is also λ 1 According to the composition theorem of Poisson process, the arrival process of user job tasks in the map queue is a Poisson process composed of three Poisson processes, with the parameter λ+λ 2 +λ 3 Therefore, we can list the equations to calculate the job arrival rate of the map queue and the reduce queue:

[0075]

[0076] By calculating the equation group, we can get the expression of job arrival rate under different situations:

[0077]

[0078] Optionally, there is an optional method for scheduling job tasks in a queuing network model diagram, such as Figure 4 As shown in the figure, we can understand the queuing network model more clearly: the initial job rate is λ, entering the Map queue, then entering the Map server, then entering the Reduce queue, and then entering the Reduce server. The job rate entering the Map queue is λ 1 , which includes the initial rate λ, the first rate λ 2 and the second rate λ 3 ; The rate of entering the Reduce queue is λ 4 .

[0079] From the above content, it can be seen that the job task scheduling system determines the queuing network model based on the initial rate, first rate, second rate, third rate of job tasks arriving at the big data platform and the three probability events of job tasks after leaving the first type of service station, and then determines the job task arrival rate and average workload of the big data platform. This can improve the intelligence and efficiency of the big data platform in resource management and job scheduling, and ensure the efficiency and timeliness of job task processing.

[0080] In an optional embodiment, the job task scheduling system first determines the average processing time of the job task at the ith service station based on the average number of job tasks in the ith service station, the average rate at which the job tasks arrive at the ith service station, and the processing speed of the job tasks at the ith service station, wherein the ith service station is any first-class service station or second-class service station, and then obtains the transfer probability of the job task from the ith service station to the jth service station, wherein the jth service station is a service station different from the ith service station, and then obtains the transfer time of the job task from the ith service station to the jth service station, and then determines the average time it takes for the job task to arrive at the ith service station and leave the big data platform based on the transfer time, the transfer probability and the average processing time of the job task at the ith service station, and then determines the arrival rate and average workload of the job tasks on the big data platform based on the queuing network model and the average time it takes for the job task to arrive at the ith service station and leave the big data platform.

[0081] Optionally, the average number of tasks in the i-th service station is denoted by L si ; The average rate at which the job arrives at the i-th service station is denoted by λ i ; The processing speed of the i-th service station for the job task is denoted by μ i .

[0082] Optionally, the average processing time of the job task at the i-th service station is denoted as T qi Where T qi It can be calculated by the following formula:

[0083]

[0084] Optionally, the transfer probability of a task from the i-th service station to the j-th service station is denoted as q ij ; The transfer time of the task from the i-th service station to the j-th service station is denoted as T ij .

[0085] Optionally, the average time it takes for a task to arrive at the i-th service station and leave the big data platform is denoted as T i .

[0086] Among them, T i It can be calculated by the following formula:

[0087]

[0088] Among them, j=1,2,3,...,M represents all nodes.

[0089] Optionally, use T lIt represents the average time interval from when the user job task arrives at the map node to when it leaves the network, and is represented by T 2 It represents the average time interval from when the user job task arrives at the reduce node to when it leaves the network. In the established network system,

[0090] Optionally, users arriving from outside must first enter the map node, so the time a job stays in the entire scheduling system can be expressed as T 1 In order to find the time T that a job stays in the entire scheduling system, 1 , and make the following derivation: First, according to the derivation, the equation group can be listed:

[0091]

[0092] where q 22 =0, and after solving, we get

[0093] Then you can make

[0094] Solving the above formula, we can get:

[0095]

[0096] Among them, the arrival rate λ map is the number of jobs arriving at the service station map per unit time; the completion rate μ map is the number of jobs that the service station map can complete per unit time, and the arrival rate λ reduce is the number of jobs that arrive at the service station reduce per unit time; the completion rate μ reduce It is the number of jobs that can be completed by the server reducer per unit time.

[0097] Optionally, the time series model can be used to approximately accurately predict the arrival rate of user job tasks in a certain period of time. The first-order autoregressive model can be used to predict the resource requirements of different jobs. The model can be described as:

[0098]

[0099] in, Represents a time series {a i}; e is white noise, which is set to 0 here. represents the predicted value; ρ(1) represents the autocorrelation function, which is defined as follows:

[0100]

[0101] where σ 2 Represents the time series {ai The time series {λ} and {w} represent the average user arrival rate and average workload in a series of consecutive time slices, respectively.

[0102] From the above content, it can be seen that the job task scheduling system determines the average time it takes for a job task to arrive at the i-th service station and leave the big data platform through the transfer time, transfer probability and the average processing time of the job task at the i-th service station, and then determines the arrival rate and average workload of the job tasks on the big data platform, thereby achieving forward-looking and accurate prediction of the platform's resource requirements, and providing a scientific basis for dynamic reservation of resources and efficient scheduling of jobs.

[0103] In an optional embodiment, the job task scheduling system first obtains the arrival rate of the k-th priority job task when it enters the first type of service station, wherein the k-th priority is any priority, and then obtains the arrival rate of the k-th priority job task when it enters the second type of service station, and then obtains the first function and the second function, wherein the first function is used to characterize the correlation between the processing rate of the k-th priority job task at the first type of service station and the amount of resources allocated to the job task by the first type of service station; the second function is used to characterize the correlation between the processing rate of the k-th priority job task at the second type of service station and the amount of resources allocated to the job task by the second type of service station, and then determines the task processing time of the k-th priority job task according to the first function, the second function, the arrival rate of the k-th priority job task when it enters the first type of service station and the arrival rate when it enters the second type of service station, and then determines the reserved processing time and reserved resource amount allocated by the big data platform to the k-th priority job task according to the task processing time of the k-th priority job task.

[0104] Optionally, the arrival rate of the k-th priority task entering the first-class service station is denoted as λ k1 .

[0105] Optionally, the arrival rate of the k-th priority task entering the second-class service station is denoted as λ k2 .

[0106] Optionally, the map and reduce server processing rates μ for priority k jobs k1 and μ k2 The number of slots s allocated to the map and reduce phases of the job k1 and k2 The relationship between k1 =f1(s k1 ) (i.e. the first function) and μ k2 =f2(s k2)(i.e. the second function) is used to represent it.

[0107] Optionally, the task processing time of the kth priority task can be recorded as T k , which can be expressed as:

[0108] From the above content, it can be seen that the job task scheduling system can cope with the complex challenges of resource allocation of big data processing platforms through priority cascade analysis, construction of association relationship functions, dynamic prediction of processing time, and scientific configuration of reserved processing time and reserved resource volume, and can also achieve efficient processing of multi-priority job tasks.

[0109] In an optional embodiment, the job task scheduling system first obtains the total amount of allocated resources of the big data platform and the maximum processing time set for the job tasks of each priority level, and then determines the reserved processing time and reserved resource amount allocated by the big data platform for the job tasks of the kth priority level based on the total amount of allocated resources, the maximum processing time set for the job tasks of each priority level, and the task processing time of the job tasks of the kth priority level.

[0110] Optionally, based on the task processing time of the k-th priority job task, determine the reserved processing time and reserved resource amount allocated by the big data platform to the k-th priority job task, and ensure that the stay of jobs of different priorities is lower than the given deadline and the total number of slots is limited.

[0111] Alternatively, the problem of ensuring that jobs of different priorities stay below a given deadline and the total number of slots is limited can be described using the following formula:

[0112]

[0113] The number of slots in the reduce phase can be allocated according to the ratio of slot allocation in the map phase, that is, s k1 =m*s k2 , the above formula (10) can be solved by binary search, with Sreduce and Smap representing the total number of slots in the reduce phase and map phase respectively, and the parameter m representing their ratio. represents the kth given deadline.

[0114] From the above content, it can be seen that the job task scheduling system determines the reserved processing time and reserved resource amount allocated by the big data platform for the k-th priority job task based on the total amount of allocated resources, the maximum processing time set for the job tasks of each priority level, and the task processing time of the k-th priority job task, ensuring that the stay of jobs of different priorities is lower than the given deadline and the total number of slots is limited.

[0115] In an optional embodiment, the job task scheduling system first obtains the amount of resources occupied by the task queue where each job task is located, wherein the task queue where each job task is located includes multiple job tasks with the same priority level as the job task, and then determines the resource hunger of the task queue based on the reserved resources and the occupied resources of the task queue where each job task is located, wherein the resource hunger is positively correlated with the resource scheduling priority, and then performs resource scheduling for each job task based on the resource hunger of the task queue where each job task is located and the reserved processing time allocated for each job task.

[0116] Optionally, the amount of resources that each job task has occupied in the task queue, that is, the number of map slots and reduce slots actually occupied by the job in the priority k queue at a certain moment, can be recorded as mAssigned k and rAssigned k , which change dynamically and reflect the number of jobs currently being processed and resource usage.

[0117] Optionally, the hunger degree is the ratio of the number of resources that should be allocated but are not actually allocated to the number of resources that should be allocated. Each priority queue is set with a hunger degree, and when a free slot arrives, it will be allocated to the priority queue with the highest hunger degree.

[0118] Optionally, when scheduling resources, the system needs to reserve a certain number of slots to ensure that each priority queue has enough slots available at all times. k Greater than s k1 If the shared resource is exhausted, the excess will occupy the quota of shared resources. When the shared resource is exhausted, the new free slot will be allocated only to mAssigned k Less than s k1 If all mAssigned k Less than s k1 If none of the queues need new slots, the new free slots will be reserved.

[0119] Optionally, there is an example of a resource scheduling algorithm: taking the priority queue set I and the new free slot newSlot as input, the hunger degree of all priority queues is calculated first. Mapshared represents the remaining shared slots. If Mapshared is equal to 0, all priorities whose number of allocated slots is greater than the number of slots to be allocated are deleted. Then all priorities are sorted according to their hunger degree and stored in a queue sortedQueue. Finally, a priority k with the highest hunger degree and meeting the localization condition is selected from the sortedQueue queue, and the slot is assigned to k. If there is no queue that meets the localization condition, the slot is assigned to the priority with the highest hunger degree. The scheduling algorithm in the Reduce phase is similar to that in the Map phase.

[0120] From the above content, it can be seen that the job task scheduling system determines the resource hunger of the task queue by the amount of resources already occupied by the task queue where each job task is located, and then performs resource scheduling for each job task based on the resource hunger of the task queue where each job task is located and the reserved processing time allocated to each job task. It can realize intelligent configuration of resources, ensure that each job task obtains the required resources within its reserved processing time, and improve the efficiency of job scheduling and service quality.

[0121] In an optional embodiment, the job task scheduling system first determines the resource hunger of the task queue in the first category of service stations based on the reserved resource amount of the task queue where each job task is located in the first category of service stations and the amount of resources already occupied by the task queue in the first category of service stations, and then and / or determines the resource hunger of the task queue in the second category of service stations based on the reserved resource amount of the task queue where each job task is located in the second category of service stations and the amount of resources already occupied by the task queue in the second category of service stations.

[0122] Optionally, the resource hunger degree of the task queue is divided into a map hunger degree mHungryDegree (i.e., the resource hunger degree of the task queue in the first class service station is determined according to the reserved resource amount of the task queue where each job task is located in the first class service station and the resource amount already occupied by the task queue in the first class service station) and a reduce hunger degree rHungryDegree (i.e., the resource hunger degree of the task queue in the second class service station is determined according to the reserved resource amount of the task queue where each job task is located in the second class service station and the resource amount already occupied by the task queue in the second class service station), which can be calculated using the following formula:

[0123] mHungryDegree k =1-mAssigned k / s k1 (11)

[0124] rHungryDegree k =1-rAssigned k / s k2 (12)

[0125] From the above content, it can be seen that the job task scheduling system not only optimizes the allocation of platform resources and improves the efficiency of job task processing, but also ensures dynamic reservation and intelligent scheduling of resources by independently calculating the resource hunger of the job task queue in the Map job station and the Reduce job station.

[0126] It can be seen from the above content that according to the technical solution of this application, at least the following technical effects can be achieved:

[0127] 1. Effectively improve the efficiency of multi-priority big data job scheduling and operation, and enhance the resource supply and scheduling capabilities of the big data platform.

[0128] 2. It can support the big data platform's ability to support business scenarios with high timeliness requirements, and assist the business end-to-end in forming high-timeliness data perception and analysis and decision-making capabilities.

[0129] In an optional embodiment, Figure 5 The flow chart of the overall technical solution of the model of an optional method for scheduling job tasks according to the embodiment of the present application is shown, which can more clearly understand the overall technical solution of the model of the method for scheduling job tasks: first, the queuing network is modeled, and the task scheduling model is modeled using the queuing network model, and then the job arrival rate and average workload are predicted. Through the time series model, the user job task arrival rate in a certain time period can be approximately and accurately predicted. Then the job resources are allocated, and finally, according to the resource allocation algorithm, the concept of hunger degree in the scheduling algorithm is introduced to dynamically schedule the tasks.

[0130] According to another aspect of the embodiment of the present application, a scheduling device for job tasks is also provided, wherein: Figure 6 is a schematic diagram of an optional scheduling device for job tasks according to an embodiment of the present application, such as Figure 6 As shown, the scheduling device for job tasks includes: a first determining unit 601 , a second determining unit 602 , and a processing unit 603 .

[0131] The first determination unit 601 determines the arrival rate and average workload of the job tasks of the big data platform, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the job tasks received by the big data platform during the preset time period; the second determination unit 602 determines the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities according to the arrival rate and average workload of the job tasks; the processing unit 603 performs resource scheduling for each job task received by the big data platform during the preset time period according to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities.

[0132] Optionally, the first determination unit 601 includes: a first acquisition subunit, a first determination subunit, and a second determination subunit. The first acquisition subunit is used to acquire the initial rate at which the job task arrives at the big data platform; the first determination subunit is used to determine three probability events of the job task after leaving the first type of service station, wherein the first probability event represents the probability of the job task entering the second type of service station, the second probability event represents the probability of the job task leaving the big data platform, and the third probability represents the probability of the job task entering a new first type of service station. The first type of service station is used to convert the job task into an intermediate data set in the form of a key-value pair, and the second type of service station is used to merge and process the intermediate data with the same primary key; the second determination subunit is used to determine the arrival rate and average workload of the job task of the big data platform according to the initial rate at which the job task arrives at the big data platform and the three probability events of the job task after leaving the first type of service station.

[0133] Optionally, the second determination subunit includes: a first processing module, a second processing module, a third processing module, a first determination module and a second determination module. The first processing module is used to take the rate of the job tasks leaving the first type of service station when entering the task queue of the first type of service station as the first rate; the second processing module is used to take the rate of the job tasks leaving the second type of service station when entering the task queue of the first type of service station as the second rate; the third processing module is used to take the rate of the job tasks leaving the first type of service station when entering the task queue of the second type of service station as the third rate; the first determination module is used to determine the queue network model according to the initial rate of the job tasks arriving at the big data platform, the first rate, the second rate, the third rate and the three probability events of the job tasks after leaving the first type of service station, wherein the queue network model is used to simulate the process of the job tasks arriving at the big data platform according to the Poisson process of the initial rate through the function form, and the service time of the big data platform for the job tasks is an independently distributed random variable; the second determination module is used to determine the arrival rate and average workload of the job tasks of the big data platform according to the queue network model.

[0134] Optionally, the second determination module includes: a first determination submodule, a first acquisition submodule, a second acquisition submodule, a second determination submodule and a third determination submodule. Among them, the first determination submodule is used to determine the average processing time of the job task in the i-th service station according to the average number of job tasks in the i-th service station, the average rate at which the job tasks arrive at the i-th service station, and the processing speed of the i-th service station for the job tasks, wherein the i-th service station is any first-class service station or second-class service station; the first acquisition submodule is used to obtain the transfer probability of the job task from the i-th service station to the j-th service station, wherein the j-th service station is a service station different from the i-th service station; the second acquisition submodule is used to obtain the transfer time of the job task from the i-th service station to the j-th service station; the second determination submodule is used to determine the average time taken for the job task to leave the big data platform from arriving at the i-th service station according to the transfer time, the transfer probability and the average processing time of the job task at the i-th service station; the third determination submodule is used to determine the arrival rate and average workload of the job task of the big data platform according to the queuing network model and the average time taken for the job task to leave the big data platform from arriving at the i-th service station.

[0135] Optionally, the second determining unit 602 includes: a second acquiring subunit, a third acquiring subunit, a fourth acquiring subunit, a third determining subunit and a fourth determining subunit. Among them, the second acquisition subunit is used to obtain the arrival rate of the job task of the kth priority when entering the first type of service station, wherein the kth priority is any priority; the third acquisition subunit is used to obtain the arrival rate of the job task of the kth priority when entering the second type of service station; the fourth acquisition subunit is used to obtain the first function and the second function, wherein the first function is used to characterize the correlation between the processing rate of the job task of the kth priority at the first type of service station and the amount of resources allocated to the job task by the first type of service station; the second function is used to characterize the correlation between the processing rate of the job task of the kth priority at the second type of service station and the amount of resources allocated to the job task by the second type of service station; the third determination subunit is used to determine the task processing time of the job task of the kth priority according to the first function, the second function, the arrival rate of the job task of the kth priority when entering the first type of service station and the arrival rate when entering the second type of service station; the fourth determination subunit is used to determine the reserved processing time and reserved resource amount allocated by the big data platform to the job task of the kth priority according to the task processing time of the job task of the kth priority.

[0136] Optionally, the fourth determination subunit includes: a first acquisition module and a third determination module. The first acquisition module is used to acquire the total amount of allocated resources of the big data platform and the maximum processing time set for each priority job task; the third determination module is used to determine the reserved processing time and reserved resource amount allocated by the big data platform for the kth priority job task according to the total amount of allocated resources, the maximum processing time set for each priority job task, and the task processing time of the kth priority job task.

[0137] Optionally, the processing unit 603 includes: a fifth acquisition subunit, a fifth determination subunit, and a first processing subunit. The fifth acquisition subunit is used to obtain the amount of resources occupied by the task queue where each job task is located, wherein the task queue where each job task is located includes multiple job tasks with the same priority as the job task; the fifth determination subunit is used to determine the resource hunger of the task queue according to the reserved resources and the occupied resources of the task queue where each job task is located, wherein the resource hunger is positively correlated with the resource scheduling priority; the first processing subunit is used to perform resource scheduling for each job task according to the resource hunger of the task queue where each job task is located and the reserved processing time allocated to each job task.

[0138] Optionally, the fifth determination subunit includes: a fourth determination module and a fifth determination module. The fourth determination module is used to determine the resource hunger of the task queue in the first type of service station according to the reserved resource amount of the task queue where each job task is located in the first type of service station and the resource amount already occupied by the task queue in the first type of service station; the fifth determination module is used to determine the resource hunger of the task queue in the second type of service station according to the reserved resource amount of the task queue where each job task is located in the second type of service station and the resource amount already occupied by the task queue in the second type of service station.

[0139] An embodiment of the present application may provide an electronic device, Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 Only one is shown) a processor, a memory, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0140] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] It can be understood by those skilled in the art that Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 7 The structure of the electronic device is not limited. Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 7 Different configurations are shown.

[0142] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0143] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned method for scheduling job tasks.

[0144] According to another aspect of the present application, a computer program product is also provided, wherein the computer program product includes computer instructions, wherein when the computer instructions are executed, the device where the computer program product is located executes the above-mentioned scheduling method for job tasks.

[0145] The above-mentioned embodiments or examples disclosed in the present application are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection disclosed in the present application. In the absence of contradiction, each step in a certain embodiment or example in the present application can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0146] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0147] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0152] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for scheduling a job task, characterized in that: include: Determine the arrival rate and average workload of the job tasks of the big data platform, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the big data platform processing the received job tasks within the preset time period; Determine, according to the arrival rate of the job tasks and the average workload, the reserved processing time and the reserved resource amount allocated by the big data platform to each job task corresponding to different priorities; According to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed on each job task received by the big data platform within a preset time period.

2. The method for scheduling a job task according to claim 1, characterized in that: Determine the arrival rate and average workload of the big data platform's job tasks, including: Obtaining an initial rate at which job tasks arrive at the big data platform; Determine three probability events of the job task after leaving the first type of service station, wherein the first probability event represents the probability of the job task entering the second type of service station, the second probability event represents the probability of the job task leaving the big data platform, and the third probability represents the probability of the job task entering a new first type of service station, the first type of service station is used to convert the job task into an intermediate data set in the form of a key-value pair, and the second type of service station is used to merge the intermediate data with the same primary key; The arrival rate and average workload of the job tasks on the big data platform are determined based on the initial rate at which the job tasks arrive at the big data platform and the three probability events of the job tasks after leaving the first type of service station.

3. The method for scheduling job tasks according to claim 2, characterized in that: According to the initial rate at which the job tasks arrive at the big data platform and the three probability events of the job tasks after leaving the first type of service station, the arrival rate and average workload of the job tasks of the big data platform are determined, including: The rate at which the job tasks leaving the first type of service station enter the task queue of the first type of service station is taken as a first rate; The rate at which the job tasks leaving the second-category service station enter the task queue of the first-category service station is taken as the second rate; The rate at which the job tasks leaving the first type of service station enter the task queue of the second type of service station is used as a third rate; Determine a queuing network model according to the initial rate at which the job tasks arrive at the big data platform, the first rate, the second rate, the third rate, and three probability events of the job tasks after leaving the first type of service station, wherein the queuing network model is used to simulate, in a functional form, the process in which the job tasks arrive at the big data platform according to the Poisson process of the initial rate, and the service time of the big data platform for the job tasks is an independently distributed random variable; The arrival rate of job tasks and the average workload of the big data platform are determined according to the queuing network model.

4. The method for scheduling a job task according to claim 3, characterized in that: Determining the arrival rate and average workload of the job tasks of the big data platform according to the queuing network model includes: According to the average number of job tasks in the ith service station, the average rate at which the job tasks arrive at the ith service station, and the processing speed of the ith service station for the job tasks, determine the average processing time of the job tasks at the ith service station, wherein the ith service station is any one of the first-category service station or the second-category service station; Obtaining a transfer probability of the job task from the i-th service station to the j-th service station, wherein the j-th service station is a service station different from the i-th service station; Obtaining the transfer time of the job task from the i-th service station to the j-th service station; Determine the average time taken for the job task to arrive at the i-th service station and leave the big data platform according to the transfer time, the transfer probability and the average processing time of the job task at the i-th service station; According to the queuing network model and the average time taken for the job task to arrive at the i-th service station and leave the big data platform, the arrival rate and average workload of the job tasks of the big data platform are determined.

5. The method for scheduling a job task according to claim 1, characterized in that: Determining the reserved processing time and the reserved resource amount allocated by the big data platform to each job task corresponding to different priorities according to the arrival rate of the job task and the average workload, including: Obtaining the arrival rate of the k-th priority job tasks when entering the first type of service station, wherein the k-th priority is any priority; Obtaining the arrival rate of the k-th priority job task when entering the second type of service station; Obtaining a first function and a second function, wherein the first function is used to characterize the correlation between the processing rate of the job task of the kth priority at the first type of service station and the amount of resources allocated to the job task by the first type of service station; and the second function is used to characterize the correlation between the processing rate of the job task of the kth priority at the second type of service station and the amount of resources allocated to the job task by the second type of service station; Determine the task processing time of the k-th priority task according to the first function, the second function, the arrival rate of the k-th priority task when entering the first type of service station, and the arrival rate of the k-th priority task when entering the second type of service station; According to the task processing duration of the job task of the kth priority, the reserved processing time and the reserved resource amount allocated by the big data platform to the job task of the kth priority are determined.

6. The method for scheduling work tasks according to claim 5, characterized in that: Determining the reserved processing time and the reserved resource amount allocated by the big data platform to the k-th priority job task according to the task processing time of the k-th priority job task, including: Obtain the total amount of allocated resources of the big data platform and the maximum processing time set for each priority level of the job task; The reserved processing time and reserved resource amount allocated by the big data platform to the job task of the kth priority are determined based on the total amount of allocated resources, the maximum processing time set for the job task of each priority level, and the task processing time of the job task of the kth priority level.

7. The method for scheduling a job task according to claim 1, characterized in that: According to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed on each job task received by the big data platform within a preset time period, including: Obtaining the amount of resources occupied by the task queue where each job task is located, wherein the task queue where each job task is located includes multiple job tasks with the same priority level as the job task; Determine the resource hunger of the task queue according to the reserved resource amount and the occupied resource amount of the task queue where each job task is located, wherein the resource hunger is positively correlated with the resource scheduling priority; Resource scheduling is performed on each job task according to the resource hunger degree of the task queue where each job task is located and the reserved processing time allocated to each job task.

8. The method for scheduling work tasks according to claim 7, characterized in that: Determining the resource hunger of the task queue according to the reserved resource amount and the occupied resource amount of the task queue where each job task is located includes: Determine the resource hunger of the task queue in the first type of service station according to the reserved resource amount of the task queue where each job task is located in the first type of service station and the resource amount already occupied by the task queue in the first type of service station; and / or; The resource hunger degree of the task queue in the second type of service station is determined according to the reserved resource amount of the task queue where each job task is located in the second type of service station and the resource amount already occupied by the task queue in the second type of service station.

9. A scheduling device for a job task, characterized in that: include: A first determination unit determines the arrival rate and average workload of the job tasks of the big data platform, wherein the arrival rate represents the arrival rate of the job tasks of the big data platform within a preset time period, and the average workload represents the resource demand corresponding to the job tasks received by the big data platform during the preset time period; A second determination unit determines, according to the arrival rate of the job tasks and the average workload, the reserved processing time and the reserved resource amount allocated by the big data platform to each job task corresponding to different priorities; The processing unit performs resource scheduling for each job task received by the big data platform within a preset time period according to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the method for scheduling a job task as described in any one of claims 1 to 8.

11. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method for scheduling job tasks as described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method for scheduling job tasks described in any one of claims 1 to 8 are implemented.

Images