A task scheduling method and device, electronic equipment and storage medium

Abstract

The application discloses a task scheduling method and device, electronic equipment and storage medium, and relates to the technical field of big data calculation, and aims to solve the problem that the scheduling of tasks with different weight levels is not reasonable when a YARN computing cluster performs tasks at the present stage. The method comprises the following steps: obtaining a task group of tasks to be scheduled; wherein the task group comprises a plurality of tasks to be scheduled; determining an initial execution queue of the task group; wherein the initial execution queue is used to indicate the execution order of the plurality of tasks to be scheduled; in the case that the tasks to be scheduled included in the task group increase or decrease, determining a first weight value of each task to be scheduled in the task group; determining an updated execution queue of the task group according to the first weight value of each task to be scheduled in the task group; and executing each task to be scheduled in the task group according to the updated execution queue. The application is used for task scheduling of a YARN computing cluster.

Description

Technical Field

[0001] This application relates to the field of big data computing, and in particular to a task scheduling method, apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, with the explosive growth of online data, big data technology has been increasingly widely used in scenarios such as analysis, prediction, and even decision-making. The underlying technology of big data is often based on measurement report (MR) tasks to achieve various business objectives.

[0003] MapReduce (MR) task technology typically requires a scheduler that considers the weight of each computing node and distributes the task to different nodes in the computing cluster for execution. Currently, such a scheduler is generally a computing cluster managed by another resource negotiator (YARN).

[0004] The current task scheduling scheme for YARN computing clusters is not reasonable enough for scheduling tasks with different weight levels. Some low-weight tasks will not be allocated resources for a long time, causing these tasks to fail. Summary of the Invention

[0005] This application provides a task scheduling method, apparatus, electronic device, and storage medium to solve the problem that the scheduling of tasks with different weight levels is not reasonable when YARN computing clusters perform task execution at present.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides a task scheduling method, comprising: obtaining a task group of tasks to be scheduled; wherein the task group includes multiple tasks to be scheduled; determining an initial execution queue of the task group; wherein the initial execution queue is used to indicate the execution order of the multiple tasks to be scheduled; determining a first weight value for each task to be scheduled in the task group when the number of tasks to be scheduled in the task group increases or decreases; determining an updated execution queue of the task group based on the first weight value of each task to be scheduled in the task group; and executing each task to be scheduled in the task group based on the updated execution queue.

[0008] Based on the above technical solution, this application can map task groups to the resources of a computing cluster, assigning a task group to a specific resource partition of the computing cluster; and when computing resources are insufficient, whenever there are changes in the tasks in the execution queue of a task group, the execution weights of the tasks in the task group are updated according to a preset weight update mechanism, thereby reordering the execution queue of the task group, providing users with the ability to execute priority tasks first, and solving the problem that in a shared computing cluster, the isolation of task resources for a single user leads to unreasonable scheduling of tasks with different weight levels.

[0009] In one possible implementation, obtaining the task group of the task to be scheduled specifically includes: in response to the user's first operation, determining the attribute feature information of the task to be scheduled; wherein the attribute feature information of the task to be scheduled includes one or more of the following: task content information, execution time information, preset weight value, and belonging task group information; and publishing the task to be scheduled to the task group corresponding to the task to be scheduled based on the attribute feature information of the task to be scheduled.

[0010] In one possible implementation, determining the initial execution queue of the task group specifically includes: determining the tasks to be sorted based on the execution time information of the tasks to be scheduled; wherein the tasks to be sorted are the scheduled tasks that need to be executed at the initial moment; and determining the initial execution queue in descending order according to the preset weight values ​​of the tasks to be sorted.

[0011] In one possible implementation, the first weight value of each task to be scheduled in the task group is determined, satisfying the following formula:

[0012] W = W0 * (n + 1) 2

[0013] Where W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue.

[0014] In one possible implementation, the update execution queue is sorted in descending order according to the first weight value of the tasks to be scheduled. Based on the update execution queue, each task to be scheduled in the task group is executed, specifically including: submitting the task to be scheduled at the head of the update execution queue to the computing cluster and determining the task execution result.

[0015] Secondly, this application provides a task scheduling apparatus, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire a task group of tasks to be scheduled; wherein the task group includes multiple tasks to be scheduled; the processing unit is configured to determine an initial execution queue of the task group; wherein the initial execution queue is used to indicate the execution order of the multiple tasks to be scheduled; the processing unit is further configured to determine a first weight value for each task to be scheduled in the task group when the number of tasks to be scheduled in the task group increases or decreases; the processing unit is further configured to determine an updated execution queue of the task group based on the first weight value of each task to be scheduled in the task group; the processing unit is further configured to execute each task to be scheduled in the task group according to the updated execution queue.

[0016] In one possible implementation, the processing unit is further configured to respond to the user's first operation by determining the attribute characteristics of the task to be scheduled; wherein the attribute characteristics of the task to be scheduled include one or more of the following: task content information, execution time information, preset weight value, and belonging task group information; the processing unit is further configured to publish the task to be scheduled to the task group corresponding to the task to be scheduled based on the attribute characteristics of the task to be scheduled.

[0017] In one possible implementation, the processing unit is further configured to determine the tasks to be sorted based on the execution time information of the tasks to be scheduled; wherein the tasks to be sorted are the scheduled tasks that need to be executed at the initial moment; the processing unit is further configured to determine the initial execution queue in descending order based on the preset weight values ​​of the tasks to be sorted.

[0018] In one possible implementation, the first weight value of each task to be scheduled in the task group is determined, satisfying the following formula:

[0019] W = W0 * (n + 1) 2

[0020] Where W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue.

[0021] In one possible implementation, the processing unit is also used to submit the scheduled task at the head of the update execution queue to the computing cluster and determine the task execution result.

[0022] Thirdly, this application provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by an electronic device of this application, cause the electronic device to perform the task scheduling method described in the first aspect and any possible implementation thereof.

[0023] Fourthly, this application provides an electronic device, including: a processor and a memory; wherein the memory is used to store one or more programs, the one or more programs including computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform the task scheduling method as described in the first aspect and any possible implementation thereof.

[0024] Fifthly, this application provides a computer program product containing instructions that, when executed on a computer, cause the electronic device of this application to perform the task scheduling method as described in the first aspect and any possible implementation thereof.

[0025] Sixthly, this application provides a chip system applied to a task scheduling device; the chip system includes one or more interface circuits and one or more processors. The interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the task scheduling device and send the signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the task scheduling device performs a task scheduling method as described in the first aspect and any possible design of the first aspect. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the architecture of a task scheduling device provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram of the architecture of another task scheduling device provided in the embodiments of this application;

[0028] Figure 3 A flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0029] Figure 4 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0030] Figure 5 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the structure of a task scheduling device provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of another task scheduling device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In this article, the character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can be understood as A or B.

[0035] The terms "first" and "second" in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first edge service node" and "second edge service node" are used to distinguish different edge service nodes, not to describe a characteristic order of edge service nodes.

[0036] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0037] Furthermore, in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplarily" or "for example" is intended to present concepts in a concrete manner.

[0038] To facilitate understanding of the technical solution of this application, the technical terms involved in this application are introduced below:

[0039] 1. Measurement report (MR)

[0040] MR refers to the transmission of data every 480 milliseconds on the traffic channel (470 milliseconds on the signaling channel), and this data can be used for network evaluation and optimization.

[0041] By processing the collected measurement data using MR technology, it can be used to evaluate the entire wireless network environment, replacing a large number of routine drive tests and fixed-point tests, thus saving communication operation and maintenance costs. Furthermore, evaluating the network based on measurement reports from actual user calls is more targeted than drive tests and fixed-point measurements; it also allows for analysis of this collected measurement data to determine user behavior patterns, cell distribution, and other information, which is more conducive to the formulation of network optimization strategies.

[0042] Typical applications of MR technology include: cell traffic distribution analysis, real-time assessment of wireless coverage, handover process analysis, investigation of malicious network use, statistics of paging messages, and statistics of SMS information.

[0043] 2. Yet another resource negotiator (YARN)

[0044] YARN is a new Hadoop resource manager. It is a general-purpose resource management system that provides unified resource management and scheduling for upper-layer applications. Its introduction has brought great benefits to computing clusters in terms of utilization, unified resource management and data sharing.

[0045] Based on the preceding text, MR task technology generally requires a scheduling mechanism. The scheduling mechanism considers the weight of each computing node and places the task on different nodes of the computing cluster for execution. Currently, such a scheduling mechanism is generally a computing cluster managed by YARN.

[0046] YARN provides three queue modes to handle the execution order of YARN jobs when cluster resources are insufficient. The most widely used mode is the elastic queue mode. The elastic queue mode provides a queue for each YARN user, which corresponds to a portion of the computing resources in the YARN cluster. After a user has used up their portion of resources, they wait in their own queue. In this way, the resources of each user are isolated from each other to a certain extent and do not affect each other.

[0047] In this application, a YARN computing cluster is used to execute user-published computing tasks.

[0048] The above explains the technical terms used in this application.

[0049] In recent years, with the explosive growth of online data, big data technology has been increasingly widely used in scenarios such as analysis, prediction, and even decision-making. The underlying technology of big data is often based on measurement report (MR) tasks to achieve various business objectives.

[0050] MapReduce (MR) task technology typically requires a scheduler that considers the weight of each computing node and distributes the task to different nodes in the computing cluster for execution. Currently, such a scheduler is generally a computing cluster managed by another resource negotiator (YARN).

[0051] Current task scheduling schemes for YARN computing clusters have the following drawbacks:

[0052] (1) Only single-layer resource isolation is supported for users. Sometimes, only one resource and one queue are allocated to users at the YARN layer. Users cannot further divide the computing resources they own. Furthermore, since users only have one queue and adopt the first-in-first-out method, if users have tasks with high resource overhead and long execution time, subsequent tasks will not be executed.

[0053] (2) Even if YARN allocates multiple queues to users, since the queues use a first-in-first-out method to execute YARN jobs, users cannot be required to execute tasks according to priority.

[0054] (3) Some vendors implement priority queues on the basis of YARN scheduling, but the task weights in the priority queue are not updated, resulting in low-priority tasks not being executed.

[0055] It is evident that the existing task scheduling schemes for YARN computing clusters have shortcomings in their scheduling of tasks with different weight levels. This results in some low-weight tasks being unable to be allocated resources for extended periods, leading to task failures.

[0056] To address the shortcomings of the existing technology, this application provides a task scheduling method and apparatus that can map task groups to the resources of a computing cluster, assigning a task group to a specific resource partition of the computing cluster. When computing resources are insufficient, whenever there are changes in the tasks in the execution queue of a task group, the execution weights of the tasks in the task group are updated according to a preset weight update mechanism, thereby reordering the execution queue of the task group. This provides users with the ability to execute priority tasks first, solving the problem that in a shared computing cluster, the isolation of task resources for a single user leads to unreasonable scheduling of tasks with different weight levels.

[0057] The embodiments provided in this application will now be described in detail with reference to the accompanying drawings.

[0058] For example, such as Figure 1 As shown, Figure 1The present application provides an architecture diagram of a task scheduling device 10, which includes: a job addition module 11, a job publishing module 12, a job submission module 13, and a computing cluster module 14.

[0059] The task addition module 11 is used to respond to user operations and determine the attribute characteristics of the user's requested task (i.e., the task to be scheduled, which will be referred to as the task to be scheduled hereafter). It can be understood that the task addition module 11 can be equipped with an interactive device that enables interaction with the user, such as a touch screen or a semantic recognition system, so as to realize the function of determining the task to be scheduled in response to the user's operation.

[0060] For example, the attribute feature information may include one or more of the following: task content information, execution time information, preset weight value, and assigned task group information. Specifically, the task content information is used to reflect the specific content of the task to be scheduled, such as the method of calculation of the data required; the execution time information is used to indicate the specific time when the task to be scheduled starts and the completion of the task within a certain market; the preset weight value is used to characterize the priority of the task to be scheduled; and the assigned task group information is used to characterize which task group the task to be scheduled can be assigned to.

[0061] The job publishing module 12 is used to publish the tasks to be scheduled to the job submission module 13 based on the execution time information.

[0062] The job submission module 13 is used to determine the execution queue based on the attribute characteristics of the task to be scheduled, and send each task in the task group to the computing cluster module 14 according to the execution queue.

[0063] For example, such as Figure 2 As shown, the job submission module 13 may include the following functional sub-modules: task group management module 131, queue management module 132, and weight calculation module 133.

[0064] The task group management module 131 is used to manage the task groups corresponding to the tasks to be scheduled. Specifically, it includes determining the initial task group and updating the tasks to be scheduled included in the task group after a task to be scheduled is submitted for execution.

[0065] The queue management module 132 is used to manage the execution queues corresponding to task groups. Specifically, it includes determining the execution queue corresponding to each task to be scheduled in the task group, and updating the execution queue according to the weight value calculated by the weight calculation module 133 when there are changes in the task group.

[0066] The weight calculation module 133 is used to calculate the weight value of each task to be scheduled in the task group according to a preset calculation formula when there are additions or deletions of tasks included in the task group.

[0067] In one possible implementation, the weight calculation module 133 calculates the weight value of each task to be scheduled in the task, satisfying the following formula:

[0068] W = W0 * (n + 1) 2

[0069] Where W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue.

[0070] The above describes one possible architecture for the submodules included in the job submission module 13.

[0071] The computing cluster module 14 is used to execute the tasks to be scheduled submitted by the job submission module 13 and determine the final task execution result.

[0072] In different application scenarios, the job addition module 11, job publishing module 12, job submission module 13, and computing cluster module 14 can be deployed in different devices included in the task scheduling device 10, or they can be integrated into the same device included in the task scheduling device 10. This application does not make any specific limitations on this.

[0073] When the job creation module 11, job publishing module 12, job submission module 13, and computing cluster module 14 are integrated into the same device within the task scheduling device 10, the communication method between these modules is the same as that between modules within the same device. In this case, the communication process between the four modules is the same as the communication process between them when they are independent of each other.

[0074] The above describes the system architecture of a task scheduling method provided in this application.

[0075] The task scheduling method provided in this application will be described in detail below with reference to the accompanying drawings.

[0076] It should be noted that,

[0077] For example, such as Figure 3 As shown, the task scheduling method provided in this application specifically includes the following steps:

[0078] S301, The task scheduling device obtains the task group of the task to be scheduled.

[0079] The task group includes multiple tasks to be scheduled. These tasks are those input by the user through the task scheduling device.

[0080] Optionally, the task scheduling device responds to the user's operation and determines the attribute characteristics of the task to be scheduled.

[0081] For example, the attribute feature information may include one or more of the following: task content information, execution time information, preset weight value, and assigned task group information. Specifically, the task content information is used to reflect the specific content of the task to be scheduled, such as the method of calculation of the data required; the execution time information is used to indicate the specific time when the task to be scheduled starts and the completion of the task within a certain market; the preset weight value is used to characterize the priority of the task to be scheduled; and the assigned task group information is used to characterize which task group the task to be scheduled can be assigned to.

[0082] In one possible implementation, the task scheduling device determines the task group based on the attribute characteristics of the tasks to be scheduled. It should be noted that the specific process by which the task scheduling device determines the task group based on the attribute characteristics of the tasks to be scheduled is described in S401-S402 below, and will not be repeated here.

[0083] In one possible implementation, S301 can be executed by the job addition module, job release module and task group management module in the job submission module of the task scheduling device described above, so that the task scheduling device can obtain the task group of the task to be scheduled.

[0084] S302, The task scheduling device determines the initial execution queue of the task group.

[0085] The initial execution queue is used to indicate the execution order of multiple tasks to be scheduled. For example, when the computing resources of the computing cluster are sufficient, the task scheduling device submits the task at the head of the initial execution queue to the computing cluster for execution.

[0086] In one possible implementation, the task scheduling device determines the initial execution queue of the task group based on the attribute characteristics of the tasks to be scheduled. It should be noted that the specific process by which the task scheduling device determines the initial execution queue of the task group based on the attribute characteristics of the tasks to be scheduled is described in S501-S502 below, and will not be repeated here.

[0087] In one possible implementation, S302 can be specifically executed by the queue management module in the task scheduling device described above, so that the task scheduling device determines the initial execution queue of the task group.

[0088] S303. When the number of tasks to be scheduled in a task group increases or decreases, the task scheduling device determines the first weight value of each task to be scheduled in the task group.

[0089] Understandably, when the task scheduler submits the task at the head of the queue to the computing cluster based on the initial execution queue, the total number of tasks to be scheduled in the task group will decrease; conversely, when the user adds new tasks to be scheduled based on the task scheduler's job addition module, the total number of tasks to be scheduled in the task group will increase.

[0090] In response to the two situations mentioned above, the task scheduling device will recalculate the weight value of each task to be scheduled in the task group after the change in the number of tasks, which is the first weight value.

[0091] For example, the task scheduling device determines that the first weight value of each task to be scheduled in the task group satisfies the following formula:

[0092] W = W0 * (n + 1) 2

[0093] Where W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue.

[0094] In one possible implementation, S303 can be specifically executed by the task group management module and weight calculation module in the task scheduling device described above, so that the task scheduling device can determine the first weight value of each task to be scheduled in the task group when the number of tasks to be scheduled in the task group increases or decreases.

[0095] S304. The task scheduling device determines the update execution queue of the task group based on the first weight value of each task to be scheduled in the task group.

[0096] Optionally, the task scheduler determines the update execution queue of the task group based on the descending order of the first weight value of each task to be scheduled. Thus, when the computing resources of the computing cluster are sufficient, the task scheduler will submit the task at the head of the update execution queue to the computing cluster for execution.

[0097] In one possible implementation, S304 can be specifically executed by the job submission module in the task scheduling device described above, so as to determine the update execution queue of the task group based on the first weight value of each task to be scheduled in the task group.

[0098] S305. The task scheduling device executes each scheduled task in the task group according to the updated execution queue.

[0099] For example, in conjunction with the method for determining the update execution queue in S304, the task scheduling device specifically submits the task to be scheduled at the head of the update execution queue to the computing cluster and determines the task execution result.

[0100] Optionally, the computing cluster in this application embodiment may be a YARN computing cluster or other computing clusters with task computing capabilities.

[0101] In one possible implementation, S305 can be specifically executed by the job submission module in the task scheduling device described above, so that the task scheduling device executes each scheduled task in the task group according to the updated execution queue.

[0102] Based on the above technical solution, this application embodiment can map task groups to the resources of the computing cluster, assigning a task group to a specific resource partition of the computing cluster; and when computing resources are insufficient, whenever there are changes in the tasks in the execution queue of a task group, the execution weights of the tasks in the task group are updated according to a preset weight update mechanism, thereby reordering the execution queue of the task group, providing users with the ability to execute priority tasks first, and solving the problem that in a shared computing cluster, the isolation of task resources for a single user leads to unreasonable scheduling of tasks with different weight levels.

[0103] For example, combined Figure 4 In the task scheduling method provided in this application, the task scheduling device determines the task group based on the attribute characteristic information of the task to be scheduled, including the following steps:

[0104] S401, The task scheduling device responds to the user's first operation and determines the attribute characteristic information of the task to be scheduled.

[0105] Optionally, the user's first operation may include swiping, clicking, or voice control on the interactive device set by the task scheduling device, and this application embodiment does not specifically limit this.

[0106] In one possible implementation, S401 can be specifically executed by the job addition module in the task scheduling device described above, so that the task scheduling device responds to the user's first operation and determines the attribute feature information of the task to be scheduled.

[0107] S402. The task scheduling device publishes the task to be scheduled to the task group corresponding to the task to be scheduled based on the attribute characteristics information of the task to be scheduled.

[0108] Optionally, the task scheduling device may publish the task to be scheduled to the task group corresponding to the task based on the task group information in the attribute feature information of the task to be scheduled.

[0109] In one possible implementation, S402 can be specifically executed by the job publishing module in the task scheduling device described above, so that the task scheduling device publishes the task to be scheduled to the task group corresponding to the task to be scheduled according to the attribute feature information of the task to be scheduled.

[0110] Based on the above technical solution, the embodiments of this application can publish the user's newly added tasks to the corresponding task group, which facilitates the smooth progress of the subsequent task scheduling process.

[0111] For example, combined Figure 5 The task scheduling method provided in this application, specifically determining the initial execution queue of a task group based on the attribute characteristics of the tasks to be scheduled, includes the following steps:

[0112] S501. The task scheduling device determines the tasks to be sorted based on the execution time information of the tasks to be scheduled.

[0113] Among them, the tasks to be ordered are the scheduled tasks that need to be executed at the initial moment. That is, if the execution time information of the task to be ordered indicates that the time when the task to be ordered needs to start execution is the current moment, then the task to be ordered is determined as a task to be ordered.

[0114] In one possible implementation, S501 can be specifically executed by the job publishing module in the task scheduling device described above, so that the task scheduling device can determine the tasks to be sorted based on the execution time information of the tasks to be scheduled.

[0115] S502. The task scheduling device determines the initial execution queue in descending order based on the preset weight values ​​of the tasks to be sorted.

[0116] In one possible implementation, S502 can be specifically executed by queue management in the task scheduling device described above, so that the task scheduling device determines the initial execution queue in descending order according to the preset weight values ​​of the tasks to be sorted.

[0117] Based on the above technical solution, the embodiments of this application can determine the initial execution queue of the task group according to the attribute feature information of the task to be scheduled, which facilitates the smooth progress of the subsequent task scheduling process.

[0118] This application embodiment can divide the task scheduling device into functional modules or functional units according to the above method examples. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0119] For example, such as Figure 6The diagram shown is a possible structural schematic of a task scheduling device according to an embodiment of this application. The task scheduling device 600 includes: an acquisition unit 601 and a processing unit 602.

[0120] The acquisition unit 601 is used to acquire a task group of tasks to be scheduled. The task group includes multiple tasks to be scheduled.

[0121] Processing unit 602 is used to determine the initial execution queue of the task group. The initial execution queue is used to indicate the execution order of multiple tasks to be scheduled.

[0122] The processing unit 602 is further configured to determine a first weight value for each scheduled task in the task group when the number of scheduled tasks included in the task group increases or decreases.

[0123] The processing unit 602 is also used to determine the update execution queue of the task group based on the first weight value of each task to be scheduled in the task group.

[0124] The processing unit 602 is also used to execute each scheduled task in the task group according to the updated execution queue.

[0125] Optionally, the processing unit 602 is further configured to, in response to the user's first operation, determine the attribute characteristics of the task to be scheduled. The attribute characteristics of the task to be scheduled include one or more of the following: task content information, execution time information, preset weight value, and task group information.

[0126] Optionally, the processing unit 602 is further configured to publish the task to be scheduled to the task group corresponding to the task to be scheduled based on the attribute feature information of the task to be scheduled.

[0127] Optionally, the processing unit 602 is further configured to determine the tasks to be sorted based on the execution time information of the tasks to be scheduled. The tasks to be sorted are the scheduled tasks that need to be executed at the initial moment.

[0128] Optionally, the processing unit 602 is also used to determine the initial execution queue in descending order based on the preset weight values ​​of the tasks to be sorted.

[0129] Optionally, the processing unit 602 is also used to submit the scheduled task at the head of the update execution queue to the computing cluster and determine the task execution result.

[0130] Optionally, the task scheduling device 600 may further include a storage unit ( Figure 6 (shown in dashed box) The storage unit stores a program or instruction. When the acquisition unit 601 and the processing unit 602 execute the program or instruction, the task scheduling device can execute the task scheduling method described in the above method embodiment.

[0131] also, Figure 6 The technical effects of the task scheduling device can be referred to the technical effects of the task scheduling method described in the above embodiments, and will not be repeated here.

[0132] For example, Figure 7 This is a schematic diagram illustrating another possible structure for task scheduling involved in the above embodiments. For example... Figure 7 As shown, the task scheduling device 700 includes a processor 702.

[0133] The processor 702 is used to control and manage the actions of the task scheduling, such as executing the steps performed by each unit in the task scheduling device 600, and / or other processes used to execute the technical solutions described herein.

[0134] The processor 702 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the contents of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the contents of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0135] Optionally, the task scheduler 700 may further include a communication interface 703, a memory 701, and a bus 704. The communication interface 703 is used to support communication between the task scheduler 700 and other network entities. The memory 701 is used to store the program code and data of the task scheduler.

[0136] The memory 701 may be a memory used in task scheduling. The memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0137] The 704 bus can be an Extended Industry Standard Architecture (EISA) bus, etc. The 704 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0138] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and module described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0139] This application provides a computer program product containing instructions. When the computer program product is run on the electronic device of this application, it causes the computer to execute the uninstallation strategy determination method described in the above method embodiment.

[0140] This application also provides a computer-readable storage medium storing instructions. When a computer executes these instructions, the electronic device of this application performs each step of the task scheduling execution in the method flow shown in the above method embodiments.

[0141] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing, or any other form of computer-readable storage medium in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task scheduling method applied to a YARN computing cluster, characterized in that, The method includes: Obtain a task group for tasks to be scheduled; wherein, the task group includes multiple tasks to be scheduled; the task group corresponds to a resource of the YARN computing cluster, and one task group corresponds to a resource partition of the YARN computing cluster; Determine the initial execution queue for the task group; wherein the initial execution queue is used to indicate the execution order of the plurality of scheduled tasks; When the number of scheduled tasks included in the task group increases or decreases, a first weight value is determined for each scheduled task in the task group; the first weight value satisfies the following formula: W=W0 (n+1) 2 Wherein, W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue; The update execution queue of the task group is determined based on the first weight value of each task to be scheduled in the task group; According to the updated execution queue, each of the scheduled tasks in the task group is executed.

2. The method according to claim 1, characterized in that, The process of obtaining the task group to be scheduled specifically includes: In response to the user's first operation, the attribute feature information of the task to be scheduled is determined; wherein, the attribute feature information of the task to be scheduled includes one or more of the following: task content information, execution time information, preset weight value, and belonging task group information; Based on the attribute characteristics of the task to be scheduled, the task to be scheduled is published to the task group corresponding to the task to be scheduled.

3. The method according to claim 2, characterized in that, Determining the initial execution queue of the task group specifically includes: Based on the execution time information of the tasks to be scheduled, the tasks to be sorted are determined; wherein, the tasks to be sorted are the scheduled tasks that need to be executed at the initial moment. The initial execution queue is determined by descending order of the preset weight values ​​of the tasks to be sorted.

4. The method according to claim 1, characterized in that, The update execution queue is arranged in descending order according to the first weight value of the tasks to be scheduled. The step of executing each task to be scheduled in the task group according to the update execution queue specifically includes: The task to be scheduled at the head of the update execution queue is submitted to the computing cluster to determine the task execution result.

5. A task scheduling device applied to a YARN computing cluster, characterized in that, The task scheduling device includes: an acquisition unit and a processing unit. The acquisition unit is used to acquire a task group of tasks to be scheduled; wherein, the task group includes multiple tasks to be scheduled; the task group corresponds to the resources of the YARN computing cluster, and one task group corresponds to a certain resource partition of the YARN computing cluster; The processing unit is configured to determine the initial execution queue of the task group; wherein the initial execution queue is used to indicate the execution order of the plurality of tasks to be scheduled; The processing unit is further configured to determine a first weight value for each task to be scheduled in the task group when the number of tasks to be scheduled in the task group increases or decreases; the first weight value satisfies the following formula: W=W0 (n+1) 2 Wherein, W represents the first weight value of the task to be scheduled, W0 represents the initial weight value of the task to be scheduled, and n represents the number of tasks to be scheduled that have been executed in the initial execution queue; The processing unit is further configured to determine the update execution queue of the task group based on the first weight value of each task to be scheduled in the task group; The processing unit is further configured to execute each of the scheduled tasks in the task group according to the updated execution queue.

6. The task scheduling device according to claim 5, characterized in that, The processing unit is further configured to respond to the user's first operation and determine the attribute feature information of the task to be scheduled; wherein the attribute feature information of the task to be scheduled includes one or more of the following: task content information, execution time information, preset weight value, and belonging task group information. The processing unit is further configured to publish the task to be scheduled to the task group corresponding to the task to be scheduled based on the attribute feature information of the task to be scheduled.

7. The task scheduling device according to claim 6, characterized in that, The processing unit is further configured to determine the tasks to be sorted based on the execution time information of the tasks to be scheduled; wherein the tasks to be sorted are the tasks to be scheduled that need to be executed at the initial moment. The processing unit is further configured to determine the initial execution queue by descending the preset weight values ​​of the tasks to be sorted.

8. The task scheduling device according to claim 5, characterized in that, The processing unit is also used to submit the scheduled task at the head of the update execution queue to the computing cluster and determine the task execution result.

9. An electronic device, characterized in that, include: A processor and a memory; wherein the memory is used to store computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform the task scheduling method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed by an electronic device, enable the electronic device to perform the task scheduling method as described in any one of claims 1-4.

Images