Task processing method, device, server and storage medium

Abstract

The present disclosure relates to a task processing method, device, server and storage medium. The method comprises: determining a rerun sub-job according to metadata of a rerun job submitted by a target account, obtaining a first concurrency number of target tasks in the rerun sub-job to be submitted to a workflow scheduling system, and a second concurrency number of tasks recorded in a target sliding window, and determining whether to submit the target tasks in the rerun sub-job to the workflow scheduling system according to the first concurrency number and the second concurrency number, so as to avoid users from frequently submitting rerun jobs and reduce resource waste.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a task processing method, device, server, and storage medium. Background Art

[0002] Task rerunning refers to the operation of backtracking historical data. Cascade rerunning means that after scheduling a task and backtracking historical data, the downstream tasks of this task must also be scheduled and the corresponding data must be backtracked. At present, when cascade rerunning involves many upstream and downstream tasks, the traditional method will split a job submitted by the user into multiple sub-jobs to rerun the tasks. However, since each sub-job is not controlled, it will lead to a large number of tasks being rerun at the same time. If there is no restriction, when users frequently submit rerun jobs, it will cause a waste of resources. Summary of the invention

[0003] The present disclosure provides a task processing method, device, server and storage medium to at least solve the problem in the related art that users frequently submit re-run jobs, which will cause Yarn resources to be occupied and cause resource waste. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a task processing method, including:

[0005] Determine the re-run sub-job based on the metadata of the re-run job submitted by the target account;

[0006] Obtaining a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job and a second concurrent number of tasks recorded in the target sliding window;

[0007] According to the first concurrent number and the second concurrent number, it is determined whether to submit the target task in the re-run sub-job to the workflow scheduling system.

[0008] In an exemplary embodiment, determining the re-run sub-job according to the metadata of the re-run job submitted by the target account includes:

[0009] Get the metadata of the rerun job submitted by the target account;

[0010] The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

[0011] In an exemplary embodiment, determining whether to submit the target task in the rerun sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number includes:

[0012] If the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0013] In an exemplary embodiment, the method further comprises:

[0014] If the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, updating the first waiting time of the target task;

[0015] When the first waiting time is reached, the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number is returned to be executed.

[0016] In an exemplary embodiment, the determining to submit the target task in the re-run sub-job to the workflow scheduling system also includes:

[0017] Obtaining resource usage of the workflow scheduling system;

[0018] If the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0019] In an exemplary embodiment, the method further comprises:

[0020] If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, updating the second waiting time of the target task;

[0021] When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0022] In an exemplary embodiment, after the resource usage rate of the workflow scheduling system is less than a second preset threshold, before the determining to submit the target task in the re-run sub-job to the workflow scheduling system, the method further includes:

[0023] Get resource usage of resource management system;

[0024] If the resource usage rate of the resource management system is less than a third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0025] In an exemplary embodiment, the method further comprises:

[0026] If the resource usage rate of the resource management system is greater than or equal to the third preset threshold, updating the third waiting time of the target task;

[0027] When the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0028] In an exemplary embodiment, the method further comprises:

[0029] Determine the number of concurrent tasks based on the task profile, resource utilization of the workflow scheduling system, resource utilization of the resource management system, and preset influencing factors;

[0030] The second concurrent number of tasks recorded in the target sliding window is updated according to the concurrent number of tasks.

[0031] According to a second aspect of an embodiment of the present disclosure, there is provided a task processing device, including:

[0032] A first determination unit is configured to determine a re-run sub-job according to metadata of the re-run job submitted by the target account;

[0033] An acquiring unit is configured to acquire a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job and a second concurrent number of tasks recorded in the target sliding window;

[0034] The second determining unit is configured to determine whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number.

[0035] In an exemplary embodiment, the first determining unit is further configured to execute:

[0036] Get the metadata of the rerun job submitted by the target account;

[0037] The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

[0038] In an exemplary embodiment, the second determining unit is further configured to execute:

[0039] If the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0040] In an exemplary embodiment, the second determining unit is further configured to execute:

[0041] If the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, updating the first waiting time of the target task;

[0042] When the first waiting time is reached, the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number is returned to be executed.

[0043] In an exemplary embodiment, the second determining unit is further configured to execute:

[0044] Obtaining resource usage of the workflow scheduling system;

[0045] If the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0046] In an exemplary embodiment, the second determining unit is further configured to execute:

[0047] If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, updating the second waiting time of the target task;

[0048] When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0049] In an exemplary embodiment, the second determining unit is further configured to execute:

[0050] Get resource usage of resource management system;

[0051] If the resource usage rate of the resource management system is less than a third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0052] In an exemplary embodiment, the second determining unit is further configured to execute:

[0053] If the resource usage rate of the resource management system is greater than or equal to the third preset threshold, updating the third waiting time of the target task;

[0054] When the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0055] In an exemplary embodiment, the task processing device further includes:

[0056] A third determining unit is configured to determine the number of concurrent tasks according to the task profile, the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system and a preset influencing factor;

[0057] An updating unit is configured to update a second concurrent number of tasks recorded in the target sliding window according to the concurrent number of tasks.

[0058] According to a third aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device performs the task processing method described in any one of the embodiments of the first aspect.

[0059] According to a fourth aspect of an embodiment of the present disclosure, a server is provided, including:

[0060] processor;

[0061] a memory for storing instructions executable by the processor;

[0062] The processor is configured to execute the instructions to implement the task processing method described in any one of the embodiments of the first aspect.

[0063] According to a fifth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of a server, the server can execute the task processing method described in any one of the embodiments of the first aspect.

[0064] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: determining the rerun sub-job according to the metadata of the rerun job submitted by the target account, obtaining the first concurrent number of the target task to be submitted to the workflow scheduling system in the rerun sub-job, and the second concurrent number of the task recorded in the target sliding window, and determining whether to submit the target task in the rerun sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number. Among them, determining whether to submit the target task in the rerun sub-job to the workflow scheduling system is determined by the first concurrent number of the target task to be submitted to the workflow scheduling system and the second concurrent number of the task recorded in the target sliding window, thereby avoiding users from frequently submitting rerun jobs and reducing resource waste.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0067] Figure 1 The present invention is an application environment diagram of a task processing method according to an exemplary embodiment.

[0068] Figure 2 The figure is a flowchart of a task processing method according to an exemplary embodiment.

[0069] Figure 3 The figure is a block diagram of a task processing device according to an exemplary embodiment.

[0070] Figure 4 The figure is a diagram showing the internal structure of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0073] The task processing method provided by the present disclosure can be applied to Figure 1In the application environment shown. The terminal 110 interacts with the server 120 through a network. The terminal 110 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 120 may be implemented with an independent server or a server cluster consisting of multiple servers. The terminal 110 determines the metadata of the rerun job in response to the submission operation of the rerun job of the target account. The server 120 determines the rerun sub-job based on the metadata of the rerun job submitted by the target account, obtains the first concurrent number of the target task to be submitted to the workflow scheduling system in the rerun sub-job, and the second concurrent number of the task recorded in the target sliding window, and determines whether to submit the target task in the rerun sub-job to the workflow scheduling system based on the first concurrent number and the second concurrent number.

[0074] Figure 2 is a flowchart of a task processing method according to an exemplary embodiment. Figure 2 As shown, the task processing method is used to Figure 1 The server in the process includes the following steps:

[0075] In step S21, the re-run sub-job is determined according to the metadata of the re-run job submitted by the target account;

[0076] Among them, a job refers to the basic unit of work submitted by the target account to the workflow scheduling system, which is the sum of the work required by the target account to be done by the computer during a transaction processing or calculation process. A rerun job refers to a job that requires processing or calculation of historical data.

[0077] Among them, metadata, also known as intermediate data and relay data, is data that describes data, mainly information that describes data attributes. The metadata in the disclosed embodiment includes the root task list of the rerun job, the start time of the rerun job, the end time of the rerun job, and the size of the time slice of the rerun job.

[0078] In an optional embodiment, determining the re-run sub-job according to the metadata of the re-run job submitted by the target account includes:

[0079] Get the metadata of the rerun job submitted by the target account;

[0080] The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

[0081] In the disclosed embodiment, the job can be divided according to the size of the time slice to obtain a plurality of sub-jobs. The division of the job to obtain the sub-jobs belongs to the prior art and will not be described in detail here. The job is divided to obtain a plurality of sub-jobs, and different sub-jobs are assigned to different clusters respectively, which can improve the execution efficiency.

[0082] In step S22, a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job and a second concurrent number of tasks recorded in the target sliding window are obtained;

[0083] Among them, a task is the smallest granularity of executable entities in workflow scheduling. The metadata of a task includes the user identifier Owner, the development group ID GroupId, and the re-run queue to be submitted. A sub-job contains one or more tasks and task downstreams. What is finally submitted to the workflow scheduling system is the task granularity (including the start and end time of the divided sub-jobs, and the number of concurrent tasks allowed to be executed by the workflow scheduling system at the same time). The workflow scheduling system is a system that schedules tasks.

[0084] In the disclosed embodiment, the metadata of the rerun job submitted by the target account includes a root task list. According to the root task list and the task dependency graph, all downstream tasks (also referred to as rerun subgraphs) that need to be rerun can be calculated. The rerun job is divided according to the time slices in the metadata, and the rerun sub-job is determined. When the dependent upstream task in the rerun sub-job is empty or all the dependent upstream tasks are successfully executed, it means that the downstream task is ready and submitted to the workflow scheduling system, and the task to be submitted to the workflow scheduling system in the rerun sub-job is called the target task.

[0085] The task dependency graph includes the dependency relationship between tasks and a graph composed of multiple tasks. The task dependency graph is calculated based on all the tasks in the workflow scheduling system and the relationship between the tasks.

[0086] Among them, when the target task in the re-run sub-job is submitted to the workflow scheduling system, the task will be stored in the corresponding sliding time window. The second concurrency number of the task recorded in the target sliding window represents the concurrency number of tasks that have been successfully submitted to the workflow scheduling system.

[0087] Furthermore, the implementation method of obtaining the second concurrent number of tasks recorded in the target sliding window is:

[0088] Determine the first time to obtain the target task to be submitted to the workflow scheduling system in the rerun sub-job, pre-set the size of the target sliding window to a preset time period, which includes a start time and an end time, then use the first time as the end time of the target sliding window, and determine the start time of the sliding window based on the preset time period and the end time. For example, pre-set the size of the target sliding window to 10 minutes, determine the first time to obtain the target task to be submitted to the workflow scheduling system in the rerun sub-job to 15:20 on April 20, 2020, then determine 15:20 on April 20, 2020 as the end time of the target sliding window, and determine the start time of the sliding window to be 15:10 on April 20, 2020 as the start time of the sliding window based on the preset time period and the end time. Afterwards, determine the second concurrent number of tasks recorded within the time period.

[0089] In step S23, it is determined whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number.

[0090] In the embodiment of the present disclosure, whether to submit the target task in the re-run sub-job to the workflow scheduling system is determined by comparing the sum of the first concurrent number and the second concurrent number with a preset threshold.

[0091] In an optional embodiment, if the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0092] Wherein, the first preset threshold value represents the submission limit value of the concurrent number of tasks. If the sum of the first concurrent number and the second concurrent number is less than the first preset threshold value, it means that after the target task to be submitted is submitted, the concurrent number of tasks does not exceed the submission limit value, and the target task in the re-run sub-job is still allowed to be submitted to the workflow scheduling system. Through this method, users can be controlled to submit re-run jobs frequently, limit the system's submission of re-run tasks, and reduce resource waste.

[0093] In an optional embodiment, if the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, the first waiting time of the target task is updated; when the first waiting time is reached, return to execute the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system based on the first concurrent number and the second concurrent number.

[0094] In the disclosed embodiment, if the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold value, it means that after the target task to be submitted is submitted, the concurrent number of the task exceeds the submission limit value, and the target task in the re-run sub-job is not allowed to be submitted to the workflow scheduling system. At this time, it is necessary to set a waiting time (i.e., the first waiting time) for the above-mentioned target task. When the first waiting time is reached, the sum of the first concurrent number of the target task and the second concurrent number of the task recorded in the target sliding window and the subsequent steps are recalculated. According to the above method, users can be prevented from frequently submitting re-run jobs, limiting the submission of re-run tasks by the system, and reducing resource waste.

[0095] The above-mentioned task processing method determines the rerun sub-job according to the metadata of the rerun job submitted by the target account, obtains the first concurrent number of the target task to be submitted to the workflow scheduling system in the rerun sub-job, and the second concurrent number of the task recorded in the target sliding window, and determines whether to submit the target task in the rerun sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number. Among them, whether to submit the target task in the rerun sub-job to the workflow scheduling system is determined by the first concurrent number of the target task to be submitted to the workflow scheduling system and the second concurrent number of the task recorded in the target sliding window, thereby avoiding users from frequently submitting rerun jobs and reducing resource waste.

[0096] In an exemplary embodiment, the task processing method includes the following steps:

[0097] The first step is to determine the re-run sub-job based on the metadata of the re-run job submitted by the target account.

[0098] The second step is to obtain the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0099] In the third step, if the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, the first waiting time of the target task is updated.

[0100] The fourth step is, when the first waiting time is reached, returning to the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number.

[0101] The contents described in the first to fourth steps in the embodiment of the present disclosure are consistent with those described in the above embodiments and will not be repeated here.

[0102] In a fifth step, if the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, the resource usage rate of the workflow scheduling system is obtained.

[0103] Among them, the fifth step and the third step are in parallel relationship.

[0104] In the embodiment of the present disclosure, if the sum of the first concurrent number and the second concurrent number is less than the first preset threshold, it is also necessary to obtain the resource utilization rate of the workflow scheduling system. According to the resource utilization rate of the workflow scheduling system, it is determined whether to submit the target task in the re-run sub-job to the workflow scheduling system.

[0105] In the sixth step, if the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0106] Among them, the second preset threshold is a critical value for measuring the resource usage of the workflow scheduling system. When the resource utilization rate of the workflow scheduling system is less than the second preset threshold, it means that the resource utilization rate of the workflow scheduling system is not particularly large and can continue to be used. When the resource utilization rate of the workflow scheduling system is greater than or equal to the second preset threshold, it means that the resource utilization rate of the workflow scheduling system is relatively large and cannot continue to be used. Through this method, tasks can be submitted only when the resource utilization rate of the workflow scheduling system is less than the second preset threshold, avoiding submitting too many tasks and causing task congestion. If it is not managed for a long time, it will affect the execution of other tasks.

[0107] Step 7: If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, update the second waiting time of the target task.

[0108] Step 8. When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks in the re-run sub-job to be submitted to the workflow scheduling system and the second concurrent number of tasks recorded in the target sliding window.

[0109] In the disclosed embodiment, the second preset threshold represents the submission limit value of the concurrent number of tasks. If the resource utilization rate of the workflow scheduling system is greater than or equal to the second preset threshold value, it means that after the target task to be submitted is submitted, the concurrent number of tasks exceeds the submission limit value, and the target task in the re-run sub-job is not allowed to be submitted to the workflow scheduling system. At this time, it is necessary to set a waiting time (i.e., the second waiting time) for the above-mentioned target task. When the second waiting time is reached, the sum of the first concurrent number of the target task and the second concurrent number of the tasks recorded in the target sliding window and the subsequent steps are recalculated. According to the above method, users can be prevented from frequently submitting re-run jobs, reducing the resource utilization rate of the workflow scheduling system, limiting the submission of re-run tasks by the system, and reducing resource waste.

[0110] In an exemplary embodiment, the task processing method includes the following steps:

[0111] The first step is to determine the re-run sub-job based on the metadata of the re-run job submitted by the target account.

[0112] The second step is to obtain the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0113] In the third step, if the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, the first waiting time of the target task is updated.

[0114] The fourth step is, when the first waiting time is reached, returning to the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number.

[0115] In a fifth step, if the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, the resource usage rate of the workflow scheduling system is obtained.

[0116] Among them, the fifth step and the third step are in parallel relationship.

[0117] Step 6: If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, update the second waiting time of the target task.

[0118] Step 7. When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0119] The contents described in the first to seventh steps in the embodiment of the present disclosure are consistent with the corresponding contents described in the above embodiments and will not be repeated here.

[0120] In step 8, if the resource utilization rate of the workflow scheduling system is less than the second preset threshold, the resource utilization rate of the resource management system is obtained.

[0121] Among them, the eighth step and the sixth step are in parallel relationship.

[0122] In the ninth step, if the resource utilization rate of the resource management system is less than the third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0123] The third preset threshold is a critical value for measuring the resource usage of the resource management system (Yarn resource management system). When the resource usage rate of the resource management system is less than the third preset threshold, it means that the resource usage rate of the resource management system is not particularly large and can continue to be used. When the resource usage rate of the resource management system is greater than or equal to the third preset threshold, it means that the resource usage rate of the resource management system is relatively large and cannot continue to be used.

[0124] Among them, the Yarn resource management system is a general resource management system that can provide unified resource management and scheduling for upper-layer applications. Map Reduce tasks and Spark tasks can be run on the Yarn platform. Yarn resources are divided according to queues. Each business has its own re-running queue, and the allocation of resources for each queue may be different. If a queue has fewer resources and many tasks are submitted to this queue, the queue will be seriously congested. If it is not managed for a long time, it will affect the execution of other tasks.

[0125] In the disclosed embodiment, when the resource utilization rate of the resource management system is less than the third preset threshold, it means that the resource utilization rate of the resource management system is not particularly large, the resources are not tight, and they can continue to be used, and tasks can continue to be submitted to the workflow scheduling system. Through this method, tasks can be submitted only when the resource utilization rate of the resource management system is less than the third preset threshold, avoiding submitting too many tasks and causing task congestion. If it is not managed for a long time, it will affect the execution of other tasks.

[0126] In step 10, if the resource utilization rate of the resource management system is greater than or equal to the third preset threshold, the third waiting time of the target task is updated.

[0127] In the eleventh step, when the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks in the re-run sub-job to be submitted to the workflow scheduling system and the second concurrent number of tasks recorded in the target sliding window.

[0128] In the disclosed embodiment, the third preset threshold represents the submission limit value of the concurrent number of tasks. If the resource utilization rate of the resource management system is greater than or equal to the third preset threshold value, it means that after the target task to be submitted is submitted, the concurrent number of tasks exceeds the submission limit value, and the target task in the re-run sub-job is not allowed to be submitted to the workflow scheduling system. At this time, it is necessary to set a waiting time (i.e., the third waiting time) for the above-mentioned target task. When the third waiting time is reached, the sum of the first concurrent number of the target task and the second concurrent number of the tasks recorded in the target sliding window and the subsequent steps are recalculated. According to the above method, users can be prevented from frequently submitting re-run jobs, reducing the resource utilization rate of the resource management system and reducing resource waste.

[0129] In an exemplary embodiment, the method further comprises:

[0130] Determine the concurrent number of tasks based on the task portrait, the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system and preset influencing factors; and update the second concurrent number of tasks recorded in the target sliding window based on the concurrent number of tasks.

[0131] Among them, the task portrait represents indicators such as the number of concurrent tasks and task resource consumption. The preset impact factor is a preset threshold value, which can be modified according to actual conditions. Optionally, the task portrait corresponds to impact factor 1, the resource utilization rate of the workflow scheduling system corresponds to impact factor 2, and the resource utilization rate of the resource management system corresponds to impact factor 3, then task portrait * impact factor 1 + task resource consumption and other indicators * impact factor 2 + resource utilization rate of the resource management system * impact factor 3 = number of concurrent tasks.

[0132] In the disclosed embodiment, after the target task is submitted to the workflow scheduling system, the number of concurrent tasks is determined according to the task portrait, the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system, and the preset influencing factors. The second concurrent number of tasks recorded in the target sliding window is updated to the number of concurrent tasks, so that the metadata of the rerun job submitted by the target account can be re-determined, and the first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job is obtained, as well as the second concurrent number of tasks recorded in the target sliding window (the number of concurrent tasks at this time is the number of concurrent tasks determined according to the task portrait, the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system, and the preset influencing factors), and the subsequent steps are continued to avoid users from frequently submitting rerun jobs, reducing the resource utilization rate of the resource management system, and reducing resource waste.

[0133] It should be understood that although Figure 2The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0134] Figure 3 is a block diagram of a task processing device according to an exemplary embodiment. Figure 3 The device includes a first determining unit 31, an acquiring unit 32 and a second determining unit 33.

[0135] The first determining unit 31 is configured to determine the re-run sub-job according to the metadata of the re-run job submitted by the target account;

[0136] The acquisition unit 32 is configured to execute acquisition of a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job, and a second concurrent number of tasks recorded in the target sliding window;

[0137] The second determining unit 33 is configured to determine whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number.

[0138] In an exemplary embodiment, the first determining unit 31 is further configured to execute:

[0139] Get the metadata of the rerun job submitted by the target account;

[0140] The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

[0141] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0142] If the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0143] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0144] If the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, updating the first waiting time of the target task;

[0145] When the first waiting time is reached, the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number is returned to be executed.

[0146] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0147] Obtaining resource usage of the workflow scheduling system;

[0148] If the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0149] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0150] If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, updating the second waiting time of the target task;

[0151] When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0152] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0153] Get resource usage of resource management system;

[0154] If the resource usage rate of the resource management system is less than a third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

[0155] In an exemplary embodiment, the second determining unit 33 is further configured to execute:

[0156] If the resource usage rate of the resource management system is greater than or equal to the third preset threshold, updating the third waiting time of the target task;

[0157] When the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

[0158] In an exemplary embodiment, the task processing device further includes:

[0159] A third determining unit is configured to determine the number of concurrent tasks according to the task profile, the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system and a preset influencing factor;

[0160] An updating unit is configured to update a second concurrent number of tasks recorded in the target sliding window according to the concurrent number of tasks.

[0161] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0162] Figure 4 4 is a block diagram of a device 400 for task management according to an exemplary embodiment. For example, the device 400 may be a server. Figure 4 , the device 400 includes a processing component 420, which further includes one or more processors, and a memory resource represented by a memory 422 for storing instructions, such as applications, that can be executed by the processing component 420. The application stored in the memory 422 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 420 is configured to execute the instructions to perform the above-mentioned task management method.

[0163] The device 400 may also include a power component 424 configured to perform power management of the device 400, a wired or wireless network interface 426 configured to connect the device 400 to a network, and an input / output (I / O) interface 428. The device 400 may operate based on an operating system stored in the memory 422, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0164] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 422 including instructions, which can be executed by a processor of the device 400 to perform the above method. The storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0165] In an exemplary embodiment, a server is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the task processing method described in any one of the embodiments.

[0166] In an exemplary embodiment, a storage medium is provided. When instructions in the storage medium are executed by a processor of a server, the server is enabled to execute the task processing method described in any one of the embodiments.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0168] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A task processing method, characterized in that: include: Determine the re-run sub-job based on the metadata of the re-run job submitted by the target account; Obtain a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job, and a second concurrent number of tasks recorded in the target sliding window, wherein the second concurrent number indicates the concurrent number of tasks that have been successfully submitted to the workflow scheduling system within a preset period of time; Compare the sum of the first concurrent number and the second concurrent number with a preset threshold, and determine whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the comparison result; Determine the number of concurrent tasks according to the task portrait and the preset impact factor corresponding to the task portrait, the resource utilization rate of the workflow scheduling system and the preset impact factor corresponding to the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system and the preset impact factor corresponding to the resource utilization rate of the resource management system; The second concurrent number of tasks recorded in the target sliding window is updated according to the concurrent number of tasks.

2. The task processing method according to claim 1, characterized in that: The step of determining the re-run sub-job according to the metadata of the re-run job submitted by the target account includes: Get the metadata of the rerun job submitted by the target account; The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

3. The task processing method according to claim 1, characterized in that: The comparing the sum of the first concurrent number and the second concurrent number with a preset threshold, and determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the comparison result, includes: If the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

4. The task processing method according to claim 3, characterized in that: The method further comprises: If the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, updating the first waiting time of the target task; When the first waiting time is reached, the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number is returned to be executed.

5. The task processing method according to claim 3, characterized in that: The step of determining to submit the target task in the re-run sub-job to the workflow scheduling system also includes: Obtaining resource utilization of the workflow scheduling system; If the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

6. The task processing method according to claim 5, characterized in that: The method further comprises: If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, updating the second waiting time of the target task; When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

7. The task processing method according to claim 5, characterized in that: After the resource usage rate of the workflow scheduling system is less than the second preset threshold, before the determination to submit the target task in the re-run sub-job to the workflow scheduling system, the method further includes: Get resource usage of resource management system; If the resource usage rate of the resource management system is less than a third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

8. The task processing method according to claim 7, characterized in that: The method further comprises: If the resource usage rate of the resource management system is greater than or equal to the third preset threshold, updating the third waiting time of the target task; When the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

9. A task processing device, characterized in that: include: A first determination unit is configured to determine a re-run sub-job according to metadata of the re-run job submitted by the target account; An acquisition unit is configured to execute acquisition of a first concurrent number of target tasks to be submitted to the workflow scheduling system in the rerun sub-job, and a second concurrent number of tasks recorded in the target sliding window, wherein the second concurrent number indicates the concurrent number of tasks that have been successfully submitted to the workflow scheduling system in a preset time period; A second determining unit is configured to compare the sum of the first concurrent number and the second concurrent number with a preset threshold, and determine whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the comparison result; The third determination unit is configured to determine the number of concurrent tasks according to the task portrait and the preset impact factor corresponding to the task portrait, the resource utilization rate of the workflow scheduling system and the preset impact factor corresponding to the resource utilization rate of the workflow scheduling system, the resource utilization rate of the resource management system and the preset impact factor corresponding to the resource utilization rate of the resource management system; An updating unit is configured to update a second concurrent number of tasks recorded in the target sliding window according to the concurrent number of tasks.

10. The task processing device according to claim 9, characterized in that: The first determining unit is further configured to execute: Get the metadata of the rerun job submitted by the target account; The rerun job is divided according to the time slices in the metadata to determine the rerun sub-jobs.

11. The task processing device according to claim 9, characterized in that: The second determining unit is further configured to execute: If the sum of the first concurrent number and the second concurrent number is less than a first preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

12. The task processing device according to claim 11, characterized in that: The second determining unit is further configured to execute: If the sum of the first concurrent number and the second concurrent number is greater than or equal to the first preset threshold, updating the first waiting time of the target task; When the first waiting time is reached, the step of determining whether to submit the target task in the re-run sub-job to the workflow scheduling system according to the first concurrent number and the second concurrent number is returned to be executed.

13. The task processing device according to claim 11, characterized in that: The second determining unit is further configured to execute: Obtaining resource utilization of the workflow scheduling system; If the resource usage rate of the workflow scheduling system is less than a second preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

14. The task processing device according to claim 13, characterized in that: The second determining unit is further configured to execute: If the resource usage rate of the workflow scheduling system is greater than or equal to the second preset threshold, updating the second waiting time of the target task; When the second waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

15. The task processing device according to claim 13, characterized in that: The second determining unit is further configured to execute: Get resource usage of resource management system; If the resource usage rate of the resource management system is less than a third preset threshold, it is determined to submit the target task in the re-run sub-job to the workflow scheduling system.

16. The task processing device according to claim 15, characterized in that: The second determining unit is further configured to execute: If the resource usage rate of the resource management system is greater than or equal to the third preset threshold, updating the third waiting time of the target task; When the third waiting time is reached, return to execute the step of obtaining the first concurrent number of target tasks to be submitted to the workflow scheduling system in the re-run sub-job and the second concurrent number of tasks recorded in the target sliding window.

17. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the task processing method according to any one of claims 1 to 8.

18. A storage medium, when instructions in the storage medium are executed by a processor of a server, enables the server to execute the task processing method according to any one of claims 1 to 8.

Images