Task scheduling method, device, equipment and storage medium

Abstract

The present application provides a task scheduling method, device, equipment and storage medium. It relates to the field of resource management technology. The method includes: obtaining the task to be run and the configuration information of the task to be run; the configuration information of the task to be run includes the computing resources required to run the task to be run; judging whether the computing resources required for the task to be run are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; if the computing resources required for the task to be run are greater than the remaining resources of the computing device, then continue to execute the judgment step until the computing resources required for the task to be run are less than or equal to the remaining resources of the computing device; create a thread for the task to be run, and allocate computing resources to the thread to run the task to be run. The method of the present application solves the problem that a large number of threads created in the prior art are in a waiting state, causing the task to be in an idle running state.

Description

Technical Field

[0001] The present application relates to the field of resource management technology, and in particular to a task scheduling method, device, equipment and storage medium. Background Art

[0002] Task scheduling refers to the automatic execution of tasks based on a preset time point, a preset time interval, or a preset number of executions.

[0003] Task scheduling is to create threads for tasks and allocate computing resources to the created threads to run tasks. After creating threads for tasks, the task scheduling center will assign tasks to computing devices. If the computing device does not have free resources to run the task, the task will be in a waiting state for a long time and wait until the computing node has sufficient resources before executing the assigned task.

[0004] This will cause a large number of created threads to be in a waiting state, causing the task to run in vain. Summary of the invention

[0005] The present application provides a task scheduling method, apparatus, device and storage medium, which are used to solve the problem in the prior art that a large number of threads created are in a waiting state, causing tasks to be in an idle running state.

[0006] In a first aspect, the present application provides a task scheduling method, comprising: obtaining a task to be run and configuration information of the task to be run; the configuration information of the task to be run includes the computing resources required to run the task to be run; judging whether the computing resources required for the task to be run are less than or equal to the remaining resources of a computing device; the computing device is used to perform task processing; if the computing resources required for the task to be run are greater than the remaining resources of the computing device, continuing to execute the judging step until the computing resources required for the task to be run are less than or equal to the remaining resources of the computing device; creating a thread for the task to be run, and allocating computing resources to the thread to run the task to be run.

[0007] In a second aspect, the present application provides a task scheduling device, comprising: an acquisition module, used to acquire a task to be run and configuration information of the task to be run; the configuration information of the task to be run includes the computing resources required to run the task to be run; a judgment module, used to judge whether the computing resources required for the task to be run are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; if the computing resources required for the task to be run are greater than the remaining resources of the computing device, continue to execute the judgment step until the computing resources required for the task to be run are less than or equal to the remaining resources of the computing device; an allocation module, used to create a thread for the task to be run, and allocate computing resources to the thread to run the task to be run.

[0008] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0010] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0011] The task scheduling method, apparatus, device and storage medium provided by the present application are as follows: obtaining the tasks to be run and the configuration information of the tasks to be run; the configuration information of the tasks to be run includes the computing resources required to run the tasks to be run; judging whether the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; if the computing resources required for the tasks to be run are greater than the remaining resources of the computing device, then the judging step is continued until the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device; the tasks to be run are assigned to the computing device to run the tasks to be run. Since after the tasks to be run are obtained, when it is determined that the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device, the tasks to be run are assigned to the computing device to run the tasks to be run. Therefore, it is possible to solve the problem that a large number of created threads are in a waiting state, causing the tasks to be in an empty running state. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 An application scenario diagram of task scheduling in the prior art;

[0014] Figure 2 This is an application scenario diagram of the task scheduling method applicable to the embodiment of the present application;

[0015] Figure 3 The process of the task scheduling method provided in the embodiment of the present application Figure 1 ;

[0016] Figure 4 The process of the task scheduling method provided in the embodiment of the present application Figure 2 ;

[0017] Figure 5 The process of the task scheduling method provided in the embodiment of the present application Figure 3 ;

[0018] Figure 6 An example diagram of an operation plan tree including multiple subtasks provided in an embodiment of the present application;

[0019] Figure 7 The process of the task scheduling method provided in the embodiment of the present application Figure 4 ;

[0020] Figure 8 An application scenario diagram of ETL task scheduling in the prior art;

[0021] Fig. 9 An example diagram of a task scheduling method in a parallel task queue provided in an embodiment of the present application;

[0022] Fig.10 An example diagram of a task scheduling method in a serial task queue provided in an embodiment of the present application;

[0023] Fig.11 A schematic diagram of the structure of a task scheduling device provided in an embodiment of the present application;

[0024] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] Terminology explanation:

[0028] ETL: It is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end.

[0029] Platform: refers to the system architecture.

[0030] Figure 1 This is an application scenario diagram of task scheduling in the prior art. Figure 1 As shown, the application scenario includes: a task queue 11, a scheduling center 12 and at least one computing device 13; wherein the task queue includes multiple tasks ( Figure 1 Each rectangular box in FIG.

[0031] Task scheduling refers to the automatic execution of tasks based on a preset time point, a preset time interval, or a preset number of executions.

[0032] In the prior art, the scheduling center 12 is used to obtain tasks from the task queue and create threads, and to assign the created threads to computing devices. After the scheduling center assigns the created threads to computing devices, if the computing devices do not have enough resources to run the tasks assigned to the threads, the tasks will be hung on the computing nodes for waiting, which is equivalent to the tasks running in vain. When the computing devices finish executing the currently executed tasks and release the resources, they will execute the assigned tasks. In this way, a large number of created threads will be in a waiting state, which will cause the tasks to be in an vain state, thereby affecting the efficiency of task execution.

[0033] In response to the above technical problems, the present application proposes the following technical concept: after obtaining the task to be run, first determine whether the computing device has sufficient resources to run the task to be run. If the computing device has sufficient resources to run the task to be run, create a thread for the task to be run, and allocate computing resources to the thread to run the task to be run. In this way, the problem that a large number of threads created in the prior art are in a waiting state, causing the task to be in an idle state, can be solved.

[0034] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0035] Figure 2 FIG. 1 is an application scenario diagram of the task scheduling method applicable to the embodiment of the present application. Figure 2 As shown, the application scenario includes: a source end 21, a scheduling center 22, a destination end 23 and a computing device 24. Among them, the source end 21 includes one or more business system databases, each of which includes source data. The destination end 23 includes a data warehouse or other databases. The scheduling center 22 can be an ETL scheduling tool, which can be a server or a server cluster including multiple servers. The computing device 24 can be a computing cluster including multiple computers, each of which can be regarded as a computing node.

[0036] The user configures the configuration information of the ETL task scheduling through the command line and generates a configuration file. The scheduling center 22 reads the configuration file and generates tasks to be executed according to the configuration information in the configuration file. The tasks to be executed include tasks of extracting data from the source end 21, tasks of converting the data extracted from the source end 21, and tasks of loading the converted data to the destination end 23.

[0037] Among them, the configuration information of task scheduling includes the entities in the job, the job running frequency, the dependency between jobs, the dependency between jobs and time points, the dependency between jobs and external data, and the dynamic parameter transmission of jobs.

[0038] The entities in a job refer to the threads that execute the job. For example, the number of threads can be set, and the number of threads determines the number of tasks that the computing device can execute in parallel.

[0039] The job run frequency refers to how often the job runs.

[0040] Inter-job dependency refers to the dependency relationship between jobs. For example, if there are jobs A and B, and the running of job A depends on the execution result of job B, then jobs A and B can be considered to have a dependency relationship.

[0041] Job-time dependency means that the job needs to be completed before a specified time.

[0042] The dependency between a job and external data means that the execution of a job depends on external data. The external data can be data from the source or data from other stages. For example, if a job in the conversion stage depends on the result data obtained by executing the tasks in the extraction stage, the result data obtained by executing the tasks in the extraction stage is the external data that the job in the conversion stage depends on.

[0043] Dynamic parameter transfer of a job means that a job passes the configuration information of a job that depends on it to a job that depends on it. For example, suppose there are jobs A and B, and the running of job A depends on the result data obtained after the execution of job B. When job B passes the result data obtained after the execution of job B to job A, it can also pass the configuration information of job B to job A.

[0044] The above descriptions of configuration information all use jobs as examples. However, in reality, a job can include one or more tasks, and a task is the execution object of a computing node. Therefore, the configuration information of task scheduling can also be understood as the configuration information for each task in a job, that is, the entity in a job refers to the thread that executes each task in each job. Inter-job dependency refers to the dependency between tasks in each job. Job-time point dependency means that each task in each job needs to be completed before a specified time point. Job-external data dependency means that the execution of each task in each job needs to rely on external data. Dynamic parameter transmission of a job means that a task passes the configuration information of a task that depends on it to a task that depends on it.

[0045] based on Figure 2 In the application scenario shown, an embodiment of the present application also provides a task scheduling method. Figure 3 The process of the task scheduling method provided in the embodiment of the present application Figure 1 .like Figure 3 As shown, the task scheduling method includes:

[0046] S301, obtaining tasks to be run and configuration information of the tasks to be run in a task queue; the configuration information of the tasks to be run includes computing resources required to run the tasks to be run.

[0047] The execution subject of the method of this embodiment may be Figure 2 The dispatch center shown in .

[0048] In this embodiment, task processing can be performed simultaneously through parallel task queues and serial task queues. Among them, when a task in a parallel task queue is executed, even if the current task in the parallel task queue has not been completed, the next task can be started. When a task in a serial task queue is executed, the current task must be completed before the next task can be started.

[0049] Optionally, obtaining the task to be run includes: obtaining the task to be run from a task queue; the task queue includes a parallel task queue or a serial task queue.

[0050] Before this step, you need to configure the computing resources required for each task. The computing resources required for each task can be configured through configuration files, which use commands to configure configuration information.

[0051] S302: Determine whether the computing resources required for the task to be executed are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing.

[0052] In this embodiment, the resources of the computing device refer to the CPU, memory and other resources of the computing device. The computing device includes multiple computing nodes, and the resources corresponding to the computing nodes that are running tasks among the multiple computing nodes are used resources, while the resources corresponding to the idle computing nodes that are not running tasks are remaining resources.

[0053] Before this step, it also includes:

[0054] Step a1, determining the total computing resources of the computing device and the used resources at the current moment;

[0055] Step a2: Subtract the currently used resources from the total computing resources of the computing device to obtain the remaining resources of the computing device.

[0056] S303: If the computing resources required by the task to be executed are greater than the remaining resources of the computing device, continue to execute the judgment step until the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device.

[0057] This step is to determine whether there are sufficient computing resources in the computing device to process the pending tasks before assigning them to the computing device. If there are sufficient computing resources in the computing device to process the pending tasks, the pending tasks will be assigned to the computing device, otherwise, the pending tasks will be stored in the task queue.

[0058] S304: Create a thread for the task to be run, and allocate computing resources to the thread to run the task to be run.

[0059] This step is to assign the task to be run to one of the computing nodes so that the computing node runs the task to be run. Specifically, a thread is created for the task to be run, and the task to be run is run through the thread.

[0060] This embodiment obtains the tasks to be run and the configuration information of the tasks to be run; the configuration information of the tasks to be run includes the computing resources required to run the tasks to be run; it is determined whether the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; if the computing resources required for the tasks to be run are greater than the remaining resources of the computing device, the judgment step is continued until the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device; a thread is created for the tasks to be run, and computing resources are allocated to the threads to run the tasks to be run. Since after the tasks to be run are obtained, when it is determined that the computing resources required for the tasks to be run are less than or equal to the remaining resources of the computing device, a thread is created for the tasks to be run, and computing resources are allocated to the threads to run the tasks to be run. Therefore, the problem that a large number of created threads are in a waiting state, causing the tasks to be in an empty running state, can be solved, and the effect of task operation management through resource estimation can be achieved.

[0061] Figure 4 The process of the task scheduling method provided in the embodiment of the present application Figure 2 .like Figure 4 As shown, the task scheduling method includes the following steps:

[0062] S401, obtaining a target task; the target task is a task that has reached its task execution time.

[0063] This embodiment obtains the target task from the task list. Before obtaining the target task from the task list, it also includes: generating a task list according to the running frequency of the task. Specifically, it determines whether the current moment has reached the running time of the task according to the running frequency of the task. If so, the task is added to the task list; otherwise, the task is not processed. Among them, the running frequency of the task is configured through commands and stored in the configuration information in the configuration file.

[0064] Among them, the task list can be understood as a task plan list. Exemplarily, a task plan list for each day can be generated according to the configuration information of the task, and the tasks included in the task plan list are the tasks that need to be executed on that day. For example, the running frequency of task 1 is to be executed once at 13:15 every day, then when it reaches 13:15 every day, it can be determined that task 1 needs to be executed, and task 1 is added to the task list.

[0065] S402: Determine whether the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue.

[0066] The maximum number of tasks in a task queue refers to the maximum number of tasks that the task queue can accommodate. It is also the maximum number of tasks that can be added to the task queue from the task list. For example, if the maximum number of tasks in a task queue is 100, and the number of tasks to be run currently in the task queue is 90, it is determined that the queue to be run can add the target task.

[0067] S403: If the number of all tasks to be run in the task queue is greater than or equal to the maximum number of tasks in the task queue, continue to execute the judgment step until the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue.

[0068] In this embodiment, the maximum number of tasks in the task queue is configured in advance by means of commands and stored in a configuration file.

[0069] S404: Add the target task to the task queue.

[0070] Specifically, add the target task to the task queue, including:

[0071] Step b1, obtaining configuration information of the target task; the configuration information of the target task includes an identifier of the target task, and the identifier of the target task is used to indicate that the target task is added to a parallel task queue or a serial task queue.

[0072] Exemplarily, the target task identifier can be set to 1, and the corresponding operation is to add the target task to the parallel task queue when the target task's running time is reached, and the target task identifier can be set to 0, and the corresponding operation is to add the target task to the serial task queue when the target task's running time is reached.

[0073] It should be understood that the operations indicated by 1 and 0 above are only exemplary, and it is also possible to set the target task's identifier to 0, and correspondingly add the target task to the parallel task queue when the target task's runtime is reached, and set the target task's identifier to 1, and correspondingly add the target task to the serial task queue when the target task's runtime is reached. This embodiment does not limit this.

[0074] Before step b1, the identification of the task needs to be configured. The identification of the target task can be configured according to the computing resources and running frequency required by the task. Specifically, the identification of the task whose required computing resources are greater than the preset resources and whose running frequency is less than the preset frequency can be set to 1, while the identification of the task whose required computing resources are less than or equal to the preset resources and whose running frequency is greater than or equal to the preset frequency can be set to 0.

[0075] Step b2: If the identifier of the target task indicates that the target task is to be added to the parallel task queue, then the target task is added to the parallel task queue.

[0076] Step b3: If the identifier of the target task indicates that the target task is to be added to the serial task queue, then the target task is added to the serial task queue.

[0077] During the execution of a task, there are cases of successful execution and failed execution. In the case of task execution failure, an automatic scheduling strategy can be set to automatically schedule the failed task to re-run. Specifically, a failed task queue can be provided to temporarily cache the failed task, and when the current time reaches the preset failed task scheduling time, the failed task is obtained from the failed task queue and re-added to the parallel task queue. The process specifically includes the following steps:

[0078] Step c1: Determine whether the task to be run is successful.

[0079] Step c2: If the task to be run is successfully run, the task to be run is removed from the task queue.

[0080] Step c3: If the task to be run fails and the number of failures reaches a preset number of failures, the task to be run is added to a failed task queue.

[0081] As described above, the task queue includes a parallel task queue and a serial task queue. This embodiment includes the following two optional implementations:

[0082] In an optional implementation, for a parallel task queue, if the task to be run is successfully executed, the task to be run is removed from the parallel task queue; if the task to be run fails to execute and the number of failures reaches a preset number of failures, the task to be run is added to the failed task queue.

[0083] In another optional implementation, for a serial task queue, if the task to be run is successfully executed, the task to be run is removed from the serial task queue; if the task to be run fails to execute and the number of failures reaches a preset number of failures, the task to be run is added to a failed task queue.

[0084] Among them, in the above two optional implementations, adding the to-be-run tasks to the failed task queue can be understood as removing the to-be-run tasks from the parallel task queue or the serial task queue, and adding the to-be-run tasks to the failed task queue.

[0085] After adding the task to be run to the failed task queue, the following steps may also be included:

[0086] Step c4: Determine whether the current time has reached the scheduled time of the failed task.

[0087] Step c5: If the current time reaches the scheduling time of the failed task, the failed task is obtained from the failed task queue and added to the parallel task queue.

[0088] Among them, getting failed tasks from the failed task queue and adding them to the parallel task queue includes:

[0089] Step c51: Determine whether the computing resources required by the failed task are less than or equal to the remaining resources of the computing device.

[0090] Step c52: If the computing resources required by the failed task are greater than the remaining resources of the computing device, continue to execute the judgment step c51 until the computing resources required by the failed task are less than or equal to the remaining resources of the computing device.

[0091] Step c53: Allocate the failed task to a computing device to run the failed task.

[0092] Furthermore, since execution failure still occurs after adding failed tasks in the failed queue to the parallel task queue, the failed tasks may be added to the serial task queue when the number of times the failed tasks are added to the failed queue exceeds a preset number.

[0093] Step c5: If the current time reaches the scheduled time of the failed task, the failed task is obtained from the failed task queue and added to the parallel task queue, so that the failed task can be automatically re-run.

[0094] Based on the above embodiments, the tasks in the parallel task queue or serial task queue may include subtasks, that is, the tasks to be executed are task sets. For this type of tasks, the present application proposes the following embodiments:

[0095] Figure 5 The process of the task scheduling method provided in the embodiment of the present application Figure 3 .like Figure 5 As shown, the task scheduling method includes the following steps:

[0096] S501. Determine M target subtasks that need to be executed currently according to the execution order of N subtasks; M is an integer greater than or equal to 1 and less than or equal to N.

[0097] In this embodiment, the configuration information of the task to be run also includes: the task to be run includes N subtasks, the execution order between the N subtasks, and the identifier of each subtask, and the identifier of each subtask is used to characterize whether the subtask is a serial task or a parallel task; N is an integer greater than or equal to 2.

[0098] Before step S501, the following steps may also be included:

[0099] Step d1: If the task to be run includes subtasks, an operation plan tree structure of the task to be run can be generated according to the configuration information of the task to be run; wherein the operation plan tree of the task to be run includes N nodes, and the N nodes correspond to the N subtasks respectively.

[0100] The execution order of the N subtasks may be determined according to the operation plan tree structure of the task to be executed.

[0101] Step d2: If the subtask corresponding to the parent node is executed successfully, then the subtask corresponding to at least one child node of the parent node is added to the task queue.

[0102] Specifically, the subtask is added to the parallel task queue or the serial task queue according to the identifier of the subtask corresponding to the child node of the parent node.

[0103] Step d3: When all subtasks corresponding to all nodes in the operation plan tree are executed successfully, it is determined that the task to be executed is executed successfully, and the task to be executed is removed from the task queue.

[0104] S502: Add the target subtask to a parallel task queue or a serial task queue according to the identifier of the target subtask.

[0105] S503: Determine whether all N subtasks are executed successfully.

[0106] S504: If all N subtasks are executed successfully, it is determined that the task to be executed is executed successfully.

[0107] S505: If any one of the N subtasks is not successfully executed, it is determined that the execution of the task to be executed has failed.

[0108] The present embodiment is described below with reference to an example:

[0109] Figure 6 An example diagram of an operation plan tree including multiple subtasks provided in an embodiment of the present application. Figure 6 As shown, assume that there is a task A, which includes subtask 1, subtask 2, subtask 3 and subtask 4. The dependency relationship between the four subtasks is: after subtask 1 is completed, subtask 2 is executed, and after subtask 3 is completed, subtask 4 is executed, where subtask 2 and subtask 3 can be executed in parallel. In the task scheduling process, the tasks to be executed include:

[0110] Step A: Execute subtask 1.

[0111] Step B: If subtask 1 is executed successfully, subtask 2 and subtask 3 are executed.

[0112] Step C: If both subtask 2 and subtask 3 are successfully executed, then execute subtask 4.

[0113] Step D: If subtask 4 is executed successfully, it is determined that the task to be executed is executed successfully.

[0114] Figure 7 The process of the task scheduling method provided in the embodiment of the present application Figure 4 .like Figure 7 As shown, the task scheduling method includes the following steps:

[0115] S701. Obtain configuration information for task processing; the configuration information for task processing includes a preset occupancy rate of computing resources of a computing device.

[0116] S702: Determine the occupancy rate of computing resources by X processes of the computing device.

[0117] S703. If the occupancy rate of the computing resources by the X processes of the computing device is greater than or equal to the preset occupancy rate of the computing resources of the computing device, the X processes of the computing device are sorted in descending order of the occupancy rate of the computing resources, and the tasks to be run corresponding to the first Y processes are stopped, so that the occupancy rate of the computing resources by the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device; each process includes one or more tasks to be run.

[0118] Specifically, step S703 includes the following steps:

[0119] Step e1, determining whether the occupancy rate of the computing resources of the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device;

[0120] Step e2: If the occupancy rate of the computing resources by the X processes of the computing device is greater than or equal to the preset occupancy rate of the computing resources of the computing device, the X processes of the computing device are sorted in descending order of the occupancy rate of the computing resources, and the to-be-run tasks corresponding to the process with the largest occupancy rate of the computing resources are stopped, and the judgment step is continued until the occupancy rate of the computing resources by the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device.

[0121] Step e3: Add the stopped tasks to be run to the failed task queue.

[0122] Optionally, in order to avoid direct task abandonment and thus task execution failure, after step S703, the to-be-run tasks corresponding to the process occupying the most resources may be added to the failed task queue to improve the task execution success rate.

[0123] For example, the preset occupancy rate of the computing resources of the computing device is 90%, and the sum of the memory occupancy rates of all processes on the computing device is 95%. Then all the processes on the 100 computing nodes are sorted in descending order according to the memory occupancy rate, and the tasks corresponding to the processes with the largest memory occupancy rate are stopped, thereby releasing computing resources.

[0124] This embodiment determines whether the occupancy rate of the computing resources of the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device. If the occupancy rate of the computing resources of the X processes of the computing device is greater than or equal to the preset occupancy rate of the computing resources of the computing device, the X processes of the computing device are sorted in descending order of the computing resource occupancy rate, and the to-be-run tasks corresponding to the process with the largest computing resource occupancy rate are stopped, and the judgment step is continued until the occupancy rate of the computing resources of the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device. In this way, the use of computing resources can be monitored during the task running process to prevent the computing resources from being occupied by one of the tasks for a long time, resulting in an excessively high occupancy rate of the computing resources, which affects the processing efficiency of other tasks. Or, the problem of task processing timeout caused by too many tasks leading to the even distribution of computing resources.

[0125] On the basis of the above embodiment, after allocating the tasks to be executed to the computing device to execute the tasks to be executed, the method of this embodiment further includes the following steps:

[0126] Step f1, obtaining task running results of multiple tasks to be run within a preset time period; the task running results include whether the tasks to be run are run successfully.

[0127] The preset time period may be in days, and the task running results of all tasks in the task list of each day are formed into a task running list and sent to the terminal device of the target object.

[0128] The task running list may include the total number of tasks running on that day, the total number of tasks running successfully, and the total number of tasks running failed.

[0129] Step f2: Generate a task running list according to the task running results of multiple tasks to be run within a preset time period.

[0130] Step f3: Send the task running list to the terminal device of the target object; the target object is the object for which the task to be run is arranged.

[0131] On the basis of the above embodiment, the task running result of the task to be run also includes: whether the task to be run is completed according to the agreed time; then the method of this embodiment also includes:

[0132] Step g1: Determine whether the actual arrival time of the data corresponding to the task to be run does not exceed the agreed time.

[0133] For example, assuming that the task to be run is a task in the extraction phase, and the data corresponding to the task to be run is data from the source end, it is to determine whether the data from the source end arrives at the agreed time.

[0134] Step g2: If the actual arrival time of the data corresponding to the task to be run does not exceed the agreed time, an alarm message indicating that the data of the task to be run has arrived late is generated.

[0135] Step g3: Generate a task running result of the task to be run according to the alarm information of delayed arrival of the data of the task to be run.

[0136] For example, assuming that the agreed time for the data corresponding to Task 1 is 13:15 every day, if the actual arrival time of the data corresponding to Task 1 does not exceed 13:15 every day, it is determined that the data of the task to be run has not been delayed, and the task running result of the task to be run can be directly generated. If the actual arrival time of the data corresponding to Task 1 is after 13:15 every day, it is determined that the data of Task 1 has been delayed.

[0137] It should be understood that before adding a task to the task list, it is necessary to determine whether the data corresponding to the task has arrived. If the data corresponding to the task has arrived, the task will be added to the task list, otherwise, the task will not be processed. The delayed arrival of the data corresponding to the task does not affect the execution of the task. When the task reaches the task running time and the data corresponding to the task has not arrived, it will continue to determine whether the data corresponding to the task has arrived until the data corresponding to the task arrives, and the task will be added to the task queue.

[0138] When the running status of the daily tasks is counted, if the data corresponding to the task arrives late, the alarm information of the delayed arrival of the data corresponding to the task is added to the task running results of the tasks to be run, and a task running list for the day is generated.

[0139] Figure 8 This is an application scenario diagram of the ETL task scheduling of the existing technology. Figure 8 As shown, the application scenario includes: a source end 81, a terminal device 82, a configuration database 83 and a destination end 84.

[0140] The source end 81 includes one or more business system databases, each of which includes multiple source data. The destination end 84 includes a data warehouse or other databases.

[0141] The ETL scheduling tool is provided on the terminal device 82 , and the terminal device 82 includes a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0142] The configuration database 83 stores configuration information for ETL task scheduling, including the running frequency of ETL tasks, the dependency between ETL tasks and time points, the execution order between ETL tasks, the dependency between ETL tasks, and the dynamic parameter transmission between ETL tasks.

[0143] In the prior art, the ETL scheduling tool reads configuration information from the configuration database 83 , extracts source data from the source end 11 according to the configuration information, transforms the source data, and loads the source data to the destination end 84 .

[0144] Therefore, in order to adapt to various architecture platforms, various scheduling tools need to be coupled with the database, and each scheduling tool will specify a fixed type of configuration database. For example, the configuration database 83 specified by the ETL scheduling tool of the B / S architecture is an oral database; the configuration database 83 specified by the ETL scheduling tool of the C / S architecture is a mysql database. When the ETL scheduling tool of the B / S architecture needs to use a mysql type configuration database, it is also necessary to install some files and modify the source code before it can be used. Therefore, it is impossible to achieve scheduling between heterogeneous databases, resulting in a large reliance on the database for the scheduling tool, and the inability to be used directly across platforms, resulting in low scheduling flexibility.

[0145] In addition, an ETL process includes multiple stages, for example, extraction is one stage, conversion is one stage, and loading is one stage. An ETL process may be implemented using one technology type or different technology types. For scenarios where different technology types of ETL implementation methods are required, for example, the extraction stage uses the ETL implementation method of the sas technology type, the conversion stage uses the ETL implementation method of the python technology type, and the loading stage uses the ETL implementation method of the datastage technology type. Since each stage uses a different technology type, each stage needs to be implemented by a separate system. For example, the ETL scheduling tool extracts data from the source end through the sas technology type to obtain the first intermediate result data, and sends the first intermediate result data to the system of the conversion stage. The system of the conversion stage performs data conversion according to the first intermediate result data through the python technology type to obtain the second intermediate result data, and sends the second intermediate result data to the system of the loading stage. The system of the loading stage loads the second intermediate result data to the destination end through the datastage technology type. Therefore, it is impossible to use multiple technology types of ETL implementation methods in one system in the prior art.

[0146] In response to the above technical problems, the present application proposes the following technical ideas: Taking into account the universality of cross-platform use, the configuration information of ETL task scheduling is configured in the form of a configuration file. The configuration file can be stored in a storage path of the system's hardware device. Therefore, the configuration information can be configured without relying on a database. In addition, the configuration information is configured in the configuration file in the form of a command line. The ETL implementation methods of different technical types in the ETL scheduling tool can be understood as using different languages ​​for data processing, but when they are executed on a computer, they are ultimately executed through the basic command line method. Therefore, the configuration of the configuration information in the form of a command line can adapt to a variety of ETL implementation methods of different technical types.

[0147] The configuration information of the task to be run is obtained, including: obtaining the configuration information of the task to be run from a configuration file; wherein the configuration information is configured in the configuration file in a command line manner.

[0148] Specifically, the user configures the configuration information of the task through a terminal device in the form of commands and stores it as a configuration file. The configuration file can be stored in any storage path of the system, and any path is a path that can be accessed by the network.

[0149] After configuring the configuration information of the task, you can also view, change, modify, and other operations on the configuration information in the configuration file. Specifically, it is to receive the target operation for the configuration file input by the user through the terminal device; the target operation includes: at least one of viewing, adding, deleting, and modifying; according to the target operation, perform the corresponding operation on the configuration file. The terminal device in this embodiment can be a scheduling center. For example, during the task scheduling process, the user needs to modify the configuration information of the task according to the task running request, or view the configuration information of the task. Then, by entering the target operation for the configuration file on the terminal device, at least one of the operations of viewing, adding, deleting, and modifying the configuration information can be achieved. It should be understood that the terminal device of this embodiment can be one of the servers of the scheduling center.

[0150] On the basis of the above embodiment, in order to ensure the security of the configuration file, the configuration file can also be encrypted and stored in one of the servers of the dispatch center. When the user wants to view or modify the configuration information in the configuration file, he can enter the password in the password prompt box provided by the graphical user interface of the server, and after successful decryption, view and modify the configuration information in the configuration file, where the modification operation includes adding and deleting. If the server detects that the user has performed a modification operation on the configuration information in the configuration file, the modified configuration file is encrypted and stored.

[0151] In the above embodiment, it has been introduced that before adding a task to the task list, it is necessary to determine whether the data corresponding to the task has arrived. If the data corresponding to the task has arrived, the task will be added to the task list, otherwise, the task will not be processed. However, in actual applications, there may be the following scenario, that is, after the data corresponding to the task has arrived, the user may manually set the task to be re-executed and obtain new data for the task. In order to make the data obtained when the task is executed the latest data, in this embodiment, a thread is created for the task to be run, and computing resources are allocated to the thread to run the task to be run, including:

[0152] h1. Determine whether external data on which the task to be run depends is received; external data is data obtained by running the task on which the task to be run depends;

[0153] h2. If external data that the task to be run depends on is received, the task to be run is allocated to the computing device to run the task to be run.

[0154] This embodiment can further reduce the situation where tasks run in vain by determining whether external data on which the task to be run depends is received when it is determined that the computing resources required for the task to be run is less than or equal to the remaining resources of the computing device, and by creating a thread for the task to be run and allocating computing resources to the thread to run the task to be run when the external data on which the task to be run depends is received, thereby further reducing the situation where tasks run in vain.

[0155] The following is a detailed introduction to the execution process of tasks in parallel task queues, failed task queues, and serial task queues through some specific examples:

[0156] Fig. 9 This is an example diagram of a task scheduling method in a parallel task queue provided in an embodiment of the present application. Fig. 9 As shown, the task scheduling method includes the following steps:

[0157] S901: Determine whether the end of the parallel task queue has been reached.

[0158] Among them, S901 is intended to determine whether there are available tasks to be run in the parallel task queue.

[0159] S902: If the end of the parallel task queue has been reached, the process ends.

[0160] If the end of the parallel task queue has been reached, it means that there are no available tasks to be run in the parallel task queue.

[0161] S903: If the task has not reached the end of the parallel task queue, the task to be executed is obtained from the parallel task queue.

[0162] If the end of the parallel task queue is not reached, it means that there are available tasks to be run in the parallel task queue.

[0163] S904: Determine whether the computing resources required for the task to be executed are less than or equal to the remaining resources of the computing device.

[0164] S905. If the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device, a thread is created for the task to be executed, and the task to be executed is executed through the thread, and the process returns to step S901.

[0165] In this embodiment, after a thread is created for a task to be executed, the next task to be executed can be obtained from the parallel task queue without determining whether the task has been executed.

[0166] S906: After the task is executed, remove the task from the parallel task queue.

[0167] The task execution is completed, including: the task execution is successful, or the number of task failures reaches a preset number.

[0168] Fig.10 This is an example diagram of a task scheduling method in a serial task queue provided in an embodiment of the present application. Fig.10 As shown, the task scheduling method includes the following steps:

[0169] S1001, determining whether the serial task queue is an empty queue;

[0170] S1002. If the serial task queue is an empty queue, obtain the serial task from the task list and add it to the serial task queue, and obtain the task to be run from the serial task queue;

[0171] S1003, if the serial task queue is not an empty queue, determine whether the end of the serial task queue has been reached;

[0172] S1004, if the task has not reached the end of the serial task queue, then obtain the task to be run from the serial task queue;

[0173] S1005, if the end of the serial task queue has been reached, then end;

[0174] S1006: Determine whether the computing resources required for the task to be executed are less than or equal to the remaining resources of the computing device.

[0175] S1007: If the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device, a thread is created for the task to be executed, and the task to be executed is executed through the thread.

[0176] S1008, determining whether the task to be run is in an execution state;

[0177] S1009: If the task to be run is in the execution state, obtain the next task to be run from the serial task queue.

[0178] S1010, if the task to be run is not in the execution state, determining whether the task to be run is executed successfully;

[0179] Among them, not being in the execution state can be understood as being in the execution completion state.

[0180] S1011. If the task to be run is successfully executed, the task to be run is removed from the serial task queue.

[0181] S1012: If the task to be run fails, determine whether the maximum number of failures has been reached;

[0182] S1013: If the maximum number of failures is reached, the task to be run is removed from the serial task queue.

[0183] S1014: If the maximum number of failures has not been reached, create a thread for the task to be run, allocate computing resources to the thread to run the task to be run, and obtain the next task to be run from the serial task queue.

[0184] Fig.11 This is a schematic diagram of the structure of the task scheduling device provided in the embodiment of the present application. Fig.11 As shown, the task scheduling device includes: an acquisition module 1101, a judgment module 1102 and an allocation module 1103;

[0185] The acquisition module 1101 is used to acquire the task to be run and the configuration information of the task to be run; the configuration information of the task to be run includes the computing resources required to run the task to be run;

[0186] The judgment module 1102 is used to judge whether the computing resources required by the task to be run are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; if the computing resources required by the task to be run are greater than the remaining resources of the computing device, the judgment step is continued until the computing resources required by the task to be run are less than or equal to the remaining resources of the computing device;

[0187] The allocation module 1103 is used to create a thread for the task to be run, and allocate computing resources to the thread to run the task to be run.

[0188] In some embodiments, the acquisition module 1101 acquires the task to be executed, specifically including: acquiring the task to be executed from a task queue; the task queue includes a parallel task queue or a serial task queue.

[0189] In some embodiments, the configuration information of the task scheduling also includes the maximum number of tasks in the task queue; the device also includes: an adding module 1104; an acquiring module 1101, which is also used to acquire a target task; the target task is a task that has reached the task execution time; a judging module 1102, which is also used to judge whether the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue; if the number of all tasks to be run in the task queue is greater than or equal to the maximum number of tasks in the task queue, the judging step continues to be executed until the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue; an adding module 1104 is used to add the target task to the task queue.

[0190] In some embodiments, the adding module 1104 adds the target task to the task queue, specifically including: obtaining configuration information of the target task; the configuration information of the target task includes an identifier of the target task, and the identifier of the target task is used to indicate that the target task is added to a parallel task queue or a serial task queue; if the identifier of the target task indicates that the target task is added to a parallel task queue, then the target task is added to the parallel task queue; if the identifier of the target task indicates that the target task is added to a serial task queue, then the target task is added to the serial task queue.

[0191] In some embodiments, the device also includes: a determination module 1105 and a removal module 1106; the determination module 1105 is used to determine whether the task to be run has been successfully executed; the removal module 1106 is used to remove the task to be run from the task queue if the task to be run has been successfully executed; the adding module 1104 is also used to add the task to be run to the failed task queue if the task to be run fails and the number of failures reaches a preset number of failures.

[0192] In some embodiments, the configuration information of the task to be run also includes: the task to be run includes N subtasks, the execution order between the N subtasks, and the identifier of each subtask, and the identifier of each subtask is used to characterize whether the subtask is a serial task or a parallel task; N is an integer greater than or equal to 2, and the determination module 1105 is also used to determine the M target subtasks that need to be executed currently according to the execution order between the N subtasks; M is an integer greater than or equal to 1 and less than or equal to N; the adding module 1104 is also used to add the target subtask to the parallel task queue or the serial task queue according to the identifier of the target subtask; the determination module 1105 is also used to determine whether all the N subtasks are executed successfully; and, if all the N subtasks are executed successfully, it is determined that the task to be run is executed successfully.

[0193] In some embodiments, the device further includes: a stop operation module 1107; the acquisition module 1101 is also used to obtain configuration information of task processing; the configuration information of task processing includes a preset occupancy rate of computing resources of the computing device; the determination module 1105 is also used to determine the occupancy rate of computing resources of the computing device; the stop operation module 1107 is used to sort the X processes of the computing device in descending order of computing resource occupancy rate if the occupancy rate of computing resources of the computing device is greater than or equal to the preset occupancy rate of computing resources of the computing device, and stop the tasks to be run corresponding to the first Y processes; each process includes one or more tasks to be run.

[0194] In some embodiments, the adding module 1104 is further used to add the to-be-run tasks corresponding to the process occupying the most resources to the failed task queue.

[0195] In some embodiments, the device also includes: a generation module 1108 and a sending module 1109; the acquisition module 1101 is also used to obtain the task running results of multiple tasks to be run within a preset time period; the task running results include whether the tasks to be run are successfully run; the generation module 1108 is used to generate a task running list according to the task running results of multiple tasks to be run within the preset time period; the sending module 1109 is used to send the task running list to the terminal device of the target object; the target object is the object for arranging the tasks to be run.

[0196] In some embodiments, the task running result of the task to be run also includes: whether the task to be run is completed according to the agreed time; the determination module 1105 is also used to determine whether the actual arrival time of the data corresponding to the task to be run does not exceed the agreed time; the generation module 1108 is also used to generate an alarm message of delayed arrival of the data of the task to be run if the actual arrival time of the data corresponding to the task to be run does not exceed the agreed time; and, based on the alarm message of delayed arrival of the data of the task to be run, generate the task running result of the task to be run.

[0197] In some embodiments, the acquisition module 1101 acquires the configuration information of the task to be run, specifically including: acquiring the configuration information of the task to be run from a configuration file; wherein the configuration file configures the configuration information in a command line manner.

[0198] In some embodiments, the apparatus further includes: a receiving module 1110, for receiving a target operation for the configuration file input by a user through a terminal device; the target operation includes: at least one of viewing, adding, deleting and modifying; and an execution module 1111, for performing a corresponding operation on the configuration file according to the target operation.

[0199] In some embodiments, the allocation module 1103 creates a thread for the task to be run and allocates computing resources to the thread to run the task to be run, specifically including: determining whether external data on which the task to be run depends is received; the external data is data obtained by running the task on which the task to be run depends; if the external data on which the task to be run depends is received, the task to be run is allocated to the computing device to run the task to be run.

[0200] The task scheduling device provided in the embodiment of the present application can be used to execute the technical solution of the task scheduling method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0201] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the allocation module 1103 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above allocation module 1103. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0202] Fig.12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.12 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0203] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the scheme in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0204] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0205] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0206] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0207] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0208] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the task scheduling method in the above embodiment.

[0209] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the task scheduling method of the above embodiment.

[0210] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the task scheduling method in the above embodiment can be implemented.

[0211] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0212] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A task scheduling method, characterized in that: include: Acquire a task to be run and configuration information of the task to be run; the configuration information of the task to be run includes computing resources required to run the task to be run; Determining whether the computing resources required for the task to be executed are less than or equal to the remaining resources of a computing device used for task processing; If the computing resources required by the task to be executed are greater than the remaining resources of the computing device, continue to perform the judgment step until the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device; Creating a thread for the task to be run, and allocating computing resources to the thread to run the task to be run; The step of creating a thread for the task to be executed and allocating computing resources to the thread to execute the task to be executed includes: Determine whether external data on which the task to be run depends is received; the external data is data obtained by running the task on which the task to be run depends; If external data on which the task to be run depends is received, a thread is created for the task to be run, and computing resources are allocated to the thread to run the task to be run; The obtaining of the configuration information of the task to be run includes: Acquire the configuration information of the task to be run from a configuration file, wherein the configuration file is stored in a storage path of a hardware device of the system; wherein the configuration information is configured in the configuration file in a command line manner; The method further comprises: Acquire configuration information of task processing; the configuration information of task processing includes a preset occupancy rate of computing resources of the computing device; Determine the occupancy rate of computing resources by X processes of the computing device; If the occupancy rate of the computing resources by the X processes of the computing device is greater than or equal to the preset occupancy rate of the computing resources of the computing device, the X processes of the computing device are sorted in descending order of the occupancy rate of the computing resources, and the tasks to be run corresponding to the first Y processes are stopped, so that the occupancy rate of the computing resources by the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device; each process includes one or more tasks to be run; Add the stopped pending tasks to the failed task queue.

2. The method according to claim 1, characterized in that The step of obtaining the task to be executed includes: The to-be-run task is obtained from a task queue; the task queue includes a parallel task queue or a serial task queue.

3. The method according to claim 2, characterized in that The configuration information of the task scheduling also includes the maximum number of tasks in the task queue; before obtaining the tasks to be run, the method further includes: Obtain a target task; the target task is a task that reaches a task execution time; Determine whether the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue; If the number of all tasks to be run in the task queue is greater than or equal to the maximum number of tasks in the task queue, continue to perform the judging step until the number of all tasks to be run in the task queue is less than the maximum number of tasks in the task queue; Add the target task to the task queue.

4. The method according to claim 3, characterized in that The step of adding the target task to the task queue includes: Acquire configuration information of the target task; the configuration information of the target task includes an identifier of the target task, and the identifier of the target task is used to indicate that the target task is added to a parallel task queue or a serial task queue; If the identifier of the target task indicates that the target task is to be added to a parallel task queue, then the target task is added to the parallel task queue; If the identifier of the target task indicates that the target task is to be added to a serial task queue, the target task is added to the serial task queue.

5. The method according to claim 2, characterized in that: After creating a thread for the task to be run and allocating computing resources to the thread to run the task to be run, the method further includes: Determine whether the task to be executed is executed successfully; If the task to be run is successfully run, the task to be run is removed from the task queue; If the task to be run fails and the number of failures reaches a preset number of failures, the task to be run is added to a failed task queue.

6. The method according to claim 2, characterized in that The configuration information of the task to be run also includes: the task to be run includes N subtasks, the execution order between the N subtasks, and the identifier of each subtask, and the identifier of each subtask is used to indicate whether the subtask is a serial task or a parallel task; N is an integer greater than or equal to 2, and the method further includes: According to the execution order of the N subtasks, the M target subtasks to be executed are determined; the M is an integer greater than or equal to 1 and less than or equal to N; According to the identifier of the target subtask, the target subtask is added to the parallel task queue or the serial task queue; Determine whether all the N subtasks are successfully executed; If all of the N subtasks are executed successfully, it is determined that the task to be executed is executed successfully.

7. The method according to any one of claims 1 to 6, characterized in that: After creating a thread for the task to be run and allocating computing resources to the task to be run to run the task to be run, the method further includes: Obtaining task execution results of multiple tasks to be executed within a preset time period; the task execution results include whether the tasks to be executed are executed successfully; Generate a task running list according to the task running results of the multiple tasks to be run within the preset time period; The task execution list is sent to a terminal device of a target object; the target object is an object for which the task to be executed is arranged.

8. The method according to claim 7, characterized in that The task execution result of the task to be executed also includes: whether the task to be executed is completed according to the agreed time; the method also includes: Determine whether the actual arrival time of the data corresponding to the task to be executed does not exceed the agreed time; If the actual arrival time of the data corresponding to the task to be run does not exceed the agreed time, an alarm message indicating that the data of the task to be run has arrived late is generated; The task running result of the task to be run is generated according to the alarm information of delayed arrival of the data of the task to be run.

9. The method according to claim 1, characterized in that: The method further comprises: Receiving a target operation for the configuration file input by a user through a terminal device; the target operation includes: at least one of viewing, adding, deleting and modifying; According to the target operation, a corresponding operation is performed on the configuration file.

10. A task scheduling device, characterized in that: include: An acquisition module, used to acquire a task to be run and configuration information of the task to be run; the configuration information of the task to be run includes computing resources required to run the task to be run; A judgment module, used to judge whether the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device; the computing device is used to perform task processing; If the computing resources required by the task to be executed are greater than the remaining resources of the computing device, continue to perform the judgment step until the computing resources required by the task to be executed are less than or equal to the remaining resources of the computing device; An allocation module, used for creating a thread for the task to be run, and allocating computing resources to the thread to run the task to be run; The allocation module is specifically used to: determine whether external data on which the task to be run depends is received; the external data is data obtained by running the task on which the task to be run depends; if the external data on which the task to be run depends is received, allocate the task to be run to the computing device to run the task to be run; The acquisition module is specifically used to acquire the configuration information of the task to be run from a configuration file, and the configuration file is stored in a storage path of a hardware device of the system; wherein the configuration information is configured in the configuration file in a command line manner; The acquisition module is further used to acquire configuration information of task processing; the configuration information of task processing includes a preset occupancy rate of computing resources of the computing device; A determination module, used to determine the occupancy rate of computing resources by X processes of the computing device; a stop operation module, configured to sort the X processes of the computing device in descending order of computing resource occupancy rate, and stop the tasks to be run corresponding to the first Y processes, if the occupancy rate of the computing resources by the X processes of the computing device is greater than or equal to the preset occupancy rate of the computing resources of the computing device, so that the occupancy rate of the computing resources by the X processes of the computing device is less than the preset occupancy rate of the computing resources of the computing device; each process includes one or more tasks to be run; Add a module to add stopped pending tasks to the failed task queue.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images