Task scheduling method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose a task scheduling method, apparatus, computer equipment and storage medium. The scheme determines batch tasks to be processed, splits the batch tasks according to the data attributes of the task data in the batch tasks, and saves the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory; queries the data directory through a preset main processor, and when there is a task directory to be processed in the data directory, obtains the task processing status of at least one sub-processor; based on the task processing status of at least one sub-processor, determines the target sub-processor corresponding to the task directory to be processed according to the preset task scheduling logic; assigns the subtasks to be processed under the task directory to be processed to the target sub-processor, so that the target sub-processor processes the subtasks to be processed, and sends the subtasks to be processed after the task processing is completed to the downstream module, thereby improving the processing efficiency when multiple tasks are processed.

Description

Technical Field

[0001] The present application relates to the technical field of task scheduling, and in particular to a task scheduling method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of society, when multiple process canvases are running at the same time, since there are multiple sub-process tasks in one process task, and the data processed by different sub-process tasks and the time for processing the data are different, when processing multiple tasks, the processor can usually only be arranged to process the tasks in a specific order. However, the scale of different tasks and the processing capacity of the processor are different, which leads to low processing efficiency when processing multiple tasks. Summary of the Invention

[0003] The embodiments of the present application provide a task scheduling method, apparatus, computer equipment, and storage medium, which can improve the processing efficiency when processing multiple tasks.

[0004] The present invention provides a method for scheduling tasks, including:

[0005] Determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks;

[0006] Splitting the batch tasks according to the data attributes of the task data in the batch tasks, and saving the resulting subtasks into a task directory according to the data attributes to obtain a data directory, wherein each subtask in the task directory in the data directory satisfies a preset matching condition in terms of the data attributes;

[0007] querying the data directory through a preset main processor, and obtaining a task processing status of at least one secondary processor when a task directory to be processed exists in the data directory;

[0008] Based on the task processing status of at least one of the above-mentioned sub-processors, determining a target sub-processor corresponding to the above-mentioned task directory to be processed according to a preset task scheduling logic;

[0009] The to-be-processed subtasks under the to-be-processed task directory are allocated to the target sub-processor, so that the target sub-processor processes the to-be-processed subtasks and sends the to-be-processed subtasks after task processing to the downstream module.

[0010] Accordingly, an embodiment of the present application further provides a task scheduling device, including:

[0011] A task determination module is used to determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks;

[0012] A task splitting module is used to split the batch tasks according to the data attributes of the task data in the batch tasks, and save the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory, and each subtask under the task directory in the data directory meets the preset matching conditions on the data attributes;

[0013] a status acquisition module, configured to query the data directory through a preset main processor, and obtain the task processing status of at least one secondary processor when a task directory to be processed exists in the data directory;

[0014] a processor determining module, configured to determine, based on a task processing state of at least one of the sub-processors and in accordance with a preset task scheduling logic, a target sub-processor corresponding to the task directory to be processed;

[0015] The task assignment module is used to assign the pending subtasks under the pending task directory to the target subprocessor so that the target subprocessor processes the pending subtasks and sends the pending subtasks to the downstream module after the task processing is completed.

[0016] In some embodiments, the task scheduling apparatus further includes:

[0017] A record adding module is used to add a target task record corresponding to the task directory to be processed in the preset task status monitoring table when there is a task directory to be processed in the data directory;

[0018] The signal updating module is used to determine the task status of the above-mentioned to-be-processed subtask according to the heartbeat signal updated by the above-mentioned target sub-processor at the target task record in the above-mentioned task status monitoring table.

[0019] In some embodiments, the task scheduling apparatus further includes:

[0020] A status determination module is configured to determine that the task status of the subtask to be processed is a processing failure if the heartbeat signal at the target task record is not updated within a preset time;

[0021] The state acquisition module is further used to reacquire the task processing state of at least one sub-processor, and based on the reacquired task processing state of at least one sub-processor, determine the sub-processor that processes the above-mentioned sub-task to be processed according to the preset task scheduling logic.

[0022] In some embodiments, the processor determination module includes:

[0023] A quantity determination unit, configured to determine the number of tasks to be processed by the current task according to the heartbeat signal corresponding to each task record in the task status monitoring table;

[0024] The processor determining unit is used to determine the target sub-processor corresponding to the above-mentioned task directory to be processed according to the preset task scheduling logic based on the task processing status of at least one of the above-mentioned sub-processors when the above-mentioned task quantity is less than the preset quantity threshold.

[0025] In some embodiments, the task splitting module includes:

[0026] A data storage unit is used to save the task data corresponding to the subtasks obtained by splitting into a directory folder of the task directory corresponding to the above subtasks;

[0027] The above-mentioned task allocation module includes:

[0028] The data acquisition unit is used to assign the pending subtasks under the above-mentioned pending task directory to the above-mentioned target sub-processor, so that the above-mentioned target sub-processor obtains the task data corresponding to the above-mentioned pending subtasks from the directory folder corresponding to the above-mentioned pending task directory, and performs task processing on the task data of the above-mentioned pending subtasks.

[0029] In some embodiments, the data attribute includes a time attribute, and the task splitting module further includes:

[0030] The task splitting unit is used to split the task corresponding to the task data with consistent time attributes into a subtask according to the time attributes of the task data in the batch tasks.

[0031] In some embodiments, the task scheduling apparatus further includes:

[0032] The time acquisition module is used to obtain the registration time of each registered processor;

[0033] The election module is used to select the processor with the earliest registration time as the main processor based on the registration time of each processor;

[0034] The processor determination module is further configured to determine processors other than the main processor among the registered processors as secondary processors.

[0035] Correspondingly, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes any task scheduling method provided in any embodiment of the present application.

[0036] Correspondingly, an embodiment of the present application further provides a storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the above task scheduling method.

[0037] An embodiment of the present application determines batch tasks to be processed and obtains data attributes of task data of the above batch tasks; splits the above batch tasks according to the data attributes of the task data in the above batch tasks, and saves the subtasks obtained by splitting into a task directory according to the above data attributes to obtain a data directory, and each subtask under the task directory in the above data directory meets the preset matching conditions on the above data attributes; queries the above data directory through a preset main processor, and when there is a task directory to be processed in the above data directory, obtains the task processing status of at least one sub-processor; based on the task processing status of at least one of the above sub-processors, determines the target sub-processor corresponding to the task directory to be processed according to the preset task scheduling logic; assigns the sub-tasks to be processed under the above task directory to be processed to the above target sub-processor, so that the above target sub-processor performs task processing on the sub-tasks to be processed, and sends the sub-tasks to be processed after task processing is completed to the downstream module, thereby improving the processing efficiency when multiple tasks are processed by task splitting and scheduling the sub-processors corresponding to the tasks through task scheduling logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present application.

[0040] Figure 2 A schematic diagram of a task state machine change provided in an embodiment of the present application.

[0041] Figure 3 This is a structural block diagram of a task scheduling device provided in an embodiment of the present application.

[0042] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of this application.

[0044] The embodiments of the present application provide a task scheduling method, apparatus, storage medium and computer equipment. Specifically, the task scheduling method of the embodiments of the present application can be executed by a computer device, wherein the computer device can be a server, or a terminal or other device. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a desktop computer, a laptop computer, a tablet computer, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0045] For example, the computer device may be a terminal, which may determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks; split the batch tasks according to the data attributes of the task data in the batch tasks, and save the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory, and each subtask under the task directory in the data directory meets the preset matching conditions on the data attributes; query the data directory through a preset main processor, and when there is a task directory to be processed in the data directory, obtain the task processing status of at least one sub-processor; based on the task processing status of at least one sub-processor, determine the target sub-processor corresponding to the task directory to be processed according to the preset task scheduling logic; assign the subtasks to be processed under the task directory to be processed to the target sub-processor, so that the target sub-processor performs task processing on the subtasks to be processed, and sends the subtasks to be processed after task processing is completed to the downstream module.

[0046] Based on the above problems, embodiments of the present application provide a task scheduling method, apparatus, computer device, and storage medium, which can improve the processing efficiency when processing multiple tasks.

[0047] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0048] An embodiment of the present application provides a task scheduling method, which can be executed by a terminal or a server. The embodiment of the present application takes the task scheduling method executed by a terminal as an example for explanation.

[0049] See also Figure 1 , Figure 1 This is a flow chart of a task scheduling method provided in an embodiment of the present application. The specific process of the task scheduling method can be as follows:

[0050] 101. Determine the batch tasks to be processed, and obtain data attributes of the task data of the batch tasks.

[0051] Among them, the above-mentioned batch tasks to be processed are tasks corresponding to multiple currently running process canvases. Since a process canvas has different sub-processes, the sub-process is a task, and different sub-processes correspond to different execution data processing times. Therefore, the terminal processes the batch tasks accordingly by obtaining the data attributes of the task data corresponding to each task in the batch task.

[0052] The data attributes include, but are not limited to, the process canvas where the task is located, the policy granularity required to execute the task, the channel for obtaining task data, the type of task processing result delivery, the canvas type of the process canvas where the task is located, and time attributes. The time attributes include the data processing time of the task data, which is equivalent to the time when the task processing result is delivered.

[0053] 102. Split the batch tasks according to the data attributes of the task data in the batch tasks, and save the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory, and each subtask under the task directory in the data directory meets the preset matching conditions in the data attributes.

[0054] In this embodiment, the terminal can improve the flexibility of task processing by splitting the above-mentioned batch tasks, facilitate concurrent processing of the above-mentioned batch tasks, and reduce the latency of task processing. In this embodiment, the terminal splits the batch tasks by at least one data attribute corresponding to the task data of each task in the above-mentioned batch tasks, which not only improves the efficiency of task splitting, but also reduces resource usage during task processing, thereby improving the overall timeliness. For example, the attribute values ​​of the data attributes of the task data of each task in the batch task are split into a subtask, and the number of tasks in the subtask is at least one.

[0055] Specifically, the data attribute includes a time attribute, and the splitting of the batch tasks based on the data attribute of the task data in the batch tasks may include: splitting the tasks corresponding to the task data with consistent time attributes into subtasks based on the time attribute of the task data in the batch tasks. Whether the task data are consistent may be determined based on the attribute value of the time attribute.

[0056] For example, assume that the current batch task corresponds to two process canvases, namely process canvas A and process canvas B, and there are two sub-processes in process canvas A and process canvas B, and the data processing time of the two sub-process task data in process canvas A is 3 days and 5 days respectively, and the data processing time of the two sub-process task data in process canvas B is 7 days and 5 days respectively, then the tasks corresponding to the task data with consistent time attributes are split into one subtask, that is, the sub-process with a data processing time of 5 days for the task data in process canvas A and the sub-process with a data processing time of 5 days for the task data in process canvas B are split into one subtask.

[0057] In this embodiment, after splitting the above-mentioned batch tasks, the terminal saves the subtasks obtained by splitting into a task directory according to the data attributes used in the splitting to obtain a data directory. The subtasks under each task directory in the above-mentioned data directory meet the preset matching conditions on the above-mentioned data attributes, thereby realizing monitoring of the subtasks obtained by splitting through the data directory, and also obtaining the task data of the subtasks based on the data directory. For example, if the data attribute of a task directory is set to the data processing time of the task data, and the attribute value of the data attribute is 7 days, then the data processing time of the task data of the subtask saved in the task directory is 7 days.

[0058] In some embodiments, when splitting the batch tasks based on the data attributes of the task data in the batch tasks, the terminal may split the batch tasks at the finest level based on the data attributes of the task data. Furthermore, when splitting the batch tasks, the terminal may also determine at least one data attribute required for splitting the batch tasks based on business requirements.

[0059] In some embodiments, saving the split subtasks into a task directory according to the data attributes to obtain a data directory may include: saving the task data corresponding to the split subtasks into a directory folder of the task directory corresponding to the subtasks.

[0060] Specifically, after a batch task is split, the task data corresponding to the resulting subtasks can be stored in the distributed file system HDFS based on the split results. That is, the task data corresponding to the resulting subtasks is stored in a directory folder corresponding to the task directory in the HDFS data directory based on the data attributes used during the split. This allows data to be retrieved from the directory folder corresponding to the subtask when needed. Files on HDFS can be split into multiple corresponding directory folders according to the task directory to improve concurrency during concurrent processing.

[0061] In some embodiments, after the task data corresponding to the split subtasks are saved in a specific task directory in the data directory, a corresponding key attribute index may be set to facilitate accurate search for tasks through the key attribute index.

[0062] 103. Query the data directory through a preset main processor, and when there is a task directory to be processed in the data directory, obtain the task processing status of at least one secondary processor.

[0063] In this embodiment, in order to achieve high availability, the terminal can set up a task scheduling module for deploying multiple services, such as task query service and task state machine management service. The terminal can perform task query service through the main processor in the task scheduling module, that is, query the above-mentioned data directory to obtain tasks that are not currently being processed, that is, the task directory existing in the data directory is the task directory to be processed, and the subtasks corresponding to the task directory to be processed are the tasks to be processed, so that the terminal can determine the task directory corresponding to the task that currently needs to be processed through the main processor. In addition, the terminal obtains the task processing status of at least one sub-processor in the task scheduling module to facilitate the scheduling of tasks corresponding to the task directory to be processed. Among them, the above-mentioned task scheduling module is constructed through a stateful java service. For the sake of high availability, the terminal can start two or more scheduling modules at the same time.

[0064] The main processor is primarily used to execute task scheduling logic, i.e., perform task query services and assign the queried tasks to the secondary processors, as well as to manage the task state machine, i.e., manage the status of task processing. The secondary processor is primarily used to perform task processing, thereby implementing at least one operation such as adding, deleting, checking, and modifying the relevant status data of each task through the task scheduling module. The main processor can periodically perform task query services through polling to determine whether there is a task directory to be processed by scanning the data directory.

[0065] The task processing status of the secondary processor is used to indicate the load of the secondary processor, and the task processing status includes but is not limited to the number of currently unprocessed tasks, the progress of currently processed tasks, the number of failed tasks, etc.

[0066] In some embodiments, before querying the above-mentioned data directory through a preset main processor, it may also include: the terminal can use the distributed lock zookeeper or redis to implement the master election service in the task scheduling module to elect the leader role from each processor, that is, the above-mentioned main processor. For example, the terminal implements the master election function through the temporary node of zookeeper.

[0067] In some embodiments, the terminal can determine the main processor through the registration time of each processor, that is, before querying the above-mentioned data directory through the preset main processor, it can also include: the terminal obtains the registration time of each registered processor, and selects the processor with the earliest registration time as the main processor based on the registration time of each processor, and determines the processors other than the above-mentioned main processor among the registered processors as secondary processors.

[0068] 104. Based on the task processing status of at least one of the sub-processors, determine a target sub-processor for processing the to-be-processed task directory according to a preset task scheduling logic.

[0069] In this embodiment, the terminal uses the main processor to schedule tasks corresponding to the task directory to be processed according to the preset task scheduling logic to determine the target sub-processor corresponding to the above-mentioned task directory to be processed, thereby realizing flexible task allocation and reasonable configuration of processor resources, thereby reducing resource occupancy and improving the timeliness of task processing.

[0070] The task scheduling logic can be determined based on at least one parameter of the number of tasks under the task directory to be processed, the user scale corresponding to the tasks under the task directory to be processed, and the load condition. The load condition can be the load condition of the sub-processor. Thus, by comprehensively considering the conditions of the task directory to be processed and the processing capacity of the sub-processor, batch tasks can be processed, for example, multiple tasks to be processed for multiple different tenants.

[0071] 105. Allocate the pending subtasks in the pending task directory to the target subprocessor, so that the target subprocessor processes the pending subtasks and sends the pending subtasks after task processing to the downstream module.

[0072] In this embodiment, after the terminal determines the target sub-processor, the terminal can realize task processing of the pending sub-tasks under the pending task directory by assigning the pending sub-tasks under the pending task directory to the target sub-processor, and after the target sub-processor completes the processing of the pending sub-tasks, the pending sub-tasks that have been processed are sent to the downstream module, that is, the processing results of the pending sub-tasks that have been processed are sent to the downstream module, thereby finally realizing the task processing of the pending sub-tasks and the logic of sending the processing results.

[0073] Specifically, the sub-processor can implement task processing for the tasks assigned to the sub-processor through Flink's batch processing mode. That is, in this mode, the sub-processor reads the task data of each task and performs task processing on the task data of each task respectively.

[0074] In some embodiments, to facilitate monitoring, the secondary processor in the task scheduling module may periodically update the task progress while processing a task, so that the primary processor in the task scheduling module can determine whether the current task is being processed. This allows the primary processor to promptly detect and reschedule any failed task when task processing fails. The secondary processor may update the task progress by updating a heartbeat signal.

[0075] In some embodiments, if the task data corresponding to the split subtasks is saved in the directory folder of the task directory corresponding to the above subtasks, the above-mentioned to-be-processed subtasks under the above-mentioned to-be-processed task directory are assigned to the above-mentioned target sub-processor, so that the above-mentioned target sub-processor obtains the corresponding task data from the directory folder when performing task processing on the above-mentioned to-be-processed subtasks, thereby improving the acquisition speed of task data of each task.

[0076] Specifically, it may include: the terminal assigns the to-be-processed subtasks under the above-mentioned to-be-processed task directory to the above-mentioned target sub-processor, so that the above-mentioned target sub-processor obtains the task data corresponding to the above-mentioned to-be-processed subtasks from the directory folder corresponding to the above-mentioned to-be-processed task directory, and performs task processing on the task data of the above-mentioned to-be-processed subtasks.

[0077] In some embodiments, in order to improve the control ability of the main processor over the task, after querying the above-mentioned data directory through the preset main processor, it can also include: when there is a pending task directory in the above-mentioned data directory, the terminal adds the target task record corresponding to the above-mentioned pending task directory in the preset task status monitoring table, so that the terminal can then assign the pending subtasks under the pending task directory to the target sub-processor. The target sub-processor can regularly update specific information at the target task record, and then determine the state machine changes of the task based on the specific information updated by the target sub-processor at the target task record to realize the monitoring of the task.

[0078] The task status monitoring table may be a MySQL data table.

[0079] Correspondingly, after allocating the pending subtasks under the above-mentioned pending task directory to the above-mentioned target sub-processor, it can also include: determining the task status of the above-mentioned pending subtasks based on the heartbeat signal updated by the above-mentioned target sub-processor at the target task record in the above-mentioned task status monitoring table, which is the above-mentioned specific information, and the task status of the above-mentioned pending subtasks can be saved through the MySQL data table.

[0080] It is understandable that each time the terminal schedules a task, it will perform at least one operation such as adding, deleting, modifying, or checking the task record in the task status monitoring table.

[0081] Among them, the above-mentioned task status includes but is not limited to normal task processing status, task processing failure status, task timeout status, processing completion status, etc., that is, if the target task record updates the heartbeat signal within a certain period of time, it means that the task corresponding to the target task record is being processed, and the task status of the task is not in the normal processing state; if the target task record does not update the heartbeat signal within a certain period of time, it means that the target task record has lost the heartbeat, and the task that has lost the heartbeat needs to be rescheduled to realize the cleaning of expired data; if the task processing is completed, the task status of the task is updated to the processing completion status.

[0082] For example, Figure 2 As shown, after the task is started for processing, if the task updates the heartbeat signal regularly, the task status of the task is "Running"; if the task processing is completed, the task status of the task changes from "Running" to "Finished", indicating that the task processing is completed; if the heartbeat signal of the task is not updated within a certain period of time, the task status of the task changes from "Running" to "Default", indicating that the task processing has failed; further, if the sub-processor is again assigned to process the task whose heartbeat signal has not been updated within a certain period of time, the task status of the task changes from "Default" to "Running", indicating that the task is re-processed.

[0083] In some embodiments, after assigning the pending subtasks under the pending task directory to the target sub-processor, the method may further include: if a heartbeat signal detected at the target task record is not updated within a preset time, determining that the task status of the pending subtask is a processing failure. The terminal needs to reacquire the task processing status of at least one sub-processor, and based on the reacquired task processing status of at least one sub-processor, finally determine the sub-processor to process the pending subtask according to a preset task scheduling logic.

[0084] In some embodiments, the above-mentioned determination of the target sub-processor corresponding to the above-mentioned task directory to be processed based on the task processing status of at least one of the above-mentioned sub-processors according to the preset task scheduling logic may include: the terminal determines the number of tasks to be processed by the current task based on the heartbeat signal corresponding to each task record in the above-mentioned task status monitoring table. In order to prevent the cluster resources from being exhausted, if the above-mentioned number of tasks is equal to the preset number threshold, task allocation will not be performed, that is, the target sub-processor corresponding to the above-mentioned task directory to be processed will not be determined according to the preset task scheduling logic. When the above-mentioned number of tasks is less than the preset number threshold, the terminal determines the target sub-processor corresponding to the above-mentioned task directory to be processed according to the preset task scheduling logic based on the task processing status of at least one of the above-mentioned sub-processors.

[0085] An embodiment of the present application discloses a task scheduling method, which includes: determining batch tasks to be processed, obtaining data attributes of task data of the above batch tasks; splitting the above batch tasks according to the data attributes of the task data in the above batch tasks, and saving the subtasks obtained by splitting into a task directory according to the above data attributes to obtain a data directory, and each subtask under the task directory in the above data directory meets a preset matching condition on the above data attributes; querying the above data directory through a preset main processor, and when there is a task directory to be processed in the above data directory, obtaining the task processing status of at least one sub-processor; based on the task processing status of at least one sub-processor, determining the target sub-processor corresponding to the task directory to be processed according to the preset task scheduling logic; allocating the subtasks to be processed under the task directory to be processed to the above target sub-processor, so that the above target sub-processor performs task processing on the subtasks to be processed, and sending the subtasks to be processed after task processing is completed to the downstream module, thereby improving the processing efficiency when multiple tasks are processed.

[0086] To facilitate better implementation of the task scheduling method provided in the embodiment of the present application, the embodiment of the present application also provides a task scheduling device based on the above task scheduling method. The meanings of the terms are the same as those in the above task scheduling method, and the specific implementation details can be referred to the description in the method embodiment.

[0087] See also Figure 3 , Figure 3 This is a structural block diagram of a task scheduling device provided in an embodiment of the present application, the device comprising:

[0088] The task determination module 301 is used to determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks;

[0089] The task splitting module 302 is configured to split the batch task according to the data attributes of the task data in the batch task, and save the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory, wherein each subtask under the task directory in the data directory meets a preset matching condition in terms of the data attributes;

[0090] A status acquisition module 303 is configured to query the data directory through a preset main processor, and when there is a task directory to be processed in the data directory, acquire the task processing status of at least one secondary processor;

[0091] The processor determining module 304 is configured to determine a target sub-processor for processing the task list based on a task processing state of at least one sub-processor and a preset task scheduling logic;

[0092] The task assignment module 305 is used to assign the pending subtasks in the pending task directory to the target subprocessor, so that the target subprocessor processes the pending subtasks and sends the completed pending subtasks to the downstream module.

[0093] In some embodiments, the task scheduling apparatus further includes:

[0094] A record adding module is used to add a target task record corresponding to the task directory to be processed in the preset task status monitoring table when there is a task directory to be processed in the data directory;

[0095] The signal updating module is used to determine the task status of the above-mentioned to-be-processed subtask according to the heartbeat signal updated by the above-mentioned target sub-processor at the target task record in the above-mentioned task status monitoring table.

[0096] In some embodiments, the task scheduling apparatus further includes:

[0097] A status determination module is configured to determine that the task status of the subtask to be processed is a processing failure if the heartbeat signal at the target task record is not updated within a preset time;

[0098] The state acquisition module is further used to reacquire the task processing state of at least one sub-processor, and based on the reacquired task processing state of at least one sub-processor, determine the sub-processor that processes the above-mentioned sub-task to be processed according to the preset task scheduling logic.

[0099] In some embodiments, the processor determination module 304 includes:

[0100] A quantity determination unit, configured to determine the number of tasks to be processed by the current task according to the heartbeat signal corresponding to each task record in the task status monitoring table;

[0101] The processor determining unit is used to determine the target sub-processor corresponding to the above-mentioned task directory to be processed according to the preset task scheduling logic based on the task processing status of at least one of the above-mentioned sub-processors when the above-mentioned task quantity is less than the preset quantity threshold.

[0102] In some embodiments, the task splitting module 302 includes:

[0103] A data storage unit is used to save the task data corresponding to the subtasks obtained by splitting into a directory folder of the task directory corresponding to the above subtasks;

[0104] The task allocation module 305 includes:

[0105] The data acquisition unit is used to assign the pending subtasks under the above-mentioned pending task directory to the above-mentioned target sub-processor, so that the above-mentioned target sub-processor obtains the task data corresponding to the above-mentioned pending subtasks from the directory folder corresponding to the above-mentioned pending task directory, and performs task processing on the task data of the above-mentioned pending subtasks.

[0106] In some embodiments, the data attribute includes a time attribute, and the task splitting module 302 further includes:

[0107] The task splitting unit is used to split the task corresponding to the task data with consistent time attributes into a subtask according to the time attributes of the task data in the batch tasks.

[0108] In some embodiments, the task scheduling apparatus further includes:

[0109] The time acquisition module is used to obtain the registration time of each registered processor;

[0110] The election module is used to select the processor with the earliest registration time as the main processor based on the registration time of each processor;

[0111] The processor determination module is further configured to determine processors other than the main processor among the registered processors as secondary processors.

[0112] The embodiment of the present application discloses a task scheduling device, which includes a task determination module 301 for determining a batch of tasks to be processed and obtaining data attributes of the task data of the batch tasks; a task splitting module 302 for splitting the batch tasks according to the data attributes of the task data in the batch tasks, and saving the subtasks obtained by splitting into a task directory according to the data attributes to obtain a data directory, wherein each subtask under the task directory in the data directory meets a preset matching condition on the data attributes; a status acquisition module 303 for querying the data directory through a preset main processor, and when there is a task directory to be processed in the data directory, obtaining the task processing status of at least one sub-processor; a processor determination module 304 for determining a target sub-processor corresponding to the task directory to be processed according to a preset task scheduling logic based on the task processing status of at least one sub-processor; and a task allocation module 305 for allocating the subtasks to be processed under the task directory to be processed to the target sub-processor, so that the target sub-processor performs task processing on the subtasks to be processed and sends the subtasks to be processed after the task processing is completed to the downstream module. In this way, the processing efficiency of multiple tasks is improved.

[0113] Accordingly, the embodiment of the present application also provides a computer device, which may be a terminal. Figure 4 As shown, Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 400 includes a processor 401 having one or more processing cores, a memory 402 having one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 is electrically connected to the memory 402. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation of the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0114] The processor 401 is the control center of the computer device 400. It uses various interfaces and lines to connect the various parts of the entire computer device 400. By running or loading software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the computer device 400 and processes data, thereby monitoring the computer device 400 as a whole.

[0115] In the embodiment of the present application, the processor 401 in the computer device 400 loads instructions corresponding to one or more application processes into the memory 402 according to the following steps, and the processor 401 runs the application stored in the memory 402 to implement various functions:

[0116] Determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks;

[0117] Splitting the batch tasks according to the data attributes of the task data in the batch tasks, and saving the resulting subtasks into a task directory according to the data attributes to obtain a data directory, wherein each subtask in the task directory in the data directory satisfies a preset matching condition in terms of the data attributes;

[0118] querying the data directory through a preset main processor, and obtaining a task processing status of at least one secondary processor when a task directory to be processed exists in the data directory;

[0119] Based on the task processing status of at least one of the above-mentioned sub-processors, determining a target sub-processor corresponding to the above-mentioned to-be-processed task directory according to a preset task scheduling logic;

[0120] The to-be-processed subtasks under the to-be-processed task directory are allocated to the target sub-processor, so that the target sub-processor processes the to-be-processed subtasks and sends the to-be-processed subtasks after task processing to the downstream module.

[0121] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0122] Optional, such as Figure 4 As shown, the computer device 400 further includes: a touch screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. Among them, the processor 401 is electrically connected to the touch screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407 respectively. Those skilled in the art will understand that Figure 4 The computer device structure shown in the figure does not constitute a limitation to the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0123] The touch display screen 403 can be used for displaying a graphical user interface and receiving the operation instructions that the user acts on the graphical user interface. The touch display screen 403 can include a display panel and a touch panel. Wherein, the display panel can be used for displaying the message input by the user or the message provided to the user and the various graphical user interfaces of the computer device, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured using forms such as a liquid crystal display (LCD), an organic light emitting diode (OLED), and an organic light-emitting diode (OLED). The touch panel can be used for collecting the touch operation of the user thereon or near it (such as the user uses any suitable object or attachment such as a finger, a stylus on the touch panel or near the touch panel), and generates corresponding operation instructions, and the operation instruction executes corresponding program. Optionally, the touch panel can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch message from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 401, and can receive the command sent by the processor 401 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 401 to determine the type of touch event, and then the processor 401 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 403 to realize input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize input and output functions. That is, the touch display screen 403 can also be used as part of the input unit 406 to realize the input function.

[0124] The radio frequency circuit 404 may be used to transmit and receive radio frequency signals, so as to establish wireless communication with a network device or other computer device through wireless communication, and to transmit and receive signals between the network device or other computer device.

[0125] Audio circuit 405 can be used to provide an audio interface between the user and the computer device through a speaker and microphone. Audio circuit 405 can convert received audio data into electrical signals and transmit them to the speaker, which then converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 405 and converted into audio data. The audio data is then output to processor 401 for processing, then transmitted via RF circuit 404 to, for example, another computer device, or to memory 402 for further processing. Audio circuit 405 may also include an earphone jack to allow communication between external headphones and the computer device.

[0126] The input unit 406 may be configured to receive input digital, character messages, or user feature messages (such as fingerprint, iris, facial messages, etc.), and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls.

[0127] Power supply 407 is used to supply power to various components of computer device 400. Optionally, power supply 407 can be logically connected to processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 407 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0128] although Figure 4 Not shown in the figure, the computer device 400 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] As can be seen from the above, the computer device provided in this embodiment obtains the data attributes of the task data of the batch tasks by determining the batch tasks to be processed; splits the batch tasks according to the data attributes of the task data in the batch tasks, and saves the subtasks obtained by splitting into the task directory according to the above data attributes to obtain a data directory, and each subtask under the task directory in the data directory meets the preset matching conditions on the above data attributes; queries the data directory through the preset main processor, and obtains the task processing status of at least one sub-processor when there is a task directory to be processed in the data directory; based on the task processing status of at least one sub-processor, determines the target sub-processor corresponding to the task directory to be processed according to the preset task scheduling logic; assigns the subtasks to be processed under the task directory to be processed to the target sub-processor, so that the target sub-processor performs task processing on the subtasks to be processed, and sends the subtasks to be processed after task processing is completed to the downstream module.

[0131] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0132] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute the steps of any of the task scheduling methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:

[0133] Determine the batch tasks to be processed and obtain the data attributes of the task data of the batch tasks;

[0134] Splitting the batch tasks according to the data attributes of the task data in the batch tasks, and saving the resulting subtasks into a task directory according to the data attributes to obtain a data directory, wherein each subtask in the task directory in the data directory satisfies a preset matching condition in terms of the data attributes;

[0135] querying the data directory through a preset main processor, and obtaining a task processing status of at least one secondary processor when a task directory to be processed exists in the data directory;

[0136] Based on the task processing status of at least one of the above-mentioned sub-processors, determining a target sub-processor corresponding to the above-mentioned to-be-processed task directory according to a preset task scheduling logic;

[0137] The to-be-processed subtasks under the to-be-processed task directory are allocated to the target sub-processor, so that the target sub-processor processes the to-be-processed subtasks and sends the to-be-processed subtasks after task processing to the downstream module.

[0138] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0139] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0140] Since the computer program stored in the storage medium can execute the steps in any task scheduling method provided in the embodiments of the present application, the beneficial effects that can be achieved by any task scheduling method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0141] The above is a detailed introduction to a task scheduling method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A task scheduling method, characterized in that: The method comprises: Determine a batch task to be processed, and obtain data attributes of task data of the batch task, wherein the data attributes include a time attribute; According to the time attribute of the task data in the batch task, the task corresponding to the task data with the same time attribute is split into a subtask, and the subtasks obtained by splitting are saved in the task directory according to the data attribute to obtain a data directory, and each subtask under the task directory in the data directory meets the preset matching condition on the data attribute; querying the data directory through a preset main processor, and when there is a pending task directory in the data directory, obtaining the task processing status of at least one secondary processor, and adding a target task record corresponding to the pending task directory to a preset task status monitoring table; Based on the task processing state of at least one of the secondary processors, determining a target secondary processor corresponding to the to-be-processed task directory according to a preset task scheduling logic; Allocating the pending subtasks under the pending task directory to the target subprocessor, so that the target subprocessor processes the pending subtasks, and sending the pending subtasks after task processing to the downstream module; determining the task status of the to-be-processed subtask according to a heartbeat signal updated by the target subprocessor at the target task record in the task status monitoring table; If the heartbeat signal at the target task record is not updated within the preset time, the task status of the pending subtask is determined to be processing failure; The task processing state of at least one secondary processor is reacquired, and based on the reacquired task processing state of the at least one secondary processor, a secondary processor that processes the to-be-processed subtask is determined according to a preset task scheduling logic.

2. The method according to claim 1, characterized in that The determining, based on the task processing state of at least one of the secondary processors and according to a preset task scheduling logic, a target secondary processor corresponding to the to-be-processed task directory comprises: Determine the number of tasks to be processed by the current task based on the heartbeat signal corresponding to each task record in the task status monitoring table; When the number of tasks is less than a preset number threshold, a target secondary processor for processing the to-be-processed task directory is determined according to a preset task scheduling logic based on a task processing state of at least one secondary processor.

3. The method according to claim 1, characterized in that The step of saving the split subtasks into a task directory according to the data attributes to obtain a data directory includes: Save the task data corresponding to the split subtasks into the directory folder of the task directory corresponding to the subtasks; The allocating the to-be-processed subtasks under the to-be-processed task directory to the target subprocessor so that the target subprocessor processes the to-be-processed subtasks includes: The to-be-processed subtasks under the to-be-processed task directory are allocated to the target subprocessor, so that the target subprocessor obtains task data corresponding to the to-be-processed subtasks from the directory folder corresponding to the to-be-processed task directory, and performs task processing on the task data of the to-be-processed subtasks.

4. The method according to any one of claims 1 to 3, characterized in that Before querying the data directory through a preset main processor, the method further includes: Get the registration time of each registered processor; According to the registration time of each processor, the processor with the earliest registration time is selected as the main processor; Processors other than the main processor among the registered processors are determined as sub-processors.

5. A task scheduling device, characterized in that: The device comprises: A task determination module, configured to determine a batch of tasks to be processed and obtain data attributes of task data of the batch tasks, wherein the data attributes include a time attribute; A task splitting module is used to split the task corresponding to the task data with the same time attribute into a subtask according to the time attribute of the task data in the batch task, and save the subtasks obtained by splitting into a task directory according to the data attribute to obtain a data directory, wherein each subtask under the task directory in the data directory meets the preset matching condition on the data attribute; a status acquisition module, configured to query the data directory through a preset main processor, and when a task directory to be processed exists in the data directory, acquire a task processing status of at least one secondary processor; A record adding module is used to add a target task record corresponding to the task directory to be processed in a preset task status monitoring table when there is a task directory to be processed in the data directory; a processor determining module, configured to determine, based on a task processing state of at least one of the sub-processors and according to a preset task scheduling logic, a target sub-processor corresponding to the to-be-processed task directory; a task assignment module, configured to assign the pending subtasks under the pending task directory to the target subprocessor, so that the target subprocessor processes the pending subtasks, and sends the pending subtasks after task processing to the downstream module; a signal updating module, configured to determine the task status of the to-be-processed subtask according to a heartbeat signal updated by the target subprocessor at the target task record in the task status monitoring table; A status determination module is configured to determine that the task status of the to-be-processed subtask is a processing failure if it is detected that the heartbeat signal at the target task record has not been updated within a preset time; The state acquisition module is further used to reacquire the task processing state of at least one sub-processor, and based on the reacquired task processing state of at least one sub-processor, determine the sub-processor that processes the to-be-processed sub-task according to a preset task scheduling logic.

6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the task scheduling method according to any one of claims 1 to 4 is implemented.

7. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the task scheduling method according to any one of claims 1 to 4.

Images