Job scheduling method and system based on cloud platform

Abstract

The embodiment of the invention discloses a job scheduling method and system based on a cloud platform, and relates to the technical field of batch job scheduling. The method comprises the steps that a to-be-processed task is received through a receiver, a task file of the to-be-processed task is stored in an information storage of a storage unit, and a preprocessing task corresponding to the to-be-processed task is established in a preprocessing queue of the storage unit; if a distributed lock for a preprocessing stage is preempted in a lock memory of a storage unit through a preemptor of a preprocessing unit, determining a target preprocessing task in a preprocessing queue and sending the target preprocessing task to a divider of the preprocessing unit; a target task file in an information memory is segmented through a divider of a preprocessing unit, obtained sub-task files are placed in the information memory, and processing tasks corresponding to the sub-task files are established in a processing queue of a storage unit; and processing the target processing task in the processing queue of the storage unit through a processing unit.

Description

Technical Field

[0001] The present invention relates to the technical field of batch job scheduling, and in particular to a job scheduling method and system based on a cloud platform. Background Art

[0002] When performing batch job scheduling, traditional distributed batch job scheduling methods support cluster deployment and distributed task scheduling. Most of these methods are based on database-based task registration and discovery. When the cluster is deployed in the scheduling center, it is necessary to ensure that the cluster machine clocks are consistent, and distributed scheduling of jobs must be implemented based on hard coding such as server IDs.

[0003] The existing batch job scheduling framework relies on special hardware configurations such as hard coding, and some mainstream cloud platform products do not have fixed hardware coding, so there is a problem that the batch job scheduling framework is difficult to directly go to the cloud. Summary of the invention

[0004] The present invention provides a job scheduling method and system based on a cloud platform. When performing job scheduling, it is not associated with hardware configurations such as IP and machine hard coding, and can be deployed in a cloud environment and virtual machines under the cloud, thereby improving the applicability to the cloud.

[0005] According to one aspect of the present invention, a job scheduling method based on a cloud platform is provided, which is executed by a job scheduling system, wherein the job scheduling system is configured in a node of the cloud platform, and the job scheduling system comprises: a receiver, a preprocessing unit, a processing unit, and a storage unit; the method comprises:

[0006] Receiving the task to be processed through the receiver, storing the task file of the task to be processed in the information storage unit, and establishing a pre-processing task corresponding to the task to be processed in the pre-processing queue of the storage unit;

[0007] If the distributed lock for the preprocessing stage is preempted in the lock memory of the storage unit by the preprocessing unit preemptor, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to the splitter of the preprocessing unit;

[0008] The target task file in the information storage device is segmented by a segmenter of the preprocessing unit, the subtask files obtained by segmentation are placed in the information storage device, and a processing task corresponding to the subtask file is established in the processing queue of the storage unit; the target task file is a task file corresponding to the target preprocessing task;

[0009] The target processing task in the processing queue of the storage unit is processed by the processing unit.

[0010] According to another aspect of the present invention, there is provided a job scheduling system based on a cloud platform, the job scheduling system being configured in a node of the cloud platform, the job scheduling system comprising: a receiver, a preprocessing unit, a processing unit and a storage unit;

[0011] The receiver is used to receive the task to be processed, store the task file of the task to be processed in the information storage unit, and establish a pre-processing task corresponding to the task to be processed in the pre-processing queue of the storage unit;

[0012] The preemptor of the preprocessing unit is used to preempt the distributed lock for the preprocessing stage in the lock memory of the storage unit, and is also used to determine the target preprocessing task in the preprocessing queue and send the target preprocessing task to the splitter of the preprocessing unit;

[0013] The splitter of the preprocessing unit is used to split the target task file located in the information storage device, place the subtask files obtained by the splitting in the information storage device, and establish a processing task corresponding to the subtask file in the processing queue of the storage unit; the target task file is the task file corresponding to the target preprocessing task;

[0014] The processing unit is used to process the target processing tasks in the processing queue of the storage unit.

[0015] According to the technical solution of the embodiment of the present application, the job scheduling method is executed by a job scheduling system, and the job scheduling system is configured in a node of a cloud platform. The job scheduling system includes: a receiver, a preprocessing unit, a processing unit and a storage unit; the method includes: receiving a task to be processed through a receiver, storing the task file of the task to be processed in an information storage device of a storage unit, and establishing a preprocessing task corresponding to the task to be processed in a preprocessing queue of the storage unit; if a distributed lock for the preprocessing stage is preempted in a lock storage device of a storage unit through a preprocessing unit preemptor, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to a splitter of the preprocessing unit; the target task file located in the information storage device is split by the splitter of the preprocessing unit, the subtask files obtained by the split are placed in the information storage device, and a processing task corresponding to the subtask file is established in the processing queue of the storage unit; the target task file is a task file corresponding to the target preprocessing task; the target processing task in the processing queue of the storage unit is processed by the processing unit. When performing job scheduling, this technical solution is not related to hardware configurations such as IP and machine hard coding, and can be deployed in cloud environments and virtual machines under the cloud, thereby improving its applicability to the cloud.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a flow chart of a job scheduling method based on a cloud platform provided according to Embodiment 1 of the present application;

[0019] Figure 2 is a flow chart of a job scheduling method based on a cloud platform provided according to Embodiment 2 of the present application;

[0020] Figure 3 is a flowchart of an exemplary receiver provided according to Embodiment 2 of the present application;

[0021] Figure 4 is a workflow diagram of an exemplary preprocessing unit provided according to Embodiment 2 of the present application;

[0022] Figure 5 is a workflow diagram of an exemplary processing unit provided according to Embodiment 2 of the present application;

[0023] Figure 6 is a workflow diagram of a preemptor of an exemplary preprocessing unit provided according to Embodiment 2 of the present application;

[0024] Figure 7 is a workflow diagram of a state guardian of an exemplary pre-processing unit provided according to the second embodiment of the present application;

[0025] Figure 8 It is a structural diagram of a job scheduling system based on a cloud platform provided according to Embodiment 3 of the present application;

[0026] Fig. 9 It is a structural diagram of another job scheduling system provided according to Example 3 of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 A flowchart of a job scheduling method based on a cloud platform is provided for the first embodiment of the present application. The embodiment of the present application can be applied to the situation of scheduling batch job tasks, which is executed by a job scheduling system. The job scheduling system is configured in a node of the cloud platform. The job scheduling system includes: a receiver, a preprocessing unit, a processing unit and a storage unit.

[0031] Specifically, the job scheduling system can be configured in each node of the cloud platform, and each node can execute the method. In the job scheduling system, the receiver is used to accept batch job tasks and store the task files of the batch job tasks in the specified path of the information storage device of the storage unit. The preprocessing unit includes: a preemptor, a splitter and a state guardian; the preemptor is used to preempt the distributed lock for the preprocessing stage; the splitter is used to split the task file; the processing unit includes: a preemptor, a processor and a state guardian; the preemptor is used to preempt the distributed lock for the task processing stage; the processor is used to process the subtask file; it should be noted that the storage unit includes: an information storage device, a lock storage device, a preprocessing queue and a processing queue. The information storage device is used to store the task file and the split subtask file; the lock storage device is used to place the distributed lock information; the preprocessing queue is used to place the preprocessing task; the processing queue is used to place the processing task. The information in the storage unit can be changed and read by any node of the cloud platform configured with the job scheduling system.

[0032] like Figure 1 As shown, the method includes:

[0033] S110, receiving a task to be processed through a receiver, storing a task file of the task to be processed in an information storage device of a storage unit, and establishing a preprocessing task corresponding to the task to be processed in a preprocessing queue of the storage unit.

[0034] Among them, the tasks to be processed may be batch job tasks, that is, the amount of tasks to be processed is relatively large. In the embodiment of the present application, batch job tasks are split and then processed.

[0035] Exemplarily, the receiver is deployed in a cluster, and the receiver receives tasks to be processed through an API, and stores the task files of the tasks to be processed in an information storage device of a storage unit through the receiver, and establishes a preprocessing task corresponding to the tasks to be processed in a preprocessing queue of the storage unit through the receiver, and sets the timestamp, job status, job progress and other information of the preprocessing task.

[0036] S120, if the distributed lock for the preprocessing stage is preempted in the lock memory of the storage unit by the preemptor of the preprocessing unit, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to the splitter of the preprocessing unit.

[0037] Among them, there are two types of distributed locks in the embodiment of the present application, one is a distributed lock for the preprocessing stage, and the other is a distributed lock for the task processing stage, which are different. For the distributed lock in the preprocessing stage, each node can preempt the distributed lock through the preemptor, and at the same time, only one node can preempt the distributed lock to avoid the situation where multiple nodes preprocess the same preprocessing task.

[0038] Specifically, if the distributed lock for the preprocessing stage is preempted in the lock memory of the storage unit by the preprocessing unit's preemptor, the target preprocessing task is determined in the preprocessing queue based on the relevant information of each preprocessing task (such as timestamp, job status, etc.); and then the target preprocessing task is sent to the splitter of the preprocessing unit so that the splitter splits the target preprocessing task.

[0039] S130, segmenting the target task file in the information storage device through the segmenter of the preprocessing unit, placing the segmented subtask files in the information storage device, and establishing a processing task corresponding to the subtask file in the processing queue of the storage unit.

[0040] Among them, the target task file is the task file corresponding to the target preprocessing task.

[0041] Specifically, a file path is determined in the relevant information of the target preprocessing task in the preprocessing queue through the splitter of the preprocessing unit, and a target task file is determined in the information storage device of the storage unit based on the file path; the target task file is split and processed to obtain multiple subtask files, and for each subtask file, a processing task corresponding to the subtask file is established in the processing queue of the storage unit.

[0042] S140: Processing the target processing task in the processing queue of the storage unit by the processing unit.

[0043] Specifically, the processing unit determines the target processing task in the processing queue of the storage unit, and determines the subtask file corresponding to the target processing task in the information storage device, and then processes the target processing task.

[0044] According to the technical solution of the embodiment of the present application, the job scheduling method is executed by a job scheduling system, and the job scheduling system is configured in a node of a cloud platform. The job scheduling system includes: a receiver, a preprocessing unit, a processing unit and a storage unit; the method includes: receiving a task to be processed through a receiver, storing the task file of the task to be processed in an information storage device of a storage unit, and establishing a preprocessing task corresponding to the task to be processed in a preprocessing queue of the storage unit; if a distributed lock for the preprocessing stage is preempted in a lock storage device of a storage unit through a preprocessing unit preemptor, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to a splitter of the preprocessing unit; the target task file located in the information storage device is split by the splitter of the preprocessing unit, the subtask files obtained by the split are placed in the information storage device, and a processing task corresponding to the subtask file is established in the processing queue of the storage unit; the target task file is a task file corresponding to the target preprocessing task; the target processing task in the processing queue of the storage unit is processed by the processing unit. When performing job scheduling, this technical solution is not related to hardware configurations such as IP and machine hard coding, and can be deployed in cloud environments and virtual machines under the cloud, thereby improving its applicability to the cloud.

[0045] Embodiment 2

[0046] Figure 2 A flowchart of a cloud platform-based job scheduling method provided in Example 2 of the present application, which is optimized based on the above-mentioned example.

[0047] like Figure 2 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0048] S210, receiving the task to be processed through the receiver, and storing the task file of the task to be processed in the information storage unit.

[0049] S220: Determine, by means of a receiver, a receiving time of the task to be processed.

[0050] S230, establishing a preprocessing task corresponding to the to-be-processed task in the preprocessing queue of the storage unit through the receiver, setting the job status of the preprocessing task to unfinished, and setting the timestamp of the preprocessing task to the receiving time.

[0051] Specifically, a preprocessing task corresponding to the task to be processed is established in the preprocessing queue of the storage unit through the receiver. Since the preprocessing task has not been preprocessed yet, the job status of the preprocessing task is set to incomplete, the timestamp of the preprocessing task is set to the receiving time, and the segmentation progress of the preprocessing task is set to 0.

[0052] S240: If the distributed lock for the preprocessing stage is preempted in the lock memory of the storage unit by the preemptor of the preprocessing unit, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to the splitter of the preprocessing unit.

[0053] Specifically, sending the target preprocessing task to the splitter of the preprocessing unit may be: sending the associated information such as the file path and name of the target preprocessing task to the splitter of the preprocessing unit.

[0054] In an embodiment of the present application, optionally, a target preprocessing task is determined in a preprocessing queue through a preemptor of a preprocessing unit, including: a preprocessing task in the preprocessing queue whose timestamp is before the current time, whose job status is incomplete and which is at the head of the queue is determined as a target preprocessing task through the preemptor of the preprocessing unit.

[0055] Exemplarily, in the preprocessing queue, there may be preprocessing tasks whose timestamps are before the current time, the current time, and after the current time. The embodiment of the present application determines the preprocessing task whose job status is incomplete, whose timestamp is before the current time, and whose timestamp is the earliest (that is, at the head of the queue) as the target preprocessing task. The target preprocessing task determined in this way is a task that has never been preprocessed; or it is a task that has been preprocessed by a node but the preprocessing has not been completed.

[0056] S250, changing the timestamp of the target preprocessing task in the preprocessing queue to a time after the current time is extended by a first preset time period through the preemptor of the preprocessing unit, and releasing the distributed lock for the preprocessing stage.

[0057] Specifically, if the target preprocessing task has been sent to the splitter of the preprocessing unit through the preprocessing unit's preemptor, the target preprocessing task is already in the splitting process. In this case, the timestamp of the target preprocessing task in the preprocessing queue is changed to a time after the current time is extended by a first preset time length (for example, 30 minutes), and then the distributed lock is released. Even if other nodes preempt the distributed lock for the preprocessing stage, since the timestamp of the task is later than the current time, it will not be repeatedly acquired by other nodes.

[0058] In this solution, each node can only be seized by one node at the same time. After a node seizes the distributed lock for the preprocessing stage, it determines the target preprocessing task in the preprocessing list and changes the timestamp of the target preprocessing task, thereby releasing the distributed lock for the preprocessing stage. After other nodes seize the distributed lock for the preprocessing stage again, they will determine a new target preprocessing task in the preprocessing list again, so that each preprocessing task can be preprocessed in sequence.

[0059] S260, sending the target preprocessing task to the state daemon of the preprocessing unit through the preemptor of the preprocessing unit.

[0060] Specifically, while the target preprocessing task is sent to the splitter of the preprocessing unit through the preemptor of the preprocessing unit, the target preprocessing task is also sent to the state guardian of the preprocessing unit through the preemptor of the preprocessing unit, so that when the target preprocessing task is in the preprocessing process, the timestamp of the state guardian is always later than the current time, that is, the task will not be acquired by other nodes.

[0061] S270, adjusting the timestamp of the target preprocessing task in the preprocessing queue through the status guardian of the preprocessing unit so that the time of the timestamp is after the current time, until receiving a message from the splitter of the preprocessing unit that the target task file has been split.

[0062] If the status guardian of the preprocessing unit receives a target preprocessing task sent by the preemptor of the preprocessing unit, the timestamp of the target preprocessing task in the preprocessing queue is adjusted so that the time of the timestamp is after the current time; if a message is received from the splitter of the preprocessing unit that the target task file has been split, the timestamp of the target preprocessing task in the preprocessing queue is stopped.

[0063] In an embodiment of the present application, optionally, the timestamp of the target preprocessing task in the preprocessing queue is adjusted by the status guard of the preprocessing unit, including: every time the second preset time period passes, the status guard of the preprocessing unit determines whether the time difference between the timestamp of the target preprocessing task in the preprocessing queue and the current time is less than a third preset time period; if less than, the timestamp of the target preprocessing task in the preprocessing queue is changed to a time after the current time is extended by the first preset time period; the first preset time period is greater than the third preset time period; the third preset time period is greater than the second preset time period.

[0064] Exemplarily, the second preset time length is 1 minute, and the third preset time length is 20 minutes, that is, every 1 minute, the status guardian of the preprocessing unit determines whether the time difference between the timestamp of the target preprocessing task in the preprocessing queue and the current time is less than 20 minutes; if less, the timestamp of the target preprocessing task in the preprocessing queue is changed to a time 30 minutes later than the current time.

[0065] This scheme is set up in such a way that during the preprocessing process (i.e., during the segmentation process) of the target preprocessing task, since the timestamp is always later than the current time, even if a node seizes the distributed lock for the preprocessing stage, it will not be acquired by other nodes. In addition, if the current node is down, after waiting for the first preset time, the timestamp time will be earlier than the new current time, and the preprocessing task can still be acquired and preprocessed by other nodes.

[0066] S280, segmenting the target task file in the information storage device through the segmenter of the preprocessing unit, placing the segmented subtask files in the information storage device, and establishing a processing task corresponding to the subtask file in the processing queue of the storage unit.

[0067] In an embodiment of the present application, optionally, the target task file located in the information storage device is segmented by a segmenter of a preprocessing unit, and the subtask files obtained by the segmentation are placed in the information storage device, including: segmenting the target task file located in the information storage device according to a target segmentation method by a segmenter of the preprocessing unit to obtain multiple subtask files, and placing the multiple subtask files obtained by the segmentation in the information storage device; synchronizing the segmentation progress of the target task file in the preprocessing queue of the storage unit by the segmenter of the preprocessing unit.

[0068] Exemplarily, when the target task file located in the information storage device is segmented according to the target segmentation method by the segmenter of the preprocessing unit, it can be: according to the specified splitting method (such as fixed size segmentation, segmentation by province and city, segmentation by the last digit of the mobile phone number, etc.) and preprocessing method (such as screening out illegal messages and discarding them, judging whether it is a file to be processed at a specified time according to the content of the file header, if it is a file to be processed at a specified time, it will not be processed, the timestamp will be changed to the specified processing time, and it will be put back into the preprocessing queue and the task will be ended).

[0069] In an embodiment of the present application, optionally, a processing task corresponding to the subtask file is established in the processing queue of the storage unit through the splitter of the preprocessing unit, including: establishing a processing task corresponding to the subtask file in the processing queue of the storage unit through the splitter of the preprocessing unit, and setting the job status of the processing task to incomplete; if it is determined through the splitter of the preprocessing unit that the subtask file has a specified processing time, then setting the timestamp of the processing task corresponding to the subtask file in the processing queue of the storage unit to the specified processing time; otherwise, setting the timestamp of the processing task corresponding to the subtask file according to the associated information of the subtask file.

[0070] Exemplarily, when setting the timestamp of the processing task corresponding to the subtask file according to the associated information of the subtask file, the time when the segmentation of the subtask file is completed may be set as the timestamp of the processing task corresponding to the subtask file.

[0071] In the embodiment of the present application, an exemplary workflow diagram of a receiver can be referred to Figure 3 An exemplary workflow diagram of the pre-processing unit can be found in Figure 4 An exemplary processing unit workflow diagram can be found at Figure 5 An exemplary workflow diagram of the preemptor of the preprocessing unit can be found in Figure 6 An exemplary workflow diagram of the state keeper of the pre-processing unit can be found in Figure 7 .

[0072] S290, processing the target processing task in the processing queue of the storage unit through the processing unit.

[0073] In an embodiment of the present application, optionally, a target processing task in a processing queue of the storage unit is processed by a processing unit, including: preempting a distributed lock for a task processing stage in a lock memory of the storage unit by a preemptor of the processing unit; if a distributed lock for a task processing stage is preempted by a preemptor of the processing unit, a processing task in the processing queue whose timestamp is before the current time, whose job status is incomplete and which is at the head of the queue is determined as a target processing task, and the target processing task is sent to a processor in the processing unit; and a subtask file corresponding to the target processing task located in an information memory is processed by the processor of the processing unit.

[0074] Specifically, similar to the pre-processing unit, only one node can successfully seize the distributed lock of the task processing stage at the same time. If the preemptor of a processing unit seizes the distributed lock of the task processing stage, the processing task with a timestamp before the current time, an unfinished job status and at the head of the queue in the processing queue is determined as the target processing task. Then, the processor of the processing unit processes the subtask file corresponding to the target processing task in the information storage.

[0075] In an embodiment of the present application, optionally, after a processing task in the processing queue whose timestamp is before the current time, whose job status is incomplete and which is at the head of the queue is determined as a target processing task by a preemptor of the processing unit, the method further includes: sending the target processing task to a status guardian in the processing unit by the preemptor of the processing unit; adjusting the timestamp of the target processing task by the status guardian of the processing unit so that the time of the timestamp is after the current time, until a message is received from the processor of the processing unit that the target processing task has been processed.

[0076] Exemplarily, every time the second preset time period has passed, the status guardian of the processing unit determines whether the time difference between the timestamp of the target processing task in the processing queue and the current time is less than a third preset time period; if it is, the timestamp of the target processing task in the processing queue is changed to a time after the current time is extended by the first preset time period; the first preset time period is greater than the third preset time period; the third preset time period is greater than the second preset time period.

[0077] Exemplarily, the second preset time length is 1 minute, and the third preset time length is 20 minutes, that is, every 1 minute, the status guardian of the processing unit determines whether the time difference between the timestamp of the target processing task in the processing queue and the current time is less than 20 minutes; if less, the timestamp of the target processing task in the processing queue is changed to a time 30 minutes later than the current time.

[0078] This scheme is set up in such a way that when the target processing task is being processed (i.e., being processed by the processor), since the timestamp is always later than the current time, even if a node seizes the distributed lock for the processing stage, it will not be acquired by other nodes. In addition, if the current node is down, after waiting for the first preset time, the timestamp time will be earlier than the new current time, and the processing task can still be acquired and processed by other nodes.

[0079] In an embodiment of the present application, optionally, when the subtask file corresponding to the target processing task located in the information storage device is processed by the processor of the processing unit, the method also includes: synchronizing the processing progress of the target processing task in the processing queue of the storage unit by the processor of the processing unit.

[0080] Specifically, the embodiment of the present application synchronizes the processing progress of the target processing task in the processing queue of the storage unit through the processor of the processing unit. If a node crashes, since the processing progress of the target processing task is placed in the processing queue, after the target processing task is acquired by other nodes, it can be processed again based on the processing progress, thereby avoiding the problem of repeated processing and saving processing time.

[0081] The technical scheme of the embodiment of the present application can support cluster deployment and task scheduling on the cloud. All processing modules can be expanded and enhanced at any time. The job scheduling system can be deployed in the newly added node, and then the job scheduling method described in the embodiment of the present application is realized, and the reliability is strong. The technical scheme of the embodiment of the present application is not related to IP, machine hard coding, etc., and both cloud environment and virtual machine under the cloud can be deployed, and the applicability is wide. In the embodiment of the present application, all server resources of the receiver, pre-processing unit, processing unit and storage unit are effectively utilized, and redundant idle disaster recovery resources do not need to be reserved, which is relatively more resource-saving. In the embodiment of the present application, a single exception of each unit node can be automatically restored (i.e., if a pre-processing task or processing task is interrupted, as time goes by, the timestamp of the interrupted task will be earlier than the timestamp of other tasks, and can be obtained and processed again by other nodes), without manual intervention, and long jobs can be split into short jobs and processed by multiple nodes, and resource utilization is high.

[0082] Embodiment 3

[0083] Figure 8 This is a schematic diagram of the structure of a cloud platform-based job scheduling system provided in the third embodiment of the present application. The job scheduling system is configured in a node of the cloud platform. The system can execute the cloud platform-based job scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 8 As shown, the system includes: a receiver 310, a pre-processing unit 320, a processing unit 330 and a storage unit 340;

[0084] The receiver 310 is used to receive the task to be processed, store the task file of the task to be processed in the information storage of the storage unit 340, and establish a pre-processing task corresponding to the task to be processed in the pre-processing queue of the storage unit 340;

[0085] The preemptor of the preprocessing unit 320 is used to preempt the distributed lock for the preprocessing stage in the lock memory of the storage unit 340, and is also used to determine the target preprocessing task in the preprocessing queue and send the target preprocessing task to the splitter of the preprocessing unit;

[0086] The splitter of the preprocessing unit 320 is used to split the target task file in the information storage, place the subtask files obtained by the splitting in the information storage, and establish a processing task corresponding to the subtask file in the processing queue of the storage unit 340; the target task file is the task file corresponding to the target preprocessing task;

[0087] The processing unit 330 is used to process the target processing task in the processing queue of the storage unit 340 .

[0088] Fig. 9 A structural schematic diagram of another job scheduling system provided in an embodiment of the present application, wherein a receiver is used to receive job tasks; a preemptor of a preprocessing unit is used to preempt a distributed lock for a preprocessing stage; the splitter is used to split the task file; a state guardian of the preprocessing unit is used to guard the timestamp of a target preprocessing task; the processing unit comprises: a preemptor, a processor and a state guardian; the preemptor of the processing unit is used to preempt a distributed lock for a task processing stage; the processor is used to process a subtask file; and the state guardian of the processing unit is used to guard the timestamp of a target processing task. The storage unit comprises: an information storage, a lock storage, a preprocessing queue and a processing queue. The information storage is used to store task files and split subtask files; the lock storage is used to place distributed lock information; the preprocessing queue is used to place preprocessing tasks; and the processing queue is used to place processing tasks. The information in the storage unit can be modified and read by any node of a cloud platform configured with a job scheduling system.

[0089] In the embodiment of the present application, optionally, the receiver is further used for:

[0090] Determining a receiving time of the task to be processed;

[0091] Establishing a preprocessing task corresponding to the to-be-processed task in the preprocessing queue of the storage unit, setting the job status of the preprocessing task to unfinished, and setting the timestamp of the preprocessing task to the receiving time;

[0092] Accordingly, the preemptor of the preprocessing unit is also used for:

[0093] Change the timestamp of the target preprocessing task in the preprocessing queue to a time after the current time is extended by a first preset time period, and release the distributed lock for the preprocessing stage;

[0094] Sending the target preprocessing task to a state daemon of a preprocessing unit;

[0095] Correspondingly, the status guardian of the preprocessing unit is used to adjust the timestamp of the target preprocessing task in the preprocessing queue so that the timestamp is after the current time until a message is received from the splitter of the preprocessing unit indicating that the target task file has been split.

[0096] In the embodiment of the present application, optionally, the preemptor of the preprocessing unit is further used to:

[0097] A preprocessing task in the preprocessing queue whose timestamp is before the current time, whose job status is unfinished and which is at the head of the queue is determined as a target preprocessing task.

[0098] In the embodiment of the present application, optionally, the state guardian of the pre-processing unit is further used to:

[0099] Every time the second preset time has passed, determining whether the time difference between the timestamp of the target preprocessing task in the preprocessing queue and the current time is less than a third preset time;

[0100] If it is less than, the timestamp of the target preprocessing task in the preprocessing queue is changed to the time after the current time is extended by the first preset time length; the first preset time length is greater than the third preset time length; the third preset time length is greater than the second preset time length.

[0101] In the embodiment of the present application, optionally, the segmenter of the pre-processing unit is further used for:

[0102] The target task file in the information storage device is divided into a plurality of subtask files according to the target division method, and the plurality of subtask files obtained by the division are placed in the information storage device;

[0103] The segmentation progress of the target task file is synchronized in the preprocessing queue of the storage unit.

[0104] In the embodiment of the present application, optionally, the segmenter of the pre-processing unit is further used for:

[0105] Creating a processing task corresponding to the subtask file in the processing queue of the storage unit, and setting the job status of the processing task to unfinished;

[0106] If it is determined that the subtask file has a specified processing time, setting the timestamp of the processing task corresponding to the subtask file in the processing queue of the storage unit to the specified processing time;

[0107] Otherwise, the timestamp of the processing task corresponding to the subtask file is set according to the associated information of the subtask file.

[0108] In the embodiment of the present application, optionally, the preemptor of the processing unit is further used to:

[0109] Seizing a distributed lock for a task processing stage in a lock memory of the storage unit;

[0110] If the distributed lock for the task processing stage is seized, the processing task whose timestamp is before the current time, whose job status is unfinished and located at the head of the queue in the processing queue is determined as the target processing task, and the target processing task is sent to the processor in the processing unit;

[0111] Correspondingly, the processor of the processing unit is also used to process the subtask file corresponding to the target processing task located in the information storage.

[0112] In the embodiment of the present application, optionally, the preemptor of the processing unit is further used to:

[0113] Sending the target processing task to a status daemon in the processing unit;

[0114] Correspondingly, the status guardian of the processing unit is also used to adjust the timestamp of the target processing task so that the time of the timestamp is after the current time, until a message is received from the processor of the processing unit indicating that the target processing task has been completed.

[0115] In an embodiment of the present application, optionally, the processor of the processing unit is also used to synchronize the processing progress of the target processing task in the processing queue of the storage unit.

[0116] A cloud platform-based job scheduling system provided in an embodiment of the present application can execute a cloud platform-based job scheduling method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A job scheduling method based on a cloud platform, characterized in that: The method is executed by a job scheduling system, which is configured in a node of a cloud platform. The job scheduling system includes: a receiver, a preprocessing unit, a processing unit, and a storage unit. The method includes: Receiving the task to be processed through the receiver, storing the task file of the task to be processed in the information storage unit, and establishing a pre-processing task corresponding to the task to be processed in the pre-processing queue of the storage unit; If the distributed lock for the preprocessing stage is preempted in the lock memory of the storage unit by the preprocessing unit preemptor, a target preprocessing task is determined in the preprocessing queue and the target preprocessing task is sent to the splitter of the preprocessing unit; The target task file in the information storage device is segmented by a segmenter of the preprocessing unit, the subtask files obtained by segmentation are placed in the information storage device, and a processing task corresponding to the subtask file is established in the processing queue of the storage unit; the target task file is a task file corresponding to the target preprocessing task; The target processing task in the processing queue of the storage unit is processed by the processing unit.

2. The method according to claim 1, characterized in that The receiver establishes a preprocessing task corresponding to the to-be-processed task in a preprocessing queue of a storage unit, including: Determining, by a receiver, a receiving time of the task to be processed; Establishing a preprocessing task corresponding to the to-be-processed task in a preprocessing queue of a storage unit through a receiver, setting a job status of the preprocessing task to unfinished, and setting a timestamp of the preprocessing task to a receiving time; Correspondingly, after the preemptor of the preprocessing unit determines the target preprocessing task in the preprocessing queue, the method further includes: The preemptor of the preprocessing unit changes the timestamp of the target preprocessing task in the preprocessing queue to a time after the current time is extended by a first preset time period, and releases the distributed lock for the preprocessing stage; Sending the target preprocessing task to the state daemon of the preprocessing unit through the preemptor of the preprocessing unit; The state daemon of the preprocessing unit adjusts the timestamp of the target preprocessing task in the preprocessing queue so that the timestamp is after the current time until a message is received from the splitter of the preprocessing unit indicating that the target task file has been split.

3. The method according to claim 2, characterized in that The target preprocessing task is determined in the preprocessing queue by the preprocessing unit preemptor, including: The preemptor of the preprocessing unit determines the preprocessing task in the preprocessing queue, whose timestamp is before the current time, whose job status is unfinished and which is at the head of the queue as the target preprocessing task.

4. The method according to claim 2, characterized in that: The timestamp of the target preprocessing task in the preprocessing queue is adjusted through the status daemon of the preprocessing unit, including: Every time the second preset time has passed, the state guardian of the preprocessing unit determines whether the time difference between the timestamp of the target preprocessing task in the preprocessing queue and the current time is less than a third preset time; If it is less than, the timestamp of the target preprocessing task in the preprocessing queue is changed to the time after the current time is extended by the first preset time length; the first preset time length is greater than the third preset time length; the third preset time length is greater than the second preset time length.

5. The method according to claim 1, characterized in that The target task file in the information storage device is segmented by a segmenter of the preprocessing unit, and the subtask files obtained by segmentation are placed in the information storage device, including: The target task file in the information storage device is segmented by a segmenter of a preprocessing unit according to a target segmentation method to obtain a plurality of subtask files, and the plurality of subtask files obtained by segmentation are placed in the information storage device; The segmentation progress of the target task file is synchronized in the preprocessing queue of the storage unit through the segmenter of the preprocessing unit.

6. The method according to claim 1, characterized in that The process of creating a processing task corresponding to the subtask file in the processing queue of the storage unit by the splitter of the preprocessing unit includes: Creating a processing task corresponding to the subtask file in the processing queue of the storage unit through the splitter of the preprocessing unit, and setting the job status of the processing task to unfinished; If it is determined by the splitter of the pre-processing unit that the sub-task file has a specified processing time, then in the processing queue of the storage unit, the timestamp of the processing task corresponding to the sub-task file is set to the specified processing time; Otherwise, the timestamp of the processing task corresponding to the subtask file is set according to the associated information of the subtask file.

7. The method according to claim 6, characterized in that Processing the target processing task in the processing queue of the storage unit by the processing unit includes: Preempting a distributed lock for a task processing stage in a lock memory of the storage unit by a preemptor of the processing unit; If the distributed lock for the task processing stage is preempted by the preemptor of the processing unit, the processing task whose timestamp is before the current time, whose job status is unfinished and located at the head of the queue in the processing queue is determined as the target processing task, and the target processing task is sent to the processor in the processing unit; The subtask file corresponding to the target processing task located in the information storage is processed by the processor of the processing unit.

8. The method according to claim 7, characterized in that After determining, by the preemptor of the processing unit, a processing task in the processing queue whose timestamp is before the current time, whose job status is unfinished and which is at the head of the queue as a target processing task, the method further comprises: Sending the target processing task to a state daemon in the processing unit through a preemptor of the processing unit; The timestamp of the target processing task is adjusted by the status daemon of the processing unit so that the time of the timestamp is after the current time until a message is received from the processor of the processing unit indicating that the processing of the target processing task has been completed.

9. The method according to claim 7, characterized in that: When the subtask file corresponding to the target processing task located in the information storage is processed by the processor of the processing unit, the method further includes: The processing progress of the target processing task is synchronized in the processing queue of the storage unit through the processor of the processing unit.

10. A job scheduling system based on a cloud platform, characterized in that: The job scheduling system is configured in a node of the cloud platform, and the job scheduling system includes: a receiver, a preprocessing unit, a processing unit, and a storage unit; The receiver is used to receive the task to be processed, store the task file of the task to be processed in the information storage unit, and establish a pre-processing task corresponding to the task to be processed in the pre-processing queue of the storage unit; The preemptor of the preprocessing unit is used to preempt the distributed lock for the preprocessing stage in the lock memory of the storage unit, and is also used to determine the target preprocessing task in the preprocessing queue and send the target preprocessing task to the splitter of the preprocessing unit; The splitter of the preprocessing unit is used to split the target task file located in the information storage device, place the subtask files obtained by the splitting in the information storage device, and establish a processing task corresponding to the subtask file in the processing queue of the storage unit; the target task file is the task file corresponding to the target preprocessing task; The processing unit is used to process the target processing tasks in the processing queue of the storage unit.

Images