Task scheduling method and device based on XML file, equipment and storage medium

Abstract

The invention provides a task scheduling method and device based on an XML file, equipment and a storage medium. Comprising the steps that a task scheduling platform is constructed, scheduling information of the task scheduling platform is stored through an XML file, and the task scheduling platform comprises serial group nodes and parallel group nodes; obtaining a to-be-processed task; determining relevance of the to-be-processed tasks; and allocating the associated to-be-processed tasks to the serial group nodes for sequential processing, and allocating the irrelevant to-be-processed tasks to the parallel group nodes for simultaneous execution.

Description

Technical Field

[0001] The present invention belongs to the field of task scheduling, and in particular relates to a task scheduling method and device, equipment and storage medium based on XML files. Background Art

[0002] Task scheduling refers to the process of automatically executing tasks based on a given time point, time interval, or number of executions. It involves the management of the allocation and execution order of computing tasks on computing resources. Its main goal is to improve the operating efficiency and performance of the system and effectively utilize computing resources by reasonably arranging the execution order of tasks.

[0003] With the gradual use of data warehouses, the basic scheduling platform of data warehouse systems faces many challenges. A large number of task scheduling can no longer be carried out well through timers (crontab), and the execution order of related tasks cannot be provided. It is often the case that ETL tasks may fail to execute due to data source problems, network failures or code errors, and the automatic retry mechanism and error alarm mechanism cannot be implemented. In addition, since some foreign scheduling platforms need to store scheduling information in the database, it brings great deployment pressure to scheduling. Summary of the invention

[0004] The present invention provides a task scheduling method and device, equipment and storage medium based on XML files, which can improve the timeliness of task scheduling.

[0005] On the one hand, a task scheduling method based on an XML file is provided, comprising: Build a task scheduling platform and store the scheduling information of the task scheduling platform through XML files. The task scheduling platform includes serial group nodes and parallel group nodes. Get pending tasks; Determine the relevance of pending tasks; Assign related pending tasks to serial group nodes for sequential processing, and assign unrelated pending tasks to parallel group nodes for simultaneous execution.

[0006] Optionally, when a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The parallel group node determines the processing order of the multiple tasks to be processed according to the order in which the multiple tasks to be processed arrive.

[0007] Optionally, when a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The parallel group node determines the processing order of the multiple tasks to be processed according to the task processing durations of the multiple tasks to be processed.

[0008] Optionally, when a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: According to the importance of the tasks to be processed, priorities are assigned to the tasks to be processed, and multiple tasks to be processed are processed in sequence according to the priorities.

[0009] Optionally, it also includes: The priorities of pending tasks are adjusted according to their historical execution periods.

[0010] Optionally, when a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: Build a queue of pending tasks. Each pending task in the queue gets CPU time in turn according to the time slice. After the time slice is used up, the pending task is put back to the end of the queue and waits for the next round of scheduling.

[0011] Optionally, when a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The execution order of the tasks to be processed is determined according to the response ratio of the tasks to be processed, where the response ratio is (waiting time of the tasks to be processed+required service time of the tasks to be processed) / required service time of the tasks to be processed.

[0012] On the other hand, a task scheduling device based on an XML file is provided, comprising: The platform construction module is used to construct a task scheduling platform and store the scheduling information of the task scheduling platform through an XML file. The task scheduling platform includes a serial group node and a parallel group node. The acquisition module is used to obtain tasks to be processed; A determination module, used to determine the relevance of tasks to be processed; The allocation module is used to allocate related pending tasks to serial group nodes for sequential processing, and allocate unrelated pending tasks to parallel group nodes for simultaneous execution.

[0013] On the other hand, an electronic device is provided, comprising the task scheduling device based on the XML file as described above.

[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is executed by a processor to implement the XML file-based task scheduling method as described in any one of the above items.

[0015] The technical solution provided by the present invention has the following beneficial effects: 1. The scheduling information is saved in XML files. The entire scheduling platform does not need to install a database. The application is simple to build and is very suitable for daily calls of small batch jobs. At the same time, the XML editing mode is provided to meet the structured thinking of developers. The entire tool greatly facilitates daily development work.

[0016] 2. A serial-parallel scheduling group is proposed, which treats related tasks as a serial group, and can complete the scheduling and execution of related tasks in an orderly manner. Multiple unrelated tasks or serial groups can be formed into a parallel group, which increases the concurrency of scheduling tasks and processes multiple unrelated tasks at the same time, thereby greatly improving the timeliness of scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a task scheduling method based on XML files provided by the present invention.

[0019] Figure 2 A schematic diagram of a task scheduling platform provided by the present invention.

[0020] Figure 3 This is an example diagram of an XML tag provided by the present invention.

[0021] Figure 4 This is a structural block diagram of a task scheduling device based on XML files provided by the present invention.

[0022] Figure 5 This is a structural block diagram of an electronic device provided by the present invention.

[0023] The reference numerals are as follows: 21: Platform building module; 22: Acquisition module; 23: Determination module; 24: Allocation module; 31: processor; 32: memory. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In response to the above problems, we have uniformly sorted out the ETL development task process, abstracted the tasks, used XML technology to store shared data in plain text format, and established a task scheduling platform based on XML technology. During development, the operator only needs to modify the XML task configuration file and configure the task information to be added through the graphical interface to achieve rapid development and deployment of scheduling tasks. It can realize automatic redo of task operation errors and operation log recording, and has complete core scheduling functions, which can realize serial, parallel, dependency, mutual exclusion, execution plan, timing, fault tolerance, loop, conditional branch, remote, load balancing, custom conditions and other functions.

[0026] Figure 1 This is a flow chart of a task scheduling method based on XML files provided by the present invention. Figure 1 ,include: S11. Build a task scheduling platform and store the scheduling information of the task scheduling platform through an XML file. The task scheduling platform includes a serial group node and a parallel group node.

[0027] Figure 2 A schematic diagram of a task scheduling platform provided by the present invention. Figure 2 , where group0 is a serial group, Group01 is a parallel group node, and Group011 is a serial group node.

[0028] Among them, beginjob, endjob, job1, job2, job3, and job4 are tasks to be processed respectively.

[0029] Among them, beginjob and endjob are associated with pending tasks under Group01 and are assigned to the serial group for processing in sequence, while job1, job2, and Group011 are not associated, so they are assigned to multiple nodes under the serial group for processing at the same time. Job3 and job4 under Group011 are associated, so they are assigned to the same serial group node for processing in sequence.

[0030] In this embodiment, Figure 2 The implementation process of the scheduling platform in is as follows: The first step is to label the process nodes and execute the following common XML tags for daily task scheduling. Figure 3 .

[0031] Step 2: Create a process file named test.XML and edit the file information according to Figure 3 The process information shown, the specific contents of the edited process are as follows: <serial> <name> Group0< / name> <sh> <name> beginjob< / name> <desc> Start the task< / desc> < / sh> <parallal> <name> Group01< / name> <sh> <name> job2< / name> <para> busdate< / para> < / sh> <sh> <name> job3< / name> <para> busdate< / para> < / sh> <parallal> <name> group011< / name> <proc> <name> job3< / name> < / proc> <proc> <name> job4< / name> < / proc> < / parallal> < / parallal> <sh> <name> endjob< / name> <desc> Ending the task< / desc> < / sh> < / serial> The third step is to use parXML test.XML to load the process information after editing the file, and register the process in the parainfo.txt file. Label the process 10. Then use startflow 10 to start the process and complete the batch running of related tasks.

[0032] S12: Obtain tasks to be processed.

[0033] S13. Determine the relevance of the tasks to be processed.

[0034] S14, assigning the associated tasks to be processed to the serial group nodes for sequential processing, and assigning the unrelated tasks to be processed to the parallel group nodes for simultaneous execution.

[0035] In this embodiment, the parallel group node can determine the processing order of the multiple tasks to be processed according to the order in which the multiple tasks to be processed arrive. The processing order of the tasks to be processed is determined by a first-come, first-served (FCFS) method, which is conducive to fairness in the processing order.

[0036] In this embodiment, the parallel group node can also determine the processing order of multiple pending tasks according to the task processing time of the multiple pending tasks. It is mainly divided into two types: non-preemptive and preemptive. The preemptive type is also called the shortest remaining time first algorithm (SRTF). This method can reduce the average waiting time.

[0037] In this embodiment, the parallel group node can also assign priorities to the tasks to be processed according to their importance, and process multiple tasks to be processed in sequence according to their priorities. Each task has a priority, and the task with the highest priority gets the CPU execution time. The priority can be static (determined when created and remains unchanged) or dynamic (may change during the task running process). Task scheduling can be performed according to the importance of the task, thereby ensuring that more important and more urgent tasks can be executed first.

[0038] In this embodiment, when determining the processing order of the tasks to be processed according to the priorities, it also includes: The priorities of pending tasks are adjusted according to their historical execution periods.

[0039] In this embodiment, the priority of the pending tasks is modified based on both the priority of the pending tasks and the processing time of the pending tasks. For example, if the processing cycle of some pending tasks with high priorities is too long, the priority of the pending tasks with shorter processing cycles can be increased to ensure the efficiency of task execution.

[0040] In this embodiment, it is also possible to: construct a pending task queue, each pending task in the pending task queue obtains CPU time in turn according to the time slice, and after the time slice is used up, the pending task is put back to the end of the pending task queue to wait for the next round of scheduling. This method has high fairness, and each pending task can get the same time slice; it has good activity, and each pending task can always remain active. The selection of the time slice size will not affect the system performance and efficiency.

[0041] In this embodiment, the execution order of the tasks to be processed can also be determined according to the response ratio of the tasks to be processed, where the response ratio is (waiting time of the tasks to be processed + required service time of the tasks to be processed) / required service time of the tasks to be processed. This processing method can improve the response speed of the system.

[0042] The present invention provides a task scheduling method based on XML files, which has the following beneficial effects: 1. The scheduling information is saved in XML files. The entire scheduling platform does not need to install a database. The application is simple to build and is very suitable for daily calls of small batch jobs. At the same time, the XML editing mode is provided to meet the structured thinking of developers. The entire tool greatly facilitates daily development work.

[0043] 2. A serial-parallel scheduling group is proposed, which treats related tasks as a serial group, and can complete the scheduling and execution of related tasks in an orderly manner. Multiple unrelated tasks or serial groups can be formed into a parallel group, which increases the concurrency of scheduling tasks and processes multiple unrelated tasks at the same time, thereby greatly improving the timeliness of scheduling.

[0044] Compared with the related art, the technical advantages of the present invention are: First, a breakthrough in the concept of overall information organization of task operations. From simple information recording to structured and regularized "code" information, in the traditional task operation information organization method, a single operation is regarded as an independent design unit, while in this case design, one or more structures are used as design units. The traditional focus is to regard each operation as an individual, while this case is more inclined to a larger whole. From the perspective of the overall characteristics of a task, this solution designs the corresponding calling rules and expresses them in XML format.

[0045] Second, shift the concept from configuration to development. Traditional job information organization is expressed in record form, while this case uses structured code with certain grammatical rules to express it. Records use the concept of configuration, while codes obviously use the concept of development.

[0046] Figure 4 This is a structural diagram of a task scheduling device based on XML files provided by the present invention. Figure 4 ,include: The platform construction module 21 is used to construct a task scheduling platform and store the scheduling information of the task scheduling platform through an XML file. The task scheduling platform includes a serial group node and a parallel group node; An acquisition module 22 is used to acquire tasks to be processed; A determination module 23, used to determine the relevance of the tasks to be processed; The allocation module 24 is used to allocate the related tasks to be processed to the serial group nodes for sequential processing, and allocate the unrelated tasks to be processed to the parallel group nodes for simultaneous execution.

[0047] Figure 5 This is a structural block diagram of an electronic device provided in this embodiment. Figure 5 , electronic devices may include Figure 4 The task scheduling device based on XML file. Generally, the electronic device includes: a processor 31 and a memory 32.

[0048] The processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 31 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 31 may also include a main processor and a coprocessor, the main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. The memory 32 may include one or more computer-readable storage media, which may be non-transitory. The memory 32 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 32 is used to store at least one instruction, which is used to be executed by the processor 31 to implement the XML file-based task scheduling method performed by an electronic device provided in the method embodiment of the present application.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task scheduling method based on XML file, characterized in that: include: Build a task scheduling platform and store the scheduling information of the task scheduling platform through XML files. The task scheduling platform includes serial group nodes and parallel group nodes. Get pending tasks; Determine the relevance of pending tasks; Assign related pending tasks to serial group nodes for sequential processing, and assign unrelated pending tasks to parallel group nodes for simultaneous execution.

2. The XML file-based task scheduling method according to claim 1, characterized in that: When a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The parallel group node determines the processing order of the multiple tasks to be processed according to the order in which the multiple tasks to be processed arrive.

3. The XML file-based task scheduling method according to claim 1, characterized in that: When a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The parallel group node determines the processing order of the multiple tasks to be processed according to the task processing durations of the multiple tasks to be processed.

4. The XML file-based task scheduling method according to claim 1, characterized in that: When a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: According to the importance of the tasks to be processed, priorities are assigned to the tasks to be processed, and multiple tasks to be processed are processed in sequence according to the priorities.

5. The XML file-based task scheduling method according to claim 4, characterized in that: Also includes: The priorities of pending tasks are adjusted according to their historical execution periods.

6. The XML file-based task scheduling method according to claim 1, characterized in that: When a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: Build a queue of pending tasks. Each pending task in the queue gets CPU time in turn according to the time slice. After the time slice is used up, the pending task is put back to the end of the queue and waits for the next round of scheduling.

7. The XML file-based task scheduling method according to claim 1, characterized in that: When a parallel group node processes multiple pending tasks, the process of determining the processing order of the multiple pending tasks is as follows: The execution order of the tasks to be processed is determined according to the response ratio of the tasks to be processed, where the response ratio is (waiting time of the tasks to be processed+required service time of the tasks to be processed) / required service time of the tasks to be processed.

8. A task scheduling device based on XML files, characterized in that: include: The platform construction module is used to construct a task scheduling platform and store the scheduling information of the task scheduling platform through an XML file. The task scheduling platform includes a serial group node and a parallel group node. The acquisition module is used to obtain tasks to be processed; A determination module, used to determine the relevance of tasks to be processed; The allocation module is used to allocate related pending tasks to serial group nodes for sequential processing, and allocate unrelated pending tasks to parallel group nodes for simultaneous execution.

9. An electronic device, characterized in that: It comprises the task scheduling device based on XML file as described in claim 8.

10. A computer-readable storage medium, characterized in that: At least one program code is stored in the computer-readable storage medium, and the program code is executed by a processor to implement the XML file-based task scheduling method as described in any one of claims 1 to 7.

Images