Distributed task scheduling system and method
By decomposing financial data analysis tasks into sub-tasks and scheduling according to system resources, the problem of low execution efficiency in the existing technology is solved, efficient distributed task scheduling and flexible selection of programming languages is realized, and the financial data analysis tasks are improved. Execution efficiency.
Patent Information
- Application Number
- CN202210310344.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-28
AI Technical Summary
The prior art fails to fully consider the characteristics of the task in the distributed scheduling of financial data analysis tasks, resulting in low execution efficiency and the inability to flexibly choose programming languages and schedule according to system resources.
A distributed task scheduling system is proposed. By decomposing financial data analysis tasks into multiple subtasks, configuring the interface information and parameter information of the execution unit of each subtask, and publishing them to the task queue, and determining whether to execute immediately based on system resources , obtain the execution unit through the service registration center and start in parallel to improve efficiency.
It realizes efficient distributed scheduling of financial data analysis tasks, supports flexible selection of programming languages, optimizes task execution based on system resources, and improves execution efficiency and system robustness.
Smart Images

Figure CN114637589B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed technology, and in particular to a distributed task scheduling system and method. Background Art
[0002] At present, financial market business is becoming more and more active, and there are more and more scenarios involving analysis of financial data analysis tasks in related business systems. Therefore, these analysis tasks need to be executed in large quantities. The existing technology generally schedules a series of analysis tasks through a batch job system according to a set driving relationship to schedule each job in the batch job system to realize the data extraction, conversion and loading process. However, since the characteristics of each analysis task are not fully considered, the execution efficiency of the analysis task cannot be improved in a targeted manner. Summary of the invention
[0003] The embodiment of the present invention provides a distributed task scheduling system for implementing distributed scheduling for financial data analysis tasks with high execution efficiency. The system includes:
[0004] A first configuration module is used to decompose the financial data analysis task into multiple sub-analysis tasks, and configure the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task;
[0005] The task publishing module is used to publish financial data analysis tasks to the pending task queue;
[0006] The scheduling module is used to determine whether the financial data analysis task should be executed immediately according to the current system resources after monitoring that there is a new financial data analysis task in the pending task queue. If so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task;
[0007] The execution module is used to obtain the execution results of each execution unit based on the parameter information of the execution unit; and analyze the execution results of the financial data analysis task according to the execution results of each execution unit.
[0008] The embodiment of the present invention provides a distributed task scheduling method for implementing distributed scheduling for financial data analysis tasks with high execution efficiency. The method includes:
[0009] Decomposing the financial data analysis task into a plurality of sub-analysis tasks, and configuring the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task;
[0010] Publish financial data analysis tasks to the pending task queue;
[0011] After monitoring that there is a new financial data analysis task in the pending task queue, determine whether the financial data analysis task is to be executed immediately according to the current system resources. If so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task;
[0012] Based on the parameter information of the execution units, each execution unit is started in parallel to obtain the execution result of each execution unit;
[0013] Analyze the execution results of the financial data analysis task according to the execution results of each execution unit.
[0014] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned distributed task scheduling method when executing the computer program.
[0015] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed task scheduling method is implemented.
[0016] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the distributed task scheduling method is implemented.
[0017] In an embodiment of the present invention, a first configuration module is used to decompose a financial data analysis task into multiple sub-analysis tasks, and configure the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task; a task publishing module is used to publish the financial data analysis task to a pending task queue; a scheduling module is used to determine whether the financial data analysis task is to be executed immediately based on current system resources after monitoring that there is a new financial data analysis task in the pending task queue, and if so, query the service registration center based on the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task; an execution module is used to start each execution unit in parallel based on the parameter information of the execution unit to obtain the execution result of each execution unit; and analyze the execution result of the financial data analysis task based on the execution result of each execution unit. In the above process, the financial data analysis tasks are analyzed and split, and a unified mechanism is provided to configure the execution units of each analysis task. The execution units of the sub-analysis tasks are executed concurrently with high efficiency. The execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task, thereby supporting the selection of a programming language that meets the computing requirements to implement the execution unit, giving full play to the advantages of various languages. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0019] Figure 1 Schematic diagram of a distributed task scheduling system in an embodiment of the present invention Figure 1 ;
[0020] Figure 2 Schematic diagram of a distributed task scheduling system in an embodiment of the present invention Figure 3 ;
[0021] Figure 3 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 1 ;
[0022] Figure 4 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 3 ;
[0023] Figure 5 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 4 ;
[0024] Figure 6 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 5 ;
[0025] Figure 7 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0027] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.
[0028] The inventors found that in the prior art, most of the distributed scheduling for financial data analysis tasks did not split the tasks. In the financial field, the same type of analysis tasks may include several independent parts. For example, in the analysis of the future situation of a batch of bonds, after selecting the start analysis date and the end analysis date, when conducting the analysis, the business not only pays attention to accounting indicators such as average size and average interest rate, but also hopes to value the bonds according to the yield curve. These two analysis requirements are independent of each other. By splitting the analysis tasks by analysis type and processing them concurrently, this analysis process can be completed efficiently.
[0029] In addition, the inventors found that for multiple execution units of the same task in the same system, the existing technology cannot flexibly select programming languages according to the characteristics of the tasks. In the financial field, there are many kinds of analysis tasks, and the independent units of each task have their own characteristics. The flexible selection of appropriate programming languages can give full play to the advantages of various programming languages and improve execution efficiency.
[0030] The inventors found that there is no effective monitoring execution for each execution unit, and it cannot be flexibly scheduled according to system resources.
[0031] Based on the above situation, the present invention proposes a distributed task scheduling system, which first splits the analysis tasks, supports flexible selection of programming languages in the coding implementation link for each execution unit, and flexibly schedules according to system resources and task running conditions.
[0032] Figure 1 Schematic diagram of a distributed task scheduling system in an embodiment of the present invention Figure 1 ,like Figure 1 As shown, the system includes:
[0033] The first configuration module 101 is used to decompose the financial data analysis task into multiple sub-analysis tasks, and configure the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task;
[0034] The task publishing module 102 is used to publish the financial data analysis task to the pending task queue;
[0035] The scheduling module 103 is used to determine whether the financial data analysis task is to be executed immediately according to the current system resources after monitoring that there is a new financial data analysis task in the pending task queue. If so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task;
[0036] The execution module 104 is used to start each execution unit in parallel based on the parameter information of the execution unit, obtain the execution result of each execution unit; and analyze the execution result of the financial data analysis task according to the execution result of each execution unit.
[0037] In addition, the first configuration module can also configure the task type of the financial data analysis task and the sub-analysis task, and obtain the unique identification of the task type for subsequent data statistics. The first configuration module can be displayed to the user in the form of an interface to facilitate user editing.
[0038] The execution module can start each execution unit in parallel in the cluster environment through remote scheduling. Each execution unit can be a microservice to further improve the execution efficiency. Microservices are generally managed through the service registration center. Therefore, according to the interface information of the execution unit of each sub-analysis task, the service registration center is queried to obtain the execution unit of each sub-analysis task.
[0039] The scheduling module determines whether the financial data analysis task should be executed immediately based on the current system resources. The current system resources mainly refer to the thread pool, that is, whether there are idle threads in the thread pool. If not, the execution is suspended. Otherwise, it can be executed immediately. Of course, other system resources can also be judged.
[0040] It should be noted that the execution units with the same functions have different operating efficiency when implemented in different programming languages. For example, for financial analysis tasks, Python language has more complete third-party library functions, which can more conveniently implement functions; C / C++ is more efficient and can complete analysis tasks quickly. Therefore, the execution unit of each sub-analysis task is implemented through the selected programming language, and the programming language meets the computing requirements of the sub-analysis task.
[0041] In one embodiment, the scheduling module is further used to:
[0042] When it is determined according to current system resources that the financial data analysis task is not to be executed immediately, after a set time interval, it is determined according to current system resources whether the financial data analysis task is to be executed immediately.
[0043] In one embodiment, the execution module is specifically used for:
[0044] Add sub-analysis tasks with association relationships to an execution group;
[0045] Execute multiple execution groups in parallel.
[0046] The above-mentioned association includes a pre-association relationship and a post-association relationship. For example, the analysis task is divided into sub-analysis task 1, sub-analysis task 2, sub-analysis task 3, sub-analysis task 4, and sub-analysis task 5. Sub-analysis task 1 and sub-analysis task 2 are associated with each other, and sub-analysis task 2 needs to be executed after sub-analysis task 1 is executed. In this case, sub-analysis task 1 and sub-analysis task 2 are added to an execution group, and sub-analysis task 3, sub-analysis task 4, and sub-analysis task 5 are respectively in one execution group, so the execution groups can be executed in parallel.
[0047] In one embodiment, the execution module is specifically used for:
[0048] Before starting each execution unit in parallel, an execution record is generated;
[0049] After starting each execution unit in parallel, updating the execution result of each execution unit to the execution record;
[0050] Based on the execution record, the execution result of the financial data analysis task is analyzed.
[0051] In one embodiment, each sub-analysis task has multiple execution units, and the multiple execution units are spare to improve the robustness of the system.
[0052] Figure 2 Schematic diagram of a distributed task scheduling system in an embodiment of the present invention Figure 2 In one embodiment, the system further includes a monitoring module 201, which is used to:
[0053] Monitor the execution status of the execution unit;
[0054] When the execution unit fails to execute the sub-analysis task, determining a post-associated sub-analysis task of the sub-analysis task corresponding to the execution unit;
[0055] Stop the post-correlation sub-analysis task.
[0056] Through the above embodiments, unnecessary sub-analysis tasks can be avoided.
[0057] The monitoring module can also read the execution time of each step of each execution unit to facilitate monitoring the execution status of the sub-analysis task.
[0058] To summarize, in the system proposed in the embodiment of the present invention, the first configuration module is used to decompose the financial data analysis task into multiple sub-analysis tasks, and configure the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task; the task publishing module is used to publish the financial data analysis task to the pending task queue; the scheduling module is used to determine whether the financial data analysis task is to be executed immediately according to the current system resources after monitoring that there is a new financial data analysis task in the pending task queue, and if so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task; the execution module is used to start each execution unit in parallel and obtain the execution result of each execution unit; the execution result of the financial data analysis task is analyzed according to the execution result of each execution unit. In the above process, the financial data analysis tasks are analyzed and split, and a unified mechanism is provided to configure the execution units of each analysis task. The execution units of the sub-analysis tasks are executed concurrently with high efficiency. According to the computing requirements of each sub-analysis task, the execution unit corresponding to the programming language that meets the computing requirements is determined from multiple execution units, thereby supporting the free selection of execution units implemented by programming languages and giving full play to the advantages of various languages.
[0059] The embodiment of the present invention further proposes a distributed task scheduling method, the principle of which is similar to that of the distributed task scheduling system and will not be described in detail here.
[0060] Figure 3The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 1 ,include:
[0061] Step 301, decomposing the financial data analysis task into multiple sub-analysis tasks, and configuring the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task;
[0062] Step 302, publishing the financial data analysis task to the pending task queue;
[0063] Step 303, after monitoring that there is a new financial data analysis task in the pending task queue, determine whether the financial data analysis task is to be executed immediately according to the current system resources; if so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task;
[0064] Step 304, based on the parameter information of the execution unit, start each execution unit in parallel and obtain the execution result of each execution unit;
[0065] Step 305: Analyze the execution result of the financial data analysis task according to the execution results of each execution unit.
[0066] Figure 4 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 2 In one embodiment, the method further comprises:
[0067] Step 401, when it is determined according to current system resources that the financial data analysis task is not to be executed immediately, after a set time interval, it is determined according to current system resources whether the financial data analysis task is to be executed immediately.
[0068] In one embodiment, starting each execution unit in parallel is specifically used for:
[0069] Add sub-analysis tasks with association relationships to an execution group;
[0070] Execute multiple execution groups in parallel.
[0071] Figure 5 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 3 , in one embodiment, before starting each execution unit in parallel, it also includes: step 501, generating an execution record;
[0072] After starting each execution unit in parallel, the method further includes: step 502, updating the execution result of each execution unit to the execution record;
[0073] Analyzing the execution result of the financial data analysis task according to the execution result of each execution unit includes: analyzing the execution result of the financial data analysis task based on the execution record.
[0074] In one embodiment, each sub-analysis task has multiple execution units.
[0075] Figure 6 The process of the distributed task scheduling method in the embodiment of the present invention is as follows Figure 4 In one embodiment, the method further comprises:
[0076] Step 601, monitoring the execution status of the execution unit;
[0077] Step 602, when the execution unit fails to execute the sub-analysis task, determining the post-associated sub-analysis task of the sub-analysis task corresponding to the execution unit;
[0078] Step 603: Stop the post-association sub-analysis task.
[0079] To summarize, in the method proposed in the embodiment of the present invention, the financial data analysis task is decomposed into multiple sub-analysis tasks, and the interface information and parameter information of the execution unit of each sub-analysis task are configured; wherein, the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task; the financial data analysis task is published to the pending task queue; after monitoring that there is a new financial data analysis task in the pending task queue, it is determined whether the financial data analysis task is to be executed immediately according to the current system resources, and if so, the service registration center is queried according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task; the execution units are started in parallel to obtain the execution results of each execution unit; and the execution results of the financial data analysis task are analyzed according to the execution results of each execution unit. In the above process, the financial data analysis tasks are analyzed and split, and a unified mechanism is provided to configure the execution units of each analysis task. The execution units of the sub-analysis tasks are executed concurrently with high efficiency. According to the computing requirements of each sub-analysis task, the execution unit corresponding to the programming language that meets the computing requirements is determined from multiple execution units, thereby supporting the free selection of execution units implemented by programming languages and giving full play to the advantages of various languages.
[0080] An embodiment of the present invention further provides a computer device, Figure 7 Schematic diagram of a computer device in an embodiment of the present invention. The computer device 700 includes a memory 710, a processor 720, and a computer program 730 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 730, the above-mentioned distributed task scheduling method is implemented.
[0081] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed task scheduling method is implemented.
[0082] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the distributed task scheduling method is implemented.
[0083] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program business systems. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program business system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program business systems according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0087] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A distributed task scheduling system, It is characterized in that include: A first configuration module is used to decompose the financial data analysis task into multiple sub-analysis tasks, and configure the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task; The task publishing module is used to publish financial data analysis tasks to the pending task queue; The scheduling module is used to determine whether the financial data analysis task should be executed immediately according to the current system resources after monitoring that there is a new financial data analysis task in the pending task queue. If so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task; An execution module, which is used to start each execution unit in parallel based on the parameter information of the execution unit, and obtain the execution result of each execution unit; and analyze the execution result of the financial data analysis task according to the execution result of each execution unit; The execution module is specifically used to: add sub-analysis tasks with associated relationships to an execution group; execute multiple execution groups in parallel; It also includes a monitoring module, which is used to: monitor the execution status of the execution unit; when the execution unit fails to execute a sub-analysis task, determine the post-associated sub-analysis task of the sub-analysis task corresponding to the execution unit; and stop the post-associated sub-analysis task.
2. The system according to claim 1, It is characterized in that The scheduling module is also used to: When it is determined according to current system resources that the financial data analysis task is not to be executed immediately, after a set time interval, it is determined according to current system resources whether the financial data analysis task is to be executed immediately.
3. The system according to claim 1, It is characterized in that The execution module is specifically used for: Before starting each execution unit in parallel, an execution record is generated; After starting each execution unit in parallel, updating the execution result of each execution unit to the execution record; Based on the execution record, the execution result of the financial data analysis task is analyzed.
4. The system according to claim 1, It is characterized in that Each sub-analysis task has multiple execution units.
5. A distributed task scheduling method, It is characterized in that include: Decomposing the financial data analysis task into a plurality of sub-analysis tasks, and configuring the interface information and parameter information of the execution unit of each sub-analysis task; wherein the execution unit of each sub-analysis task is implemented by a selected programming language, and the programming language meets the computing requirements of the sub-analysis task; Publish financial data analysis tasks to the pending task queue; After monitoring that there is a new financial data analysis task in the pending task queue, determine whether the financial data analysis task is to be executed immediately according to the current system resources. If so, query the service registration center according to the interface information of the execution unit of each sub-analysis task to obtain the execution unit of each sub-analysis task; Based on the parameter information of the execution units, each execution unit is started in parallel to obtain the execution result of each execution unit; Analyze the execution results of the financial data analysis task according to the execution results of each execution unit; Starting each execution unit in parallel includes: adding sub-analysis tasks with associated relationships to an execution group; executing multiple execution groups in parallel; The method further includes: monitoring the execution status of the execution unit; when the execution unit fails to execute the sub-analysis task, determining a post-associated sub-analysis task of the sub-analysis task corresponding to the execution unit; and stopping the post-associated sub-analysis task.
6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method of claim 5 is implemented.
7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of claim 5 is implemented.
8. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to claim 5 is implemented.
