Real-time computing task management and scheduling method and system based on event driving

Abstract

The invention discloses a real-time computing task management and scheduling method and system based on event driving, and the method comprises the steps: defining a plurality of real-time computing tasks, and configuring a dependency relationship among the plurality of real-time computing tasks; obtaining a calculation result of the real-time calculation task and writing the calculation result into a data storage system; capturing a log event of which the calculation result is successfully written, and storing the log event into a opentop ic queue of kafka; the data of the opentop ic queue is consumed, and the log event is distributed to a downstream real-time calculation task after the set delay time is over. By introducing kafka and a dependency relationship, data dependency management and calculation scheduling between real-time calculation tasks are realized. The system can dynamically schedule the execution of the task according to the actual availability of the data, so that calculation errors caused by inconsistent data are avoided. In addition, the system also has relatively high flexibility and expandability, and can be conveniently expanded to a more complex calculation scene.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time data stream processing, and in particular to a method and system for event-driven real-time computing task management and scheduling. Background Art

[0002] In real-time computing environments, such as Apache Flink, there are often data dependencies between multiple tasks. For example, after Task 1 writes the calculation results to Elasticsearch (ES), Task 2 needs to rely on the calculation results of Task 1. However, due to network latency, ES index refresh mechanism, etc., the data written by Task 1 to ES may not be immediately queried by Task 2, resulting in data inconsistency and calculation errors. Summary of the invention

[0003] In view of this, the present invention proposes a method and system for real-time computing task management and scheduling based on event-driven, which can dynamically schedule the execution of tasks according to the actual availability of data, thereby avoiding calculation errors caused by inconsistent data. The present invention provides the following technical solutions:

[0004] A method for managing and scheduling real-time computing tasks based on event-driven, the method comprising:

[0005] Define multiple real-time computing tasks and configure the dependencies between them;

[0006] Obtain the calculation results of real-time computing tasks and write them into the data storage system;

[0007] Capture the log event of the successful writing of the calculation result and store the log event in the depen_topic queue of Kafka;

[0008] Consumes data from the depen_topic queue and distributes log events to downstream real-time computing tasks after the set delay time.

[0009] Furthermore, when defining a plurality of real-time computing tasks and configuring dependencies between the plurality of real-time computing tasks, the method further includes:

[0010] Dynamically generate task execution plans based on dependencies.

[0011] Furthermore, after dynamically generating the task execution plan according to the dependency relationship, the method further includes:

[0012] According to the task execution plan, the initial real-time computing task is executed.

[0013] Furthermore, the log event encapsulates the calculation result of the real-time calculation task.

[0014] Furthermore, the method of defining multiple real-time computing tasks and configuring dependencies between the multiple real-time computing tasks includes:

[0015] The dependency is a data table or a task;

[0016] When the dependency is a task, parse the table output by the dependent task;

[0017] Convert the dependency relationship between real-time computing tasks into dependency table data between tasks and tables;

[0018] Encapsulate the dependency table data as events and distribute them to the depend_topic queue;

[0019] Distribute the events in the message queue to the topic ic subscribed by the current real-time computing task to complete the configuration of the dependencies between real-time computing tasks.

[0020] Further, after consuming the data in the depen_topic queue and distributing the log event to the downstream real-time computing task after the set delay time, the method further includes:

[0021] Detecting whether the downstream real-time computing task has received the log event;

[0022] If it has been received, the downstream real-time computing task is executed; otherwise,

[0023] Wait for the completion of receiving log events and execute downstream real-time computing tasks.

[0024] The present invention further discloses a system for managing and scheduling real-time computing tasks based on event-driven, comprising:

[0025] Custom modules are used to define multiple real-time computing tasks and configure the dependencies between multiple real-time computing tasks;

[0026] A data management module is used to obtain the calculation results of real-time computing tasks and write them into the data storage system;

[0027] The event management module captures the log event of the successful writing of the calculation results and stores the log event in the depen_top ic queue of Kafka;

[0028] The data distribution module is used to consume the data in the depen_topic queue and distribute the log events to the downstream real-time computing tasks after the set delay time.

[0029] The present invention further discloses a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0030] The present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0031] The present invention further discloses a computer program product, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0032] According to the technical solution of the present invention, by introducing Kafka and dependency relationships, data dependency management and computation scheduling between real-time computing tasks are realized. The system can dynamically schedule the execution of tasks according to the actual availability of data, thereby avoiding computation errors caused by inconsistent data. In addition, the system also has high flexibility and scalability, and can be easily expanded to more complex computing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which:

[0034] Figure 1 is a schematic diagram of a method for event-driven real-time computing task management and scheduling in an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the system structure of event-driven real-time data stream dependency management and calculation in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the implementation mode of the present application. Obviously, the described implementation mode is only a part of the implementation mode of the present application, not all the implementation modes. Based on the implementation mode in the present application, all other implementation modes obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its implementation methods, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0040] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to the specific circumstances. In addition, the term "plurality" should mean two or more than two.

[0041] It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0042] In a real-time computing environment, there is often a data dependency between multiple real-time computing tasks. That is, if Task 1 is dependent on Task 2, then the real-time computing of Task 2 depends on the real-time computing result of Task 1. However, the data transmission or query mechanism of the computing result may cause Task 2 to be unable to obtain accurate computing result data, resulting in data inconsistency and computing errors.

[0043] refer to Figure 1 In order to solve this problem, this embodiment discloses a method for managing and scheduling real-time computing tasks based on event-driven, and the method includes the following steps:

[0044] S100: define multiple real-time computing tasks, and configure dependencies between the multiple real-time computing tasks.

[0045] It should be understood that the dependency between real-time computing tasks refers to sending the calculation result of a certain real-time computing task to the downstream real-time computing task through the dependency. In the present embodiment, the real-time computing task environment adopts the Apache Flink framework, which is only exemplary and not limited in the present embodiment. Therefore, when configuring the dependency between multiple real-time computing tasks, it is also necessary to dynamically generate a task execution plan according to the dependency, and execute the initial real-time computing task according to the task execution plan, and further dynamically adjust the execution plan according to the execution status of the real-time computing task. For example, a dependency is established between task 1 and task 2, and task 3 is the downstream real-time computing task of task 2. Then the task execution plan can control task 2 to start after task 1 is calculated, and control task 3 to start after task 2 is calculated. The above is only an exemplary statement, and the specific task execution plan can be adjusted according to the application scenario.

[0046] The dependency relationship between the above-mentioned multiple real-time computing tasks can be a common data table or a task. Then, the method of configuring the dependency relationship between the multiple real-time computing tasks includes:

[0047] S110: When the dependency relationship is a task, parsing a table output by the dependent task;

[0048] S120: converting the dependency relationship between the real-time computing tasks into dependency table data between the tasks and the tables;

[0049] S130: Encapsulate the dependency table data into events and distribute them to the depend_topic queue;

[0050] S140: Distribute the events in the message queue to the topic ic subscribed by the current real-time computing task to complete the configuration of the dependency relationship between the real-time computing tasks.

[0051] It can be seen from this that after the dependency relationship between real-time computing tasks is configured, if the downstream real-time computing task wants to execute, it must obtain the event trigger of its upstream dependent task. It will not happen that task 1 has just been executed and the downstream task 2 directly queries the calculation result of task 1, thereby avoiding data query errors.

[0052] S200: Obtaining the calculation results of the real-time calculation task and writing them into the data storage system;

[0053] In this embodiment, the data storage system uses Elasticsearch (ES), which can realize the storage of calculation result data and facilitate secondary operations on the calculation result data, such as packaging, event log generation, etc. At the same time, the ES index refresh mechanism can make the calculation result data indexed instantly, and at the same time can ensure the consistency of the calculation result data.

[0054] S300: Capture the log event of the successful writing of the calculation result, and store the log event in the depen_top ic queue of Kafka;

[0055] According to the existing real-time computing task solutions, data recognition errors usually occur because the downstream real-time computing task indexes the calculation results when the calculation result data is not saved. In order to avoid this situation, the solution disclosed in this embodiment generates a log event after writing the calculation result data to ES, indicating that the calculation result is successfully written to ES. At the same time, the log event also encapsulates the calculation result data. After the log event indicating that the calculation result data is successfully written to ES is captured, Kafka is introduced as a data transfer station to realize asynchronous transmission of data between real-time computing tasks, that is, the log event is stored in the depen_top ic queue of Kafka.

[0056] S400: Consume the data in the depen_top ic queue and distribute the log events to the downstream real-time computing tasks after the set delay time ends.

[0057] As mentioned above, the real-time computing task environment of this embodiment adopts the Apache Flink framework, and relies on the window function of Flink to realize data delay, specifically, consuming the data of the depen_topic queue, and distributing the log event to the downstream real-time computing task after the set delay time ends, so as to ensure that when the real-time calculation of the example task 1 is completed, its downstream task 2 will not immediately query the calculation result of task 1, but wait for the log event after the delay time to be distributed to task 2 before starting execution. At this time, it indicates that task 1 has been determined to be executed and the calculation result is correctly stored in ES, so as to ensure the accuracy of the execution result of task 2.

[0058] After the log event is distributed to the downstream real-time computing task, the method disclosed in this embodiment further includes:

[0059] Detect whether the downstream real-time computing task has received the log event; if it has been received, execute the downstream real-time computing task; otherwise, wait for the completion of receiving the log event and execute the downstream real-time computing task.

[0060] In summary, this implementation method realizes asynchronous transmission of data between tasks by introducing Kafka as a transit station for dependent data, and realizes delayed data distribution by consuming data in the depend_topic queue of Kafka, ensuring that data is stable in ES before being used by downstream real-time computing tasks. The computing scheduling module dynamically schedules tasks according to dependencies to improve the flexibility and scalability of the system.

[0061] refer to Figure 2 This embodiment further discloses a system for managing and scheduling real-time computing tasks based on event-driven, including:

[0062] The custom module 21 is used to define multiple real-time computing tasks and configure the dependencies between the multiple real-time computing tasks; it is also used to dynamically generate a task execution plan according to the dependencies, and execute the initial real-time computing task according to the task execution plan.

[0063] For configuring the dependency between multiple real-time computing tasks, the implementation method of the custom module 21 is: the dependency is a data table or a task; when the dependency is a task, parse the table output by the dependent task; convert the dependency between the real-time computing tasks into dependency table data between the task and the table; encapsulate the dependency table data into events and distribute them to the depend_topic queue; distribute the events in the message queue to the topic ic subscribed by the current real-time computing task to complete the configuration of the dependency between the real-time computing tasks.

[0064] A data management module 22, used to obtain the calculation results of the real-time calculation task and write them into the data storage system;

[0065] The event management module 23 captures the log event of the successful writing of the calculation result and stores the log event in the depen_top ic queue of Kafka;

[0066] The data distribution module 24 is used to consume the data in the depen_topic queue and distribute the log events to the downstream real-time computing tasks after the set delay time ends; it is also used to,

[0067] Detecting whether the downstream real-time computing task has received the log event;

[0068] If it has been received, the downstream real-time computing task is executed; otherwise,

[0069] Wait for the completion of receiving log events and execute downstream real-time computing tasks.

[0070] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0071] The processor 501 and the memory 502 communicate with each other via the bus 503 ; the processor 501 is used to call the program instructions in the memory 502 to execute the method provided by the above implementation.

[0072] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the method provided by the above embodiment.

[0073] This embodiment further discloses a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0074] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0077] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for managing and scheduling real-time computing tasks based on event-driven, characterized in that: The method comprises: Define multiple real-time computing tasks and configure the dependencies between them; Obtain the calculation results of real-time computing tasks and write them into the data storage system; Capture the log event of the successful writing of the calculation result and store the log event in the depen_topic queue of Kafka; Consumes data from the depen_topic queue and distributes log events to downstream real-time computing tasks after the set delay time.

2. The method for managing and scheduling real-time computing tasks based on event-driven according to claim 1, characterized in that: When defining multiple real-time computing tasks and configuring dependencies between the multiple real-time computing tasks, the method further includes: Dynamically generate task execution plans based on dependencies.

3. The method for event-driven real-time computing task management and scheduling according to claim 2, characterized in that: After dynamically generating the task execution plan according to the dependency relationship, the method further includes: According to the task execution plan, the initial real-time computing task is executed.

4. The method for managing and scheduling real-time computing tasks based on event-driven according to claim 1, characterized in that: The log event encapsulates the calculation result of the real-time calculation task.

5. The method for managing and scheduling real-time computing tasks based on event-driven according to claim 1, characterized in that: The method of defining multiple real-time computing tasks and configuring dependencies between the multiple real-time computing tasks includes: The dependency is a data table or a task; When the dependency is a task, parse the table output by the dependent task; Convert the dependency relationship between real-time computing tasks into dependency table data between tasks and tables; Encapsulate the dependency table data as events and distribute them to the depend_topic queue; Distribute the events in the message queue to the topic subscribed by the current real-time computing task to complete the configuration of the dependencies between real-time computing tasks.

6. The method for event-driven real-time computing task management and scheduling according to claim 5, characterized in that: After consuming the data in the depen_topic queue and distributing the log event to the downstream real-time computing task after the set delay time, the method further includes: Detecting whether the downstream real-time computing task has received the log event; If it has been received, the downstream real-time computing task is executed; otherwise, Wait for the completion of receiving log events and execute downstream real-time computing tasks.

7. A system for managing and scheduling real-time computing tasks based on event-driven, characterized in that: include: Custom modules are used to define multiple real-time computing tasks and configure the dependencies between multiple real-time computing tasks; A data management module is used to obtain the calculation results of real-time computing tasks and write them into the data storage system; The event management module captures the log events of successful calculation results and stores them in the depen_topic queue of Kafka; The data distribution module is used to consume the data in the depen_topic queue and distribute the log events to the downstream real-time computing tasks after the set delay time.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images