A controllable real-time task remote execution system

Abstract

The present invention relates to a controllable real-time task remote execution system, comprising a task management module, a task scheduling module and a task execution backend; the task management module is connected to the task scheduling module, and the task management module is used to create a task and send it to the task scheduling module; the task scheduling module is connected to the task execution backend, and the task execution backend is used to obtain and execute a task scheduling object; the task execution backend is connected to the task management module TCP long connection, and the task execution backend is used to transmit the task scheduling object execution real-time status and / or real-time log information to the task management module. Through the task management unit and the task execution backend TCP long connection, the information synchronization efficiency is improved, which facilitates the task management unit to intervene in the task execution backend in time when the task execution is abnormal; and solves the problem that the information synchronization method in the prior art adopts a polling mechanism, causing information delay and execution system performance loss.

Description

Technical Field

[0001] The present invention relates to the technical field of remote task execution, and in particular to a controllable real-time remote task execution system. Background Art

[0002] In the operation and maintenance field of Internet companies, it is very common to distribute execution tasks to remote servers. On the one hand, executing commands through manual remote login is prone to human operation errors and security risks; on the other hand, as an important asset of the company's operation, the server has strict login permission settings, which makes it difficult for ordinary business department personnel to perform self-service remote real-time task execution operations.

[0003] As the operation scale and business needs of Internet companies grow, the development of task scheduling and execution systems that are autonomous and controllable in process and operation will greatly improve the efficiency and security of business departments at the self-service operation level. However, the existing task scheduling and execution systems mainly have the following technical problems:

[0004] 1) The real-time status of the task cannot be observed during the execution of the task. The polling mechanism causes information delay and low synchronization efficiency. It is impossible to intervene in time when the task is abnormal.

[0005] 2) After the task is executed, the task log is not archived, resulting in the inability to query the log information corresponding to the historical tasks;

[0006] 3) The scheduling module cannot support the execution of tasks with dependencies.

[0007] Therefore, it is necessary to provide a real-time task remote execution system that is easy to observe, controllable, can query the execution task history log and supports the execution of tasks with dependencies to solve the above technical problems. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a controllable real-time task remote execution system, which solves the problem that the information synchronization method in the prior art adopts a polling mechanism, causing information delay and execution system performance loss.

[0009] The technical effects of the present invention are achieved as follows:

[0010] A controllable real-time task remote execution system, including a task management module, a task scheduling module and a task execution backend;

[0011] The task management module is connected to the task scheduling module, and the task management module is used to create tasks and send them to the task scheduling module;

[0012] The task scheduling module is connected to the task execution backend, and the task execution backend is used to obtain and execute task scheduling objects;

[0013] The task execution backend and the task management module are TCP long connected, and the task execution backend is used to transmit the real-time status and / or real-time log information of the task scheduling object execution to the task management module.

[0014] Furthermore, the task management module includes a log processing module, and the log processing module is connected to the task execution backend TCP long connection.

[0015] The log processing module is configured to archive the real-time log information corresponding to the task scheduling object after the task scheduling object is executed.

[0016] Furthermore, it also includes a first server and a second server, the first server is used to receive and store the log information transmitted by the log processing module, and the second server is used to receive the remote execution command sent by the target backend to execute the task scheduling object.

[0017] Furthermore, the real-time log information includes task ID and / or time and / or server IP.

[0018] Furthermore, the task scheduling module is used to distribute task scheduling objects to the task execution backend.

[0019] The task scheduling module is configured to determine whether a task depends on other tasks after obtaining the task sent by the task management module, and put the task into a task list when the task does not depend on other tasks, and put the task into a dependent task relationship tree when the task depends on an unfinished task.

[0020] Furthermore, the task execution backend includes multiple backends, and the multiple backends are connected to the task scheduling module.

[0021] The backend is set to be a target backend when the backend is in an idle state, and the target backend is used to obtain and execute the task scheduling object from the task list.

[0022] Furthermore, the target backend is configured to report a status to the task management module before executing the task scheduling object, wherein the status is that the task scheduling object is executed at the target backend.

[0023] Furthermore, the task management module is configured to find a target backend for executing the task scheduling object corresponding to the task scheduling object based on the relationship mapping table, and send a control instruction to the task execution backend to control the pause, termination or continuation of the task scheduling object.

[0024] Furthermore, the target backend is configured to send a request instruction to notify the task scheduling module that the task scheduling object has been completed when the execution of the task scheduling object is completed and the task scheduling object is dependent on the first task.

[0025] Furthermore, the task scheduling module is configured to traverse all tasks that the first task depends on after receiving the request instruction. If all tasks that the first task depends on are completed, the first task is moved from the dependent task relationship tree to the task queue through the task scheduling module.

[0026] As described above, the present invention has the following beneficial effects:

[0027] 1) Through the TCP long connection between the task management unit and the task execution backend, it is possible to observe real-time logs during task execution and synchronize the real-time status of task execution, thereby improving the efficiency of information synchronization; at the same time, it is convenient for the task management unit to intervene in the task execution backend in a timely manner when the task execution is abnormal, thereby solving the problem of information delay and execution system performance loss caused by the polling mechanism used in the information synchronization method in the prior art.

[0028] 2) The log processing module is used to archive the logs of completed tasks, which improves the reading performance of historical logs.

[0029] 3) By setting up a task scheduling module, the execution of tasks with dependencies can be supported, thereby achieving controllability of task timing.

[0030] 4) The log information transmitted by the log processing module is received and stored by the first server, thereby improving the query efficiency of historical log information. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0032] Figure 1 A schematic diagram of the structure of a controllable real-time task remote execution system provided in an embodiment of this specification;

[0033] Figure 2 A flowchart of the communication process between the task execution backend and the task management module provided in the embodiment of this specification;

[0034] Figure 3 A flowchart of the task scheduling execution process provided in the embodiments of this specification.

[0035] Among them, the reference numerals in the figure correspond to:

[0036] Task management module 1, log processing module 11, task scheduling module 2, task execution backend 3, first server 4, second server 5. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0039] At present, the task scheduling execution system uses a polling mechanism during the task execution process, which causes the task management module to receive information delays and cannot observe the real-time status of the task. It cannot intervene in time when the task is abnormal, and the task log is not archived after the task execution is completed, resulting in the inability to query the log information of historical tasks.

[0040] Therefore, the present specification provides a controllable real-time task remote execution system. Through the task management unit and the task execution backend TCP long connection, it is possible to observe the real-time log during the task execution process, synchronize the real-time status of the task execution, and improve the information synchronization efficiency; at the same time, it is convenient for the task management unit to intervene in the task execution backend in time when the task execution is abnormal, which solves the problem of information delay and execution system performance loss caused by the polling mechanism used in the information synchronization method in the prior art. The log processing module is used to archive the logs of the tasks that have been executed, thereby increasing the reading performance of the historical logs. By setting the task scheduling unit, the execution of tasks with dependencies can be supported, thereby realizing the controllability of the task timing. The log information transmitted by the log processing module is received and stored by the first server, thereby improving the query efficiency of the historical log information.

[0041] The embodiments of this specification provide a controllable real-time task remote execution system, such as Figure 1 As shown, it includes a task management module 1, a task scheduling module 2 and a task execution backend 3;

[0042] The task management module 1 is connected to the task scheduling module 2. The task management module 1 is used to create tasks and send them to the task scheduling module 2.

[0043] The task scheduling module 2 is connected to the task execution backend 3, and the task execution backend 3 is used to obtain and execute the task scheduling object;

[0044] The task execution backend 3 and the task management module 1 have a TCP long connection, and the task execution backend 3 is used to transmit the real-time status and / or real-time log information of the task scheduling object execution to the task management module 1.

[0045] In a specific implementation manner, the real-time log information includes task ID and / or time and / or server IP.

[0046] Specifically, the user creates a real-time task through the task management module 1 and sends it to the task scheduling module 2. The task execution backend 3 obtains the task scheduling object from the real-time task of the task scheduling module 2 and executes it. While the task execution backend 3 executes the task scheduling object, the task execution backend 3 and the task management module 1 establish a TCP long connection, so that during the task execution process, the task execution backend 3 can send control instructions to the task management module 1 to complete the suspension, termination or continuation of the task scheduling object, so that in the event of abnormal task execution, the task execution backend 3 can intervene in the task management module 1 in time to ensure the high effectiveness of the completion of the epic task.

[0047] Among them, TCP long connection means that after the task execution backend 3 is started, according to the TCP listening address of the task management module 1 recorded in the configuration file of the task execution backend 3, a TCP long connection is established with the task management module 1 to push the real-time status and real-time log information of the task scheduling object execution to the task management module 1, thereby realizing the synchronization of the real-time status and real-time log information of the task execution. The task execution backend 3 includes multiple backends. When multiple backends execute multiple task scheduling objects correspondingly, the multiple backends simultaneously establish a TCP long connection with the task management module 1, and the task management module 1 records the task scheduling object corresponding to each backend execution as a relationship mapping table and stores it in the memory.

[0048] Through the TCP long connection between the task management unit 1 and the task execution backend 3, real-time logs can be observed during the task execution process, and the real-time status of the task execution can be synchronized, thereby improving the efficiency of information synchronization; at the same time, it is convenient for the task management unit 1 to intervene in the task execution backend in time when the task execution is abnormal, thereby solving the problem that the information synchronization method in the prior art adopts a polling mechanism, causing information delays and execution system performance losses.

[0049] like Figure 2 As shown, in this embodiment, the communication mechanism between the task execution backend 3 and the task management module 1 includes the following steps:

[0050] 1) After the task execution backend 3 is started, a TCP long connection is established with the task management module 1 according to the address of the task management module 1 recorded in the configuration file;

[0051] 2) After the TCP connection with the task management module 1 is successfully established, the status information of the task execution backend 3 is pushed to the task management module 1;

[0052] 3) The task management module 1 registers the status information of the task execution backend 3 and pushes the request address of the task scheduling module 2 to the task execution backend 3;

[0053] 4) The task execution backend 3 sends a request to the request address of the task scheduling module 2 to obtain and execute the task scheduling object;

[0054] 5) After the task execution backend 3 is terminated, the TCP connection with the task management module 1 is disconnected, and the task management module 1 updates the status information of the task execution backend 3.

[0055] In a specific implementation, the task management module 1 includes a log processing module 11, and the log processing module 11 and the task execution backend 3 have a TCP long connection.

[0056] The log processing module 11 is configured to archive the real-time log information corresponding to the task scheduling object after the task scheduling object is executed.

[0057] Specifically, the user can view the real-time log information corresponding to the task scheduling object through the log processing module 11, realizing the real-time observation of the tasks executed by the task execution backend 3. And the log processing module can also archive the logs of the completed tasks, improving the reading performance of the historical logs.

[0058] In a specific implementation, it also includes a first server 4 and a second server 5, the first server 4 is used to receive and store log information transmitted by the log processing module 11, and the second server 5 is used to receive remote execution commands sent by the target backend to execute task scheduling objects.

[0059] Among them, the log processing module of the first server transmits log information to the first server based on the SFTP protocol, and the target backend sends a remote execution command based on the SSH protocol to control the second server to execute the task scheduling object.

[0060] The log processing module is used to archive the logs of the completed tasks, thereby improving the reading performance of the historical logs. The first server receives and stores the log information transmitted by the log processing module, thereby improving the query efficiency of the historical log information.

[0061] In this embodiment, the log processing process of the task scheduling object includes the following steps:

[0062] 1) The task execution backend 3 sends the real-time log information of the task scheduling object to the log processing module 11, and the real-time log information includes the task ID, time and server IP;

[0063] 2) The log processing management module 11 receives the real-time log information, checks whether the task ID has a corresponding local temporary log file handle, and if not, creates a new temporary log file corresponding to the task ID and binds the file handle of the task ID to the relationship mapping table;

[0064] 3) After the task execution is completed, the task execution backend 3 sends a log end signal;

[0065] 4) The log processing module 11 receives the log end signaling, closes the file handle, starts the synchronization task to archive the file to the first server, and deletes the local temporary log file;

[0066] 5) During the task execution process, the log processing module 11 can be connected via websocket to read real-time log information. Every time the log processing module 11 receives a new log information, it will broadcast it via websocket and distribute the new log information to the websocket connection that views the current task ID;

[0067] 6) After the task is executed, you can directly access the static log through the first service address for viewing.

[0068] Specifically, the local temporary log file is used to store the real-time log information received by the log processing management module 11. The relationship mapping table includes the correspondence between the file handle of the task ID and the task scheduling object. The user end connects to the log processing module 11 through the websocket, and can read the temporary log file corresponding to the task scheduling object through the file handle of the task ID. Among them, the connection through the websocket communication protocol can be used for full-duplex communication between the user end and the log processing module 11. The relationship mapping table also includes the correspondence between the task scheduling object and the target backend. Through the relationship mapping table, the target backend that executes the task scheduling object task corresponding to the task scheduling object can be found to control the pause, termination or continuation of the task scheduling object.

[0069] In a specific implementation, the task scheduling module 2 is used to distribute the task scheduling object to the task execution backend 3.

[0070] The task scheduling module 2 is configured to determine whether the task depends on other tasks after obtaining the task sent by the task management module 1. When the task does not depend on other tasks, the task is placed in the task list; when the task depends on an unfinished task, the task is placed in the dependent task relationship tree. By setting up the task scheduling module 2, the execution of tasks with dependencies can be supported, thereby achieving controllability of task timing.

[0071] In a specific implementation manner, the task execution backend 3 includes multiple backends, and the multiple backends are connected to the task scheduling module 2.

[0072] The backend is set to be the target backend when the backend is in an idle state. The target backend is used to obtain and execute the task scheduling object from the task list.

[0073] In a specific implementation manner, the target backend is configured to report a status to the task management module 1 before executing the task scheduling object, and the status is that the task scheduling object is executed in the target backend.

[0074] In a specific implementation, the task management module 1 is configured to find the target backend of the task scheduling object corresponding to the task scheduling object based on the relationship mapping table, and send a control instruction to the task execution backend 3 to control the pause, termination or continuation of the task scheduling object.

[0075] In a specific implementation manner, the target backend is configured to send a request instruction to notify the task scheduling module 2 that the task scheduling object has been completed when the task scheduling object is executed and the task scheduling object is dependent on the first task.

[0076] In a specific implementation, the task scheduling module 2 is configured to traverse all tasks that the first task depends on after receiving a request instruction. If all tasks that the first task depends on are completed, the task scheduling module 2 moves the first task from the dependent task relationship tree to the task queue.

[0077] Specifically, the task queue is a redis task queue, and the dependent task relationship tree represents the dependency relationship between tasks. The task scheduling module 2 has many tasks. When there is a dependency between tasks, for example, B depends on A, then B can be executed only after A is executed. That is, through the dependent task relationship tree, it can be determined that A is the parent node task of B. For example, A depends on C, then C is the grandparent node task of B.

[0078] Figure 3 Flow chart of the task scheduling execution process. In this embodiment, the task scheduling object is task A, the first task is task B, and the scheduling execution process of the real-time task specifically includes the following steps:

[0079] 1) Task management module 1 creates a real-time task and sends it to task scheduling module 2;

[0080] 2) Task scheduling module 2 obtains the task. If the task does not depend on other tasks, it is directly placed in the redis task queue; if the task depends on other unfinished tasks, it is placed in the dependent task relationship tree;

[0081] 3) When the target backend in the task execution backend 3 is in an idle state, it requests a new task from the task scheduling module 2. The scheduling module consumes task A from the redis task queue and returns it to the target backend. The target backend obtains task A, reports the status to the task management module 1, and starts execution immediately.

[0082] 4) If task A is completed, the task management module 1 is notified of the task execution status through a TCP long connection;

[0083] 5) If task A is completed and is dependent on other tasks, then the task scheduling module 2 is notified through an http request that task A has been completed;

[0084] 6) Task scheduling module 3 receives the notification that task A has been executed, and compares the dependent task relationship tree. If task A is executed successfully, it proceeds to step 7), otherwise it proceeds to step 8);

[0085] 7) Traverse task B that depends on task A. If all tasks that task B depends on have been successfully executed, remove task B from the dependency tree and add it to the redis task queue. Otherwise, skip the processing.

[0086] 8) Traverse task B that depends on task A. If task B and all its grandparent node tasks are removed from the dependency tree, mark the status as "execution condition not met" and report the status to task management module 1.

[0087] It should be noted that the above sequence of the embodiments of this specification is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps loaded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0089] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A controllable real-time task remote execution system, characterized in that: It includes a task management module, a task scheduling module, a task execution backend and a first server; The task management module is connected to the task scheduling module, and the task management module is used to create tasks and send them to the task scheduling module; The task scheduling module is connected to the task execution backend, and the task execution backend is used to obtain and execute task scheduling objects; The task execution backend is based on the TCP listening address of the task management module recorded in the configuration file and the TCP long connection of the task management module; the task execution backend includes multiple backends, and the multiple backends correspond to executing multiple task scheduling objects; The task execution backend is used to transmit the real-time status and / or real-time log information of the task scheduling object execution to the task management module; The task management module records each task scheduling object corresponding to the backend execution as a relationship mapping table and stores it in the memory; The task management module finds the target backend for executing the task scheduling object corresponding to the task scheduling object based on the relationship mapping table; the target backend is the backend in an idle state; the task execution backend sends a control instruction to the task management module to control the pause, termination or continuation of the task scheduling object; Wherein, the task management module includes a log processing module; the log processing module is used to archive the real-time log information corresponding to the task scheduling object after the execution of the task scheduling object is completed; the real-time log information includes the task ID and / or time and / or server IP; the log processing management module receives the real-time log information, checks whether the task ID has a corresponding local temporary log file handle, and if not, creates a new temporary log file corresponding to the task ID, and binds the file handle of the task ID to the relationship mapping table; Wherein, the local temporary log file is used to store the real-time log information received by the log processing management module; the relationship mapping table includes the correspondence between the file handle of the task ID and the task scheduling object, and the correspondence between the task scheduling object and the target backend; The first server is used to receive and store the log information transmitted by the log processing module.

2. The real-time task remote execution system according to claim 1, characterized in that: It also includes a second server (5) for receiving a remote execution command sent by the target backend to execute the task scheduling object.

3. The real-time task remote execution system according to claim 1, characterized in that: The task scheduling module is used to distribute task scheduling objects to the task execution backend, The task scheduling module is configured to determine whether a task depends on other tasks after obtaining the task sent by the task management module, and to put the task into a task list when the task does not depend on other tasks, and to put the task into a dependent task relationship tree when the task depends on an unfinished task; the target backend obtains and executes the task scheduling object from the task list.

4. The real-time task remote execution system according to claim 3 is characterized in that: The target backend is configured to report a status to the task management module before executing the task scheduling object, wherein the status is that the task scheduling object is executed on the target backend.

5. The real-time task remote execution system according to claim 3, characterized in that: The target backend is configured to send a request instruction to notify the task scheduling module that the task scheduling object has been completed when the task scheduling object is executed and the task scheduling object is dependent on the first task.

6. The real-time task remote execution system according to claim 5, characterized in that: The task scheduling module is configured to traverse all tasks that the first task depends on after receiving the request instruction. If all tasks that the first task depends on are completed, the task scheduling module moves the first task from the dependent task relationship tree to the task queue.

Images