Job flow implementation method and scheduling system based on MQ and relational database
By adopting the operation flow implementation method based on MQ and relational databases in the banking system, the problem that the existing technology cannot support complex task scheduling at the same time is solved, and the automatic operation and efficient scheduling of the operation flow is realized.
Patent Information
- Application Number
- CN202210425439.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-04-21
AI Technical Summary
The existing technology is difficult to support synchronous, asynchronous tasks, job breakpoint continuation and failed task jumps at the same time, and cannot meet the complex operational needs in the banking system.
The job flow implementation method based on MQ and relational database is adopted to realize automatic operation and flexible scheduling of the job flow through message queue communication between the job trigger module, the job scheduling module and the task execution module.
It realizes the automatic operation of job flow, supports synchronous, asynchronous tasks, breakpoint continuation and failed task jumps, improving job efficiency and flexibility.
Smart Images

Figure CN115048195B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of job scheduling control, and in particular to a job flow realization method and a scheduling system based on MQ and a relational database. Background Art
[0002] In banking systems, it is often necessary to complete a set of transactions (tasks) in sequence on an operation object (such as a text).
[0003] After searching, Chinese patent CN112613840B discloses a method for job scheduling control, which includes: in response to a task execution instruction, obtaining a preset workflow task status table; determining the job status of the task to be executed according to the task execution instruction and the task status table; determining the job scheduling order according to the job status of the task to be executed; and controlling the task to be executed to execute the task to be executed according to the job scheduling order. The invention determines the job scheduling order through the job status information of the task to be executed, and the job scheduling order can be determined by the task execution instruction and the preset workflow task status table, thereby obtaining the job scheduling of the task to be executed. The invention obtains the job scheduling from the job status information, and executes the task to be executed according to the job scheduling order, which can reduce and shorten the task operation time and improve the operation efficiency.
[0004] However, the above technologies are difficult to support synchronous and asynchronous tasks, job breakpoint resumption and failed task jump at the same time. It is urgent to design a job dispatching system and job flow implementation method to realize the above functions. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a job flow implementation method and scheduling system based on MQ and relational database. The method can realize automatic operation of the job flow by simply configuring various tasks to be executed sequentially, and can simultaneously support synchronous and asynchronous tasks, job breakpoint continuation and failed task jump.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] According to a first aspect of the present invention, a data flow implementation method based on MQ and a relational database is provided, the method comprising the following steps:
[0008] Step S1, the job trigger module initiates a new task and writes the MQ message corresponding to the first task in the configured job schedule into the task queue;
[0009] Step S2, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, finds the position of the MQ message task in the job schedule, writes the execution MQ message to the execution queue, and updates and registers the new task according to the execution status, and initiates the next task MQ;
[0010] Step S3: The task execution module reads the MQ message from the execution queue, and after the execution is completed, replies with the execution result synchronously or asynchronously.
[0011] Preferably, step S2 includes the following sub-steps:
[0012] Step S21, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, and forks the child process for processing;
[0013] Step S22, the subprocess parses the read MQ message according to the job scheduling interface, and performs a null value judgment on the task ID in the task execution record table; if the task ID is null, query the first task information of the corresponding job in the job schedule table according to the job number, and then go to step S26, otherwise, go to step S23;
[0014] Step S23, determine whether there is a corresponding task in the task execution record table, if not, it is regarded as an invalid MQ message, and go to step S27; if there is a corresponding task in the task execution record table, go to step S24;
[0015] Step S24, determine whether the reply item content is a reply message, if so, go to step S25; if not, write the MQ message in the execution queue according to the task execution interface. At this time, if the task queried from the job schedule is a synchronous execution task, read the reply message from the reply queue, parse the MQ message according to the job scheduling interface, and go to step S25, otherwise go to step S27;
[0016] Step S25, change the status of the task, and query the next task information or failed task information from the job schedule according to the current status, and then go to step S26;
[0017] Step S26, generate a task ID for the new task, insert the new task into the task execution record table, and write an MQ message in the task queue according to the job scheduling interface;
[0018] Step S27, exit the subprocess.
[0019] Preferably, the fields of the job schedule include job number, task number, task name, synchronous / asynchronous flag, and failure jump task number; wherein "job number + task number" is the primary key.
[0020] Preferably, the fields of the task execution record table include task ID, operation object name, job number, task number, task name, start time, end time and task execution status; wherein the task ID is the primary key.
[0021] Preferably, the fields corresponding to the job scheduling interface include the name of the operation object, the task ID, the job number and the reply content;
[0022] The task ID is generated by the job scheduling module sequence. The task ID of each task is unique. If the task ID is a null value, the task is the first task in the job flow. The reply content is success, failure and new task.
[0023] Preferably, the fields corresponding to the job scheduling interface include operation object name, task ID, transaction code and trigger time.
[0024] Preferably, the MQ message read by the task queue in step S2 includes a first MQ message written by the job triggering module, a second MQ message written by the job scheduling module, and a third MQ message written by the task execution module;
[0025] The first MQ message is used to trigger the first task of the workflow; the second MQ message is used to trigger a new task that has been recorded but not yet executed; and the third MQ message is an asynchronous task reply message.
[0026] Preferably, the specific transactions executed by the task execution module include extensible C, Java and Shell scripts.
[0027] Preferably, if an abnormal interruption of the work flow occurs, the MQ message is rewritten into the task queue through an external program, and the corresponding task is re-executed.
[0028] According to a second aspect of the present invention, there is provided a job scheduling system based on MQ and a relational database, comprising:
[0029] The job trigger module is used to initiate a new task and write the first task MQ message in the job schedule into the task queue;
[0030] The job scheduling module is used to read the MQ message from the task queue, find the position of the MQ message task in the job schedule, write the execution MQ message to the execution queue, and update the registration of the new task according to the execution status, and initiate the next task MQ;
[0031] The task execution module is used to read MQ messages from the execution queue and reply to the execution results synchronously or asynchronously after the execution is completed.
[0032] Compared with the prior art, the present invention has the following advantages:
[0033] 1) The job flow implementation method designed by the present invention realizes the automatic operation of the job flow by simply configuring various tasks to be executed in sequence, and supports synchronous tasks, asynchronous tasks, failed task jumps, and job abnormal breakpoint continuation;
[0034] 2) The method of the present invention is based on MQ and relational database, and each module performs input and output through queues, with low coupling and strong portability;
[0035] 3) The job schedule and task execution record table of the present invention are configurable, and the configuration between modules is simple, which reduces the development workload. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Implementation process for the workflow of the present invention Figure 1 ;
[0037] Figure 2 Implementation process for the workflow of the present invention Figure 2 . DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0039] Example
[0040] Terminology explanation:
[0041] Job: A job contains multiple tasks. When all tasks are completed, the job is completed.
[0042] Synchronous task: After the job scheduler initiates a task (writes an MQ message in the queue), it still needs to wait for the execution result of the task. This task is called a synchronous task. If the task execution time is short, it can be designed as a synchronous task;
[0043] Asynchronous task: After the job scheduler initiates a task (writes an MQ message in the queue), it can continue to perform other operations without waiting for the execution result of the task. This task is called an asynchronous task. Since the job scheduler does not need to wait for the execution result of the asynchronous task and can release resources immediately, some tasks that take a long time to execute can be designed as asynchronous.
[0044] First, a system embodiment of the present invention is given, which is a job flow scheduling system based on MQ and relational database, including a job triggering module, a job scheduling module and a task execution module. The modules communicate with each other through MQ messages, specifically:
[0045] Job trigger, used to initiate a new task and write the first task MQ message in the job schedule into the task queue;
[0046] The job scheduling module is used to read the MQ message from the task queue, find the position of the MQ message task in the job schedule, write the execution MQ message to the execution queue, update the registration of the new task according to the execution status, and initiate the next task MQ;
[0047] The task execution module is used to read MQ messages from the execution queue. After the execution is completed, the execution results are replied synchronously or asynchronously based on MQ and relational database.
[0048] The job schedule is configured with tasks that need to be executed sequentially.
[0049] Next, a method embodiment of the present invention is given, which is a data flow implementation method based on MQ and a relational database. The method comprises the following steps:
[0050] Step S1, the job trigger module initiates a new task and writes the MQ message corresponding to the first task in the configured job schedule into the task queue;
[0051] Step S2, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, finds the position of the MQ message task in the job schedule, writes the execution MQ message to the execution queue, and updates and registers the new task according to the execution status, and initiates the next task MQ, specifically:
[0052] Step S21, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, and forks the child process for processing;
[0053] Step S22, the subprocess parses the read MQ message according to the job scheduling interface, and performs a null value judgment on the task ID in the task execution record table; if the task ID is null, query the first task information of the corresponding job in the job schedule table according to the job number, and then go to step S26, otherwise, go to step S23;
[0054] Step S23, determine whether there is a corresponding task in the task execution record table, if not, it is regarded as an invalid MQ message, and go to step S27; if there is a corresponding task in the task execution record table, go to step S24;
[0055] Step S24, determine whether the reply item content is a reply message, if so, go to step S25; if not, write the MQ message in the execution queue according to the task execution interface. At this time, if the task queried from the job schedule is a synchronous execution task, read the reply message from the reply queue, parse the MQ message according to the job scheduling interface, and go to step S25, otherwise go to step S27;
[0056] Step S25, change the status of the task, and query the next task information or failed task information from the job schedule according to the current status, and then go to step S26;
[0057] Step S26, generate a task ID for the new task, insert the new task into the task execution record table, and write an MQ message in the task queue according to the job scheduling interface;
[0058] Step S27, exit the subprocess;
[0059] Step S3: The task execution module reads the MQ message from the execution queue, and after the execution is completed, replies with the execution result synchronously or asynchronously.
[0060] The following is combined with Figure 1 and Figure 2 The method of the present invention is described in detail.
[0061] [MQ Queue X: QUEUE_JOB]: The job scheduling module process monitors this queue. If there is a message, it forks a child process to process it. The messages in this queue come from three sources: MQ message 1 (written by the job trigger module to trigger the first task in the job flow), MQ message 2 (written by the job scheduling process to trigger a new task that has been recorded but not yet executed), and MQ message 3 (the reply message of the asynchronous task written by the task execution module)
[0062] [MQ Queue Y: QUEUE_EXE_REQ]: The task execution module monitors the queue. If there is a message, the corresponding transaction in the message is executed.
[0063] [MQ Queue Z: QUEUE_EXE_RSP]: For synchronous tasks, after the task execution is completed, the execution module will write the reply message to this queue.
[0064] [Data structure A - job scheduling interface]: operation object name, TASKID (generated by the job scheduling module sequence, each task has a unique TASKID. If it is empty, it is the first task in the job flow), JOBNO (job number), reply content (SUCC - success, FAIL - failure, empty - new task)
[0065] [Data Structure B-Task Execution Interface]: Operation object name, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS)
[0066]
Table 1-Task Execution Record Table
[0067] Note: Record each specific task ID, execution date, time, status, operation object, corresponding job number and task number
[0068] Fields: Task ID (TASKID, primary key), operation object name, job number, task number, task name, start time, end time, task execution status (S-success, F-failure, I-processing)
[0069] Example: The operation object of this embodiment is the text abc.xml
[0070] Table 1
[0071]
[0072]
Table 2-Work Schedule
[0073] Note: All tasks that need to be executed are configured for different jobs. Tasks are divided into synchronous and asynchronous tasks. For synchronous tasks, the job scheduling module needs to wait for the execution result; for asynchronous tasks, the job scheduling subprocess exits after the execution is triggered, and does not occupy system resources.
[0074] Fields: job number, task number, task name, synchronous / asynchronous flag, and failed jump task number; the job number + task number is the primary key; if the task is successfully executed, the next task will be executed in sequence; if the execution fails, the new task will be the failed jump number task.
[0075] Example: The tasks 0001-text splitting and 0005-account entry take a long time, so they are set as asynchronous tasks; if the execution of 0001-text splitting fails, it jumps to the task 0006-text reply.
[0076] Table 2
[0077] Job Number Task Number Task Name Synchronous and asynchronous flag Failed jump task number JOB0001 0001 Text Splitting asynchronous 0006 JOB0001 0002 answer synchronous JOB0001 0003 Missing information supplement synchronous JOB0001 0004 Type Identification synchronous JOB0001 0005 Credit asynchronous JOB0001 0006 Text Reply synchronous
[0078] [Job Trigger]: Write MQ message to [MQ Queue X: QUEUE_JOB], including: operation object name (text name), TASKID (empty, triggering the first task of the job, TASKID has not been generated yet), reply content (empty)
[0079] [Job trigger]: The resident process monitors [MQ queue X: QUEUE_JOB]. If there is a message, it forks a child process to process it.
[0080] [Job trigger (subprocess)]: Parse the MQ message and split it into object name, TASKID, JOBNO, and reply content.
[0081] (Branch 1) If TASKID is empty, generate a new TASKID based on the first task of JOBNO in [Table 2 - Job Schedule], insert it into [Table 1 - Task Execution Record Table], and write the MQ message to [MQ Queue X: QUEUE_JOB].
[0082] (Branch 2.2) If TASKID is not empty and there is no corresponding task in [Table 1-Task Execution Record Table], it is regarded as an invalid MQ message and the child process exits.
[0083] (Branch 2.1.1) If TASKID is not empty and the corresponding task can be found in [Table 1-Task Execution Record Table], and the MQ message is a reply message, then update [Table 1-Task Execution Record Table] according to the reply status, and sequentially search for the next execution task in [Table 2-Job Schedule Table] (the reply status is success), or search for the failed jump task of the task from [Table 2-Job Schedule Table] (the reply status is failure), generate TASKID for the new task, and insert the relevant information of the new task into [Table 1-Task Execution Record Table], and finally write the MQ message to [MQ Queue X: QUEUE_JOB].
[0084] (Branch 2.1.2.1) If TASKID is not empty, the corresponding task can be found in [Table 1-Task Execution Record Table], the MQ message is not a reply message, and the task (JOBNO+TASKNO) is a synchronous execution task from [Table 2-Job Schedule Table], then write an MQ message to [Queue Y: QUEUE_EXE_REQ], including: operation object name, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS), and monitor [Queue Z: QUEUE_EXE_RSP] to read the reply of the task execution module. According to the reply status, update [Table 1-Task Execution Record Table], search for the next execution task in [Table 2-Job Schedule Table] (the reply status is success), or search for the failed jump task of the task from [Table 2-Job Schedule Table] (the reply status is failure), generate TASKID for the new task, and insert the relevant information of the new task into [Table 1-Task Execution Record Table], and finally write an MQ message to [MQ Queue X: QUEUE_JOB].
[0085] (Branch 2.1.2.2) If TASKID is not empty, the corresponding task can be found in [Table 1 - Task Execution Record Table], the MQ message is not a reply message, and it is found from [Table 2 - Job Schedule Table] that the task (JOBNO+TASKNO) is an asynchronous execution task, then write an MQ message to [Queue Y: QUEUE_EXE_REQ], including: operation object name, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS), and exit the child process.
[0086] [Task execution]: The resident process listens to [MQ queue Y: QUEUE_EXE_REQ], and parses the operation object name, TASKID, transaction code, etc. from the message. Execute the corresponding task according to the transaction code. And generate a reply MQ message. For synchronous transactions, the message is written to [MQ queue Z: QUEUE_EXE_RSP], otherwise it is written to [MQ queue X: QUEUE_JOB]
[0087] Note: The specific transactions executed by this module can be C, Java, shell scripts, etc., which is extensible.
[0088] This application mainly describes the method for implementing the workflow. The specific transactions performed are not the core content of this article and will not be described in detail here.
[0089] If the job flow is abnormally interrupted, the message can be rewritten in [MQ queue X: QUEUE_JOB] through an external program to re-execute the corresponding task. The MQ message only needs to comply with [Data Structure A-Job Scheduling Interface]. For the specific implementation of the external program, refer to the existing technology.
[0090] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A data flow implementation method based on MQ and relational database, characterized in that: The method comprises the following steps: Step S1, the job trigger module initiates a new task and writes the MQ message corresponding to the first task in the configured job schedule into the task queue; the fields of the job schedule include job number, task number, task name, synchronous / asynchronous flag and failure jump task number, where "job number + task number" is the primary key; Step S2, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, finds the position of the MQ message task in the job schedule, writes the execution MQ message to the execution queue, and updates and registers the new task according to the execution status, and initiates the next task MQ, which specifically includes the following sub-steps: Step S21, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, and forks the child process for processing; wherein the MQ message read by the task queue includes the first MQ message written by the job triggering module, the second MQ message written by the job scheduling module, and the third MQ message written by the task execution module; the first MQ message is for triggering the first task of the job flow; the second MQ message is for triggering a new task that has been recorded but not yet executed; the third MQ message is an asynchronous task reply message; Step S22, the subprocess parses the read MQ message according to the job scheduling interface, and performs a null value judgment on the task ID in the task execution record table; if the task ID is null, query the first task information of the corresponding job in the job schedule table according to the job number, and then go to step S26, otherwise, go to step S23; Step S23, determine whether there is a corresponding task in the task execution record table, if not, it is regarded as an invalid MQ message, and go to step S27; if there is a corresponding task in the task execution record table, go to step S24; Step S24, determine whether the reply item content is a reply message, if so, go to step S25; if not, write the MQ message in the execution queue according to the task execution interface. At this time, if the task queried from the job schedule is a synchronous execution task, read the reply message from the reply queue, parse the MQ message according to the job scheduling interface, and go to step S25, otherwise go to step S27; Step S25, change the status of the task, and query the next task information or failed task information from the job schedule according to the current status, and then go to step S26; Step S26, generate a task ID for the new task, insert the new task into the task execution record table, and write an MQ message in the task queue according to the job scheduling interface; Step S27, exit the subprocess; Step S3: The task execution module reads the MQ message from the execution queue, and after the execution is completed, replies with the execution result synchronously or asynchronously.
2. A data flow implementation method based on MQ and relational database according to claim 1, characterized in that: The fields of the task execution record table include task ID, operation object name, job number, task number, task name, start time, end time and task execution status; wherein the task ID is the primary key.
3. The data flow implementation method based on MQ and relational database according to claim 1 is characterized in that: The fields corresponding to the job scheduling interface include the operation object name, task ID, job number and reply content; The task ID is generated by the job scheduling module sequence. The task ID of each task is unique. If the task ID is a null value, the task is the first task in the job flow. The reply content is success, failure and new task.
4. The data flow implementation method based on MQ and relational database according to claim 1, characterized in that: The fields corresponding to the job scheduling interface include operation object name, task ID, transaction code and trigger time.
5. The data flow implementation method based on MQ and relational database according to claim 1 is characterized in that: The specific transactions executed by the task execution module include extensible C, Java and Shell scripts.
6. The data flow implementation method based on MQ and relational database according to claim 1, characterized in that: If the job flow is abnormally interrupted, the MQ message is rewritten into the task queue through an external program to re-execute the corresponding task.
7. A job flow scheduling system based on MQ and relational database, characterized in that: include: The job trigger module is used to initiate a new task and write the first task MQ message in the job schedule into the task queue. The fields of the job schedule include job number, task number, task name, synchronous / asynchronous flag and failure jump task number, where "job number + task number" is the primary key; The job scheduling module is used to read the MQ message from the task queue, find the position of the MQ message task in the job schedule, write the execution MQ message to the execution queue, and update the registration of the new task according to the execution status, and initiate the next task MQ. The specific implementation process includes: 1) The parent process in the job scheduling module monitors the task queue in real time, reads MQ messages from the task queue, and forks the child process for processing; wherein the MQ messages read by the task queue include the first MQ message written by the job triggering module, the second MQ message written by the job scheduling module, and the third MQ message written by the task execution module; the first MQ message is used to trigger the first task of the job flow; the second MQ message is used to trigger a new task that has been recorded but not yet executed; the third MQ message is an asynchronous task reply message; 2) The child process parses the read MQ message according to the job scheduling interface, and makes a null value judgment on the task ID in the task execution record table; if the task ID is null, it searches the job schedule table for the first task information of the corresponding job according to the job number, and then goes to the task execution module; otherwise, it goes to 3); 3) Determine whether there is a corresponding task in the task execution record table. If not, it is considered as an invalid MQ message and go to 7); if there is a corresponding task in the task execution record table, go to 4); 4) Determine whether the reply item content is a reply message. If so, go to 5); if not, write the MQ message in the execution queue according to the task execution interface. At this time, if the task queried from the job schedule table is a synchronous execution task, read the reply message from the reply queue, parse the MQ message according to the job scheduling interface, and go to 5); otherwise, go to 7); 5) Change the status of the task, and query the next task information or the failed task information from the job schedule according to the current status, and go to 6); 6) Generate a task ID for the new task, insert the new task into the task execution record table, and write an MQ message in the task queue according to the job scheduling interface; 7) Exit the child process; The task execution module is used to read MQ messages from the execution queue and reply to the execution results synchronously or asynchronously after the execution is completed.
Citation Information
Patent Citations
A job scheduling control method, apparatus, equipment, and storage medium
CN112613840B
Task scheduling method and system
CN102981904A
Task scheduling method and system based on big data service
CN112667375A
