Workflow engine task data synchronization method and system and storage medium

Abstract

According to the workflow engine task data synchronization method disclosed by the invention, the data change in the business process is analyzed by utilizing the memory cache of the workflow engine, the changed to-be-synchronized data is processed into a specific format, then the changed to-be-synchronized data is persisted into a database along with the business process, and then the changed to-be-synchronized data is packaged into an independent executable task to be submitted to a thread pool. According to the method, independent operation is synchronously carried out on task data of different business processes, mutual influence is avoided, and the stability is high. When the independent executable tasks are executed, the data to be synchronized are pushed to a message queue, a data source or a service system, and asynchronous synchronization of task data is achieved. According to the method, the data of the workflow is transferred to other systems so as to share the query pressure and partial functions of the workflow engine, the operation and maintenance cost is reduced, and the working efficiency of the workflow engine is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and provides a task data synchronization method, storage medium and system for a workflow engine. Background Art

[0002] The workflow engine is the core technology of business process management. It exchanges data between participants in the business process and assigns tasks to different executors. As part of the business system, the workflow engine provides a decisive core solution for the business system.

[0003] After the workflow engine is put into operation, with the gradual increase in data volume and more and more business systems connected to the workflow engine, the growth of data tables leads to worse and worse task data synchronization performance of the workflow engine. The existing technology embeds new business logic into the workflow engine to improve the efficiency of task data synchronization. For example, the method of collecting task data updates by embedding the workflow engine interface is prone to omissions and program defects, and increases the development cost of the workflow engine and the difficulty of subsequent maintenance and development; for another example, the method of using a database log scanning plug-in will introduce other components to the system, increase system complexity and instability, and increase the difficulty of operation for operation and maintenance personnel.

[0004] Therefore, there is an urgent need for a workflow engine task data synchronization method that does not embed business logic to stably and efficiently improve the efficiency of task data synchronization. Summary of the invention

[0005] Purpose of the invention: In order to solve the problems of poor stability and low efficiency of existing workflow engine task data synchronization technology, the present invention provides a workflow engine task data synchronization method that does not embed business logic.

[0006] Technical solution: A workflow engine task data synchronization method, comprising the following steps:

[0007] Register a command interceptor to the workflow engine. After the business process of the workflow engine is executed, the command interceptor obtains the cache entity in the command processing thread, analyzes the data in the cache entity, and obtains one or more data synchronization tasks; registers the processing class corresponding to the data synchronization task in the workflow engine, and persists the data synchronization task to the database together with other data in the command processing thread;

[0008] The task query thread queries the data synchronization tasks in the database, locks the data synchronization tasks that are created earliest in each business process and are not locked by other service nodes, and adds the data synchronization tasks to the task processing thread pool;

[0009] The task processing thread pool obtains the processing class instance corresponding to the data synchronization task according to the processing class name bound to the data synchronization task, calls the execution method of the processing class to obtain the data to be synchronized, and sends the data to be synchronized to the message queue, data source or business system to complete the synchronization; if the task is executed successfully, the task is deleted from the database; if the execution fails, the reason is recorded and waits for retry.

[0010] Furthermore, the workflow engine uses a delayed write method to update data. Before the data is written to the database, all data is stored in the corresponding memory cache; the command interceptor obtains the cache entity after the request processing of the workflow engine is completed, and extracts the cached entity data from the memory cache of the command processing thread.

[0011] Furthermore, the analysis of the data in the cache entity first searches for modified task entity data and business process entity data in the entity data, then segments the modified entity data according to the business process instances, and finally packages the segmented formatted data into one or more data synchronization tasks.

[0012] Furthermore, the data synchronization task includes a corresponding business process instance ID, a creation time, a task processing method name, and a planned execution time of the task.

[0013] Furthermore, the task query thread periodically queries the database, first groups the data synchronization tasks in the database according to the task type and the business process instance to which the task belongs, then selects the task with the earliest creation time in each group, and finally filters out the unlocked data synchronization tasks and returns them to the task query thread.

[0014] Furthermore, the task query thread will sequentially lock the unlocked data synchronization tasks returned from the database, and add the successfully locked data synchronization tasks to the task processing thread pool.

[0015] Furthermore, after the data synchronization task is submitted to the task processing thread pool, the task processing thread pool processes the data synchronization task, including the following steps:

[0016] First, the data processing interface of the workflow engine is called to obtain the name of the processing method associated with the data synchronization task, and the processing instance is obtained from the hash map of the processing method name and the corresponding processing instance predefined by the workflow engine;

[0017] By calling the data synchronization interface of the workflow engine, the task is handed over to different types of processing instances for processing. The processing instance attempts to send the data to the message queue, data source or business system and then returns normally;

[0018] The above operations do not throw exceptions and delete the corresponding data synchronization tasks in the database.

[0019] Furthermore, if an error occurs in the processing operation of the task processing thread pool, the failure reason of the data synchronization task is recorded, a new execution time is assigned to the data synchronization task and the lock is released, and the workflow engine persists the update of the data synchronization task to the database.

[0020] The present invention also discloses a computer system, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0021] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0022] Beneficial effects: The present invention discloses a workflow engine task data synchronization method, which uses the memory cache of the workflow engine to analyze the data changes in the business process, processes the changed data to be synchronized into a specific format, and then persists it in the database along with the business process, and then packages it into an independent executable task and submits it to the task processing pool. This method performs independent operations on the task data synchronization of different business processes without affecting each other, and has good stability.

[0023] When independent executable tasks are executed, the data to be synchronized is pushed to the message queue, data source or business system to achieve asynchronous synchronization of task data. This method transfers workflow data to other systems to share the query pressure and some functions of the workflow engine, reduce operation and maintenance costs, and effectively improve the work efficiency of the workflow engine.

[0024] In summary, a workflow engine task data synchronization method of the present invention achieves near real-time workflow engine task data synchronization while ensuring data consistency, and is a task data synchronization method with good stability and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the steps of a workflow engine task data synchronization method of the present invention;

[0026] Figure 2 It is a flow chart of a workflow engine task data synchronization system according to an embodiment of the present invention;

[0027] Figure 3 is a structural diagram of the interaction between the workflow engine and other services in an embodiment of the present invention;

[0028] Figure 4 It is a graph of the interaction between different threads of the workflow engine in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are not intended to limit the scope of protection of the present invention. Figure 1 FIG. 1 is a schematic diagram showing the steps of a workflow engine task data synchronization method according to the present invention. Figure 2 The flowchart of a workflow engine task data synchronization system corresponding to the above method is shown. In this embodiment system, the above method is further refined into S01 to S11.

[0030] Figure 1 The steps of the method of the present invention are described as follows:

[0031] Register the command interceptor to the workflow engine. After the business process of the workflow engine is executed, the command interceptor obtains the cache entity in the command processing thread, analyzes the data in the cache entity, and obtains one or more data synchronization tasks; registers the processing class corresponding to the data synchronization task in the workflow engine, and persists the data synchronization task along with other data in the command processing thread to the database; this step corresponds to Figure 2 S01 to S07.

[0032] The task query thread queries the data synchronization tasks in the database, locks the earliest created data synchronization task in each business process that has not been locked by other service nodes, and adds the data synchronization task to the task processing thread pool; this step corresponds to Figure 2 S08.

[0033] The task processing thread pool obtains the processing class instance corresponding to the data synchronization task according to the processing class name bound to the data synchronization task, calls the execution method of the processing class to obtain the data to be synchronized, and sends the data to be synchronized to the message queue, data source or business system to complete the synchronization; if the task is successfully executed, the task is deleted from the database; if the execution fails, the reason is recorded and waits for retry. This step corresponds to Figure 2 S09 to S11.

[0034] Figure 4 The interactive model of the service request thread, synchronization task query thread and task execution thread pool in the embodiment of the present invention is shown. In this embodiment, it is shown that the generation, sending scheduling and execution of the process data synchronization task are all completed by different threads in the workflow engine.

[0035] This embodiment system integrates the Activiti workflow engine, and also includes the necessary functional modules such as command interception processing, data synchronization task query and task execution module required for the implementation of the present invention. Activiti is an engine that abstracts the business from a specific workflow based on a series of rules and definitions to achieve the decoupling of specific business logic and workflow. The Activiti workflow engine has been widely redeveloped and integrated as the underlying support platform of the business system with its open source code and complete documentation.

[0036] Next, combine Figure 2 The present invention is described in detail.

[0037] S01. When the system of this embodiment is started, a command interceptor is registered. The system of this embodiment adds an additional command interceptor to the workflow engine as a data synchronization interceptor to obtain the entity cache of the current thread of the workflow engine and collect data change information after the process flow logic is executed, generate synchronization tasks and persist them together in the database. Figure 3 This corresponds to the interaction mode between the system and the business system during deployment of the system in this embodiment.

[0038] When the system of this embodiment is started, it is also necessary to register corresponding cache data processing classes and synchronization task processing classes for different data synchronization strategies according to the configuration information in the system configuration file. Different data synchronization strategies correspond to different data processing requirements, such as sending task data to a message queue, synchronizing task data to Elasticsearch, or directly sending data to a business system.

[0039] The cache data processing class is used to process cache data and generate data synchronization tasks when the command interceptor intercepts commands; the synchronization task processing class is used to execute synchronization logic to complete data synchronization of data payloads. The synchronization processing strategy, cache data processing class, and synchronization task processing class correspond to each other, as follows:

[0040] First, the cache data processing class is bound to the data synchronization interceptor, and the data synchronization interceptor is configured in the Activiti workflow engine through the third-party command interceptor interface provided by the Activiti workflow engine before the workflow engine is started. The Activiti workflow engine packages the implementation of a series of process functions into commands; when executing commands, Activiti defines a series of command interceptors wrapped around the periphery of command execution to implement functions such as log printing, memory caching, and transaction control. In order to obtain the data modification in the memory after the command execution is completed but before the data is persisted to the database, the data synchronization interceptor designed by the present invention is executed before the business process data is persisted to the database to ensure that the data modification can be obtained. Each cache data processing class is responsible for processing and generating synchronization data in a specific format, and packaging the synchronization data as asynchronous tasks, which are persisted to the database together with the command processing thread in the business process. This avoids synchronizing the data to the third-party system before the database transaction is successfully submitted, and at the same time avoids data loss caused by service downtime.

[0041] Subsequently, when the system of this embodiment is started, the data synchronization task processing class is registered to the system of this embodiment, and each processing class represents a data synchronization strategy.

[0042] During the system startup process of this embodiment, a task query thread for obtaining data synchronization tasks and a task processing thread pool for processing data synchronization tasks are started at the same time. In order to distinguish it from the native asynchronous tasks of Activiti, the task query thread uses the background thread implemented by the system of this embodiment, and the task type is set to push type. Figure 4 The interactive model of the service request thread, synchronization task query thread and task execution thread pool in the embodiment of the present invention is shown. In this embodiment, it is shown that the generation, sending scheduling and execution of the process data synchronization task are all completed by different threads in the workflow engine.

[0043] S02-S03, the data synchronization interceptor intercepts the command execution process of the workflow engine. After the business process command is executed, check whether there are modified entities in the entity cache. If there are no modified entities, it means that there is no data to be synchronized, skip S04-S06 and go directly to S07; if there are modified entities, go to step S04.

[0044] S04. The data synchronization interceptor will check in turn whether the task entity and the business process instance entity have changed during the command execution process. The changes include creation, update, completion and deletion. If the data synchronization interceptor detects at least any of the above data changes, it will pass these change information to the registered cache data processing class.

[0045] The Activiti workflow engine puts each entity used into the cache and saves a hash table corresponding to the original state of each entity. At the same time, it temporarily stores entities that will be added and deleted in the future in another list. By checking whether the data in the cache exists in the added and deleted lists, it can be determined whether the data is newly added or about to be deleted; by checking the consistency of the data and the initial state, it can be determined whether the data is about to be updated. Through the above analysis, three entity lists can be obtained: new, updated, and deleted entities.

[0046] In this embodiment, the system divides entities according to types, selects entities related to tasks, business processes, business variables, and task authority links, and then divides the data according to the business process instances to which it belongs. Finally, one or more intermediate results of task data changes can be constructed: for a certain business process instance, which task data has changed during the execution of this request, whether the business process information has changed, and what changes have occurred in the variables associated with the business process.

[0047] S05-S06, the cache data processing class processes the data from the data synchronization interceptor and generates data objects for pushing to the message queue. The cache data processing class will create a push-type data synchronization task, fill in the associated creation time, expiration time, name of the corresponding synchronization task processing class, business process instance ID and other information, and serialize the object and store it in the large field table of the workflow engine, and use the configuration information field of the data synchronization task to associate the two. The cache data processing class uses the workflow interface to add the data synchronization task to the entity collection that needs to be inserted, and wait for persistence. Finally, for push-type asynchronous tasks, the workflow engine will create a listener after the transaction is normally submitted, so that the task query thread can be woken up immediately after the data is submitted.

[0048] If the program throws an exception in the subsequent execution, the data synchronization task and the updated workflow data will be rolled back together. Otherwise, after the transaction is committed normally, the transaction listener will immediately wake up the waiting task query thread.

[0049] S08, after the workflow engine is started, the task query thread regularly queries executable data synchronization tasks from the asynchronous task table of the database. In this embodiment, when the workflow engine is idle, the task query thread is in a sleep state most of the time. Each time the sleep exceeds 10 seconds or is manually awakened by other threads, the task query thread immediately performs a task query.

[0050] In each task query, the database groups all data synchronization tasks in the asynchronous task table according to the business process instance ID and the name of the synchronization task processing class according to the query SQL, selects the task with the earliest creation time in each group, and finally selects the unlocked part of these tasks and returns them to the workflow engine in pages. The task query thread updates the lock fields of these data synchronization tasks in turn, marking the locked nodes and lock timeout time to ensure that these tasks will not be repeatedly acquired and executed by other service nodes under cluster deployment, and submits all data synchronization tasks that are successfully locked to the task processing thread pool.

[0051] In particular, in this embodiment, after completing the above operations, the task query thread counts whether the number of tasks successfully submitted to the task processing thread pool has reached half of the upper limit of the single page size of the paging query. If it has reached half, it will not enter sleep mode, and continue to query executable data synchronization tasks until the number of successfully submitted results is less than half; otherwise, the data synchronization task acquisition thread will enter sleep mode until it is awakened by other threads at the next data submission or automatically awakened when the sleep timeout period is reached.

[0052] S09. The execution of the task submitted to the task processing thread pool will be completed by the asynchronous task execution module of the workflow engine. Each data synchronization task needs to specify the name of its own synchronization task processing class before submitting it to the workflow engine. The workflow engine finds the corresponding instance through the processing class name field of the data synchronization task and passes the data synchronization task to the instance.

[0053] S10. The synchronous task processing class instance processes the task, extracts the data payload and sends it to the message queue. After the locked task is processed, if the execution result is normal, it is deleted, otherwise the execution time is reset and re-executed. However, when the environment, middleware or program is blocked and the task cannot be processed within the specified time, the background thread will periodically reset the lock information of these tasks to ensure that these tasks can be obtained and re-executed by the task query thread again.

[0054] S11. Each data synchronization task contains the corresponding business process instance ID and processing method. For tasks that cannot be synchronized for a long time, the workflow engine provides an interface to query and clean up tasks in the synchronization table. The business system can regularly call the interface to clean up these unsynchronized tasks, obtain the instance information involved, and make some compensation operations, such as querying the status of the business process instance that cannot be synchronized for data synchronization, recording the instance ID involved, and other operations.

[0055] To perform multiple data synchronization operations simultaneously using the system of this embodiment, you only need to provide an implementation of a cache data processing class and an implementation of a processing class corresponding to the synchronization task. When the workflow engine is started, all these interface implementations will be registered in the system of this embodiment, and these implementations will be called when needed, and ensure that the data synchronization operations between different strategies do not affect each other.

[0056] Taking the above factors into consideration, the execution of asynchronous tasks in this embodiment is designed to be executed in time sequence within the business process instance, concurrent execution between data of different business process instances, and concurrent execution of data sent to different middleware or data sources. Compared with synchronization strictly based on the generation or modification time of task data, concurrent synchronization of data of different instances and different types of data can effectively increase the concurrency that the data synchronization module can support during business peak periods; further, it can reduce the delay of business data synchronization; in addition, concurrent synchronization of different types of data can effectively avoid the problem of blocking the execution of a certain type of task and affecting the synchronization of other types of data, thereby improving the stability of the workflow engine.

Claims

1. A workflow engine task data synchronization method, characterized in that: The steps include: Register a command interceptor to the workflow engine. After the business process of the workflow engine is executed, the command interceptor obtains the cache entity in the command processing thread, analyzes the data in the cache entity, and obtains one or more data synchronization tasks; registers the processing class corresponding to the data synchronization task in the workflow engine, and persists the data synchronization task in the database; The task query thread queries the data synchronization tasks in the database, locks the data synchronization tasks that are created earliest in each business process and are not locked by other service nodes, and adds the data synchronization tasks to the task processing thread pool; The task processing thread pool obtains the processing class instance corresponding to the data synchronization task according to the processing class name bound to the data synchronization task, calls the execution method of the processing class to obtain the data to be synchronized, and sends the data to be synchronized to the message queue, data source or business system to complete the synchronization; if the task is executed successfully, the task is deleted from the database; if the execution fails, the reason is recorded and waits for retry.

2. The data synchronization method according to claim 1, characterized in that: The workflow engine uses a delayed write method to update data. Before the data is written into the database, all data is stored in the corresponding memory cache. The command interceptor obtains the cache entity after the request processing of the workflow engine is completed, and extracts the cached entity data from the memory cache of the command processing thread.

3. The data synchronization method according to claim 2, characterized in that: The data in the analysis cache entity is first searched for modified task entity data and business process entity data in the entity data, and then the modified entity data is segmented according to the business process instance, and finally the formatted data obtained by segmentation is packaged into one or more data synchronization tasks.

4. The data synchronization method according to claim 3, characterized in that: The data synchronization task includes the corresponding business process instance ID, creation time, task processing method name and the planned execution time of the task.

5. The data synchronization method according to claim 4, characterized in that: The task query thread periodically queries the database, first groups the data synchronization tasks in the database according to the task type and the business process instance to which the task belongs, then selects the task with the earliest creation time in each group, and finally filters the unlocked data synchronization tasks and returns them to the task query thread.

6. The data synchronization method according to claim 5, characterized in that: The task query thread will sequentially lock the unlocked data synchronization tasks returned from the database, and add the successfully locked data synchronization tasks to the task processing thread pool.

7. The data synchronization method according to claim 6, characterized in that: After the data synchronization task is submitted to the task processing thread pool, the task processing thread pool processes the data synchronization task, including the following steps: First, the data processing interface of the workflow engine is called to obtain the name of the processing method associated with the data synchronization task, and the processing instance is obtained from the hash map of the processing method name and the corresponding processing instance predefined by the workflow engine; By calling the data synchronization interface of the workflow engine, the task is handed over to different types of processing instances for processing. The processing instance attempts to send the data to the message queue, data source or business system and then returns normally; The above operations do not throw exceptions and delete the corresponding data synchronization tasks in the database.

8. The data synchronization method according to claim 7, characterized in that: If an error occurs in the processing operation of the task processing thread pool, the failure reason of the data synchronization task is recorded, a new execution time is assigned to the data synchronization task and the lock is released, and the workflow engine persists the update of the data synchronization task in the database.

9. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claims 1 to 8 are implemented.

Images