Data automatic loading and caching method for object metadata driven process engine
By using an object metadata-driven process engine, process data is dynamically detected and loaded in batches, and changes are detected in real time. This solves the problems of multiple data loading and complex configuration in existing technologies, and achieves efficient process configuration and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI ZHENYUN INFORMATION TECH CO LTD
- Filing Date
- 2026-04-15
- Publication Date
- 2026-06-16
AI Technical Summary
The existing process engine can only use the field information of the current object when configuring complex object structures, which leads to multiple data loading, poor performance, and the need for personnel to manage data status, which increases complexity.
Through an object metadata-driven process engine, the system dynamically senses the business data required by the process, generates batch query requests, stores them in memory cache, and detects data changes in real time to automatically update the cache, thereby achieving intelligent loading and sharing of cross-object data.
It significantly reduces configuration complexity, improves process configuration efficiency and performance, ensures data consistency, reduces the number of database interactions, and increases process execution speed and system throughput.
Smart Images
Figure CN122222571A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of process engine technology for low-code systems, and more specifically, to a method for automatic data loading and caching in an object metadata-driven process engine. Background Technology
[0002] In low-code development platforms, the process engine serves as a core component for automating business processes, allowing users to define complex business logic by visually orchestrating a series of nodes such as data validation, business logic processing, and approval nodes.
[0003] Existing process engines can only automatically load data based on a single object. When configuring a process engine, only the data of the current input parameter object can be used. In actual application systems, the object structure is complex, and there are complex relationships between objects. When configuring, only the field information of the current object can be used, which greatly limits the configuration capabilities. Furthermore, the loading is done in multiple stages, resulting in numerous interactions with the database. The application system's workflow engine may have a very long process, with different nodes using different object fields. Each node loads data on demand, leading to the same object being loaded multiple times and requiring multiple interactions with the database, resulting in poor performance. At the same time, it is necessary to configure personnel to manage complex data states. Data may be updated during the execution of the process engine. After the preceding node of the process engine has loaded the data, it may be updated in the process engine. Subsequent nodes use the same data again. The configuration personnel need to decide whether to continue execution with the old data or to re-acquire the data, which increases the complexity of the configuration. Summary of the Invention
[0004] To overcome the aforementioned deficiencies of the prior art, this invention provides a method for automatic data loading and caching of a process engine driven by object metadata, in order to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical method: an automatic data loading and caching method for an object metadata-driven process engine, comprising the following steps: S1: Based on object metadata and process engine configuration metadata, dynamically perceive the business data that the process engine needs to load at startup and during node execution, including business data across related objects; S2: Automatically integrate the required business data, generate batch query requests, and execute batch data loading; S3: Store the batch-loaded data results in a memory cache for sharing among all nodes of the process engine; S4: Before executing a process node, check whether the cached data that the node depends on has changed; S5: If data changes are detected, S2 and S3 are automatically re-executed to load batch data and update the cache; if no changes are detected, data is directly read from the memory cache for use by the node.
[0006] Preferably, the dynamic sensing process in S1 includes: Parse the object field information referenced in the configuration metadata of the process engine to automatically identify cross-object relationships; Based on the object metadata structure, the field dependencies of related objects are recursively analyzed to generate a list of multi-object data loading requirements.
[0007] Preferably, S2 is executed when the process starts, and specifically includes: Read the complete configuration metadata of the process engine; Pre-analyze all objects and fields that may be accessed by all process nodes, and merge duplicate loading requirements; Generate optimized batch query statements to load all required data from the database at once and populate the cache.
[0008] Preferably, the dynamic sensing is performed in real time during node execution: When the process reaches a new node, the configuration metadata of the current node is parsed in real time. If the required data exceeds the existing cache range, the system will dynamically supplement the newly added data requirements. The new requirements are compared with the existing data in the cache, and incremental batch loading is performed only on the parts that are not cached or have expired.
[0009] Preferably, detecting whether the cached data on which the node depends has changed in step S4 specifically includes: After a process node is executed, monitor and record the data update operations performed by the process engine through its configuration metadata, object metadata, or the application orchestrated by the process. Based on the update operation, the affected data objects are automatically marked as invalid. Before executing subsequent nodes, check the failure status of the data they depend on; if it is failed, trigger a reload.
[0010] Preferably, detecting whether the cached data on which the node depends has changed further includes: Maintain a version identifier or last update timestamp for each data object in the cache; Before a node is executed, the version identifier of the data that the node depends on is compared with the version identifier of the corresponding data in the cache. If they are inconsistent, it is determined that the data has been changed.
[0011] Preferably, the object metadata includes: The object's field definitions, field types, and primary key information; The definition of relationships between objects, including one-to-one, one-to-many, and many-to-many relationships; Based on the object metadata containing the aforementioned relationships, the system can automatically construct cross-object data query paths at runtime.
[0012] Preferably, the configuration metadata of the process engine is used to define the logical orchestration of the process, including: the node definition, execution order and triggering conditions of the process; The nodes reference business object fields to complete their business logic, and all referenced business object fields are parsed and bound through the object metadata.
[0013] Preferably, the nodes orchestrated by the process engine include low-code applications; When the application modifies business data through its code logic, the modification operation is recorded and triggers the invalidation or update of the corresponding data in the memory cache.
[0014] Preferably, the step S4 of determining the loading strategy based on data changes further includes: Throughout the entire lifecycle of the process engine execution, data loading logs are automatically maintained, recording the scope of objects loaded in each batch, loading time, and loading results. At the same time, maintain data change logs to record data update operations; Before a node is executed, the data loading log and data change records are compared and analyzed to automatically determine whether to directly use cached data, reload some of the changed objects, or reload all of them.
[0015] The technical effects and advantages of this invention are as follows: By automatically resolving field references in the process engine configuration metadata using pre-configured object metadata, it is possible to recursively analyze the complete data dependency chain across related objects, thereby dynamically generating a data loading list. This allows configuration personnel to select the required fields based on business logic without having to write complex cross-table query SQL or manually handle data loading logic. This frees developers from writing tedious and error-prone data access code, achieving true configuration as development. It significantly reduces the technical threshold and configuration complexity of low-code process development, realizes automatic perception of process data requirements and intelligent loading across objects, and greatly improves the abstraction capability and development efficiency of process configuration. By performing full batch loading at process startup and incremental batch loading during node execution, and by merging data requirements and generating optimized batch query statements, multiple scattered database requests are integrated into a very small number of single or small requests. The result data is stored in a memory cache shared by all nodes, which fundamentally solves the performance bottleneck caused by frequent I / O operations. This makes the process execution speed not increase linearly with the number of nodes. Especially for complex and long process businesses, the performance improvement effect is significant. Through batch loading and global shared cache mechanism, the number of database interactions is significantly reduced, which greatly improves the execution performance of the process engine and the system throughput. Through a dual detection mechanism based on failure markers and version identifiers and timestamp comparisons, the system can monitor and capture data changes caused by any means, such as process configuration, object operations, or application code, in real time, and automatically and accurately mark failed data. The system can intelligently decide whether to use the cache directly, partially reload, or fully reload, thereby maximizing cache utilization while ensuring strong data consistency. This completely solves the problem of configuration personnel having to manually judge and manage data validity, simplifies process execution, and achieves high reliability. It introduces an intelligent data state management strategy to ensure the consistency of cached data, while eliminating the burden of manually maintaining data state. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the automatic data loading and caching method of the present invention. Detailed Implementation
[0017] The technical methods of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0018] I. Implementation Prerequisites and Basic Component Definitions This method is implemented based on the workflow engine of a low-code system. Before implementation, the configuration and definition of the basic components need to be completed, as follows: Object metadata configuration: Construct object metadata according to low-code framework specifications, including object field definitions, such as field name, data type, and length constraints; field types, such as string, numeric, and date; primary key information, such as a unique identifier field; and the definition of relationships between objects, including one-to-one, one-to-many, and many-to-many relationships. For example, the "order object" and "customer object" are defined as a one-to-one relationship, and the "order object" and "order detail object" are defined as a one-to-many relationship. This relationship is clearly defined through the association field in the object metadata, providing a foundation for the system to automatically build cross-object query paths at runtime.
[0019] Process engine metadata configuration: Through the flowchart visualization orchestration tool, the process engine metadata is configured to define the nodes of the process, such as data validation nodes, business processing nodes, approval nodes, and external interface call nodes; the execution order, such as serial execution and branch execution; and the triggering conditions, such as triggering the next node when a specific field value is met; the business object fields referenced by each node in the configuration process are parsed and bound through the above-configured object metadata to ensure the accuracy of field references.
[0020] Low-code application integration: Complex logic units edited in low-code form are incorporated into the process engine orchestration as process nodes. This application supports operations such as modifying and querying business data through code logic, and its data operation behavior can be monitored by the process engine.
[0021] II. Core Process Implementation Steps The core implementation process of this method is detailed below: S1: Dynamically sensing business data needs Based on the configured object metadata and process engine configuration metadata, the system dynamically senses the business data required by the process in two scenarios, including business data across related objects: Full awareness upon process startup: After the process starts, the process engine automatically reads the complete process engine configuration metadata, parses the object field information referenced in all nodes, and recursively analyzes cross-object field dependencies through the association relationships defined in the object metadata. For example, if an approval node references the "product unit price" field of the "order details object", and the "order details object" and "order object" have a one-to-many association, and the "order object" and "customer object" have a one-to-one association, the system will automatically identify the association chain and generate a multi-object data loading requirement list containing related fields of "customer object", "order object" and "order details object".
[0022] Real-time supplementary awareness during node execution: When the process reaches a new node, the process engine parses the configuration metadata of the current node in real time to verify whether the fields required by the node are already included in the existing data loading requirement list. If it finds that the required data exceeds the existing cache range, such as the current node needing to reference the "delivery status" field of "logistics object", and this field is not within the awareness range at the start of the process, the engine dynamically supplements the awareness of the new data requirement and compares it with the existing data in the cache to identify fields that are not cached or have expired.
[0023] S2: Batch integration of requirements and execution of data loading Based on the data requirements sensed by S1, batch data loading is performed according to different scenarios, as follows: Full batch loading at process startup: The process engine organizes the list of multi-object data loading requirements detected at startup, merges duplicate loading requirements, such as multiple nodes needing the "Contact Information" field of "Customer Object", and retains only one loading requirement; based on the merged requirements, it generates optimized batch query statements, such as SQL query statements with multi-table joins, and forces the query to query all fields used in the requirement list to ensure data integrity, and then sends a query request to the database at once to complete the loading of all required data.
[0024] Incremental batch loading during node execution: For newly added data requirements detected during node execution, the process engine only filters out fields that are not cached or have expired, and generates incremental batch query requests to avoid repeatedly loading existing cached data. For example, when only adding the "delivery status" field of "logistics object", only the data of this field is queried, without involving query operations of other cached objects.
[0025] S3: Memory Cache Storage and Sharing After batch loading is complete, the process engine stores the data results in a memory cache as key-value pairs (field name - field value), forming a globally used field mapping set for the flow. This memory cache is shared by all nodes of the process engine. When a node needs data, it can directly read from the cache without making a separate query request to the database, reducing the number of database interactions. Simultaneously, the cache supports copy operations, allowing field data in the cache to be copied to the corresponding business object for process execution, ensuring the normal execution of business logic.
[0026] S4: Cache Data Change Detection Before each process node is executed, the process engine automatically detects whether the cached data that the node depends on has changed. The specific implementation method is as follows: Change operation monitoring and failure marking: During node execution, the process engine monitors three types of data update operations in real time: first, data modified through process engine configuration metadata, such as updating order status in approval nodes; second, data modified through object metadata configuration, such as adjusting object field values; and third, data modified through low-code application code in process orchestration, such as updating customer information through logic code in the application. When the above update operations are detected, the data change information is automatically recorded, including the changed object, changed field, and change time, and the affected data object is marked as failure.
[0027] Version identifier / timestamp comparison: Maintain a unique version identifier for each data object in the cache, such as an auto-incrementing sequence number or the last update timestamp; before a node is executed, the process engine extracts the version identifier / timestamp of the data that the node depends on and compares it with the version identifier / timestamp of the corresponding data in the cache. If the two are inconsistent, it is determined that the data has been changed; if they are consistent, it is determined that the data has not been changed.
[0028] S5: Loading strategy execution Based on the change detection results of step S4, the process engine automatically executes the corresponding loading strategy: If data changes are detected, including being marked as invalid or having inconsistent version identifiers / timestamps, the S2 batch data loading and S3 cache update will be automatically re-executed to load the latest data and overwrite the corresponding invalid data in the cache, ensuring that the data used by the nodes is the latest valid data.
[0029] If no data change is detected, the required data for the node is read directly from the memory cache for the node to use, without needing to re-initiate a database query.
[0030] Key supplementary implementation details Cache operation strategy execution: The initial population at the start of the process employs an intelligent merging strategy: the system checks if a field with the same name already exists in the cache; only if the field value does not exist in the cache will the newly retrieved value be used for population. For fields that already exist in the cache, their original values are retained. Simultaneously, the system ensures that all fields used in the requirements list are queried and loaded to guarantee data integrity.
[0031] Post-filling after node execution follows these rules: No filling is performed if the output is empty; if the current node is a query node based on an object and the output type is an object record, post-filling is not performed; post-filling uses a full update strategy: the newly queried data completely overwrites the corresponding fields in the cache to ensure that the cached data is fully synchronized with the latest state in the database. After filling is complete, the "changed" or "invalid" flags of the relevant fields are cleared, and their status is reset to "latest," laying the foundation for correct judgment in subsequent nodes.
[0032] Data log maintenance: Throughout the entire lifecycle of the process engine execution, two types of logs are automatically maintained: one is the data loading log, which records the scope of objects loaded in each batch, loading time, loaded fields, and loading results; the other is the data change log, which records relevant information of all data update operations, including changed objects, changed fields, change time, and change operation source. The loading strategy judgment before node execution is based on the comparison and analysis results of the above two types of logs to achieve intelligent decision-making on "directly using the cache", "partial reloading", or "full reloading".
[0033] Field overwrite rules: During the cache update process, it is possible to overwrite specified fields. Only fields that do not exist in the cache or have changed will be overwritten. Unchanged fields will retain their original cache values to avoid invalid cache update operations.
[0034] Finally, it should be noted that the accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for automatic data loading and caching in an object metadata-driven workflow engine, characterized by: Including the following step: S1: Based on object metadata and process engine configuration metadata, dynamically perceive the business data that the process engine needs to load at startup and during node execution, including business data across related objects; S2: Automatically integrate the required business data, generate batch query requests, and execute batch data loading; S3: Store the batch-loaded data results in a memory cache for sharing among all nodes of the process engine; S4: Before executing a process node, check whether the cached data that the node depends on has changed; S5: If data changes are detected, automatically re-execute S2 and S3 to load batch data and update the cache; If no changes are made, data is read directly from the memory cache for use by the node.
2. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The dynamic sensing process in S1 includes: Parse the object field information referenced in the configuration metadata of the process engine to automatically identify cross-object relationships; Based on the object metadata structure, the field dependencies of related objects are recursively analyzed to generate a list of multi-object data loading requirements.
3. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: S2 is executed when the process starts, and specifically includes: Read the complete configuration metadata of the process engine; Pre-analyze all objects and fields that may be accessed by all process nodes, and merge duplicate loading requirements; Generate optimized batch query statements to load all required data from the database at once and populate the cache.
4. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The dynamic sensing is performed in real time during node execution: When the process reaches a new node, the configuration metadata of the current node is parsed in real time. If the required data exceeds the existing cache range, the system will dynamically supplement the newly added data requirements. The new requirements are compared with the existing data in the cache, and incremental batch loading is performed only on the parts that are not cached or have expired.
5. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The process of detecting whether the cached data on which the node depends has changed in S4 specifically includes: After a process node is executed, monitor and record the data update operations performed by the process engine through its configuration metadata, object metadata, or the application orchestrated by the process. Based on the update operation, the affected data objects are automatically marked as invalid. Before executing subsequent nodes, check the failure status of the data they depend on; if it is failed, trigger a reload.
6. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 5, characterized in that: The step of detecting whether the cached data on which the node depends has changed further includes: Maintain a version identifier or last update timestamp for each data object in the cache; Before a node is executed, the version identifier of the data that the node depends on is compared with the version identifier of the corresponding data in the cache. If they are inconsistent, it is determined that the data has been changed.
7. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The object metadata includes: The object's field definitions, field types, and primary key information; The definition of relationships between objects, including one-to-one, one-to-many, and many-to-many relationships; Based on the object metadata containing the aforementioned relationships, the system can automatically construct cross-object data query paths at runtime.
8. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The configuration metadata of the process engine is used to define the logical orchestration of the process, including: the node definition, execution order and triggering conditions of the process; The nodes reference business object fields to complete their business logic, and all referenced business object fields are parsed and bound through the object metadata.
9. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The nodes orchestrated by the process engine include low-code applications; When the application modifies business data through its code logic, the modification operation is recorded and triggers the invalidation or update of the corresponding data in the memory cache.
10. The method for automatic data loading and caching of an object metadata-driven process engine according to claim 1, characterized in that: The loading strategy determined in S4 based on data changes further includes: Throughout the entire lifecycle of the process engine execution, data loading logs are automatically maintained, recording the scope of objects loaded in each batch, loading time, and loading results. At the same time, maintain data change logs to record data update operations; Before a node is executed, the data loading log and data change record are compared and analyzed to automatically determine whether to directly use cached data, reload some of the changed objects, or reload all of them.