A data processing execution track tracking method and system based on a unified identifier traceId
By introducing a unified traceId identifier into the process engine, combined with event acquisition and graph storage, the problems of insufficient dynamic behavior coverage and poor adaptability to multiple heterogeneous sources in data tracing are solved. Instance-level trajectory tracing and path reconstruction of the data processing process are realized, improving fault location efficiency and data processing traceability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CIVIL AVIATION CARES OF XIAMEN LTD
- Filing Date
- 2026-02-05
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies for data tracing suffer from problems such as failure to cover dynamic behaviors, insufficient instance-level tracing capabilities, poor adaptability to multi-source heterogeneity, and failure to parse complex processes. This leads to a disconnect between data tracing results and the actual flow path, making it difficult to accurately locate faults and reconstruct data flow links in multi-threaded, concurrent, or distributed deployment scenarios.
The data processing execution trajectory tracing method based on the unified identifier traceId is adopted. By introducing the runtime traceId at the process engine level, combined with event collection, streaming aggregation and graph storage, the lineage tracing and analysis at the process instance level is realized. The traceId is generated and transmitted throughout the entire life cycle. Execution trajectory events are collected and associated, execution trajectory substructures are constructed and persisted to graph storage, and multi-dimensional query and backtracking are supported.
It enables instance-level trajectory tracking and reconstruction of data processing flows, adapts to multi-source heterogeneous data interaction scenarios, accurately reconstructs the execution path of complex processes, improves the traceability of data processing and the efficiency of fault location, reduces analysis and troubleshooting costs, and has flexible scalability.
Smart Images
Figure CN122111809A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of process engine runtime tracing, task orchestration observability and data governance technology, specifically to a data processing execution trajectory tracing method and system based on a unified identifier traceId. Background Technology
[0002] Data execution trajectory tracing is a crucial foundation for data governance, fault location, and compliance auditing. It requires accurately recording the entire flow of data from its source input, through intermediate processing and transformation, to its final output. Currently, mainstream data tracing primarily employs data lineage analysis schemes based on static analysis methods. Specifically, this involves parsing the scripts of data processing jobs (such as SQL statements and ETL configuration files), metadata mapping relationships, or dependency configurations to infer upstream and downstream dependencies of the data.
[0003] However, such static analysis solutions have significant limitations in real-world, complex business scenarios:
[0004] Dynamic behavior cannot be covered: Static analysis can only infer dependencies based on preset scripts and cannot reflect the dynamic behavior during process execution (such as branch routing, task retries, fault compensation, etc.), resulting in a disconnect between data tracing results and the actual data flow path; Insufficient instance-level traceability: It is impossible to accurately associate the full-link data of a "single process instance". In multi-threaded, concurrent or distributed deployment scenarios, it is difficult to match the upstream input and downstream output of a process execution one by one. When locating faults, a large amount of irrelevant data needs to be checked. Poor adaptability to multiple heterogeneous data sources: For multiple heterogeneous data sources such as JDBC databases, MQ message queues, file storage, and external services, static analysis is difficult to collect I / O operation records of each data source in a unified manner, and cannot completely reconstruct the data flow link across data sources; Complex process analysis failure: In complex process orchestration scenarios containing parallel gateways (Fork / Join) and inclusive gateways (conditional branches), static analysis cannot determine the actual execution path at runtime, resulting in a deviation between the lineage diagram and the actual business process. Summary of the Invention
[0005] This invention proposes a data processing execution trajectory tracking method and system based on a unified identifier traceId. It introduces a runtime, unified traceId at the process engine level and combines event collection, streaming aggregation, and graph storage to achieve runtime and process instance-level lineage tracing and analysis, thereby overcoming the limitations of static analysis.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: A data processing execution trajectory tracing method based on a unified identifier traceId includes the following steps: S1 Process Instance TraceId Generation and End-to-End Transmission: When the process engine triggers the start of a process instance, the TraceIdService generates a trace identifier (traceId) to uniquely identify the process instance. The traceId remains consistent throughout the entire lifecycle of the process instance. The traceId is injected into the execution context and forcibly carried in all task nodes and critical I / O operations through the trace transmission adapter, ensuring that all nodes of the same process instance share the same traceId. S2 Execution Event Acquisition and Transmission: During the execution of the process instance, the event collector collects execution trajectory events at key execution points of each task node and asynchronously reports the events to the event channel. The key execution points include at least the task node start (NODE_START), task node end (NODE_END), data read completion (IO_READ), data write completion (IO_WRITE), and process routing selection (GATEWAY) node. The execution trajectory events adopt the standard JSON format and include at least the following fields: eventId, traceId, flowId, nodeId, nodeType, eventType, timestamp, status, input, and output, which describe the actual execution behavior of the process instance during runtime. S3 Execution Trajectory Event Association: Receives execution trajectory events from the event channel and associates the received execution trajectory events according to the traceId, so that multiple execution trajectory events generated by parallel, asynchronous, or conditional branches in the same process instance belong to the same process instance; for parallel branch scenarios, the association is completed after all branch node events arrive at the parallel merging node; for inclusive branch scenarios, only the actually executed branch node events are associated, and subsequent processing is triggered according to "any branch completed" or business-defined rules; S4 Execution Trajectory Substructure Construction: Based on the associated execution trajectory events, construct the execution trajectory substructure at the process instance level. The execution trajectory substructure includes at least a task execution instance node (NodeExecution) and an association relationship (Edge) used to describe the execution order and data interaction relationship of each task. After the process instance ends or the lineage small graph times out, persist the execution trajectory substructure to graph storage. S5 instance-level execution path tracing and backtracking: When the process instance ends, based on the execution trajectory substructure, the actual execution path and corresponding data flow order of each task node in the process instance are restored according to the order of occurrence of the execution trajectory events; based on the traceId, multi-dimensional tracing, querying and backtracking analysis capabilities are provided, including querying the complete execution trajectory by traceId, querying upstream and downstream data flow by DatasetID, and querying historical execution metrics by nodeId; S6 Offline Compensation and Security Protection: Based on event archive storage, an offline compensation mechanism is provided, which reads events periodically or on demand and merges them by instanceId / traceId to cover or supplement the execution trajectory substructure in the graph storage; security and privacy control are achieved through data anonymization, RBAC access control and audit log recording.
[0007] Furthermore, the traceId is configured as the associated primary key of the process instance execution trajectory, used to uniformly collect multiple execution trajectory events during the process instance's operation, and to drive the construction of the execution trajectory substructure and the restoration of the execution path after the process instance ends; in step S1, the format of the traceId is TRACE-{flowIdHash}-{timestamp}-{shortUUID}-{version}, where: flowIdHash is the hash or short code of the process identifier; timestamp is the process trigger time; shortUUID is a random string used to ensure global uniqueness; and version is the process version.
[0008] Furthermore, the execution trajectory events are automatically generated at the corresponding execution time points during the operation of the process instance, and are used to characterize the execution order and data interaction behavior of each task node in the process instance. The execution trajectory events include at least task node start events, task node end events, data read events, data write events, and process routing selection events. In step S1, the trace pass-through adapter embeds a JDBC executor, an MQ-PULL executor, an MQ-PUSH executor, a plugin executor, and a Groovy script executor, automatically reads the traceId from the execution context ExecutionContext, and writes the traceId in the message header or SQL comment of the I / O operation.
[0009] Furthermore, when a process instance includes parallel execution, asynchronous execution, or conditional branch execution, the traceId is used to uniformly associate execution trajectory events from different execution branches, so that execution trajectory events belonging to the same process instance can still be grouped into the execution trajectory of the same process instance even if they are generated out of order in time.
[0010] Furthermore, the relationships within the execution trajectory substructure include: Execution order: describes the start and end order of different task execution instance nodes; Data reading relationship: The reading relationship of "Dataset → NodeExecution" is associated with the sourceType, sourceId, and recordCount attributes in the input field; Data write relationship: The write relationship of "NodeExecution → Dataset" is associated with the targetType, targetId, and recordCount attributes in the output field.
[0011] Furthermore, in step S6, the data desensitization specifically means that the event does not contain plaintext sensitive data, and when data samples need to be recorded, hash values or reversible desensitization processing are used; the RBAC access control specifically means that the trajectory query interface only allows authorized users to query trajectory data of the corresponding business domain; the audit log records the operator, operation time, and operation content of event collection, trajectory storage, and query operations to meet compliance requirements.
[0012] Furthermore, it supports flexible expansion capabilities, including field-level / record-level trajectory expansion, cross-process association expansion, and trajectory hierarchical retention strategies.
[0013] A data processing execution trajectory tracing system based on a unified identifier traceId, used to implement the aforementioned data processing execution trajectory tracing method based on a unified identifier traceId, includes: Process Engine Module: Responsible for starting process instances and defining and managing process models. The process model includes StartEvent, Task, Gateway, and EndEvent. When a process instance starts, it generates a trace identifier (traceId) to uniquely identify the process instance and injects the traceId into the execution context of the process instance. Execution trajectory event acquisition module: used to collect execution trajectory events at key execution points during the task node and gateway stages during the process instance operation, generate event data according to standard JSONSchema, and asynchronously send the events to the event channel; Event Channel Module: Implemented based on message middleware Kafka, Artemis, or RocketMQ, used for transmitting events, supporting at-least-once semantics; Execution trajectory event association module: used to associate the collected execution trajectory events based on the traceId, so that execution trajectory events belonging to the same process instance are uniformly collected; Execution trajectory construction module: used to construct execution trajectory substructures at the process instance level based on associated execution trajectory events, and persist them to graph storage after the process instance ends or the lineage minigraph times out; Execution path restoration module: used to restore the actual execution path and corresponding data flow sequence of the process instance based on the execution trajectory substructure when the process instance ends; Instance-level tracing and query module: It is used to provide multi-dimensional tracing, querying and backtracking analysis of the process instance execution process based on the traceId. It provides REST API interface and front-end visual interface, supports trajectory graph display, timeline viewing, drill-down operation and data export, and the formats include image, JSON and GraphML. Offline compensation service: Reconstructs the execution trajectory substructure based on event archive storage to supplement missing trajectory data; Security and Audit Module: Implements data anonymization, RBAC access control, and audit log recording.
[0014] Furthermore, the graph storage uses JanusGraph or Neo4j, and the metadata storage uses PostgreSQL; the traceId is configured as the associated primary key of the process instance execution trajectory, used to uniformly collect execution trajectory events among multiple parallel or asynchronous task nodes.
[0015] Furthermore, in the front-end visualization interface, task execution instance nodes (NodeExecution) are displayed as rectangles, datasets are displayed as circles or diamonds, and the relationship edges carry the number of records and time consumption attributes; the REST API interface includes functions such as querying the complete process instance execution trajectory by traceId, querying upstream by DatasetID, querying downstream by DatasetID, querying historical execution metrics by nodeId, and triggering offline trajectory reconstruction.
[0016] After adopting the above solution, the beneficial effects of the present invention are as follows: 1. Achieve instance-level trajectory tracking and reconstruction of data processing flow: Based on traceId, full-link transparent transmission and streaming aggregation accurately associate the full-link data of a single process instance, solving the problem that static analysis cannot cover dynamic behaviors (branching, retries, compensation); 2. Adapt to multi-source heterogeneous data interaction scenarios: Through a unified execution trajectory event collection and association mechanism, it supports the inclusion of multi-source heterogeneous data interaction behaviors such as JDBC databases, message queues, file storage and external services involved in the process instance into the same execution trajectory for tracking, and fully restores the data execution link across data sources; 3. Supports execution path reconstruction under complex processes: For cases involving parallel execution, asynchronous execution, or conditional execution, by uniformly associating execution trajectory events from different execution branches, it can accurately reconstruct the actual execution path that occurs during the operation of complex processes, avoiding path deviations caused by static parsing alone; 5. Enhance the traceability of the data processing process: By saving the execution trajectory substructure, the execution process can still be tracked, queried, and backtracked based on the traceId after the data processing process is completed, which facilitates fault location, operation analysis, and post-event auditing.
[0017] 5. Reduce data processing, analysis, and troubleshooting costs: The structured recording method based on execution trajectory decouples the analysis of the data processing flow from scattered logs and events, reducing manual troubleshooting costs and improving the efficiency of process analysis and problem localization; 6. Flexible scalability: The execution trajectory event types and execution trajectory substructures can be expanded according to business needs to adapt to different task types and data interaction methods, and support subsequent function evolution without affecting the existing data processing flow instance tracking capabilities. Attached Figure Description
[0018] Figure 1 This is the overall system architecture diagram of the present invention.
[0019] Figure 2 This is a timing diagram of traceId generation and transparent transmission in this invention. Detailed Implementation
[0020] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0021] This invention provides a data processing execution trajectory tracking method based on a unified identifier traceId. By generating and transmitting traceId during the execution of a process instance, collecting and associating execution trajectory events, constructing a process instance-level execution trajectory substructure, and restoring the actual execution path at the end of the process instance, the method also ensures data integrity and privacy security through offline compensation and security protection mechanisms.
[0022] A data processing execution trajectory tracing system based on a unified identifier traceId, used to implement the aforementioned data processing execution trajectory tracing method based on a unified identifier traceId, includes: like Figure 1As shown, the process engine module 100 is responsible for starting process instances and defining and managing process models. The process model includes StartEvent, Task, Gateway, and EndEvent. When a process instance starts, it generates a trace identifier (traceId) to uniquely identify the process instance and injects the traceId into the execution context of the process instance. Execution trajectory event acquisition module 200: used to collect execution trajectory events at key execution points in the task node and gateway stages during the process instance operation, generate event data according to standard JSONSchema, and asynchronously send the events to the event channel; Event Channel Module 300: Implemented based on message middleware Kafka, Artemis, or RocketMQ, used for transmitting events, and supports at-least-once semantics; Execution trajectory event association module 400: used to associate the collected execution trajectory events based on the traceId, so that execution trajectory events belonging to the same process instance are uniformly collected; Execution trajectory construction module 500: Used to construct execution trajectory substructures at the process instance level based on associated execution trajectory events, and persist them to graph storage after the process instance ends or the lineage minigraph times out; Execution path restoration module 600: used to restore the actual execution path and corresponding data flow sequence of the process instance based on the execution trajectory substructure when the process instance ends; Instance-level tracing and query module 700: It is used to provide multi-dimensional tracing, querying and backtracking analysis of the process instance execution process based on the traceId. It provides REST API interface and front-end visual interface, supports trajectory graph display, timeline viewing, drill-down operation and data export, and the formats include images, JSON and GraphML. Offline compensation service: Reconstructs the execution trajectory substructure based on event archive storage to supplement missing trajectory data; Security and Audit Module: Implements data anonymization, RBAC access control, and audit log recording.
[0023] The graph storage uses JanusGraph or Neo4j, and the metadata storage uses PostgreSQL; the traceId is configured as the associated primary key of the process instance execution trajectory, used to uniformly collect execution trajectory events among multiple parallel or asynchronous task nodes.
[0024] In the front-end visualization interface, task execution instance nodes (NodeExecution) are displayed as rectangles, datasets are displayed as circles or diamonds, and relationship edges carry the number of records and time consumption attributes; the REST API interface includes functions such as querying the complete process instance execution trajectory by traceId, querying upstream by DatasetID, querying downstream by DatasetID, querying historical execution metrics by nodeId, and triggering offline trajectory reconstruction.
[0025] The method includes the following steps: S1 Process Instance TraceId Generation and End-to-End Transmission: When the process engine starts a process instance (e.g., when a CRON timer triggers a security check log processing process every 10 minutes), a globally unique traceId is generated by TraceIdService, with the following format: TRACE-{flowIdHash}-{timestamp}-{shortUUID}-{version}, For example, TRACE-5a3f-20251004T150005Z-8a7f-v1; The traceId is injected into the execution context and is forcibly carried in all task nodes and critical I / O operations through the trace pass-through adapter. The adapter is embedded in various executors such as JDBC, MQ-PULL / PUSH, plugins, and Groovy scripts, and automatically reads the traceId from the execution context and writes it into the message header or SQL comment of the I / O operation.
[0026] S2 Execution Event Collection and Transmission: During the execution of the process instance, the event collector generates standard JSON format events at key moments of each task node and asynchronously reports them to the Kafka event channel. For example, after Task1 (JDBC) finishes reading the passenger_log, it reports the IO_READ event, which includes information such as sourceType (JDBC), sourceId (passenger_log table ID), and recordCount (1000 records) in the input field; after the Task node finishes execution, it reports the NODE_END event, which includes the execution status (success / failure) and the time taken (200ms).
[0027] S3 execution trajectory event association: The execution trajectory event association module consumes and associates events by traceId. For parallel branch scenarios, association is performed after all branch events arrive at the merging node; for inclusive branch scenarios, only the actually executed branch events are associated. For example, in the passenger flow prediction process, after three parallel batch processing tasks report events, the association module waits for all three events to arrive before proceeding with subsequent processing to ensure trajectory integrity.
[0028] S4 Execution Trajectory Substructure Construction: The execution trajectory construction module builds an execution trajectory substructure based on the associated events, including task execution instance nodes such as Task1 and Task2, as well as the execution sequence and data read / write relationships between nodes. After the process instance ends (END_EVENT) or times out, the substructure is persisted to JanusGraph graph storage, and the metadata is stored in PostgreSQL. Upsert / merge operations are used during writes to avoid duplicate data.
[0029] S5 Instance-Level Execution Path Tracing and Backtracking: After a process instance ends, the execution path restoration module reads the execution trajectory substructure from the graph storage and reconstructs the actual execution path in the order of events. Users can query the complete trajectory by entering the traceId through the REST API interface of the instance-level tracing query module, or view the trajectory graph through the front-end visualization interface, which supports timeline viewing, drill-down, and data export (image / JSON / GraphML formats).
[0030] S6 Offline Compensation and Security Protection: The offline compensation service periodically reads events from the event archive storage, merges them by traceId, and supplements them to the graph storage. For example, if an event for a certain node is missing due to network fluctuations, it is supplemented through offline compensation to ensure the integrity of the trajectory data. The security and auditing module performs hash anonymization on sensitive data in events, allowing only authorized users to query trajectory data in the civil aviation security inspection business domain, and records all query operation logs.
[0031] Application Scenario 1: Log Data Processing and Alarm Fusion in Security Inspection Systems StartEvent: Triggered by CRON every 10 minutes, generating a traceId; Task 1 (JDBC): Read passenger_log and report IO_READ and NODE_END events; Task 2 (MQ-PULL): Pull data from the alarm queue, process it, and then report the IO_READ / IO_WRITE event; Gateway: Inclusive merging, after the associated module associates two events, the constructed execution trajectory substructure clearly shows how the two data streams are aggregated and written to the alarm_report table. Users can query the complete flow path through the traceId to quickly locate the source of the alarm data.
[0032] Application Scenario 2: Passenger Flow Forecasting and Dynamic Allocation of Passenger Lanes StartEvent: Triggered by CRON every 5 minutes, generating a traceId; Three parallel batch processing tasks: read historical logs, flight plans, and external feature data respectively, and report IO_READ / NODE_END events; Plugin (feature fusion + model inference): Reads 3 temporary Datasets, reports IO_READ events, and reports IO_WRITE events after prediction is completed (writes to the security_lane_forecast table). When the threshold is predicted to be exceeded, an alarm task is triggered and written to the MQ, and the IO_WRITE / NODE_END event is reported. Users can query the traceId to reconstruct the complete execution path from raw data reading, feature fusion, model inference to alarm triggering, which facilitates analysis of the causes of prediction deviations and optimization of load allocation strategies.
[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
[0034] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of data processing execution trajectory tracking methods and systems based on the unified identifier traceId under the guidance of this patent. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this patent.
Claims
1. A data processing execution trajectory tracing method based on a unified identifier traceId, characterized in that, Includes the following steps: S1 Process Instance TraceId Generation and End-to-End Transmission: When the process engine triggers the start of a process instance, the TraceIdService generates a trace identifier (traceId) to uniquely identify the process instance. The traceId remains consistent throughout the entire lifecycle of the process instance. The traceId is injected into the execution context and forcibly carried in all task nodes and critical I / O operations through the trace transmission adapter, ensuring that all nodes of the same process instance share the same traceId. S2 Execution Event Acquisition and Transmission: During the execution of the process instance, the event collector collects execution trajectory events at key execution points of each task node and asynchronously reports the events to the event channel. The key execution points include at least the task node start (NODE_START), task node end (NODE_END), data read completion (IO_READ), data write completion (IO_WRITE), and process routing selection (GATEWAY) node. The execution trajectory events adopt the standard JSON format and include at least the following fields: eventId, traceId, flowId, nodeId, nodeType, eventType, timestamp, status, input, and output, which describe the actual execution behavior of the process instance during runtime. S3 Execution Trajectory Event Association: Receives execution trajectory events from the event channel, and associates the received execution trajectory events according to the traceId, so that multiple execution trajectory events generated by parallel, asynchronous, or conditional branches in the same process instance belong to the same process instance; For parallel branching scenarios, the association is completed at the parallel merging node after all branch node events have arrived; For scenarios involving inclusive branches, only the actual branch node events are associated, and subsequent processing is triggered according to "complete any branch" or business-defined rules; S4 Execution Trajectory Substructure Construction: Based on the associated execution trajectory events, construct the execution trajectory substructure at the process instance level. The execution trajectory substructure includes at least a task execution instance node (NodeExecution) and an association relationship (Edge) used to describe the execution order and data interaction relationship of each task. After the process instance ends or the lineage small graph times out, persist the execution trajectory substructure to graph storage. S5 instance-level execution path tracing and backtracking: When the process instance ends, based on the execution trajectory substructure, the actual execution path and corresponding data flow order of each task node in the process instance are restored according to the order of occurrence of the execution trajectory events; based on the traceId, multi-dimensional tracing, querying and backtracking analysis capabilities are provided, including querying the complete execution trajectory by traceId, querying upstream and downstream data flow by DatasetID, and querying historical execution metrics by nodeId; S6 Offline Compensation and Security Protection: Based on event archive storage, an offline compensation mechanism is provided, which reads events periodically or on demand and merges them by instanceId / traceId to cover or supplement the execution trajectory substructure in the graph storage; security and privacy control are achieved through data anonymization, RBAC access control and audit log recording.
2. The data processing execution trajectory tracing method based on a unified identifier traceId according to claim 1, characterized in that, The traceId is configured as the associated primary key of the process instance execution trajectory, used to uniformly collect multiple execution trajectory events during the process instance's operation, and to drive the construction of the execution trajectory substructure and the restoration of the execution path after the process instance ends; in step S1, the format of the traceId is TRACE-{flowIdHash}-{timestamp}-{shortUUID}-{version}, where: flowIdHash is the hash or short code of the process identifier; timestamp is the process trigger time; shortUUID is a random string used to ensure global uniqueness; version is the process version.
3. The data processing execution trajectory tracing method based on a unified identifier traceId according to claim 1, characterized in that, The execution trajectory events are automatically generated at the corresponding execution time points during the operation of the process instance, and are used to characterize the execution order and data interaction behavior of each task node in the process instance. The execution trajectory events include at least task node start events, task node end events, data read events, data write events, and process routing selection events. In step S1, the trace pass-through adapter embeds a JDBC executor, an MQ-PULL executor, an MQ-PUSH executor, a plugin executor, and a Groovy script executor, automatically reads the traceId from the execution context ExecutionContext, and writes the traceId in the message header or SQL comment of the I / O operation.
4. A data processing execution trajectory tracing method based on a unified identifier traceId according to any one of claims 1 to 3, characterized in that, When a process instance includes parallel execution, asynchronous execution, or conditional branch execution, the traceId is used to uniformly associate execution trajectory events from different execution branches, so that execution trajectory events belonging to the same process instance can still be grouped into the execution trajectory of the same process instance even if they are generated out of time.
5. A data processing execution trajectory tracing method based on a unified identifier traceId according to any one of claims 1 to 4, characterized in that, The relationships within the execution trajectory substructure include: Execution order: describes the start and end order of different task execution instance nodes; Data read relationship: The read relationship of "Dataset→NodeExecution" is associated with the sourceType, sourceId, and recordCount attributes in the input field; Data write relationship: The write relationship of "NodeExecution→Dataset" is associated with the targetType, targetId, and recordCount attributes in the output field.
6. A data processing execution trajectory tracing method based on a unified identifier traceId according to any one of claims 1 to 5, characterized in that, In step S6, the data desensitization specifically means that the event does not contain plaintext sensitive data, and when data samples need to be recorded, hash values or reversible desensitization processing are used; the RBAC access control specifically means that the trajectory query interface only allows authorized users to query trajectory data of the corresponding business domain; the audit log records the operator, operation time, and operation content of event collection, trajectory storage, and query operations to meet compliance requirements.
7. A data processing execution trajectory tracing method based on a unified identifier traceId according to any one of claims 1 to 6, characterized in that, Supports flexible expansion capabilities, including field-level / record-level trajectory expansion, cross-process association expansion, and trajectory hierarchical retention strategy.
8. A data processing execution trajectory tracking system based on a unified identifier traceId, characterized in that, A data processing execution trajectory tracing method based on a unified identifier traceId as described in any one of claims 1-7 includes: Process Engine Module: Responsible for starting process instances and defining and managing process models. The process model includes StartEvent, Task, Gateway, and EndEvent. When a process instance starts, it generates a trace identifier (traceId) to uniquely identify the process instance and injects the traceId into the execution context of the process instance. Execution trajectory event acquisition module: used to collect execution trajectory events at key execution points during the task node and gateway stages during the process instance operation, generate event data according to standard JSONSchema, and asynchronously send the events to the event channel; Event Channel Module: Implemented based on message middleware Kafka, Artemis, or RocketMQ, used for transmitting events, supporting at-least-once semantics; Execution trajectory event association module: used to associate the collected execution trajectory events based on the traceId, so that execution trajectory events belonging to the same process instance are uniformly collected; Execution trajectory construction module: used to construct execution trajectory substructures at the process instance level based on associated execution trajectory events, and persist them to graph storage after the process instance ends or the lineage minigraph times out; Execution path restoration module: used to restore the actual execution path and corresponding data flow sequence of the process instance based on the execution trajectory substructure when the process instance ends; Instance-level tracing and query module: It is used to provide multi-dimensional tracing, querying and backtracking analysis of the process instance execution process based on the traceId. It provides REST API interface and front-end visual interface, supports trajectory graph display, timeline viewing, drill-down operation and data export, and the formats include image, JSON and GraphML. Offline compensation service: Reconstructs the execution trajectory substructure based on event archive storage to supplement missing trajectory data; Security and Audit Module: Implements data anonymization, RBAC access control, and audit log recording.
9. The system according to claim 8, characterized in that, The graph storage uses JanusGraph or Neo4j, and the metadata storage uses PostgreSQL; the traceId is configured as the associated primary key of the process instance execution trajectory, used to uniformly collect execution trajectory events among multiple parallel or asynchronous task nodes.
10. The system according to claim 8, characterized in that, In the front-end visualization interface, task execution instance nodes (NodeExecution) are displayed as rectangles, datasets are displayed as circles or diamonds, and relationship edges carry the number of records and time consumption attributes; the REST API interface includes functions such as querying the complete process instance execution trajectory by traceId, querying upstream by DatasetID, querying downstream by DatasetID, querying historical execution metrics by nodeId, and triggering offline trajectory reconstruction.