Data tracing method and device based on runtime snapshot

Abstract

The invention discloses a data tracing method and device based on a runtime snapshot. Comprising the steps of generating a runtime memory snapshot file based on report traceability information; analyzing the runtime memory snapshot file, extracting all keys in the first mapping relation to construct a header character set, and querying N metadata identifiers corresponding to N header characters according to the first mapping relation; accessing a report data result storage area based on the N metadata identifiers to obtain a data value list, and rendering to generate a visual report snapshot; for each metadata identifier, extracting the metadata identifier of each metadata which directly depends on and indirectly depends on the metadata identifier from the task detail information associated with the metadata identifier, and obtaining a single metadata tree based on the hierarchical dependency relationship among the metadata; rendering and generating N metadata tree snapshots based on the N metadata trees; and displaying the metadata circulation process corresponding to the abnormal data value by skipping to display the metadata tree snapshot of the metadata identifier corresponding to the report data column where the abnormal data value is located.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data tracing method, apparatus, device, medium and product based on runtime snapshots. Background Technology

[0002] Data snapshot technology is an important data management tool in computer science. Its core lies in capturing and persisting a complete state image of a system or data at a specific point in time. This technology was initially widely used in database management systems, file storage systems, and virtualization platforms, primarily for data backup and disaster recovery, system state rollback, and historical data comparison and analysis. Its basic principle is to generate a lightweight, read-only copy of the data without interrupting the main business operations through techniques such as copy-on-write or input / output redirection. This copy accurately records the state of the data at a specific moment, providing a solid foundation for subsequent data analysis, auditing, test environment setup, and fault recovery. In enterprise applications, especially complex report generation systems, the significance of data snapshots is particularly prominent. They can solidify dynamically changing, multi-layered calculated data results, providing a potential technical path for solving data traceability problems.

[0003] However, while data snapshot technology has general value in data persistence, existing solutions have many inherent flaws and limitations when directly applied to troubleshooting data issues in complex reporting systems. This makes it difficult to efficiently and accurately pinpoint the root cause of errors in report data in the production environment. Specifically, the reporting program in the system employs a flexible metadata system. Metadata can be arbitrarily calculated and combined to generate new metadata. A single data point in the report may originate from multiple calculations of multiple data points, with calculation levels reaching multiple layers. When a data point in the report does not meet expectations, troubleshooting data problems becomes time-consuming and laborious. Therefore, how to effectively troubleshoot report data and assist in quickly locating the root cause of problems has become an urgent issue to be addressed. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data tracing method, apparatus, device, medium, and product based on runtime snapshots, which can effectively investigate report data and assist in quickly locating the root cause of problems.

[0005] In a first aspect, embodiments of this application provide a data tracing method based on runtime snapshots. The method includes: when generating a report, generating a runtime memory snapshot file associated with the report based on obtained report tracing information, wherein the report tracing information includes at least task details of all data processing tasks during report execution, a first mapping relationship between N report data columns and metadata in the report, and report data results; parsing the runtime memory snapshot file, extracting all keys from the first mapping relationship to construct a header text set, and querying N metadata identifiers corresponding to the N header texts in the header text set according to the first mapping relationship; accessing the report data result storage area in the runtime memory snapshot file based on the N metadata identifiers, querying and obtaining a list of data values ​​corresponding to each metadata identifier, obtaining a list of data values ​​corresponding to each header text, and rendering and generating a report dimension based on the header text set and the list of data values ​​corresponding to each header text. Visualize report snapshots; query task details based on N metadata identifiers. For each metadata identifier, extract the metadata identifiers of its directly and indirectly dependent metadata from its associated task details. Based on the hierarchical dependencies between metadata, recursively generate a multi-level tree structure with the metadata identifier corresponding to the table header text as the root node and the metadata identifiers of each metadata as child nodes, resulting in a single metadata tree; render N metadata tree snapshots of N metadata identifiers in the metadata dimension, where each node of the metadata tree snapshot displays at least: the metadata identifier and the data result corresponding to its associated data processing task, the task identifier, and the task execution information; respond to the user's click operation on the visual report snapshot of abnormal data value, and display the metadata tree snapshot of the metadata identifier corresponding to the metadata identifier of the report data column where the abnormal data value is located by jumping to show the metadata flow process corresponding to the abnormal data value.

[0006] Secondly, embodiments of this application provide a data tracing device based on runtime snapshots. The device includes: a generation module, used to generate a runtime memory snapshot file associated with a report based on obtained report tracing information when generating a report, wherein the report tracing information includes at least task details of all data processing tasks during report execution, a first mapping relationship between N report data columns and metadata in the report, and report data results; a parsing module, used to parse the runtime memory snapshot file, extract all keys in the first mapping relationship to construct a header text set, and query N metadata identifiers corresponding to the N header texts in the header text set according to the first mapping relationship; and a rendering generation module, used to access the report data result storage area in the runtime memory snapshot file based on the N metadata identifiers, query and obtain a list of data values ​​corresponding to each metadata identifier, obtain a list of data values ​​corresponding to each header text, and render and generate a visualization in the report dimension based on the header text set and the list of data values ​​corresponding to each header text. The report snapshot module includes a metadata tree construction module, which queries task details based on N metadata identifiers. For each metadata identifier, it extracts the metadata identifiers of its directly and indirectly dependent metadata from the associated task details. Based on the hierarchical dependencies between the metadata, it recursively generates a multi-level tree structure with the metadata identifier corresponding to the table header text as the root node and the metadata identifiers of each metadata as child nodes, resulting in a single metadata tree. The rendering and generation module is used to render and generate N metadata tree snapshots of the N metadata identifiers in the metadata dimension. Each node of the metadata tree snapshot displays at least: the metadata identifier and the data result corresponding to its associated data processing task, the task identifier, and the task execution information. The snapshot display module responds to user clicks on abnormal data values ​​in the visual report snapshot by jumping to display the metadata tree snapshot of the metadata identifier corresponding to the metadata column of the report data containing the abnormal data value, showing the metadata flow process corresponding to the abnormal data value.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the steps of the data tracing method based on runtime snapshots as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the data tracing method based on runtime snapshots as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product stored in a non-volatile storage medium, wherein when the computer program product is executed by a processor, it implements the steps of the data tracing method based on runtime snapshots as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the data tracing method based on runtime snapshots as described in the first aspect.

[0011] This application provides a data tracing method, apparatus, device, medium, and product based on runtime snapshots. It proactively acquires complete report tracing information upon report execution completion, generating a runtime memory snapshot file. This snapshot file records detailed information about the report generation process, avoiding the information loss issues associated with traditional methods relying on scattered logs and manual environment reconstruction. Based on this, by parsing the snapshot file and rendering a visual report snapshot based on a first mapping relationship, users can intuitively locate abnormal data. Furthermore, a metadata tree snapshot is recursively constructed based on metadata identifiers. This snapshot clearly presents complex, deep-level computational logic in a visual manner, making each layer of metadata computation observable, thus clearly showing the complete dependency chain and computational hierarchy from the final calculation result to the source data. Finally, a visual interactive interface is provided for multi-dimensional navigation. Click operations allow users to navigate from report data to the metadata tree snapshot, and task identifiers and execution information are displayed at each node, achieving a horizontal association between data lineage and task execution flow. This guides users to focus on the root cause of problems. Compared to traditional investigation processes that heavily rely on manual experience, this application improves the accuracy and efficiency of in-depth data problem investigation. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings of the embodiments of this application will be briefly described below.

[0013] Figure 1 This is a flowchart illustrating a data tracing method based on runtime snapshots provided in an embodiment of this application;

[0014] Figure 2 This is an exemplary schematic diagram of a metadata tree snapshot provided in an embodiment of this application;

[0015] Figure 3 This is an exemplary schematic diagram of a task tree snapshot provided in an embodiment of this application;

[0016] Figure 4 This is an exemplary schematic diagram of a task tree snapshot provided in another embodiment of this application;

[0017] Figure 5 This is an exemplary schematic diagram of a data snapshot interface provided in an embodiment of this application;

[0018] Figure 6 This is a schematic diagram of the structure of a data tracing device based on runtime snapshots provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of this application clearer and more thorough, enabling those skilled in the art to better understand and implement the principles and spirit of this application. The exemplary embodiments provided herein are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of this application.

[0021] In this document, terms such as first, second, and third are used only to distinguish one entity (or operation) from another, and are not intended to require or imply any order or relationship between these entities (or operations).

[0022] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the related technologies:

[0023] While data snapshot technology has general value in data persistence, existing technical solutions have many inherent defects and limitations when directly applied to troubleshooting data problems in complex reporting systems. This makes it impossible to efficiently and accurately pinpoint the root cause of errors in report data in the production environment, specifically in the following aspects:

[0024] Traditional database or storage snapshots primarily capture static data states, typically containing copies of the original dataset or mappings to physical storage blocks. However, the generation of report data is a complex, dynamic process involving the execution of multiple levels of data processing tasks, the application of calculation rules, and dependencies between metadata. Therefore, even with a data snapshot, it's impossible to trace which specific task generated an abnormal data point, or which preceding steps or metadata that task depended on, making troubleshooting extremely difficult.

[0025] Traditional snapshot technology focuses on data storage and recovery, rather than in-depth analysis and visual diagnostics. When errors occur in report data, operations personnel face not only the challenge of recovering the data, but more importantly, understanding the causes of the errors. Traditional methods lack the ability to transform snapshot content into multi-layered, interactive visualizations, forcing a fragmented troubleshooting process that requires constant switching between different tools and logs, resulting in inefficiency and a high reliance on the personal experience of technical professionals.

[0026] When discrepancies arise in production environment reports, the common troubleshooting method is to attempt to reconstruct the entire report generation process in an offline environment. This process not only requires synchronizing the massive amounts of basic data in the production environment but also restoring complex computational logic and execution environments. It often involves the configuration and collaboration of multiple related programs, making it time-consuming and labor-intensive. Furthermore, it's difficult to guarantee complete consistency between the offline and online production environments, which can easily lead to problems that cannot be reproduced or incorrect problem localization. Each troubleshooting session requires repeating this process, impacting problem response speed and enterprise operational efficiency.

[0027] Modern, flexible reporting systems allow for the derivation of new metadata through arbitrary combinations and calculations of existing metadata, with calculation layers reaching as many as a dozen levels. Existing snapshot technologies lack the ability to effectively record and display such deep, networked data lineages. They cannot record intermediate calculation results, nor can they parse and visualize the dependency chains between metadata, making it extremely difficult to pinpoint "which layer of data or calculation is experiencing a problem."

[0028] In summary, existing data snapshot technologies have inherent shortcomings, such as limited content dimensions, lack of analytical functions, strong environmental dependence, and insufficient support for complex data lineages, making it difficult to effectively support the needs of complex reporting systems for quickly identifying and locating data problems.

[0029] In view of the inventors’ above-mentioned research findings, the embodiments of this application provide a data tracing method, apparatus, device, medium and product based on runtime snapshots, which aims to solve at least one of the above technical problems.

[0030] The data tracing method based on runtime snapshots provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] Figure 1 This is a flowchart illustrating a data tracing method based on runtime snapshots provided in an embodiment of this application. The execution entity of this data tracing method based on runtime snapshots can be a data processing system.

[0032] The following example uses a data processing system (hereinafter referred to as the system for the purpose of avoiding redundancy) as the execution subject of the data tracing method based on runtime snapshots to illustrate the data tracing method based on runtime snapshots of this application. It should be noted that the above execution subject and application scenario do not constitute a limitation on this application.

[0033] like Figure 1 As shown, the data tracing method based on runtime snapshots provided in this application embodiment may include steps 110-160.

[0034] Step 110: If a report is generated, generate a runtime memory snapshot file associated with the report based on the obtained report traceability information;

[0035] Step 120: Parse the runtime memory snapshot file, extract all keys in the first mapping relationship to construct a header text set, and query the N metadata identifiers corresponding to the N header texts in the header text set according to the first mapping relationship;

[0036] Step 130: Access the report data result storage area in the runtime memory snapshot file based on N metadata identifiers, query and obtain the data value list corresponding to each metadata identifier, obtain the data value list corresponding to each header text, and render and generate a visual report snapshot under the report dimension based on the header text set and the data value list corresponding to each header text.

[0037] Step 140: Query task details based on N metadata identifiers. For each metadata identifier, extract the metadata identifiers of its directly dependent and indirectly dependent metadata from the task details associated with it. Based on the hierarchical dependency relationship between the metadata, take the metadata identifier corresponding to the table header text as the root node and the metadata identifiers of each metadata as child nodes, recursively generate a multi-level tree structure to obtain a single metadata tree.

[0038] Step 150: Render N metadata tree snapshots based on N metadata trees, with N metadata identifiers in the metadata dimension;

[0039] Step 160: In response to the user's click operation on the abnormal data value on the visual report snapshot, the metadata tree snapshot of the metadata identifier corresponding to the metadata column of the report data where the abnormal data value is located is displayed by jumping to show the metadata flow process corresponding to the abnormal data value.

[0040] The data tracing method based on runtime snapshots provided in this application actively acquires complete report tracing information at the end of report execution, generating a runtime memory snapshot file. This runtime memory snapshot file records detailed information about the report generation process, avoiding the information loss problems associated with traditional methods that rely on scattered logs and manual environment reconstruction. Based on this, by parsing the snapshot file and rendering a visual report snapshot based on a first mapping relationship, users can intuitively locate abnormal data. Furthermore, a metadata tree snapshot is recursively constructed based on metadata identifiers. This metadata tree snapshot clearly presents complex and deep-level computational logic in a visual manner, making each layer of metadata computation observable, thus clearly presenting the complete dependency chain and computational hierarchy from the final calculation result to the source data. Finally, a visual interactive interface is provided for multi-dimensional linking and navigation. Click operations enable navigation from report data to the metadata tree snapshot, and task identifiers and task execution information are displayed at each node, achieving a horizontal link between data lineage and task execution flow. This guides users to quickly focus on the root cause of the problem. Compared to traditional troubleshooting processes that heavily rely on manual experience, this application improves the accuracy and efficiency of deep-level data problem investigation.

[0041] The specific implementation of the above steps will be described in detail below with reference to specific embodiments.

[0042] In step 110, when a report is generated, a runtime memory snapshot file associated with the report is generated based on the obtained report traceability information.

[0043] In step 110, the report traceability information is a multi-dimensional data set that records the entire process of report data from generation to final presentation. This multi-dimensional data is generated during the execution of the report generation program and stored in the running memory. It includes at least the task details of all data processing tasks during the report execution process, the first mapping relationship between N report data columns and metadata in the report, and the report data results, where N is a positive integer.

[0044] Metadata describes the meaning, source, rules, and structure of report data columns. It connects the front-end report display (the column names seen by the user, such as "user age") with the back-end data generation logic (the task of calculating the age and the metadata definition). When a user clicks on a report column in the visualization interface, the system can immediately locate the original metadata definition that generated the data through the first mapping relationship. Then, it can recursively parse its complete data lineage and dependencies, that is, which underlying data it is derived from and which calculation rules it is derived from.

[0045] There are mapping relationships among report data columns, metadata, and data processing tasks. The first mapping relationship is between report data columns and metadata, and the second mapping relationship is between report data columns and data processing tasks. Report data columns consist of header text and data values. The header text is the title or name of the report data column, and the data values ​​are the specific content of the report data column. For example, "Total Sales" is the header text, and 50000, 42000, etc., below it are the data values. Together, they constitute a column of the report. Each report data column is associated with a data processing task. Each specific data (cell) under the same report data column is generated or provided by the same specific data processing task. For example, for the "Total Sales" data column, its associated data processing task is "Calculate Total Product Sales". Each report data column is associated with metadata. Each specific data (cell) under the same report data column must conform to the definition and rules of the metadata of its column. For example, 50000, 42000, etc., must conform to the rule defined in the "Total Sales" metadata: "The sum of the amounts of all sales records".

[0046] Based on this, each data processing task is uniquely associated with one metadata element. This metadata element is the output metadata of the data processing task and also the final result of task execution. The core purpose of a data processing task is to generate data that conforms to a certain metadata definition. For example, the task of "calculating total product sales" mentioned above aims to realize the definition of the metadata element "total sales." Although each data processing task is associated with only one metadata element, the task needs to read, reference, or use one or more other metadata elements during execution. For example, the task of "calculating total product sales" needs to rely on multiple metadata elements such as "original sales records," "currency conversion rules," and "statistical time range." These metadata elements are the source or calculation basis of the output metadata. Therefore, these data elements can be identified as the metadata elements that the associated metadata element directly or indirectly depends on. In the task details information of each data processing task, the metadata identifier of its associated metadata element can be recorded through the `elements` field, and the metadata identifier of each metadata element that the associated metadata element depends on can be recorded through the `dependElements` field.

[0047] Task details refer to the set of attributes that describe the execution process, status, performance, and relationships of each data processing task during report execution. This may include, but is not limited to, the following attributes: Task Identifier: A unique identifier for the data processing task, used to accurately locate and identify the task within the system; Task Execution Status: Describes the final execution result status of the task, such as: SUCCESS, FAIL, INIT, or RUNNING; Task Execution Time: The time consumed from the start to the end of the task, usually in milliseconds, used for performance analysis and bottleneck identification; Task Dependencies: Describes the contextual relationships of the task in the overall execution flow, including a list of identifiers of its dependent preceding tasks and a list of identifiers of its dependent subsequent tasks, which together define the execution order and data flow between tasks; Data Dependencies: Describes the list of metadata identifiers (dependElements) of the input data that the task depends on during execution, as well as the identifiers (elements) of the output metadata produced and defined by the task, clarifying the task's input and output; Task Execution Information: Dynamically changing execution context information based on the type of metadata associated with the task.

[0048] The report data result refers to the final output value collection after all data processing tasks have been completed following the report execution process. This collection is a structured data storage area, with key-value pairs as its basic unit. The key is a metadata identifier, and the value is a list of specific data values ​​corresponding to that metadata identifier. This list contains the values ​​of that metadata across all data rows in the report, serving as the data units for the final rendering and population of the visual report. Therefore, the report data result is the direct data source for generating the visual report snapshot. It is associated with the report header text through the first mapping relationship, together forming the report view seen by the end user.

[0049] Runtime memory snapshot files refer to data files formed after the report generation program finishes execution, which solidify the report source information through serialization and other means. Therefore, this file is the basic data source for all subsequent analysis and visualization operations.

[0050] Specifically, after the report is generated, the memory is not released immediately. Instead, the report source information in the memory is intercepted and collected. This information is processed by serialization libraries (such as JSON and Protocol Buffers), converted into a byte stream and written to a disk file, thereby generating a runtime memory snapshot file.

[0051] Step 120 involves parsing the runtime memory snapshot file, extracting all keys in the first mapping relationship to construct a header text set, and querying the N metadata identifiers corresponding to the N header texts in the header text set according to the first mapping relationship.

[0052] In step 120, the runtime memory snapshot file is read, and the item2Meta (first mapping relationship) structure is located first. In this first mapping relationship, the key is the data column name, corresponding to the column name of the sheet in the output Excel file, and the value is the metadata identifier, which may include at least one of metadata ID and metadata name. The system extracts all keys from the first mapping relationship to form a set of header text, and retrieves the metadata identifier (value) corresponding to each header text.

[0053] In step 130, the report data result storage area in the runtime memory snapshot file is accessed based on N metadata identifiers. The list of data values ​​corresponding to each metadata identifier is queried and obtained, and the list of data values ​​corresponding to each header text is obtained. Based on the set of header texts and the list of data values ​​corresponding to each header text, a visual report snapshot under the report dimension is rendered and generated.

[0054] In step 130, the report data result storage area refers to the structured area in the runtime memory snapshot file specifically used to store report data results. It exists in the form of key-value pairs, where the key can be a metadata identifier and the value is a list of specific data values ​​corresponding to that metadata.

[0055] Specifically, the system uses N metadata identifiers as keys to access the area (data) where the final data is stored in the snapshot file, querying and retrieving the complete list of data values ​​under each metadata identifier. Finally, the header set and the corresponding data value lists for each column are combined to render a report interface that is completely consistent with what the end user sees.

[0056] For example, the metadata identifier for the table header "Total Sales" is meta_sales_total. The system finds this identifier in the data area and retrieves its value list [10000, 12000, 11000, 10500] for rendering.

[0057] In step 140, task details are queried based on N metadata identifiers. For each metadata identifier, the metadata identifiers of its directly dependent and indirectly dependent metadata are extracted from the task details. Based on the hierarchical dependency relationship between the metadata, a multi-level tree structure is recursively generated with the metadata identifier corresponding to the table header text as the root node and the metadata identifiers of each metadata as child nodes, to obtain a single metadata tree.

[0058] Specifically, the system starts with N metadata identifiers. For each metadata identifier, it queries the `dependElements` field in its task details to recursively retrieve all its direct and indirect dependent metadata. This process continues to trace downwards until it finds the underlying metadata that no longer depends on any other metadata (such as fields read directly from the database). Ultimately, based on these dependencies, a tree structure with metadata as the root node is built from bottom to top, clearly showing the complete lineage of the data.

[0059] In some embodiments of this application, step 140 may specifically include the following steps:

[0060] For each metadata identifier, obtain the metadata identifiers of each metadata that it directly depends on from the task details information of the data processing task associated with that metadata identifier, and construct a metadata set;

[0061] Recursively parse the metadata that each metadata in the metadata set directly depends on;

[0062] Repeat this process until the underlying metadata, which does not depend on other metadata, is obtained;

[0063] Based on the parsed hierarchical dependencies, a tree structure is generated with the metadata identifier corresponding to the header text as the root node, the metadata identifier of the basic metadata as the leaf node, and the metadata identifier of the first metadata as the intermediate node. The first metadata is the metadata other than the basic metadata in each metadata.

[0064] Basic metadata refers to metadata whose dependElements property is empty. This type of metadata does not depend on any other metadata and is usually the lowest-level data obtained directly from the data source, existing as a leaf node in the metadata tree. Intermediate nodes refer to metadata nodes in the metadata tree that are neither root nodes nor leaf nodes. They are both the result of dependencies of upper-level metadata and themselves depend on one or more lower-level metadata.

[0065] Specifically, starting with a metadata identifier, the `dependElements` list in its task details is accessed to obtain all its directly dependent metadata identifiers, forming the first-level metadata set. Then, for each metadata identifier in the set, the above process is repeated to obtain its dependent metadata, until the `dependElements` list of a certain metadata is found to be empty, indicating it is basic metadata, and the recursion terminates. Based on all the parsed hierarchical relationships, a tree structure is constructed from bottom to top: with basic metadata as leaf nodes and its direct dependencies as parent nodes, layer by layer upwards, ultimately forming a complete metadata tree with the target metadata as the root node.

[0066] In one example, such as Figure 2 As shown, the metadata tree is a tree with the metadata node corresponding to "151568" as the root node, the metadata nodes corresponding to "151414", "151551" and "150962" as intermediate nodes, and the metadata node corresponding to "150415" as the leaf node.

[0067] In another example, the system begins building with `meta_profit_total` as the root node: querying its task information reveals `dependElements:["meta_sales_total","meta_cost_total"]`. Continuing to query the dependencies of `meta_sales_total`, we find `dependElements:["meta_sales_q1","meta_sales_q2"]`. Querying the dependencies of `meta_sales_q1` reveals `dependElements:[]` (empty), indicating it's basic metadata, so tracing stops. The same applies to `meta_sales_q2` and `meta_cost_total`. This ultimately forms a tree with the root node being "Total Profit," calculated from "Total Sales" and "Total Cost"; "Total Sales" is obtained by adding "Q1 Sales" and "Q2 Sales".

[0068] In this embodiment, by recursively parsing the direct and indirect dependencies of metadata identifiers down to the basic metadata, and constructing a tree structure based on hierarchical dependencies with the metadata corresponding to the table header text as the root node and the basic metadata as the leaf nodes, the generation path of any data unit in the report is fully presented. This clearly shows the complete flow and calculation process of data from the source (leaf node) to the final result (root node), facilitating user understanding and use without manual sorting, and ensuring the integrity and accuracy of the lineage. Based on this, the solution can accurately locate the specific link of abnormal data in the complex calculation chain. Errors in expression calculations (expression_vertex) or aggregation operations (aggregation_vertex) can be traced layer by layer through the tree structure, thereby improving the depth and accuracy of report data problem troubleshooting.

[0069] Step 150 involves generating N metadata tree snapshots based on N metadata trees, with N metadata identifiers in the metadata dimension.

[0070] In step 150, each node of the metadata tree snapshot displays at least: the metadata identifier and the data results corresponding to its associated data processing task, the task identifier, and the task execution information.

[0071] Specifically, the system renders the tree-shaped data structure constructed in step 140 in a graphical manner. Each node not only displays its metadata identifier but also its corresponding data value, the task identifier that generated the data, and task execution information, thus providing a rich analytical view. The data results, task identifier, and task execution information are all included in the task details information.

[0072] For example, for Figure 2 The metadata identifier for the node “150415_Social Security Contribution Type Code_Details-Social Security Details_30034” includes: Value (data value), Task (task identifier), and task execution information: db (database) and sql (query statement).

[0073] In some embodiments of this application, each data processing task is associated with a metadata, which is the output metadata of the data processing task. The task details of each data processing task include at least the following fields: task identifier, task execution information, data result, metadata identifier elements of the associated metadata, and metadata identifiers dependElements of each metadata that the associated metadata depends on.

[0074] Output metadata refers to the metadata ultimately generated and defined by the data processing task. A task has one and only one output metadata, but may depend on multiple input metadata. When generating a snapshot, the system creates a detailed object for each task and populates its key fields: task identifier (identity): a globally unique ID for the task; addition: different execution context information based on the task type; data: stores the output results after the task is executed; elements: records the identifiers of the task's output metadata; and dependElements: records a list of identifiers of all metadata that the task depends on during execution. The elements and dependElements fields clearly define the inputs and outputs of each task and are the basis for constructing task flows and data lineage.

[0075] In this embodiment, by defining specific fields in the task details information, a complete execution context is provided for each data processing task. This unified structured information recording method ensures the consistency and parsability of snapshot data, enabling the rapid acquisition of task identifiers, execution status, data results, and dependencies when troubleshooting report data issues. This provides a comprehensive and reliable diagnostic basis for locating the root cause of computational anomalies in report data.

[0076] In some embodiments of this application, the task execution information varies depending on the type of metadata associated with the data processing task. The task execution information corresponding to database type metadata includes the database name and database query statement; the task execution information corresponding to aggregation type metadata includes the aggregated metadata information and grouping condition metadata; the task execution information corresponding to MaxVersion type metadata includes the aggregated metadata information, groupBy metadata, and version number; and the task execution information corresponding to expression type metadata includes the expression content.

[0077] Specifically, the task execution information (addition) field is a dynamic structure whose content depends on the type of metadata associated with the task. It stores the contextual information necessary to understand the task execution process, and the task execution information will differ depending on the type of metadata. When the system captures a task, it populates the addition field according to its type: Database type: populates datasource (data source name) and query (query SQL); Aggregation type: populates aggregated_metadata (aggregated metadata) and groupBy (grouping condition); MaxVersion type: populates aggregated_metadata, groupBy, and version (version field); Expression type: populates expression (expression string).

[0078] As a concrete example, such as Figure 2 As shown, the task execution information for database type metadata “151414”, “150962” and “150415” includes db (database) and sql (query statement); the task execution information for aggregation type metadata “151568” includes aggr (aggregated metadata) and groupBy (grouping condition); and the task execution information for expression type metadata “151551” includes expr (expression string).

[0079] In this embodiment, by dynamically recording differentiated task execution information (such as database query statements, aggregation grouping conditions, expression content, etc.) based on metadata types, the relevance and interpretability of snapshot information are enhanced. This allows for direct identification of the specific calculation logic and execution environment of different types of metadata when analyzing report data errors. This enables rapid determination of whether the problem stems from incorrect expression definitions, aggregation rule configuration issues, or data source anomalies, providing contextual information for accurately locating the root cause of the problem and thus improving troubleshooting efficiency.

[0080] In step 160, in response to the user's click operation on the abnormal data value on the visual report snapshot, a metadata tree snapshot of the metadata identifier corresponding to the metadata column of the report data where the abnormal data value is located is displayed by jumping to show the metadata flow process corresponding to the abnormal data value.

[0081] In step 160, when a user clicks on an abnormal data value on the visual report snapshot, the system can determine the report data column where the abnormal data value is located, and determine the metadata identifier corresponding to the abnormal data value by querying the first mapping relationship, thereby automatically triggering the interface to jump and display the corresponding metadata tree snapshot, seamlessly tracing the complete generation chain of the abnormal data value.

[0082] For example, in the report snapshot, if a user notices that the "Total Profit" in the second row is displayed as -1000 (clearly abnormal), clicking on this cell will cause the system to determine, through mapping relationships, that the data in this cell comes from the metadata `meta_profit_total`. The system will then automatically render and display a snapshot of the metadata tree rooted at `meta_profit_total` on the other side of the interface or in a new page. The user can immediately see that the negative profit is due to the "Total Costs" being significantly higher than the "Total Sales," and can then trace further down to pinpoint a cost calculation task or an error in the source data input.

[0083] In some embodiments of this application, the task details information further includes the following fields: the data source to which the associated metadata belongs, previousTasks, and nextTasks. After parsing the runtime memory snapshot file, the following steps may also be included:

[0084] Parse the task containers in the runtime memory snapshot file, traverse the task details stored in the task containers, generate a corresponding task node for each data processing task, and use the task identifier associated with the data processing task as the unique node identifier of the task node.

[0085] For each data processing task, the data, datasource, previousTasks, and nextTasks fields of the task details are parsed to generate multiple child nodes for the corresponding task node of the data processing task. These child nodes include data result nodes, data source nodes, previous task nodes, and next task nodes.

[0086] The task tree for each data processing task is constructed based on the task node and its multiple child nodes, and a task tree snapshot is generated by combining the task tree and task execution information.

[0087] Specifically, the system iterates through the task containers (taskContainers), creating a task node for each task and using the task's identifier as the unique identifier for this node. This task identifier can consist of the metadata identifier associated with the data processing task and its metadata type. For each task node, other fields in its task details are further parsed, and multiple child nodes are created for it.

[0088] In this system, a task node represents a graphical entity of an independent data processing task and is the basic building block of the task tree. Child nodes are subordinate nodes used to expand and describe the detailed information of a task node. Each child node describes different aspects of the task, such as data source, execution result, and dependencies.

[0089] For example, such as Figure 3 As shown, for the task node “aggregation_vertex:146_charge category|non-fixed service fee|principal_summary-

employees

[0090] In this embodiment, by parsing the task container and constructing a task tree containing multi-dimensional sub-nodes such as data results, data sources, and preceding and succeeding tasks, the complete information of a task is displayed from different dimensions (data, source, dependency), providing multi-dimensional display of task details and achieving a visual reconstruction of the entire task execution process. This task tree snapshot can intuitively display the input and output, execution order, and resource dependencies of data processing tasks, helping troubleshooters quickly pinpoint whether report data problems are caused by computation failure, execution timeout, or abnormal collaboration between preceding and succeeding tasks, thus optimizing the analysis process of complex task flows. Furthermore, the task tree clearly shows the execution order and dependencies between tasks, which helps in analyzing performance bottlenecks (critical paths) and cascading failures caused by upstream task failures.

[0091] In some embodiments of this application, the root node and child nodes in the metadata tree snapshot are both metadata nodes. After the metadata tree snapshot showing the metadata identifier corresponding to the report data column containing the abnormal data value, the following steps may also be included:

[0092] In response to a user's click on a task identifier in an abnormal metadata node in the metadata tree snapshot, the system will redirect to display the task tree snapshot of the data processing task corresponding to the task identifier, showing the task execution details of the abnormal metadata node.

[0093] Specifically, when rendering the metadata tree snapshot, the system displays the task identifier of its associated task on each metadata node and sets it as an interactive control, such as a clickable link or button. When a user has a question about a node during the metadata tree analysis process and clicks on its task identifier, the system captures this event and automatically triggers a redirect to display the task tree snapshot of that task, showing its detailed execution information.

[0094] In this embodiment, by responding to clicks on task identifiers in metadata tree nodes and navigating to the corresponding task tree snapshot, a correlation analysis channel between data lineage and task execution links is established. This breaks down the barriers between the metadata tree and the task tree, allowing users to switch between two different perspectives: data lineage tracing and task execution details, enabling cross-dimensional analysis. This interactive design allows investigators to directly jump to the execution details of the task that generated the abnormal metadata node after discovery, immediately viewing the specific execution status and context of the task that generated it. This helps determine whether the problem stems from a task logic error or an execution anomaly, thus achieving rapid tracing from data problem symptoms to the root cause of task computation, improving the consistency and accuracy of report data problem investigation.

[0095] In some embodiments of this application, the report traceability information may further include a second mapping relationship between N report data columns and data processing tasks. After rendering and generating the visual report snapshot under the report dimension as described above, the following steps may also be included:

[0096] In response to a user's click on the abnormal header text in the visual report snapshot, the system queries the data processing task corresponding to the report data column where the abnormal header text is located from the second mapping relationship, based on the abnormal header text.

[0097] By jumping to display a snapshot of the task tree for data processing tasks, the task execution details of the report data column containing the abnormal header text can be shown.

[0098] Specifically, the second mapping relationship (item2Task[]) refers to the mapping table in the report traceability information that connects the report data column with the task that generates the data in that column. Its key is the name of the report data column and its value is the task identifier.

[0099] When rendering the report snapshot, the system binds a click event to each header text. When a user clicks on an abnormal header, the system queries the item2Task mapping table to directly find the task responsible for generating all data for that column. The system then jumps to the task tree snapshot of that task, displaying its execution details, such as execution time and status.

[0100] In this embodiment, by utilizing a second mapping relationship to achieve direct jump from abnormal header text to the corresponding data processing task, users are provided with an alternative analysis path besides clicking on specific data units. When it is suspected that the entire column of data is problematic, the header can be clicked directly to quickly locate the source task, improving troubleshooting efficiency. This emphasizes the ability to troubleshoot problems from the task execution level (such as performance and status), supplementing the shortcomings of a single data perspective. Users can directly view the task details that generated the column of data without tracing metadata dependencies layer by layer, accelerating the process of locating problems with overall column-level errors in reports (such as deviations or missing data in the entire column).

[0101] In some embodiments of this application, after rendering and generating a task tree snapshot by combining the task tree and task execution information, the following may also be included:

[0102] In response to a user's click on a preceding task node in the current task tree snapshot, the system will jump to display the task tree snapshot of the data processing task corresponding to the preceding task node, showing the task execution details of the preceding task.

[0103] Specifically, when rendering the task tree, the system renders each task identifier in the previousTasks field as a clickable preceding task node. When a user clicks on a preceding task node, the system loads and displays a snapshot of the corresponding preceding task's task tree based on that identifier, enabling an interactive process of tracing back within the task tree.

[0104] For example, such as Figure 4 As shown, for the task tree of the task node “aggregation_vertex:146_chargeable category|non-fixed service fee|principal_summary-

employees

[0105] In this embodiment, by supporting clicking on a preceding task node in the task tree snapshot and jumping to view its details, the ability to trace the task execution flow in reverse is achieved. This feature allows users to trace upstream along the task execution chain, so investigators can quickly jump from the current abnormal task to its upstream task, thereby tracing the root cause of the problem along the execution chain. This facilitates analysis of whether the problem of the current task is caused by its dependent preceding tasks, realizing complete visual tracing of the task execution flow. It is particularly suitable for handling cascading report data problems caused by upstream data errors, significantly improving the investigation efficiency in complex dependency scenarios.

[0106] In some embodiments of this application, the report tracing information also includes basic report information. Before step 160 above responds to the user's click operation on the abnormal data value on the visual report snapshot, the method may further include the following steps:

[0107] Once the snapshot archive is obtained, it is parsed to obtain multiple runtime memory snapshot files corresponding to multiple reports.

[0108] Parse each runtime memory snapshot file to obtain the report name and data dimension information of multiple reports from the basic report information;

[0109] The data snapshot interface displays the snapshot archive file name, the report names of multiple reports, and data dimension information in a tree structure;

[0110] In response to a user's click on the target report name among multiple report names, a visual snapshot of the report corresponding to that report name is displayed.

[0111] Specifically, a snapshot archive refers to a single compressed file formed by packaging and compressing runtime memory snapshot files of multiple reports, facilitating unified management, transmission, and archiving. The system supports uploading and parsing .zip format snapshot archives. After parsing, the system reads the basic report information from each snapshot file within the archive and renders a tree-structured index list on the interface. Users can switch between viewing the snapshot content of different reports by clicking on different report names in the index list. The basic report information may include, but is not limited to, the following fields: report ID, report name, report data dimension, data dimension ID, data dimension name, data dimension description, dimension type, and whether the data dimension is an array.

[0112] For example, such as Figure 5 As shown, the data snapshot interface displays ① the snapshot compressed file name, ② the report name, and ③ the data dimension information corresponding to the selected report.

[0113] In this embodiment, by parsing the snapshot compressed package and constructing a tree-structured index interface containing multiple report names and data dimensions, a unified management and comparative analysis function for batch snapshots is achieved. This allows users to load and analyze multiple snapshots simultaneously, facilitating comparative analysis of reports at different time points or different versions, greatly enhancing the tool's practicality and scalability. Users can load and switch between multiple historical snapshots simultaneously, making it easy to compare report data and task execution status at different times or versions. This helps identify potential data trend anomalies or changes in computational logic introduced by version upgrades, providing historical reference and decision support for troubleshooting report data issues across time periods.

[0114] For example, the complete data structure of the runtime memory snapshot file of this application can be shown in Table (1):

[0115] Table (1)

[0116]

[0117]

[0118] In the above data structure, field AH represents basic report information, field I-AC represents task details, field AD-AF represents the second mapping relationship, field AG-AI represents the first mapping relationship, and field AJ-AM represents the report data results.

[0119] As a specific example, based on the complete data structure of the runtime memory snapshot file shown in Table (1), the data tracing process based on runtime snapshots in this application is described in detail:

[0120] Step 1: During program execution, when generating data snapshots, multiple runtime memory snapshot files of reports can be generated simultaneously and packaged into a zip file. During parsing, the multiple snapshot files within the zip file are parsed first, such as... Figure 5 As shown, it is displayed in a tree structure. Clicking on the corresponding snapshot will further display the lighting details of that snapshot;

[0121] Step 2: Parse all the header text from the item2Meta / key(AG / AH) path of table (1) and use it to render the header. Obtain the metadata identifier corresponding to the header text from the item2Meta / key(AG / AI) path, and then use the metadata identifier to obtain the list of all data values ​​corresponding to the metadata from the item2Meta / value(AJ / AM) path, which is used to render all data rows and obtain a snapshot of the visualization report.

[0122] Step 3: Obtain the metadata identifier of the tree structure to be displayed from the item2Meta(AG) table (1), use the metadata identifier to obtain the corresponding task details from the taskContainer(I), continue to obtain the metadata (dependElements) that the metadata depends on from the task details, recursively parse, and finally obtain the tree structure and the metadata tree;

[0123] Step 4: Recursively parse all task details from table (1) taskMap(J); traverse all task elements under taskContainer / taskMap, generate a tree node for each task element, where key(K) serves as the node identifier and is also used for positioning when jumping between task nodes;

[0124] Step 5: Parse datasource(M) to obtain the data source to which the metadata of the task belongs, and generate the child node of the task node: datasource;

[0125] Step 6: Parse the elements to obtain the metadata associated with the task; parse the dependElements to obtain the metadata that the metadata associated with the task depends on.

[0126] Step 7: Parse previousTasks(P) to obtain the previous tasks that the task depends on, and generate the child node previousTasks of the above task node; parse nextTasks(P) to obtain the subsequent tasks that the task depends on, and generate the child node nextTasks of the above task node.

[0127] Step 8: Parse the data result after the task is completed from data(T) and generate the child node data of the above task node;

[0128] Step 9: Based on the metadata type corresponding to the task: For database type (rds) metadata, parse the database name (db), query statement (sql), etc., from the addition (AC); for aggregation type (aggregation) metadata, parse the necessary attributes of the aggregation type metadata from the addition (AC): group metadata (groupBy), aggregation operation (aggr), etc.; for MaxVersion type (maxversion) metadata, parse the necessary attributes of the aggregation type metadata from the addition (AC): group metadata (groupBy), aggregation operation (aggr), version (version), etc.; for expression type (expression) metadata, parse the expression (expr) from the addition (AC); for association type (extra) metadata, parse the association information from the addition (AC): dictCode, dictValue, etc.

[0129] Step 10: Based on the above task nodes and their multiple child nodes, construct the task tree for each data processing task, and combine the task tree and task execution information to render and generate a process such as... Figure 3 or Figure 4 The task tree snapshot shown;

[0130] Step 11, Troubleshooting Process: After generating the report, if the user finds data anomalies in the report, download the snapshot file of the report, parse and load it to display it in a visual format; clicking on the header of the column with the problem in the visual report snapshot will jump to the task tree snapshot corresponding to that header;

[0131] Check the task details in the task tree snapshot, and jump to the preceding and subsequent tasks as needed to further investigate and locate the problem; click on the abnormal data cell in the visualization report snapshot to jump to the metadata tree snapshot corresponding to the abnormal data cell, and investigate and locate the problem in the metadata tree; if an abnormal metadata node is found in the metadata tree, click on the task identifier link on the metadata node to jump to the corresponding task tree snapshot to further investigate the task details.

[0132] By navigating between the multi-dimensional data, configurations, and execution flows mentioned above, the root cause of problems with report data can be quickly identified and located.

[0133] Corresponding to the method embodiments of this application, this application also provides a data tracing device based on runtime snapshots.

[0134] Figure 6 This is a schematic diagram of a data tracing device based on runtime snapshots provided in an embodiment of this application. Figure 6 As shown, the data tracing device 600 based on runtime snapshots may include: a generation module 610, a parsing module 620, a rendering generation module 630, a metadata tree construction module 640, and a snapshot display module 650.

[0135] The report generation module 610 generates a runtime memory snapshot file associated with the report based on the obtained report traceability information when generating a report. The report traceability information includes at least the task details of all data processing tasks during report execution, the first mapping relationship between N report data columns and metadata, and the report data results. The parsing module 620 parses the runtime memory snapshot file, extracts all keys from the first mapping relationship to construct a header text set, and queries the N metadata identifiers corresponding to the N header texts in the header text set according to the first mapping relationship. The rendering generation module 630 accesses the report data result storage area in the runtime memory snapshot file based on the N metadata identifiers, queries and obtains the data value list corresponding to each metadata identifier, obtains the data value list corresponding to each header text, and renders and generates a visual report snapshot at the report dimension based on the header text set and the data value list corresponding to each header text. The metadata tree construction module 640... The system is used to query task details based on N metadata identifiers. For each metadata identifier, it extracts the metadata identifiers of its directly and indirectly dependent metadata from the task details. Based on the hierarchical dependency relationship between the metadata, it recursively generates a multi-level tree structure with the metadata identifier corresponding to the table header text as the root node and the metadata identifiers of each metadata as child nodes, to obtain a single metadata tree. The rendering and generation module 630 is used to render and generate N metadata tree snapshots of the N metadata identifiers in the metadata dimension based on the N metadata trees. Each node of the metadata tree snapshot displays at least: the metadata identifier and the data result corresponding to its associated data processing task, the task identifier, and the task execution information. The snapshot display module 650 is used to respond to the user's click operation on the abnormal data value on the visual report snapshot, and displays the metadata tree snapshot of the metadata identifier corresponding to the metadata identifier of the report data column where the abnormal data value is located by jumping, showing the metadata flow process corresponding to the abnormal data value.

[0136] The data tracing device based on runtime snapshots provided in this application actively acquires complete report tracing information at the end of report execution and generates a runtime memory snapshot file. This runtime memory snapshot file records detailed information about the report generation process, avoiding the information loss problems caused by relying on scattered logs and manually rebuilding the environment in traditional methods. Based on this, by parsing the snapshot file and rendering a visual report snapshot based on the first mapping relationship, users can intuitively locate abnormal data. Furthermore, a metadata tree snapshot is recursively constructed based on metadata identifiers. This metadata tree snapshot clearly presents complex and deep-level computational logic in a visual manner, making each layer of metadata computation observable, thus clearly presenting the complete dependency chain and computational hierarchy from the final calculation result to the source data. Finally, a visual interactive interface is provided for multi-dimensional association and navigation. Click operations enable navigation from report data to the metadata tree snapshot, and task identifiers and task execution information are displayed at each node, achieving a horizontal association between data lineage and task execution flow. This guides users to quickly focus on the root cause of the problem. Compared to the traditional investigation process that heavily relies on manual experience, this application can improve the accuracy and efficiency of in-depth data problem investigation.

[0137] In some embodiments of this application, each data processing task is associated with a metadata, which is the output metadata of the data processing task. The task details of each data processing task include at least the following fields: task identifier, task execution information, data result, metadata identifier elements of the associated metadata, and metadata identifiers dependElements of each metadata that the associated metadata depends on.

[0138] In some embodiments of this application, the task execution information varies depending on the type of metadata associated with the data processing task. The task execution information corresponding to database type metadata includes the database name and database query statement; the task execution information corresponding to aggregation type metadata includes the aggregated metadata information and grouping condition metadata; the task execution information corresponding to MaxVersion type metadata includes the aggregated metadata information, groupBy metadata, and version number; and the task execution information corresponding to expression type metadata includes the expression content.

[0139] In some embodiments of this application, the task details information further includes the following fields: the data source to which the associated metadata belongs, previousTasks, and nextTasks. It also includes a task tree construction module, specifically used for: after parsing the runtime memory snapshot file, parsing the task containers in the runtime memory snapshot file, traversing the task details information stored in the task containers, generating a corresponding task node for each data processing task, and using the task identifier associated with the data processing task as the unique node identifier for that task node; for each data processing task, parsing the data, datasource, previousTasks, and nextTasks fields of the task details information, generating multiple child nodes for the corresponding task node of the data processing task, including data result nodes, data source nodes, previous task nodes, and next task nodes; constructing a task tree for each data processing task based on the task node and its multiple child nodes, and rendering a task tree snapshot by combining the task tree and task execution information.

[0140] In some embodiments of this application, the root node and child nodes in the metadata tree snapshot are both metadata nodes. The snapshot display module 650 is further configured to: after displaying the metadata tree snapshot of the metadata identifier corresponding to the metadata identifier of the report data column where the abnormal data value is located, respond to the user's click operation on the task identifier in the abnormal metadata node in the metadata tree snapshot, and display the task tree snapshot of the data processing task corresponding to the task identifier by jumping to display the task execution details corresponding to the abnormal metadata node.

[0141] In some embodiments of this application, the report traceability information also includes a second mapping relationship between N report data columns and data processing tasks. The snapshot display module 650 is further configured to: after rendering and generating a visual report snapshot in the report dimension, in response to the user's click operation on the abnormal header text on the visual report snapshot, query the data processing task corresponding to the report data column where the abnormal header text is located from the second mapping relationship based on the abnormal header text; and display the task execution details of the report data column where the abnormal header text is located by jumping to display the task tree snapshot of the data processing task.

[0142] In some embodiments of this application, the snapshot display module 650 is further configured to: after rendering and generating a task tree snapshot by combining the task tree and task execution information, in response to the user's click operation on the preceding task node in the current task tree snapshot, display the task tree snapshot of the data processing task corresponding to the preceding task node by jumping to display the task execution details of the preceding task.

[0143] In some embodiments of this application, the report tracing information also includes basic report information. The parsing module is further configured to, before responding to a user's click operation on an abnormal data value in the visual report snapshot, parse the snapshot compressed package to obtain multiple runtime memory snapshot files corresponding to multiple reports, provided the snapshot compressed package is obtained. The parsing module is also configured to parse each runtime memory snapshot file to obtain the report name and data dimension information of multiple reports from the basic report information. The snapshot display module is further configured to display the snapshot compressed package file name, the report names of multiple reports, and the data dimension information in a tree structure on the data snapshot interface. The snapshot display module is also configured to, in response to a user's click operation on a target report name among multiple report names, display the visual report snapshot corresponding to the target report name.

[0144] The data tracing device based on runtime snapshots provided in this application embodiment can achieve... Figure 1-5 The various processes implemented by the service platform in the method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0145] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0146] like Figure 7 As shown, the electronic device 700 includes a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702.

[0147] In one example, the processor 702 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0148] Memory 701 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the data tracing method based on runtime snapshots according to the embodiments of the first aspect of this application.

[0149] The processor 702 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 701, in order to implement the data tracing method based on runtime snapshots in the embodiments of the first aspect described above.

[0150] In some examples, the electronic device 700 may also include a communication interface 703 and a bus 710. For example, Figure 7 As shown, the memory 701, processor 702, and communication interface 703 are connected through bus 710 and complete communication with each other.

[0151] The communication interface 703 is mainly used to enable communication between various modules, devices, units, and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 703.

[0152] Bus 710 includes hardware, software, or both, that couples components of electronic device 700 together. For example, and not limitingly, bus 710 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 710 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0153] The electronic device provided in this application embodiment is capable of achieving Figure 1-5The various processes implemented by the electronic device in the method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0154] In conjunction with the data tracing method based on runtime snapshots in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the steps of any of the data tracing methods based on runtime snapshots in the above embodiments.

[0155] In conjunction with the data tracing method based on runtime snapshots in the above embodiments, this application embodiment can provide a computer program product to implement this method. This (computer) program product is stored in a non-volatile storage medium, and when executed by at least one processor, it implements the steps of any of the data tracing methods based on runtime snapshots in the above embodiments.

[0156] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described data tracing method embodiment based on runtime snapshots, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0157] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0158] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0159] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0160] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0161] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0162] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for data provenance based on runtime snapshots, characterized in that, Comprise: In the case of generating a report, based on the obtained report trace information, generate the runtime memory snapshot file associated with the report, wherein the report trace information at least includes the task detail information of all data processing tasks in the report execution process, the first mapping relationship between N report data columns and metadata in the report, and the report data result; Parse the runtime memory snapshot file, extract all keys in the first mapping relationship to build a table header text set, and query the N metadata identifiers corresponding to the N table header texts in the table header text set according to the first mapping relationship; Based on the N metadata identifiers, access the report data result storage area in the runtime memory snapshot file, query and obtain the data value list corresponding to each metadata identifier, obtain the data value list corresponding to each table header text, and based on the table header text set and the data value list corresponding to each table header text, render and generate a visual report snapshot under the report dimension; Based on the N metadata identifiers, query the task detail information, for each metadata identifier, extract the metadata identifiers of each metadata directly and indirectly dependent on it from its associated task detail information, and based on the hierarchical dependency relationship between the metadata, take the metadata identifier corresponding to the table header text as the root node, and take the metadata identifiers of the metadata as the child nodes. Recursive generation of multi-level tree structure, get single metadata tree; Based on N metadata trees, render and generate N metadata tree snapshots under the metadata dimension of the N metadata identifiers, wherein each node of the metadata tree snapshot displays at least: metadata identifier and its associated data processing task corresponding data result, task identifier, task execution information; In response to the click operation of the user on the abnormal data value in the visual report snapshot, display the metadata tree snapshot of the metadata identifier corresponding to the report data column of the abnormal data value through jumping, and show the metadata flow process corresponding to the abnormal data value.

2. The method of claim 1, wherein, Based on the N metadata identifiers, query the task detail information, for each metadata identifier, extract the metadata identifiers of each metadata directly and indirectly dependent on it from its associated task detail information, and based on the hierarchical dependency relationship between the metadata, take the metadata identifier corresponding to the table header text as the root node, and take the metadata identifiers of the metadata as the child nodes. Recursive generation of multi-level tree structure, get single metadata tree, including: For each metadata identifier, obtain the metadata identifiers of each metadata directly dependent on it from the task detail information of the data processing task associated with the metadata identifier, and construct a metadata set; Recursively analyze each metadata in the metadata set that is directly dependent on the metadata itself; Repeat the process until the basic metadata that does not depend on other metadata is obtained; According to the parsed hierarchical dependency relationship, a tree structure is generated, taking the metadata identifier corresponding to the table header text as a root node, taking the metadata identifier of the basic metadata as a leaf node, and taking the metadata identifier of the first metadata as an intermediate node, wherein the first metadata is the metadata other than the basic metadata among the metadata.

3. The method of claim 1, wherein, Each data processing task is associated with a metadata, which is the output metadata of the data processing task, and the task detail information of each data processing task at least includes the following fields: task identifier identity, task execution information addition, data result data, metadata identifier elements of the associated metadata, and metadata identifier dependElements of each metadata on which the associated metadata depends.

4. The method of claim 3, wherein, The task execution information is different according to the type of the metadata associated with the data processing task, and the task execution information corresponding to the database type metadata includes database name and database query statement; the task execution information corresponding to the aggregation type metadata includes aggregated metadata information and grouping condition metadata; the task execution information corresponding to the MaxVersion type metadata includes aggregated metadata information, groupBy metadata, and version number; The task execution information corresponding to the expression type metadata includes expression content.

5. The method of claim 3, wherein, The task detail information further includes the following fields: data source datasource to which the associated metadata belongs, previous task previousTasks, and next task nextTasks, and after parsing the runtime memory snapshot file, further includes: The task container in the runtime memory snapshot file is parsed, the task detail information stored in the task container is traversed, a corresponding task node is generated for each data processing task, and the task identifier associated with the data processing task is taken as the unique node identifier of the task node; For each data processing task, the data, datasource, previousTasks, and nextTasks fields of the task detail information are parsed respectively, a plurality of child nodes are generated for the task node corresponding to the data processing task, and the plurality of child nodes include data result node, data source node, previous task node, and next task node; The task tree of each data processing task is constructed based on the task node and the plurality of child nodes thereof, and the task tree snapshot is rendered and generated in combination with the task tree and the task execution information. The root node and the child node in the metadata tree snapshot are both metadata nodes, and after displaying the metadata tree snapshot of the metadata identifier corresponding to the report data column of the abnormal data value, further includes:

6. The method of claim 5, wherein, In response to the click operation of the task identifier in the abnormal metadata node in the metadata tree snapshot by the user, the task tree snapshot of the data processing task corresponding to the task identifier is displayed by jumping, and the task execution details corresponding to the abnormal metadata node are displayed. ​ 7. The method of claim 5, wherein, The report trace information further comprises a second mapping relationship between the N report data columns and data processing tasks, and after a visual report snapshot under a report dimension is rendered and generated, the report trace information further comprises: In response to a click operation of a user on an abnormal table header text in the visual report snapshot, a data processing task corresponding to a report data column where the abnormal table header text is located is queried from the second mapping relationship based on the abnormal table header text; A task tree snapshot of the data processing task is displayed by jumping, and task execution details of the report data column where the abnormal table header text is located are displayed.

8. The method of claim 5, wherein, After a task tree snapshot is rendered and generated by combining the task tree and the task execution information, the report trace information further comprises: In response to a click operation of a user on a previous task node in the current task tree snapshot, a task tree snapshot of a data processing task corresponding to the previous task node is displayed by jumping, and task execution details of the previous task are displayed.

9. The method of claim 1, wherein, The report trace information further comprises report basic information, and before a click operation of a user on an abnormal data value in the visual report snapshot is responded to, the report trace information further comprises: In a case where the snapshot compression package is obtained, the snapshot compression package is parsed to obtain a plurality of runtime memory snapshot files corresponding to a plurality of reports; Each runtime memory snapshot file is parsed, and report names and data dimension information of the plurality of reports are obtained from the report basic information; In a data snapshot interface, a snapshot compression package file name, report names and data dimension information of the plurality of reports are displayed in a tree structure; In response to a click operation of a user on a target report name in the plurality of report names, a visual report snapshot of a report corresponding to the target report name is displayed.

10. An electronic device, comprising: The electronic device comprises a processor and a memory storing computer program instructions; and the electronic device implements the method according to any one of claims 1-9 when executing the computer program instructions.

Images