A system and method for automatically batch resetting data by blood relationship

Abstract

The application provides a system and method for automatically resetting data in batches through blood relationship, the system comprises a blood relationship information acquisition module, a blood relationship analysis and path analysis module, an object classification and grouping module, and a batch execution control module; the method comprises the following sub-steps: S1: extracting node and associated attribute information from a manifest file; S2: generating a blood relationship path set; S3: identifying nodes belonging to a real-time computing platform group; S4: classifying the nodes belonging to the real-time computing platform group according to object types, and generating corresponding deletion instructions; and S5: identifying an analytical database basic table and executing a batch empty table operation instruction; the application can identify through a blood relationship full link, clean up Doris basic table data and rebuild a real-time computing link synchronously when triggering batch resetting after data repair, ensure that upstream and downstream data are consistent from a starting point, and avoid the problem that the data are still inconsistent after repair.

Description

Technical Field

[0001] This invention relates to the field of data warehousing and real-time data processing technology, and in particular to a system and method for automatically resetting data in batches based on lineage. Background Technology

[0002] As enterprise data scale continues to grow, in order to achieve highly timely data synchronization and high-performance indicator analysis, more and more data platforms are adopting a hybrid architecture of "real-time computing + analytical database". Common technology combinations under this architecture include: using DBT as a unified data development and modeling tool, relying on RisingWave to build real-time data sources and real-time models (Table / Materialized View), and transmitting the processed results data to the downstream in real time through the Sink mechanism. At the same time, Apache Doris is used to carry real-time or near-real-time data and provides indicator services to the outside world through its core capabilities such as aggregated tables and materialized views.

[0003] The existing data processing method is as follows: the data building tool dbt is used as the orchestration entry point to execute the RisingWave related scripts. After receiving the instruction, RisingWave creates a Source to access the original data source and transmits the data in the RisingWave model table to the downstream Apache Doris database in real time through the Sink mechanism. The Apache Doris database further processes the received data according to its own aggregate table and materialized view processing capabilities and provides indicator services to the outside world.

[0004] Existing data processing methods have the following drawbacks:

[0005] Data inconsistency after original data repair: When errors occur in the original data and are repaired on the data source side, the data that has been synchronized to the Apache Doris database and its corresponding materialized views or aggregate tables cannot detect the data changes and correct them, resulting in inaccurate results for downstream metrics that rely on this data, affecting business decisions.

[0006] Manual repair and maintenance is costly and inefficient: When data is abnormal, it requires multiple manual steps to repair, such as stopping the sink, cleaning up the Doris table data, deleting and rebuilding the RisingWave table structure and the source. The process is cumbersome and prone to errors, and cannot meet the needs of large-scale operation and maintenance of multiple modules and multiple sinks. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a system and method for automatically resetting data in batches based on blood relations.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A system for automatically resetting data in batches based on blood relations, the system comprising a blood relation information acquisition module, a blood relation parsing and path analysis module, an object classification and grouping module, and a batch execution control module;

[0010] The lineage information acquisition module is used to acquire the manifest file generated by the data building tool dbt during the compilation or runtime phase, and extract metadata such as model nodes, source nodes, and dependencies between nodes contained therein;

[0011] The lineage resolution and path analysis module is used to resolve the dependencies between nodes and construct a lineage path tracing upwards from the Sink node;

[0012] The object classification and grouping module is used to classify nodes in the lineage path according to data platform type and object type;

[0013] The batch execution control module performs batch operations such as deletion and clearing on target data objects in the real-time computing platform and analytical database.

[0014] This invention also proposes a method for automatically resetting data in batches based on blood relations, comprising the following sub-steps:

[0015] S1: Extract node and associated attribute information from the manifest file;

[0016] Execute the compilation or run commands for the relevant projects of the data building tool dbt. After execution, the data building tool dbt will generate the corresponding manifest file in the preset directory of its relevant projects.

[0017] The lineage information acquisition module calls the built-in API interface of the data construction tool dbt to establish a connection with the manifest file, and extracts node information and related attribute information from the manifest file. The related attribute information includes dependency relationship attributes and downstream relationship information.

[0018] The node information includes node type, node name, data platform, module, dependency attributes, and object type; the node type includes model node, source node, and sink node; the node name is a unique identifier; the data platform refers to the technology platform corresponding to the node, including real-time computing platform RisingWave, analytical database Apache Doris, etc.; the module includes transaction module, etc.; the object type includes source object, computing model object, and sink object.

[0019] The dependency attribute contains a list of upstream node names that the node depends on;

[0020] The downstream association information includes the target database name, target table name, table structure mapping relationship, and data platform identifier; among which, the data platform identifier corresponds to a unique type of target database, the target database name corresponds to a unique target table, and the target table name corresponds to a unique table structure mapping relationship.

[0021] S2: Generate a set of bloodline paths;

[0022] S21: Mark the starting node and save it to the starting node set;

[0023] Users pass parameters to the lineage analysis and path analysis module through the run-operation command of the data building tool dbt; the lineage analysis and path analysis module receives the parameters and marks the starting node of the lineage path based on the parameters, and saves all the starting nodes to the starting node set;

[0024] S22: Construct a directed acyclic graph based on the starting node;

[0025] The lineage resolution and path analysis module takes each Sink node in the set of starting nodes as the initial node, and for each initial node, reads the list of upstream node names of the dependency relationship attributes.

[0026] Create a directed edge for each upstream node name in the upstream node list, with the starting point of the directed edge being the initial node and the ending point being the upstream node corresponding to the upstream node name;

[0027] For each upstream node, repeat the above steps to create the corresponding directed edge until the upstream node name list of a certain node is empty, that is, there is no upstream node.

[0028] Integrate all the nodes involved in step S22 into a node set of a directed acyclic graph, and integrate all the directed edges involved in step S22 into a directed edge set of a directed acyclic graph.

[0029] After deduplication of the node set and directed edge set, all nodes and directed edges in the node set and directed edge set are integrated to obtain a directed acyclic graph;

[0030] S23: Traverse the directed acyclic graph to generate a set of bloodline paths;

[0031] Starting from the initial node, a traversal algorithm is used to trace the dependency relationship properties of the directed acyclic graph layer by layer: first, the direct upstream nodes of the initial node are traversed, and then the associated nodes of each upstream node are recursively traversed until there are no more upstream dependencies; all traversed nodes are collected, and after deduplication, a set of lineage paths is formed.

[0032] S3: Identify nodes belonging to the real-time computing platform group;

[0033] The object classification and grouping module traverses all nodes in the lineage path set, reads the data platform to which each node belongs one by one, filters out nodes whose data platform is RisingWave, and classifies the node into the real-time computing platform group. The real-time computing platform group includes Source nodes, model nodes, Sink nodes, etc.

[0034] S4: Classify the nodes belonging to the real-time computing platform group according to object type and generate corresponding deletion instructions;

[0035] Iterate through all nodes within the real-time computing platform group, read the object type of each node, and group them according to the object type.

[0036] Nodes of object types Sink object, Source object, and computation model object are respectively divided into the corresponding Sink object subset, Source object subset, and computation model object subset;

[0037] The object classification and grouping module uses a built-in instruction generation function. The instruction generation function reads the node names in the Sink object subset, Source object subset, and Computation Model object subset in sequence, and generates the corresponding Sink object deletion instruction, Computation Model object deletion instruction, and Source object deletion instruction.

[0038] The batch execution control module calls the built-in execution environment connection function of the dbt macro to establish communication with the real-time computing platform, and submits the three types of deletion instructions to the real-time computing platform in batches according to the order of Sink object deletion instructions, computation model object deletion instructions, and Source object deletion instructions. The real-time computing platform receives and executes the deletion instructions in batches.

[0039] S5: Identify the basic tables in the analytical database and execute batch table clearing commands;

[0040] From the nodes belonging to the real-time computing platform group obtained in step S3, select the nodes of type Sink node and extract the downstream association information of the nodes.

[0041] Based on the data platform identifier in the downstream association information, the downstream target database type is filtered as analytical database, and the target table in the analytical database is matched based on the target database name and target table name in the downstream association information; the target table contains metadata information, including table type identifier, upstream dependent table list, data write source record, etc.

[0042] The object classification and grouping module reads the metadata information of the target table and determines whether the target table is a basic data table based on the metadata information.

[0043] Specifically, if the table type identifier in the metadata information is the base table identifier, the upstream dependency list is empty, and the data write source record is the Sink node in the real-time computing platform RisingWave, then the target table is determined to be a base data table; otherwise, it is a non-base data table.

[0044] The object classification and grouping module generates batch table clearing operation instructions based on the preset syntax specifications of analytical databases;

[0045] Submit the generated batch of clear table operation instructions to the batch execution control module;

[0046] The batch execution control module establishes communication with the specified analytical database through a preset dbt macro based on pre-configured database connection parameters (address, port, permissions, etc.); it sequentially submits clear table operation instructions to the specified analytical database in batches; and the specified analytical database receives and executes the clear table operation instructions in batches.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] This invention identifies the complete dependency chain of Sink, computing model, Source, and Doris base table through full-chain identification of bloodline relationships. When batch reset is triggered after data repair, the Doris base table data is cleaned up simultaneously and the real-time computing chain is rebuilt, ensuring that upstream and downstream data are consistent from the starting point. This avoids the problem of inconsistency after repair from the root, and improves the accuracy and reliability of downstream indicator data.

[0049] This method extracts node dependencies from the manifest file through a lineage information acquisition and parsing module. It constructs a directed acyclic graph starting from a specified sink or module to identify all associated RisingWave and Doris base tables, eliminating the need for manual link analysis. The batch execution control module submits operation commands in batches via dbt macros in a preset order, eliminating the need for manual execution of each system and step. This reduces human costs and avoids errors caused by human operation. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of a method for automatically resetting data in batches based on blood relations, as proposed in this invention. Detailed Implementation

[0051] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0052] like Figure 1 As shown, a system for automatically resetting data in batches based on blood relations is provided. The system includes a blood relation information acquisition module, a blood relation parsing and path analysis module, an object classification and grouping module, and a batch execution control module.

[0053] The lineage information acquisition module is used to acquire the manifest file generated by the data building tool dbt during the compilation or runtime phase, and extract metadata such as model nodes, source nodes, and dependencies between nodes contained therein;

[0054] The lineage resolution and path analysis module is used to resolve the dependencies between nodes and construct a lineage path tracing upwards from the Sink node;

[0055] The object classification and grouping module is used to classify nodes in the lineage path according to data platform type and object type;

[0056] The batch execution control module performs batch operations such as deletion and clearing on target data objects in the real-time computing platform and analytical database.

[0057] This invention also proposes a method for automatically resetting data in batches based on blood relations, comprising the following sub-steps:

[0058] S1: Extract node and associated attribute information from the manifest file;

[0059] Execute the compilation or run commands for the relevant projects of the data building tool dbt. After execution, the data building tool dbt will generate the corresponding manifest file in the preset directory of its relevant projects.

[0060] The lineage information acquisition module calls the built-in API interface of the data building tool dbt to establish a connection with the manifest file, and extracts node information, dependency attributes, and downstream association information from the manifest file;

[0061] The node information includes node type, node name, data platform, module, dependency attributes, and object type; the node type includes model node, source node, and sink node; the node name is a unique identifier; the data platform refers to the technology platform corresponding to the node, including real-time computing platform RisingWave, analytical database Apache Doris, etc.; the module includes transaction module, etc.; the object type includes source object, computing model object, and sink object.

[0062] The dependency attribute contains a list of upstream node names that the node depends on;

[0063] The downstream association information includes the target database name, target table name, table structure mapping relationship, and data platform identifier; among which, the data platform identifier corresponds to a unique type of target database, the target database name corresponds to a unique target table, and the target table name corresponds to a unique table structure mapping relationship.

[0064] S2: Generate a set of bloodline paths;

[0065] S21: Mark the starting node and save it to the starting node set;

[0066] Users pass parameters to the lineage analysis and path analysis module through the run-operation command of the data building tool dbt; the lineage analysis and path analysis module receives the parameters and marks the starting node of the lineage path based on the parameters, and saves all the starting nodes to the starting node set;

[0067] Specifically, if the parameter is a list of Sink names, the node names of the Sink nodes contained in the Sink name list are matched with the node names of the Sink nodes extracted in step S1, and the Sink nodes in the Sink name list whose node names match exactly are marked as starting nodes and saved to the starting node set.

[0068] If the parameter is a module name, the lineage analysis and path analysis module traverses all the nodes extracted in step S1, checks the module to which each node belongs one by one, filters out the nodes whose module to which the module name in the parameter belongs and whose node type is Sink node, marks them as starting nodes, and saves them to the starting node set.

[0069] S22: Construct a directed acyclic graph based on the starting node;

[0070] The lineage resolution and path analysis module takes each Sink node in the set of starting nodes as the initial node, and for each initial node, reads the list of upstream node names of the dependency relationship attributes.

[0071] Create a directed edge for each upstream node name in the upstream node list, with the starting point of the directed edge being the initial node and the ending point being the upstream node corresponding to the upstream node name;

[0072] For each upstream node, repeat the above steps to create the corresponding directed edge until the upstream node name list of a certain node is empty, that is, there is no upstream node.

[0073] Integrate all the nodes involved in step S22 into a node set of a directed acyclic graph, and integrate all the directed edges involved in step S22 into a directed edge set of a directed acyclic graph.

[0074] After deduplication of the node set and directed edge set, all nodes and directed edges in the node set and directed edge set are integrated to obtain a directed acyclic graph;

[0075] S23: Traverse the directed acyclic graph to generate a set of bloodline paths;

[0076] Starting from the initial node, a traversal algorithm is used to trace the dependency relationship properties of the directed acyclic graph layer by layer: first, the direct upstream nodes of the initial node are traversed, and then the associated nodes of each upstream node are recursively traversed until there are no more upstream dependencies; all traversed nodes are collected, and after deduplication, a set of lineage paths is formed.

[0077] S3: Identify nodes belonging to the real-time computing platform group;

[0078] The object classification and grouping module traverses all nodes in the lineage path set, reads the data platform to which each node belongs one by one, filters out nodes whose data platform is RisingWave, and classifies the node into the real-time computing platform group. The real-time computing platform group includes Source nodes, model nodes, Sink nodes, etc.

[0079] S4: Classify the nodes belonging to the real-time computing platform group according to object type and generate corresponding deletion instructions;

[0080] Iterate through all nodes within the real-time computing platform group, read the object type of each node, and group them according to the object type.

[0081] Nodes of object types Sink object, Source object, and computation model object are respectively divided into the corresponding Sink object subset, Source object subset, and computation model object subset;

[0082] The object classification and grouping module uses a built-in instruction generation function. The instruction generation function reads the node names in the Sink object subset, Source object subset, and Computation Model object subset in sequence, and generates the corresponding Sink object deletion instruction, Computation Model object deletion instruction, and Source object deletion instruction.

[0083] Specifically, the instruction generation function pre-stores a syntax template library, which contains a mapping relationship between object types and syntax templates, i.e., different object types are bound to corresponding deletion syntax.

[0084] Generate Sink object deletion instructions: The instruction generation function identifies a subset of Sink objects and calls the corresponding deletion syntax based on the mapping relationship. The deletion syntax is: DROP SINK IF EXISTS<sink_name> Iterate through the subset of Sink objects, extract the node name of each node contained in the subset, and replace the placeholders in the deletion syntax with the node names.<sink_name> Generate the corresponding deletion command; repeat the above steps to generate the corresponding deletion command for all nodes in the Sink object subset, and summarize to obtain the Sink object deletion command.

[0085] Generate deletion commands for computational model objects: The command generation function identifies a subset of computational model objects and calls the corresponding deletion syntax based on the mapping relationship; for example, for a materialized view, use `DROP MATERIALIZED VIEW IF EXISTS`.<view_name> DROP TABLE IF EXISTS for a regular model table<model_table_name> wait;

[0086] Iterate through the subset of computational model objects, extracting the node name of each node contained in the subset. When the model type is a materialized view, replace the placeholder in the deletion syntax with the node name.<view_name> When the model type is a regular model table, replace the placeholders in the deletion syntax with the node names.<model_table_name> Repeat the above steps to generate corresponding deletion instructions for all nodes in the subset of computational model objects, and summarize them to obtain the computational model object deletion instructions.

[0087] Generate Source object deletion instructions: The instruction generation function identifies a subset of Source objects and calls the corresponding deletion syntax based on the mapping relationship. The deletion syntax is: DROP TABLE IF EXISTS<table_name> Iterate through the subset of Source objects, extract the node name of each node contained in the subset, and replace the placeholders in the deletion syntax with the node names.<table_name> Generate the corresponding deletion command; repeat the above steps to generate the corresponding deletion command for all nodes in the subset of the Source object, and summarize to obtain the Source object deletion command;

[0088] The batch execution control module calls the built-in execution environment connection function of the dbt macro to establish communication with the real-time computing platform, and submits the three types of deletion instructions to the real-time computing platform in batches according to the order of Sink object deletion instructions, computation model object deletion instructions, and Source object deletion instructions. The real-time computing platform receives and executes the deletion instructions in batches.

[0089] S5: Identify the basic tables in the analytical database and execute batch table clearing commands;

[0090] From the nodes belonging to the real-time computing platform group obtained in step S3, select the nodes of type Sink node and extract the downstream association information of the nodes.

[0091] Based on the data platform identifier in the downstream association information, the downstream target database type is filtered as analytical database, and the target table in the analytical database is matched based on the target database name and target table name in the downstream association information; the target table contains metadata information, including table type identifier, upstream dependent table list, data write source record, etc.

[0092] Furthermore, the target table has a corresponding database schema, which includes a default schema, a custom schema, etc.

[0093] The object classification and grouping module reads the metadata information of the target table and determines whether the target table is a basic data table based on the metadata information.

[0094] Specifically, if the table type identifier in the metadata information is the base table identifier, the upstream dependency list is empty, and the data write source record is the Sink node in the real-time computing platform RisingWave, then the target table is determined to be a base data table; otherwise, it is a non-base data table.

[0095] The object classification and grouping module generates batch table clearing operation instructions based on the preset syntax specifications of analytical databases;

[0096] Specifically, the syntax specification includes a preset syntax template, which is TRUNCATE TABLE. <database> . <schema>.<table_name> Replace the target database name of the base data table. <database>Replace the database schema of the target table. <schema>Replace the target table name<table_name> The corresponding clear operation instruction is obtained; the above steps are repeated for each basic data table to generate the corresponding clear operation instruction; the generated batch of clear table operation instructions are submitted to the batch execution control module.

[0097] The batch execution control module establishes communication with the specified analytical database through a preset dbt macro based on pre-configured database connection parameters (address, port, permissions, etc.); it sequentially submits clear table operation instructions to the specified analytical database in batches; and the specified analytical database receives and executes the clear table operation instructions in batches.

[0098] Furthermore, this method supports triggering batch reset operations by a list of Sink names, triggering batch reset operations by module names, controlling whether to include Source objects via parameters, and applying the same method to multiple modules or multiple Sinks simultaneously during the same execution.

[0099] This invention automatically identifies the complete dependency chain between Sink, computing model and Source by parsing the lineage relationship generated by dbt, avoiding the omission of key objects by manual judgment, thereby effectively reducing the probability of inconsistency in the upstream and downstream data status during data repair, link adjustment or anomaly recovery.

[0100] By converging the original multi-step, cross-system manual operation into a batch reset operation driven by lineage, the data link can complete cleaning and reconstruction in a shorter time, significantly shortening the time window required for data repair and recalculation.

[0101] By uniformly cleaning and resetting the basic tables of the analytical database, a consistent and controllable data input foundation is provided for the upper-level aggregate tables and materialized views, thereby improving the accuracy, stability and repeatability of the final indicator data.

[0102] This invention solidifies the identification and operation logic of data links into an automated method based on lineage, reducing reliance on the personal experience and manual judgment of maintenance personnel, and making the data repair process more standardized and reproducible.

[0103] This invention does not rely on modifying the real-time computing engine or analytical database itself, but is based on the existing DBT engineering system. It can be quickly implemented without changing the existing data development model and has high practical value.

[0104] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.< / schema> < / database> < / schema> < / database>

Claims

1. A method for automatically resetting data in batches based on blood relations, characterized in that: Includes the following steps: S1: Extract node and associated attribute information from the manifest file; Execute the compilation or run commands for the relevant projects of the data building tool dbt. After execution, the data building tool dbt will generate the corresponding manifest file in the preset directory of its relevant projects. The lineage information acquisition module calls the built-in API interface of the data construction tool dbt to establish a connection with the manifest file, and extracts node information and related attribute information from the manifest file. The related attribute information includes dependency relationship attributes and downstream relationship information. S2: Generate a set of bloodline paths; S21: Mark the starting node and save it to the starting node set; S22: Construct a directed acyclic graph based on the starting node; S23: Traverse the directed acyclic graph to generate a set of bloodline paths; S3: Identify nodes belonging to the real-time computing platform group; The real-time computing platform group includes Sink nodes, model nodes, and Source nodes; S4: Classify the nodes belonging to the real-time computing platform group according to object type and generate corresponding deletion instructions; Iterate through all nodes in the real-time computing platform group, read the object type of each node, which is Sink object, computing model object, or Source object; divide the node into the corresponding Sink object subset, computing model object subset, and Source object subset according to the object type; The object classification and grouping module uses a built-in instruction generation function to sequentially read the subsets of Sink objects, computation model objects, and Source objects. Based on the mapping relationship, it calls the corresponding deletion syntax to generate corresponding Sink object deletion instructions, computation model object deletion instructions, and Source object deletion instructions. Then, it submits the three types of deletion instructions to the real-time computing platform in batches in the order of Sink object deletion instructions, computation model object deletion instructions, and Source object deletion instructions. S5: Identify the basic tables in the analytical database and execute batch table clearing commands; From the nodes of the real-time computing platform group, select nodes of type Sink. Points, extract downstream association information of the nodes; Based on the data platform identifier in the downstream association information, the downstream target database type is filtered as analytical database, and the target tables in the analytical database are matched based on the target database name and target table name in the downstream association information; the object classification and grouping module filters out the target tables of the basic data tables. The object classification and grouping module generates batch table clearing operation instructions based on the preset syntax specifications of analytical databases and submits them to the batch execution control module; The batch execution control module submits clear table operation instructions to the specified analytical database in a sequential batch; the specified analytical database receives and executes the clear table operation instructions in batches.

2. The method for automatically resetting data in batches based on blood relations as described in claim 1, characterized in that: In step S1, the node information includes node type, node name, data platform, module, dependency attributes, and object type; the node type includes model node, source node, and sink node; the node name is a unique identifier; the data platform refers to the technology platform corresponding to the node, including the real-time computing platform RisingWave and the analytical database Apache Doris; the module includes the transaction module; the object type includes source object, computing model object, and sink object. The dependency attribute contains a list of upstream node names that a node depends on; Downstream association information includes the target database name, target table name, table structure mapping relationship, and data platform identifier; among which, the data platform identifier corresponds to a unique type of target database, the target database name corresponds to a unique target table, and the target table name corresponds to a unique table structure mapping relationship.

3. The method for automatically resetting data in batches based on blood relations as described in claim 1, characterized in that: The specific details of step S2 are as follows: S21: Mark the starting node and save it to the starting node set; Users pass parameters to the lineage analysis and path analysis module through the run-operation command of the data building tool dbt; the lineage analysis and path analysis module receives the parameters and marks the starting node of the lineage path based on the parameters, and saves all the starting nodes to the starting node set; S22: Construct a directed acyclic graph based on the starting node; The lineage resolution and path analysis module takes each Sink node in the set of starting nodes as the initial node, and for each initial node, reads the list of upstream node names of the dependency relationship attributes. Create a directed edge for each upstream node name in the upstream node list, with the starting point of the directed edge being the initial node and the ending point being the upstream node corresponding to the upstream node name; For each upstream node, repeat the above steps to create the corresponding directed edge until the upstream node name list of a certain node is empty, that is, there is no upstream node. After integrating all nodes and directed edges and removing duplicates, a directed acyclic graph is obtained. S23: Traverse the directed acyclic graph to generate a set of bloodline paths; Starting from the initial node, a traversal algorithm is used to trace the dependency relationship properties of the directed acyclic graph layer by layer: first, the direct upstream nodes of the initial node are traversed, and then the associated nodes of each upstream node are recursively traversed until there are no more upstream dependencies; all traversed nodes are collected, and after deduplication, a set of lineage paths is formed.

4. The method for automatically resetting data in batches based on blood relations as described in claim 1, characterized in that: In step S3, the object classification and grouping module traverses all nodes in the lineage path set, reads the data platform to which each node belongs one by one, filters out the nodes whose data platform is RisingWave, and classifies the node into the real-time computing platform group.

5. The method for automatically resetting data in batches based on blood relations as described in claim 1, characterized in that: In step S4, the instruction generation function pre-stores a syntax template library, which contains a mapping relationship between object types and syntax templates, i.e., different object types are bound to corresponding deletion syntax. The batch execution control module calls the built-in execution environment connection function of the dbt macro to establish communication with the real-time computing platform. The real-time computing platform receives and executes the deletion commands in batches.

6. The method for automatically resetting data in batches based on blood relations as described in claim 1, characterized in that: The specific details of step S5 are as follows: The object classification and grouping module generates batch clear table operation instructions based on the preset syntax specifications of analytical databases; and submits the generated batch clear table operation instructions to the batch execution control module. The batch execution control module establishes communication with the specified analytical database through a preset dbt macro based on pre-configured database connection parameters.

7. A system for automatically batch resetting data based on blood relations to implement the method of any one of claims 1-6, characterized in that: The system includes a bloodline information acquisition module, a bloodline parsing and path analysis module, an object classification and grouping module, and a batch execution control module; The bloodline information acquisition module is used to acquire the manifest file generated by the data building tool dbt during the compilation or runtime phase; The lineage resolution and path analysis module is used to resolve the dependencies between nodes and construct a lineage path tracing upwards from the Sink node; The object classification and grouping module is used to classify nodes in the lineage path according to data platform type and object type; The batch execution control module performs batch deletion and clearing operations on target data objects in the real-time computing platform and analytical database.

Images