Methods, apparatus, and electronic devices for building data relationship engines
By building a data relationship engine, the downstream tables and fields of big data models are automatically identified and modified, solving the problems of inefficiency and errors caused by the increase in the number of downstream data models, and achieving efficient and accurate data management.
Patent Information
- Application Number
- CN202510625794.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the number of downstream application layer data models has increased dramatically, and when upstream data models change, they require comprehensive manual evaluation and updates, resulting in low efficiency and a high risk of errors.
By building a data relationship engine, based on the metadata and data operation tasks of the big data model, the data flow is determined, downstream tables and fields that have data relationships with the target table are automatically identified, and automatic modifications are made according to hierarchical dependencies, reducing manual operations.
It enables automatic retrieval of downstream hierarchical relationships at the field level, improving data management efficiency, reducing human error, and ensuring the accuracy and stability of data management.
Smart Images

Figure CN123115278A_ABST
Abstract
Description
Technical Field
[0001] The embodiments disclosed herein relate to the field of big data management technology, and specifically to a method, apparatus, and electronic device for constructing a data relationship engine. Background Technology
[0002] With the widespread adoption of data technologies, every enterprise has established its own data warehouse architecture. However, with the rapid development of business, the general-purpose foundational model located upstream in the data chain often struggles to meet the increasingly diverse business data needs. To address this challenge, numerous downstream application-layer data models have been developed to meet various specific business requirements.
[0003] However, the inventors discovered that while this strategy solved the problem in the short term, it also brought new challenges. The number of downstream application layer models increased dramatically, and at an alarming rate. When upstream data models involving core business operations changed, it had a cascading effect on models across downstream business lines. This necessitated a comprehensive assessment of the affected scope and timely corrections and updates to relevant models to ensure the accuracy and stability of downstream model data. In the current technological environment, this large-scale model maintenance and updating work relies primarily on manual processing, which is not only inefficient but also prone to errors.
[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0006] Some embodiments of this disclosure provide methods, apparatus, electronic devices, computer-readable media, and computer program products for constructing data relationship engines to address one or more of the technical problems mentioned in the background section above.
[0007] In a first aspect, some embodiments of this disclosure provide a method for constructing a data relationship engine, including: determining the data flow direction based on the metadata and data operation tasks of a big data model; determining downstream tables that have a data relationship with a target table based on the data flow direction; determining downstream fields of a target field based on the data relationship between each field in the target table and each field in the downstream table, wherein the target field is a field in the target table; and constructing a data relationship engine for a big data model based on the downstream tables of the target table and the downstream fields of the target field.
[0008] In some embodiments, determining downstream tables that have a data relationship with the target table based on the data flow direction includes: representing the data in the data flow direction using an adjacency list method based on the relationship between the table and the data operation task in the data flow direction; and traversing the adjacency list using a defined traversal method to obtain the downstream tables that have a data relationship with the target table.
[0009] In some embodiments, traversing the adjacency list based on a set traversal method to obtain downstream tables that have data relationships with the target table includes: traversing the adjacency list using a breadth-first traversal method to capture all downstream tables of the target table; and performing hierarchical processing on the downstream tables to obtain a hierarchical downstream table with the target table as the first layer.
[0010] In some embodiments, determining the downstream fields of the target field based on the data relationship between each field in the target table and each field in the downstream table includes: traversing and determining the downstream fields of the target field based on the relationship between each field in each table and the data operation task in the data flow; and performing hierarchical processing on the downstream fields to obtain hierarchical downstream fields with the target field as the first layer.
[0011] In some embodiments, the method further includes: for each downstream field of the target field, determining whether the data of the downstream field needs to be modified based on the operation content of the data operation task corresponding to the downstream field; and for downstream fields of the target field that need to be modified, modifying the data of the downstream fields to be modified sequentially according to the hierarchy of each downstream field.
[0012] In some embodiments, determining whether the data of a downstream field needs to be modified based on the operation content of the data operation task corresponding to the downstream field includes: acquiring and parsing the script data of the data operation task corresponding to the downstream field, determining the operation content of the data operation task; classifying the operation content of the data operation task according to a preset classification rule; and determining whether the data of the downstream field needs to be modified based on the classification result.
[0013] In some embodiments, the data of the downstream fields to be modified are modified sequentially according to the hierarchy of each downstream field, including: prioritizing the downstream fields according to their hierarchy, wherein the downstream fields with smaller hierarchy numbers have higher priority; and modifying the script data of the downstream fields to be modified sequentially in descending order of priority.
[0014] In some embodiments, the method further includes: for each downstream field to be modified, obtaining database attribute information of a target upstream table, wherein the target upstream table is the upstream table located in the table where the downstream field is located; generating prompt information based on the database attribute information, the original script data of the downstream field, and the information to be modified; and obtaining the modification content information of the downstream field based on the prompt information and the large language model.
[0015] In some embodiments, the data relationship engine for building a big data model includes: a data relationship engine for building a big data model based on a graph, wherein the types of nodes in the graph include at least one of the following: tables, fields, and data operation tasks, and the types of edges in the graph include at least one of the following: contain, read, write, and transform; and for field nodes in the graph, their edges are added with operation content and hierarchical attributes.
[0016] In some embodiments, the method further includes: in response to detecting a data update in the big data model, using the table and field containing the updated data as query metadata; inputting the query metadata into a data relationship engine, outputting downstream task data to be modified that is affected by the query metadata, and corresponding modification information; and displaying the content output by the data relationship engine for manual review.
[0017] Secondly, some embodiments of this disclosure provide an apparatus for constructing a data relationship engine, including: a data flow direction determination unit configured to determine the data flow direction based on metadata and data operation tasks of a big data model; a downstream table determination unit configured to determine downstream tables that have a data relationship with a target table based on the data flow direction; a downstream field determination unit configured to determine downstream fields of a target field based on the data relationship between each field in the target table and each field in the downstream table, wherein the target field is a field in the target table; and an engine construction unit configured to construct a data relationship engine for a big data model based on the downstream tables of the target table and the downstream fields of the target field.
[0018] In some embodiments, the downstream table determination unit includes: a data representation subunit configured to represent the data in the data flow using an adjacency list method based on the relationship between the table in the data flow and the data operation task; and a data traversal subunit configured to traverse the adjacency list using a set traversal method to obtain the downstream tables that have a data relationship with the target table.
[0019] In some embodiments, the data traversal subunit is further configured to traverse the adjacency list using a breadth-first traversal method, capture all downstream tables of the target table, and perform hierarchical processing on the downstream tables to obtain a hierarchical downstream table with the target table as the first layer.
[0020] In some embodiments, the downstream field determination unit is further configured to traverse and determine the downstream fields of the target field according to the relationship between each field in each table and the data operation task in the data flow; and to perform hierarchical processing on the downstream fields to obtain a hierarchical downstream field with the target field as the first layer.
[0021] In some embodiments, the data relationship engine construction apparatus further includes: a modification determination unit configured to determine whether the data of each downstream field of the target field needs to be modified based on the operation content of the data operation task corresponding to the downstream field; and a data modification unit configured to modify the data of the downstream fields to be modified sequentially according to the hierarchy of each downstream field.
[0022] In some embodiments, the modification determination unit is further configured to acquire and parse the script data of the data operation task corresponding to the downstream field, determine the operation content of the data operation task, classify the operation content of the data operation task according to a preset classification rule, and determine whether the data of the downstream field needs to be modified based on the classification result.
[0023] In some embodiments, the data modification unit is further configured to prioritize downstream fields according to their hierarchy, wherein the downstream fields with smaller hierarchy numbers have higher priority; and to modify the script data of the downstream fields to be modified in descending order of priority.
[0024] In some embodiments, the data relationship engine building apparatus further includes a content modification generation unit, configured to obtain database attribute information of a target upstream table for each downstream field to be modified, wherein the target upstream table is the upstream table located in the table where the downstream field is located; generate prompt information based on the database attribute information, the original script data of the downstream field, and the information to be modified; and obtain the modification content information of the downstream field based on the prompt information and the large language model.
[0025] In some embodiments, the engine building unit is further configured as a data relationship engine for building big data models based on graphs, wherein the types of nodes in the graph include at least one of the following: tables, fields, and data operation tasks, and the types of edges in the graph include at least one of the following: contain, read, write, and transform; and for field nodes in the graph, their edges are added with operation content and hierarchical attributes.
[0026] In some embodiments, the data relationship engine building apparatus further includes an online analysis unit configured to, in response to detecting a data update in the big data model, use the table and field containing the updated data as query metadata; input the query metadata into the data relationship engine, output the downstream task data to be modified affected by the query metadata, and the corresponding modification information; and display the output of the data relationship engine for manual review.
[0027] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data relationship engine construction method described in any of the implementations of the first aspect above.
[0028] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for constructing a data relationship engine as described in any of the implementations of the first aspect above.
[0029] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method for constructing a data relationship engine as described in any of the implementations of the first aspect above.
[0030] The various embodiments disclosed above have the following beneficial effects: the data relationship engine construction method of some embodiments of this disclosure can realize automatic retrieval of downstream hierarchical relationships at the field level, reducing manual operations. Specifically, in the current technical environment, large-scale data model maintenance and updating work mainly relies on manual processing. This not only requires a large investment of human resources, but also suffers from problems such as low efficiency and susceptibility to errors.
[0031] Based on this, the data relationship engine construction method of some embodiments of this disclosure can capture metadata and data operation task-related information based on the task scheduling center of the big data model, thereby determining the data flow direction of the big data model. According to the data flow direction, downstream tables with data dependencies on the target table can be identified first, thus determining table-level dependencies. Next, based on the data relationships between fields in the target table and fields in the downstream tables, downstream fields with data dependencies on the target field can be identified, thus determining field-level dependencies. In this way, by analyzing the downstream tables of each table and the downstream fields of each field in the big data model, a data relationship engine for the big data model can be constructed. This data relationship engine enables upward (upstream) and downward (downstream) tracing, as well as automatic retrieval of downstream dependencies at the field level. This not only greatly reduces manual operations and improves the efficiency of data management, but also reduces or avoids human error, ensuring the accuracy of data management. Attached Figure Description
[0032] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0033] Figure 1 This is a flowchart of some embodiments of the method for constructing the data relationship engine disclosed herein;
[0034] Figure 2 These are flowcharts of some other embodiments of the method for constructing the data relationship engine disclosed herein;
[0035] Figure 3A These are schematic diagrams illustrating some application scenarios of the construction method of this publicly disclosed data relationship engine;
[0036] Figure 3B These are some other application scenarios illustrating the construction method of this publicly disclosed data relationship engine;
[0037] Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the data relationship engine construction apparatus disclosed herein;
[0038] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation
[0039] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0040] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0041] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0042] Furthermore, the terms “a” and “a plurality” used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as “one or more”.
[0043] Figure 1 A flow 100 illustrating some embodiments of a method for constructing a data relationship engine according to this disclosure is shown. The method may include the following steps:
[0044] Step 101: Determine the data flow direction based on the metadata and data operation tasks of the big data model.
[0045] In some embodiments, the execution entity of the data relationship engine construction method of this disclosure (such as a server or a big data model management system) can communicate with other electronic devices (such as user terminals and databases) via wired or wireless connections. Here, the execution entity can determine the data flow based on the metadata and data operation tasks of the big data model. It should be noted that metadata is typically information about the organization of data, data domains, and their relationships; it is data used to describe the data. In embodiments of this disclosure, metadata may include physical database table names, column names, field lengths, field types, constraint information, data dependencies, etc., and may also include field-level dependencies, SQL (Structured Query Language) script information, interface programs, scheduling dependencies, progress, and data update frequency, etc. The data operation tasks here can be any operation task representing data processing.
[0046] Here, the executing entity can obtain metadata and data manipulation tasks from the task scheduling center of the big data model. It's important to note that the task scheduling center is typically a tool or platform for scheduling and managing big data processing tasks, capable of scheduling and managing data model processing tasks. This center can define tasks; for example, users can define tasks to be executed through the scheduling system, including the task type, the script or program to be executed, and the location of input and output data. Additionally, it can schedule tasks, for example, automatically triggering task execution based on a preset schedule (e.g., daily, weekly) or specific conditions (e.g., data arrival, completion of prerequisite tasks). The center can also manage dependencies between tasks and perform resource allocation and optimization. As an example, the executing entity can use data crawling tools (such as Scrapy, an open-source web crawling framework) or metadata hooks to capture metadata and data manipulation tasks. This allows it to intercept data manipulation tasks from the big data task scheduling center and, based on the task scheduling information, obtain the metadata of existing models in the data mart and their direct dependencies, thereby determining the data flow and establishing and updating complete upstream and downstream data dependencies.
[0047] Understandably, data flow is typically used to record and understand the changes in data relationships between data items in a business context. Data flow can be represented graphically, showing macroscopic data transfer and changes in business logic. Taking a table as an example, the relationship between a table and a data operation task (Job) can be seen below.
[0048] Serial Number Data flow 1 TableA-Job1-TableB 2 TableA-Job2-TableC 3 TableB-Job3-TableD 4 TableB-Job4-TableE 5 TableC-Job5-TableF 6 TableD-Job6-TableG 7 TableE-Job7-TableH
[0049] Taking TableA-Job1-TableB as an example, it represents a segment of data flow: Job1 reads the contents of Table A and then writes them to Table B, that is, the downstream of A is B, and the upstream of B is A.
[0050] Step 102: Based on the data flow, determine the downstream tables that have a data relationship with the target table.
[0051] In some embodiments, the executing entity can determine downstream tables that have a data relationship with the target table based on the aforementioned data flow. This data relationship can be direct or indirect. For example, the executing entity can first search for data flows that contain the target table. Then, it can filter these data flows to identify those upstream of the target table, thus determining the downstream tables that are directly upstream of the target table—that is, the downstream tables that have a direct relationship with the target table. Afterward, these downstream tables can be used as new target tables, and the data flow can be further filtered to identify data flows upstream of the new target table, thus determining the downstream tables that have a direct relationship with the new target table. This process continues until all data flow data has been queried. It is understood that the downstream tables and target tables here typically have an indirect relationship.
[0052] Taking the data flow listed in step 101 as an example, let's assume the goal is to find all downstream dependent tables of Table B. First, we can filter out the downstream tables Table D and Table E that are directly related to Table B. Next, we can filter out the downstream table Table G that is directly related to Table D, and the downstream table Table H that is directly related to Table E. Thus, we can obtain all the downstream tables of Table B as Table D, Table E, Table G, and Table H.
[0053] In some embodiments, the executing entity may also represent the data in the data flow using an adjacency list method based on the relationship between the tables in the data flow and the data operation tasks. For example, the above data flow relationship fragments can be abstracted into a graph structure, and the data can be represented as an adjacency list: graph = {
[0054] 'Table A':['Job 1','Job 2'],
[0055] 'Job 1':['Table B'],
[0056] 'Job 2':['Table C'],
[0057] 'Table B':['Job 3','Job 4'],
[0058] 'Table C':['Job 5'],
[0059] 'Job 3':['Table D'],
[0060] 'Job 4':['Table E'],
[0061] 'Job 5':['Table F'],
[0062] 'Table D':['Job 6'],
[0063] 'Table E':['Job 7'],
[0064] 'Job 6':['Table G'],
[0065] 'Job 7':['Table H'],
[0066] }
[0067] Next, a defined traversal method can be used to traverse the adjacency list to obtain downstream tables that have data relationships with the target table. This defined traversal method can be set according to actual needs, such as Depth-First Traversal (DFS) or Breadth-First Search (BFS). As an example, the execution entity can use a breadth-first traversal method to traverse the above adjacency list, thereby capturing all downstream tables (including indirect relationships) of the target table. The downstream tables are then processed hierarchically to obtain a hierarchical downstream table with the target table as the first level.
[0068] Taking finding all downstream dependent tables of Table B as an example, all downstream tables are captured using a BFS-based approach, including forked and merged data flows. Simultaneously, the output is expanded hierarchically by creating a layered view, clearly demonstrating how Table B flows from its source to each downstream node. Furthermore, in terms of efficiency, BFS ensures that nodes are not accessed repeatedly, operating efficiently even in complex graph structures. At this point, the hierarchical downstream dependencies of Table B can be obtained:
[0069] Level 0: Table B;
[0070] Level 1:Table D(Job 3),Table E(Job 4);
[0071] Level 2: Table G (Job 6), Table H (Job 7).
[0072] Step 103: Determine the downstream field of the target field based on the data relationship between each field in the target table and each field in the downstream table.
[0073] In some embodiments, after obtaining the downstream tables of the target table, the executing entity can further determine the downstream fields of the target field based on the data relationships between the fields in the target table and the fields in the downstream tables. The target field is typically a field in the target table. It is understood that, in step 101, taking a field (col, i.e., a column) in the table as an example, the data flow relationship between the field (col) and the data operation task (Job) can be obtained.
[0074] Serial Number upper-node node edge to-node to-column 1 TableA col-mark Job0 TableB col-source 2 TableB col-name Job1 Table C col-name-sub 3 TableB col-name Job2 TableD col-name-upper 4 TableB col-id-int Job3 TableE col-id-int 5 Table C col-name-sub Job4 TableF col-result 6 TableD col-name-upper Job5 TableG col-agg 7 TableE col-id-int 8 TableF col-result 9 TableG col-agg
[0075] The upper-node column stores the table name, i.e., the source table name; the node column stores the field name, i.e., the source table field name; the edge column stores the job name, indicating the name of the data operation task; the to-node column stores the table name, i.e., the downstream table name indicated by the task, such as the downstream table name for direct writes; and the to-column column stores the downstream field name.
[0076] In some embodiments, the executing entity can determine the downstream fields of the target field by traversing the data flow and identifying the relationship between each field in each table and the data operation task. The method for determining the downstream table of the target table can also be referenced here. Optionally, the executing entity can also derive dependencies using the following method: Let G = (V, E), where G represents a graph, V represents the set of nodes, and E represents the set of edges. Let Q be the queue of nodes waiting to be traversed, and V' be the set of visited nodes.
[0077] Let s∈V, and take it as the starting point of the traversal. Then V'={s}. v is the currently visited node, and we traverse the nodes u adjacent to v. If u is not in V, add u to V'. Continue until the Q queue is completely empty, then V' is the result set.
[0078] Furthermore, the executing entity can also perform hierarchical processing on downstream fields to obtain hierarchical downstream fields with the target field as the first level. For example, if we take finding all downstream dependent fields of the `col-name` field in Table B of the above table as an example, we can obtain the hierarchical downstream dependency relationship of the `col-name` field in Table B:
[0079] Level 0:
[0080] Table B-col-name;
[0081] Level 1:
[0082] Table C–col-name-sub(Job 1),
[0083] Table D–col-name-upper(Job 2);
[0084] Level 2:
[0085] Table F–col-result(Job 4),
[0086] Table G–col-agg (Job 5).
[0087] Step 104: Based on the downstream tables of the target table and the downstream fields of the target fields, construct the data relationship engine for the big data model.
[0088] In some embodiments, the executing entity can construct a data relationship engine for the big data model based on the downstream tables of the target table obtained in step 102 and the downstream fields of the target fields obtained in step 103. The method of constructing the data relationship engine is not limited; for example, it can be constructed using relational tables. As an example, the relational tables can be named after data models of different data layers. Each relational table stores the upstream and downstream data relationships between fields in different data tables of that data model, as well as the direct dependencies between fields in tables of other data models. Indirect dependencies with other data models can be further queried from the relational tables named after those other data models.
[0089] In some alternative implementations, the executing entity can build a data relationship engine for the big data model based on a graph. The types of nodes in the graph can include at least one of the following: tables, fields, and data manipulation tasks; while the types of edges in the graph can include at least one of the following: contain, read, write, transform, etc.
[0090] As described above, the data relationship engine construction method of some embodiments of this disclosure can capture metadata and data operation task information based on the task scheduling center of the big data model, thereby determining the data flow direction of the big data model. Based on the data flow direction, downstream tables with data dependencies on the target table can be identified first, thus determining table-level dependencies. Next, based on the data relationships between fields in the target table and fields in the downstream tables, downstream fields with data dependencies on the target field can be identified, thus determining field-level dependencies. In this way, by analyzing the downstream tables of each table and the downstream fields of each field in the big data model, a data relationship engine for the big data model can be constructed. This data relationship engine enables upward (upstream) and downward (downstream) tracing, as well as automatic retrieval of downstream dependencies at the field level. This not only greatly reduces manual operations and improves data management efficiency, but also reduces or avoids human error, ensuring the accuracy of data management.
[0091] In such Figure 3A In the application scenario shown, when upstream core foundational data changes, downstream general-layer and application-layer models may also be affected. To ensure data consistency, accuracy, and stability, the data relationship engine built using the method disclosed in this paper can quickly and accurately determine the scope of affected downstream models. For example, when the data of tb0 in the fdm model changes, tb20 in the GDM model, tb11 in the ADM model, and tb1a and tb4a in the APP model will be affected.
[0092] See also Figure 2The diagram illustrates flowcharts of some other embodiments of the data relationship engine construction method disclosed herein.
[0093] After the data relationship engine is established, users can first input updated (changed) data (such as the col field of table t) into the engine through the terminal.
[0094] Next, the engine can quickly retrieve the candidate set C of all downstream models affected by the change of col in t.
[0095] Afterward, the data relationship engine can iterate through the candidate set C to determine whether the downstream model needs modification. If no modification is needed, it can continue iterating through the candidate set. If modification is required, it can output the modified content.
[0096] from Figure 2 As can be seen from this, the data relationship engine construction method of this disclosure embodiment, in order to further reduce manual operations and further improve data management efficiency, can further analyze and determine whether the data of these downstream fields need to be modified when the downstream fields of the target field are obtained.
[0097] Specifically, for each downstream field of the target field, the system first determines whether the data in that downstream field needs modification based on the operation content of the corresponding data manipulation task. As an example, the executing entity can obtain and parse the script data of the data manipulation task corresponding to the downstream field to determine the operation content of that task. The method for obtaining and parsing the script data is not limited. For example, the input SQL string can be converted into a sequence of tokens. A syntax rule file is defined, such as: `select` to locate field information, `from` to capture the input source table, and `where` to identify filtering conditions. Depending on the actual complexity of the script parsing in the data model, the syntax rule file can be expanded to cover various advanced SQL syntaxes. The token sequence is then converted into an Abstract Syntax Tree (AST) according to the syntax rules. The AST is traversed to obtain the computational operations for the direct downstream field `to-column`.
[0098] This allows us to determine whether the operation content includes a setting operation. If a setting operation is included, it means that the data in that downstream field needs to be modified. If no setting operation is included, it means that the data in that downstream field does not need to be modified.
[0099] Optionally, the executing entity can also classify the operation content of the data operation task according to preset classification rules. Then, based on the classification results, it can determine whether the data in the downstream field needs to be modified. For example, for the candidate set C mentioned above, during traversal, let the current element be c, which contains the table name, col name, and job name. Taking Table F-col-result(Job 4) as an example, by parsing the SQL script of Job4, we can obtain that the data operation task corresponding to the downstream relation field col-result is join. At this time, the data operation content can be classified into two categories. For example, 1 indicates that modification is needed, and 0 indicates that no modification is needed.
[0100] It should be noted that Category 1 typically refers to fields whose values are specifically modified, such as string substring extraction, uppercase / lowercase conversion, etc.; or the field is used as a filtering condition, such as in the WHERE clause (includes certain values), LIKE (includes a substring), etc. Category 0 typically refers to fields whose values are not specifically modified and are directly referenced; or used for aggregation counting, etc.
[0101] It should be noted that in the graph-based data relationship engine of this disclosure embodiment, the edges of field nodes in the graph can also be added with operation content and level attributes. In field-level dependency tracking, transformation operation content (transform) and level attributes are also added to each edge of the node. This information will be used to subsequently determine whether the field node data has been modified or updated.
[0102] Furthermore, based on the above analysis and judgment, for the downstream fields to be modified in the target field, the executing entity can modify the data of the downstream fields to be modified sequentially according to their hierarchy. Specifically, firstly, the downstream fields can be prioritized according to their hierarchy. The lower the hierarchy number, the higher the priority of the downstream field. Then, the script data of the downstream fields to be modified can be modified sequentially in descending order of priority. That is, those with lower hierarchy numbers are processed first. The lower the hierarchy number, the closer the node is to the upstream of the entire data flow, indicating a larger downstream impact; therefore, modifications should begin with those with lower hierarchy numbers.
[0103] In some embodiments, to improve the efficiency and accuracy of data modification, the method of this disclosure can also automatically generate specific modification suggestions. Here, for each downstream field to be modified, the database attribute information of the target upstream table can first be obtained. The target upstream table is the upstream table located in the table containing the downstream field. Next, a prompt message can be generated based on the database attribute information, the original script data of the downstream field, and the information to be modified (such as the downstream field). Then, based on the prompt message and the large language model, the modification content information of the downstream field can be obtained.
[0104] In other words, the original SQL script, specific modification requirements, and database schema information of the upstream tables can be combined into a structured prompt. This prompt is then sent to a large language model (such as an LLM) to analyze the data and output modification suggestions. Here, the original SQL script refers to the unmodified SQL script. The downstream tasks affected are the unmodified scripts. The goal is to determine whether the script needs modification, and if so, to determine where the modification should be made. The schema (database schema) is the logical framework describing the data organization structure in a database. It defines a collection of objects such as tables, views, and indexes, as well as the relationships between them (e.g., primary keys, foreign key constraints) and data types.
[0105] The method in this disclosure captures metadata changes and data operations, parses and extracts metadata features, generates data dependency analysis, and constructs a graph-based data relationship engine to achieve upward (upstream) and downward (downstream) tracing. It can automatically retrieve downstream dependency hierarchies at the field level. When a field of a basic core model changes, it can automatically detect the downstream data model processing tasks that require script modification and output the specific modifications, achieving intelligent management and maintenance of the data warehouse model.
[0106] This solution eliminates the need for extensive manual retrieval, addressing the urgent need for automated data lineage analysis and intelligent data model management in large-scale, rapidly changing data environments. It not only significantly reduces labor costs but also improves the efficiency and accuracy of data governance, providing more reliable support for data-driven decision-making.
[0107] In practical applications, if an update to data in a big data model is detected, the table and fields containing the updated data can be used as query metadata. Next, the query metadata is input into a data relational engine, which then outputs the downstream task data affected by the query metadata and the corresponding modification information. The output of the data relational engine can then be displayed for manual review.
[0108] Taking the change of the `store-name` field in `table1`, a core model in the logistics domain of a data mart, as an example. This field is frequently used in data products across various business lines and is accessed and read by a large number of downstream models. Updates necessitate unified judgments for the processing tasks of these downstream models to determine whether their processing scripts need modification. According to the method of this disclosure, the downstream model requiring modification can be located, corresponding to the relevant scheduling platform task, and the modification details can be provided. Specific steps are as follows... Figure 3B As shown:
[0109] First, all scheduling tasks under the logistics marketplace were obtained from the big data platform, totaling 5,990.
[0110] Next, the store-name of table1 was used as metadata input into the data relationship engine, resulting in 2795 downstream tasks, which greatly narrowed the scope of candidate set M, reducing the candidate set M by more than 50%.
[0111] Next, iterate through the candidate set M, read the SQL script of each model processing task, obtain the affected field col-name accessed in the task according to the lineage, parse the script, and determine whether the SQL script needs to be modified when reading col-name.
[0112] Finally, the tasks that need to be modified are output, and the modifications to these 613 tasks are manually reviewed and then submitted to the scheduling center for release.
[0113] Practical application testing revealed that the number of candidate tasks was reduced from 5990 to 613, with specific SQL modifications provided, requiring only manual review in the final stage. The method provided in this disclosure reduces the entire candidate pool by approximately 90% in a short time, significantly reducing manual retrieval and processing time and improving efficiency.
[0114] Further reference Figure 4 As a response to the above Figure 1 , 2 The implementation of the method shown herein provides some embodiments of a data relationship engine construction apparatus. These apparatus embodiments are related to... Figure 1 , 2 The methods and embodiments shown correspond to those described. This device can be specifically applied to various electronic devices.
[0115] like Figure 4As shown, the data relationship engine construction apparatus 400 in some embodiments may include: a data flow direction determination unit 401, configured to determine the data flow direction based on the metadata and data operation tasks of the big data model; a downstream table determination unit 402, configured to determine the downstream tables that have a data relationship with the target table based on the data flow direction; a downstream field determination unit 403, configured to determine the downstream fields of the target field based on the data relationship between each field in the target table and each field in the downstream table, wherein the target field is a field in the target table; and an engine construction unit 404, configured to construct the data relationship engine of the big data model based on the downstream tables of the target table and the downstream fields of the target field.
[0116] In some embodiments, the downstream table determination unit 402 may include: a data representation subunit (not shown in the figure), configured to represent the data in the data flow using an adjacency list method according to the relationship between the table in the data flow and the data operation task; and a data traversal subunit (not shown in the figure), configured to traverse the adjacency list using a set traversal method to obtain the downstream tables that have a data relationship with the target table.
[0117] In some embodiments, the data traversal subunit may be further configured to traverse the adjacency list using a breadth-first traversal method, capture all downstream tables of the target table, and perform hierarchical processing on the downstream tables to obtain a hierarchical downstream table with the target table as the first layer.
[0118] In some embodiments, the downstream field determination unit 403 may be further configured to determine the downstream fields of the target field by traversing the data flow according to the relationship between each field in each table and the data operation task; and to perform hierarchical processing on the downstream fields to obtain a hierarchical downstream field with the target field as the first layer.
[0119] In some embodiments, the data relationship engine construction apparatus 400 may further include: a modification determination unit (not shown in the figure), configured to determine whether the data of a downstream field needs to be modified for each downstream field of the target field, based on the operation content of the data operation task corresponding to the downstream field; and a data modification unit (not shown in the figure), configured to modify the data of the downstream fields to be modified sequentially according to the hierarchy of each downstream field.
[0120] In some embodiments, the modification determination unit may be further configured to acquire and parse the script data of the data operation task corresponding to the downstream field, determine the operation content of the data operation task, classify the operation content of the data operation task according to a preset classification rule, and determine whether the data of the downstream field needs to be modified based on the classification result.
[0121] In some embodiments, the data modification unit may be further configured to prioritize downstream fields according to their hierarchy, wherein the downstream fields with smaller hierarchy numbers have higher priority; and modify the script data of the downstream fields to be modified in descending order of priority.
[0122] In some embodiments, the data relationship engine construction apparatus 400 may further include a modification content generation unit (not shown in the figure), configured to, for each downstream field to be modified, obtain database attribute information of a target upstream table, wherein the target upstream table is the upstream table located in the table where the downstream field is located; generate prompt information based on the database attribute information, the original script data of the downstream field and the information to be modified; and obtain the modification content information of the downstream field based on the prompt information and the large language model.
[0123] In some embodiments, the engine building unit 404 can be further configured as a data relationship engine for building big data models based on graphs, wherein the types of nodes in the graph include at least one of the following: tables, fields, and data operation tasks, and the types of edges in the graph include at least one of the following: contain, read, write, and transform; and for field nodes in the graph, their edges are added with operation content and hierarchical attributes.
[0124] In some embodiments, the data relationship engine building apparatus 400 may further include an online analysis unit (not shown in the figure), configured to, in response to detecting a data update in the big data model, use the table and field containing the updated data as query metadata; input the query metadata into the data relationship engine, output the downstream task data to be modified affected by the query metadata, and the corresponding modification information; and display the output of the data relationship engine for manual review.
[0125] It is understandable that the units and references described in the data relation engine construction device 400 are... Figure 1 , 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the data relation engine construction device 400 and the units contained therein, and will not be repeated here.
[0126] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.
[0127] like Figure 5As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0128] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, speakers, vibrators, etc.; storage devices 508 including, for example, hard disks, disks, memory cards, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.
[0129] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this disclosure.
[0130] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0131] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.
[0132] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine the data flow direction based on the metadata and data operation tasks of the big data model; determine downstream tables that have data relationships with the target table based on the data flow direction; determine downstream fields of the target field based on the data relationships between fields in the target table and fields in the downstream tables, wherein the target field is a field in the target table; and construct a data relationship engine for the big data model based on the downstream tables of the target table and the downstream fields of the target field.
[0133] Furthermore, computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages and conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0135] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a data flow direction determination unit, a downstream table determination unit, a downstream field determination unit, and an engine building unit. The names of these units do not necessarily limit the unit itself; for example, the data flow direction determination unit may also be described as "a unit that determines the data flow direction based on the metadata and data operation tasks of a big data model."
[0136] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0137] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for constructing a data relationship engine.
[0138] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A method for constructing a data relationship engine, comprising: Determine the data flow direction based on the metadata and data operation tasks of the big data model; Based on the data flow, determine the downstream tables that have data relationships with the target table; Based on the data relationship between each field in the target table and each field in the downstream table, the downstream field of the target field is determined, wherein the target field is a field in the target table; Based on the downstream tables of the target table and the downstream fields of the target field, a data relationship engine for the big data model is constructed.
2. The method for constructing a data relationship engine according to claim 1, wherein, The step of determining downstream tables that have a data relationship with the target table based on the data flow includes: Based on the relationship between the table and the data operation task in the data flow, the data in the data flow is represented using the adjacency list method; The adjacency list is traversed using a defined traversal method to obtain downstream tables that have data relationships with the target table.
3. The method for constructing a data relationship engine according to claim 2, wherein, The step of traversing the adjacency list based on a predetermined traversal method to obtain downstream tables that have data relationships with the target table includes: The adjacency list is traversed using a breadth-first search method to capture all downstream tables of the target table. The downstream table is processed hierarchically to obtain a hierarchical downstream table with the target table as the first layer.
4. The method for constructing a data relationship engine according to claim 1, wherein, The step of determining the downstream field of the target field based on the data relationship between each field in the target table and each field in the downstream table includes: Based on the relationship between each field in each table and the data operation task in the data flow, the downstream fields of the target field are determined by traversing the data flow. The downstream fields are processed hierarchically to obtain a hierarchical downstream field with the target field as the first layer.
5. The method for constructing a data relationship engine according to claim 4, wherein, The method further includes: For each downstream field of the target field, determine whether the data of that downstream field needs to be modified based on the operation content of the data operation task corresponding to that downstream field; and For the downstream fields to be modified in the target field, the data of the downstream fields to be modified are modified sequentially according to the hierarchy of each downstream field.
6. The method for constructing a data relationship engine according to claim 5, wherein, The step of determining whether the data in the downstream field needs to be modified based on the operation content of the data operation task corresponding to the downstream field includes: Obtain and parse the script data of the data operation task corresponding to the downstream field to determine the operation content of the data operation task; Based on preset classification rules, classify the operation content of this data operation task; Determine whether the data in this downstream field needs to be modified based on the classification results.
7. The method for constructing a data relationship engine according to claim 5, wherein, The step of modifying the data of the downstream fields to be modified sequentially according to the hierarchy of each downstream field includes: The downstream fields are prioritized according to their hierarchy, with the lower the hierarchy number of the downstream field, the higher the priority. The script data of the downstream fields to be modified are modified sequentially according to their priority from high to low.
8. The method for constructing a data relationship engine according to claim 5, wherein, The method further includes: For each downstream field to be modified, obtain the database attribute information of the target upstream table, wherein the target upstream table is the upstream table located in the table containing the downstream field; Based on the database attribute information, the original script data of the downstream field, and the information to be modified, a prompt message is generated; Based on the prompt information and the large language model, the modified content information of the downstream field is obtained.
9. The method for constructing a data relationship engine according to claim 1, wherein, The data relationship engine for building big data models includes: A data relationship engine for building big data models based on graphs, wherein the types of nodes in the graph include at least one of the following: tables, fields, and data operation tasks; and the types of edges in the graph include at least one of the following: contain, read, write, and transform; and For the field nodes in the diagram, operation content and hierarchy attributes are added to their edges.
10. The method for constructing a data relationship engine according to any one of claims 1-9, wherein, The method further includes: In response to the detection of data updates in the big data model, the table and fields containing the updated data are used as query metadata; The query metadata is input into the data relationship engine, and the downstream task data to be modified that is affected by the query metadata, as well as the corresponding modification content information, is output. The output of the data relationship engine is displayed for manual review.
11. A data relationship engine construction apparatus, comprising: The data flow determination unit is configured to determine the data flow direction based on the metadata and data operation tasks of the big data model; The downstream table determination unit is configured to determine the downstream table that has a data relationship with the target table based on the data flow direction. The downstream field determination unit is configured to determine the downstream field of the target field based on the data relationship between each field in the target table and each field in the downstream table, wherein the target field is a field in the target table; The engine building unit is configured to build a data relationship engine for a big data model based on downstream tables of the target table and downstream fields of the target field.
12. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for constructing a data relationship engine as described in any one of claims 1-10.
13. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method for constructing a data relationship engine as described in any one of claims 1-10.
14. A computer program product comprising a computer program that, when executed by a processor, implements the method for constructing a data relationship engine as described in any one of claims 1-10.
