Method and device for constructing data blood relationship graph

Abstract

The application discloses a data blood relationship graph construction method and device, and the method comprises the following steps: performing N acquisition tasks in parallel, obtaining data processing logic carriers from N heterogeneous data processing source systems respectively, and obtaining initial data source files; performing a first processing operation on the initial data source files to construct a structured prompt word set of the initial data source files; inputting the structured prompt word set into a pre-trained large language model to perform a second processing operation on the structured prompt word set by the large language model, and outputting a structured field-level blood relationship pair set; and constructing a target data blood relationship graph based on the field-level blood relationship pair set. Through the application, the technical problem of low precision of the data blood relationship graph constructed in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for constructing a data lineage map. Background Technology

[0002] In related technologies, data lineage analysis methods mainly rely on parsing structured scripts in data warehouses. By extracting Structured Query Language (SQL) statements and combining lexical and syntactic analysis with metadata completion, table-level or field-level data flow graphs are constructed from data warehouse scheduling logs or script libraries. However, this approach suffers from insufficient data source coverage, resulting in low accuracy of the constructed data lineage graphs. Summary of the Invention

[0003] This application provides a method and apparatus for constructing a data kinship map, so as to at least solve the technical problem of low accuracy of data kinship maps constructed in related technologies.

[0004] According to one aspect of the embodiments of this application, a method for constructing a data lineage graph is provided, comprising: executing N acquisition tasks in parallel to obtain data processing logic carriers from N heterogeneous data processing source systems to obtain an initial data source file, wherein the data processing logic carrier is used to store information of data processing logic, and N is a natural number greater than 1; performing a first processing operation on the initial data source file to construct a structured prompt word set of the initial data source file, wherein the structured prompt words in the structured prompt word set are used to represent the relationship between a single initial data source file and an associated metadata context; inputting the structured prompt word set into a pre-trained large language model to perform a second processing operation on the structured prompt word set through the large language model to output a structured field-level lineage relationship pair set; and constructing a target data lineage graph based on the field-level lineage relationship pair set.

[0005] According to another aspect of the embodiments of this application, an apparatus for constructing a data lineage graph is also provided, comprising: a first acquisition module, configured to execute N acquisition tasks in parallel, acquiring data processing logic carriers from N heterogeneous data processing source systems respectively to obtain an initial data source file, wherein the data processing logic carrier is used to store information of data processing logic, and N is a natural number greater than 1; a first execution module, configured to perform a first processing operation on the initial data source file to construct a structured prompt word set of the initial data source file, wherein the structured prompt words in the structured prompt word set are used to represent the relationship between a single initial data source file and an associated metadata context; a first determination module, configured to input the structured prompt word set into a pre-trained large language model to perform a second processing operation on the structured prompt word set through the large language model to output a structured field-level lineage relationship pair set; and a first construction module, configured to construct a target data lineage graph based on the field-level lineage relationship pair set.

[0006] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed by a processor.

[0007] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to perform the steps of any of the above method embodiments through the computer program.

[0009] This application achieves parallel acquisition and aggregation of multi-source data by efficiently obtaining data processing logic carriers containing data processing logic from N heterogeneous data processing source systems through the parallel execution of N acquisition tasks. By constructing a structured set of prompt words representing the contextual relationship between the initial data source file and associated metadata, a large language model can perform deep semantic understanding based on rich contextual information, thereby accurately executing the second processing operation and outputting a structured set of field-level lineage relationship pairs. Finally, based on this set, a more comprehensive and accurate target data lineage relationship graph can be constructed. Therefore, it solves the technical problem of low accuracy in data lineage relationship graphs constructed in related technologies, achieving the technical effect of improving the accuracy of data lineage relationship graphs. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating an application scenario of a method for constructing a data lineage map according to an embodiment of this application;

[0011] Figure 2 This is a flowchart illustrating an optional method for constructing a data kinship map according to an embodiment of this application;

[0012] Figure 3 This is a flowchart illustrating the precise parsing method at the data lineage field level based on a large language model in this optional example;

[0013] Figure 4 This is a structural block diagram of an apparatus for constructing a data kinship map according to an embodiment of this application;

[0014] Figure 5 A schematic block diagram of a computer system architecture for implementing an electronic device according to embodiments of the present application is shown. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] According to one aspect of the embodiments of this application, a method for constructing a data kinship map is provided. Optionally, in this embodiment, the above-described method for constructing a data kinship map can be applied, but is not limited to, to applications such as... Figure 1 The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to provide data storage services for server 104.

[0018] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), mobile phone, tablet computer, etc. Server 104 may be, but is not limited to, a cloud server, server cluster, or other server types.

[0019] The method for constructing a data lineage graph according to embodiments of this application can be executed by server 104, by terminal device 102, or by both server 104 and terminal device 102. The data query method of this application according to embodiments of this application can also be executed by a client installed on the terminal device 102.

[0020] Taking the method of constructing the data kinship map in this embodiment, executed by server 104, as an example, Figure 2 This is a flowchart illustrating an optional method for constructing a data kinship map according to an embodiment of this application, as shown below. Figure 2As shown, the process of this method may include the following steps:

[0021] Step S202: Execute N acquisition tasks in parallel to obtain data processing logic carriers from N heterogeneous data processing source systems to obtain initial data source files. The data processing logic carriers are used to store information about data processing logic, and N is a natural number greater than 1.

[0022] This embodiment can be applied to the fields of data governance and data asset management, and is particularly suitable for scenarios in traditional hybrid architecture enterprises with incomplete data lineage coverage, non-standard or highly complex Structured Query Language (SQL) code, and a lack of unified scheduling logs. For example, in large financial institutions or internet companies, there are a large number of non-standard SQL statements that use SELECT statements, dynamic concatenation, or complex nesting. Traditional data warehouse scheduling log-based parsing tools cannot access this hidden data or fail to parse it. In this case, this embodiment can automatically discover and accurately construct a full-link field-level data lineage graph that runs through the business system to the data warehouse by actively collecting source code and combining it with large-scale model semantic parsing.

[0023] Optionally, in this embodiment, parallel execution refers to the system utilizing a multi-threaded, multi-process, or distributed task scheduling framework to simultaneously launch multiple independent data acquisition tasks. Each task runs independently without blocking the others, which can shorten the overall data acquisition time.

[0024] Optionally, in this embodiment, heterogeneous data processing source systems refer to a collection of systems whose data processing environments, platforms, or storage media differ significantly in terms of technology stack, protocols, data formats, or storage structures. These systems may employ different programming languages, different database types, different code hosting platforms, or different scheduling engines. For example, N heterogeneous data processing source systems may include version control systems, data warehouse scheduling engines, script management platforms, big data computing platforms, reporting tools, etc.

[0025] Optionally, in this embodiment, the data processing logic carrier refers to a file, object, or data block that physically stores data processing logic code or configuration information. It is a container or carrier of data processing logic, and its content can be parsed to extract SQL statements, data transformation rules, etc. The carrier can take various forms, including but not limited to source code files, configuration files, script files, or database logs.

[0026] Optionally, in this embodiment, the initial data source file refers to the original file content directly downloaded, captured, or read from the data processing logic carrier. At this time, the file has not yet undergone SQL extraction, cleaning, or standardization processing, and may contain non-SQL code (such as Java / Python business logic), comments, configuration items, or noisy data such as garbled characters.

[0027] Step S204: Perform a first processing operation on the initial data source file to construct a set of structured prompt words for the initial data source file, wherein the structured prompt words in the set of structured prompt words are used to represent the relationship between a single initial data source file and the associated metadata context;

[0028] Optionally, the first processing operation in this embodiment refers to the preprocessing flow of the initial data source file obtained from the heterogeneous source. The purpose is to extract valid SQL statements or data processing logic, associate them with necessary contextual information, and transform them into a format understandable by the large language model. This operation typically includes: SQL fragment extraction, noise filtering, metadata retrieval, and format standardization.

[0029] Optionally, in this embodiment, structured prompts refer to text or JavaScript Object Notation (JSON) structures that contain explicit instructions, contextual information, and input data. It is not merely simple text concatenation, but rather guides the larger model to focus on specific tasks (such as SQL completion or field mapping) through specific templates or schema definitions. In this embodiment, it refers to the input object after fusing the SQL to be parsed with the associated metadata context.

[0030] Optionally, in this embodiment, the metadata context refers to the static description information related to the tables and fields involved in the initial data source file. This includes, but is not limited to:

[0031] Technical metadata: database name, schema name, table name, field name, field type, primary key, foreign key, and comments.

[0032] Business metadata: Field business meaning, data standards, sensitivity levels, and enumeration value definitions.

[0033] Structure metadata: The structure definition of the source table and the target table, used to help the large language model understand the specific fields after the SELECT statement is expanded.

[0034] Step S206: Input the structured prompt word set into a pre-trained large language model, so as to perform a second processing operation on the structured prompt word set through the large language model and output a structured field-level lineage relationship pair set;

[0035] Optionally, in this embodiment, the pre-trained Large Language Model (LLM) refers to a general-purpose large model trained on massive amounts of code and natural language data, such as a general-purpose text model (LLMs) or a code-specific large model (LLMs). In this embodiment, the large language model may be fine-tuned with specific instructions to optimize its SQL understanding and code generation capabilities, but it does not need to be trained from scratch.

[0036] Optionally, the second processing operation in this embodiment refers to the reasoning process performed by the large language model on the input structured prompts. This includes:

[0037] Semantic understanding: Understanding the logic of SQL (such as join conditions, group filtering, and conditional aggregation).

[0038] Logical deconstruction: Identify how source fields are transformed (e.g., through calculations, aliases, functions) to map to target fields.

[0039] Structured generation: Output field-level lineage pairs according to a predefined JSON Schema or list format.

[0040] Optionally, in this embodiment, the structured field-level lineage pair set refers to a set consisting of multiple triples or tuples, representing the mapping relationship from the source field to the target field. The field-level lineage pair set can be a JSON List or array containing all the above mapping relationships.

[0041] Step S208: Construct a target data lineage graph based on the field-level lineage relationships of the set.

[0042] Optionally, in this embodiment, the field-level lineage pairs are the smallest granular units of edges or relationships in the graph. This explicitly indicates from which specific field the data flows to which specific field.

[0043] Optionally, in this embodiment, the target data lineage graph refers to a network structure formed by storing all tables and fields as nodes and lineage pairs as directed edges in a graph database.

[0044] Through the above steps, by executing N acquisition tasks in parallel, data processing logic carriers containing data processing logic can be efficiently obtained from N heterogeneous data processing source systems, realizing parallel acquisition and aggregation of multi-source data. By constructing a structured set of prompt words representing the contextual relationship between the initial data source file and associated metadata, the large language model can perform deep semantic understanding based on rich contextual information, thereby accurately executing the second processing operation and outputting a structured set of field-level lineage relationship pairs. Finally, based on this set, a more comprehensive and accurate target data lineage relationship graph can be constructed. Therefore, the technical problem of low accuracy in data lineage relationship graphs constructed in related technologies can be solved, achieving the technical effect of improving the accuracy of data lineage relationship graphs.

[0045] In one exemplary embodiment, N data acquisition tasks are executed in parallel to obtain data processing logic carriers from heterogeneous data processing source systems to obtain an initial data source file. This includes: if the data processing source system includes a data warehouse scheduling system, a source code version control system, and a script management platform, three data acquisition tasks are executed in parallel to obtain SQL scripts from the execution logs of the data warehouse scheduling system, source code files containing SQL logic from the source code version control system, and script files from the script management platform, respectively. The data warehouse scheduling system stores the data processing logic in the form of structured logs, the source code version control system stores the data processing logic in the form of structured code files, and the script management platform stores the data processing logic in the form of semi-structured script files. The initial data source file is obtained by combining the SQL scripts, the source code files, and the script files.

[0046] Optionally, in this embodiment, the data warehouse scheduling system refers to a platform used for orchestrating, scheduling, and monitoring the execution of data extraction, transformation, and loading (ETL) tasks in the data warehouse. Structured logs are task execution records kept by the data warehouse scheduling system, typically including task ID, execution time, status, and key information such as the actual executed SQL statements and input / output table names. This data is stored in a relational database or log file system and has a fixed format (such as JSON, CSV, or text with specific delimiters) for easy program parsing.

[0047] Optionally, the source code version control system in this embodiment refers to a system used to store, manage, and track the history of source code changes during software development. Structured code files refer to files written in a specific programming language (such as Java) or markup language (such as XML). In data development scenarios, these files typically contain embedded SQL statements, and their structure follows the syntax rules of the programming language.

[0048] Optionally, in this embodiment, the script management platform refers to a platform used for centralized management of various internal enterprise operation and maintenance scripts and data analysis scripts. Semi-structured script files refer to files containing text content but lacking a strict syntax tree structure, such as Shell (.sh), Perl (.pl), Python (.py), or Bash scripts. The data processing logic (SQL) is typically embedded as a string in the flow control code and needs to be extracted from context fragments using regular expressions or specific rules.

[0049] Optionally, in this embodiment, the initial data source file refers to the collection of raw files obtained after parallel acquisition. These files have not yet undergone SQL extraction, cleaning, or standardization, and retain the original format (log text, XML code, Shell scripts, etc.) of their respective source systems, serving as the input objects for the first processing operation in subsequent steps.

[0050] For example, a data warehouse scheduling log collector periodically captures and parses executed SQL scripts from the logs of ETL scheduling engines (such as Airflow and Dolphin Scheduler).

[0051] The source code repository collector connects to the enterprise's internal version control systems such as Git and SVN. It allows for filtering by repository address, branch, and file path (e.g., `...`). .sql`, .py, .java, The collector retrieves source code periodically or on a trigger-based basis using .xml files. It can identify SQL fragments within code files (such as MyBatisMapper files embedded in Java, SQLAlchemyORM statements in Python, DataFrame operations in Spark code, etc.) and extract them as independent SQL units.

[0052] The script management center collector obtains the SQL portion of Perl, Shell, and other scripts from the company's unified script management platform.

[0053] This embodiment achieves comprehensive coverage of data processing logic in structured logs, structured code, and semi-structured scripts within an enterprise by executing heterogeneous data collection tasks in parallel for a data warehouse scheduling system, source code version control system, and script management platform. This provides comprehensive and standardized data for subsequent semantic-level accurate parsing using large language models, significantly improving the completeness of constructing a data lineage graph.

[0054] In an exemplary embodiment, the first processing operation includes: extracting data processing statements from the initial data source file to obtain a set of data processing statements; associating the structured metadata of each data processing statement in the set of data processing statements to obtain a set of structured metadata; and generating the set of structured prompt words using the set of data processing statements and the set of structured metadata.

[0055] Optionally, the first processing operation in this embodiment is a preprocessing pipeline for the initial data source file obtained from the heterogeneous source. The purpose is decoupling and enhancement, that is, firstly extracting the core logic (SQL statements) from the unstructured or semi-structured file, then supplementing it with missing context information (metadata), and finally assembling it into an instruction format that the large model can understand.

[0056] Optionally, in this embodiment, the data processing statements refer to the core code snippets identified and extracted from the initial data source file (such as a Shell script, XML file, or log text) used to perform data operations. In the context of data lineage, this mainly refers to SQL statements (including SELECT, INSERT, UPDATE, DELETE, etc.) or similar data transformation logic code.

[0057] Optionally, in this embodiment, structured metadata refers to descriptive information about data tables, fields, and their attributes stored in a standardized format (such as JSON, XML, CSV). For example, technical metadata includes database name, schema name, table name, field name, field type (VARCHAR, INT, etc.), primary key, foreign key, and comments. Business metadata includes the business meaning of fields, data standards, sensitivity levels, and enumerated values.

[0058] Optionally, in this embodiment, the structured prompt set refers to a collection of multiple independent prompt objects. Each prompt object contains a fixed structure (such as a JSON schema). For example, it includes: role settings (such as SQL expert), associated metadata context, data processing statements to be parsed, and specific task instructions (such as expanding a SELECT statement and outputting lineage JSON). This structured format ensures the consistency of the large model input and facilitates subsequent parsing of the model output by the program.

[0059] This embodiment transforms unstructured or semi-structured initial data source files into a standardized set of structured prompt words through the first processing operation. This not only enables the accurate extraction of core data processing logic from heterogeneous sources, but also endows the data processing logic with rich semantic context by dynamically associating structured metadata such as table structure, field types, and business meanings, thereby improving the accuracy of field-level lineage resolution.

[0060] In an exemplary embodiment, generating the structured prompt word set using the data processing statement set and the structured metadata set includes: constructing a structured prompt word template, wherein the structured prompt word template includes a role definition area, a data context area, and an instruction constraint area; embedding the data processing statement set and the structured metadata set into the data context area to obtain a target data context area; embedding processing rules into the instruction constraint area to obtain a target instruction constraint area, wherein the processing rules include rules for standardizing data processing statements, rules for completing fields, and rules for extracting lineage relationships; assembling the role definition area, the target data context area, and the target instruction constraint area to obtain the structured prompt word set, wherein the role definition area includes task boundaries for activating the knowledge of the large language model and constraining the output format of the large language model.

[0061] Optionally, the structured prompt template in this embodiment refers to a predefined text or JSON format framework containing fixed structure fields (such as role, context, and instructions). It does not directly contain specific business data, but rather acts as a container to dynamically populate different SQL statements and metadata, ensuring that the input format sent to the large model remains consistent each time, facilitating programmatic processing and result parsing.

[0062] Optionally, in this embodiment, the role definition area is the first part of the Prompt, used to set the identity, professional field, and code of conduct of the large model.

[0063] Optionally, the task boundaries in this embodiment are used to define what the large language model does and does not do, for example, "only extract lineage relationships, do not execute SQL" or "only output JSON format, do not output explanatory text".

[0064] Optionally, in this embodiment, knowledge activation is achieved by setting an expert role (such as a senior data architect) to activate the model's relevant training knowledge in the fields of SQL semantic analysis and data governance.

[0065] Optionally, the data context area in this embodiment is the second part of the Prompt, used to provide static factual information required by the large language model for reasoning.

[0066] Optionally, in this embodiment, the target data context area refers to the metadata set embedded in the template after filtering and formatting. Presented in JSON or tabular form, it includes the structural definitions (field names, types, comments) of the source and target tables involved, forming the basis for the model's understanding of SELECT statement expansion and field mapping.

[0067] Optionally, the instruction constraint area in this embodiment is the third part of the Prompt, which is used to specify the specific logic and output specifications for the model to process data.

[0068] For example, processing rules include: normalization rules, such as converting implicit JOINs to explicit JOINs and unifying keyword capitalization; completion rules, such as expanding SELECT statements into a specific list of fields based on the metadata of the context area; and extraction rules, defining the output format of lineage relationships (such as JSON Schema) and clarifying the logic of field mapping (such as direct copying, aggregation, calculation, etc.).

[0069] Optionally, in this embodiment, the structured prompt set refers to a series of complete input objects generated after template assembly, which can be directly sent to the application programming interface (API) of a large model. Each object contains specific SQL, corresponding metadata, and unified instruction constraints.

[0070] This embodiment constructs a structured prompt word template that includes a role definition area, a data context area, and an instruction constraint area, thereby achieving the standardization and semantic enhancement of the input of the large language model, which significantly improves the accuracy of the large language model in understanding heterogeneous and non-standard SQL.

[0071] In an exemplary embodiment, the second processing operation includes: performing normalization processing on abnormal structured prompts in the structured prompt set to obtain a target structured prompt set; parsing the business logic of each target structured prompt in the target structured prompt set to obtain a business logic set, wherein the business logic includes at least one of the following: table join relationship, field mapping and calculation, filtering conditions, grouping aggregation; performing semantic understanding on each business logic in the business logic set, and outputting the set of field-level lineage pairs.

[0072] Optionally, the second processing operation in this embodiment refers to the actual large-scale model inference and post-processing of the preprocessed and templated prompt words. This includes secondary verification of input quality (anomaly handling), logical decomposition of model-generated content (business logic extraction), and semantic confirmation and formatting of the final lineage relationship (lineage relationship pair generation).

[0073] Optionally, in this embodiment, the abnormal structured prompt words refer to prompt words that cannot be correctly processed by the large model before being sent to the large model due to format errors, missing key information, excessive tokens, or failed syntax validation. For example, JSON format not closed, missing necessary fields in metadata, truncated SQL statements, etc.

[0074] Optionally, the normalization process in this embodiment refers to the remedial measures taken for abnormal prompt words. Automatic repair includes actions such as completing missing JSON parentheses, truncating excessively long metadata, and replacing illegal characters. Degradation / discarding involves marking the prompt as abnormal and discarding it if it cannot be automatically repaired (e.g., missing critical SQL logic), or logging it for manual intervention to avoid contaminating the final lineage graph.

[0075] Optionally, in this embodiment, the target structured prompt word set refers to a set of prompt words that have been normalized, are correctly formatted, have complete information, and meet the input requirements of a large model. These prompt words are the basis for subsequent parsing.

[0076] Optionally, in this embodiment, the set of business logic refers to the specific data processing behaviors identified by the large language model from SQL statements. In the context of data lineage, this mainly includes: table join relationships, which define the association keys between the source table and the target table; field mapping and calculation; filtering conditions, which, although not directly generating lineage, affect data visibility; and grouping and aggregation, which defines the logic for merging multiple source fields into a single target word source.

[0077] Optionally, in this embodiment, semantic understanding refers to the large language model not only recognizing SQL syntax but also understanding the underlying data meaning. For example, it may recognize that user_id is a primary key in the source table and a foreign key in the target table, thus confirming a one-to-many relationship; or it may recognize that CAST(age AS INT) is a type conversion rather than data loss.

[0078] Optionally, in this embodiment, the field-level lineage relationship set refers to the final output structured data list, where each element represents a specific field mapping relationship.

[0079] This embodiment achieves refined parsing and quality control from prompt words to lineage relationships through the second processing operation, thereby improving the accuracy of lineage parsing in complex SQL scenarios.

[0080] In an exemplary embodiment, the field-level lineage relationship pair set includes a source identifier, a target identifier, and a relationship indicator. The source identifier consists of a source database identifier, a source table identifier, and a source field identifier. The target identifier consists of a target database identifier, a target table identifier, and a target field identifier. The relationship indicator is used to indicate the data flow direction. The source identifier points to the target identifier to form a unidirectional data dependency link.

[0081] Optionally, in this embodiment, the field-level lineage relationship refers to the smallest granular unit describing the flow of data from the source field to the target field. It is an edge in the data lineage graph and contains complete contextual information about the data flow. The source identifier is a composite key used to uniquely identify the starting point of the data flow, consisting of the source database identifier, the source table identifier, and the source field identifier.

[0082] Optionally, in this embodiment, the unidirectional data dependency link refers to a directed graph where the data lineage always points from the source to the target. The target data only exists if the source data exists; changes in the source data may affect the target data.

[0083] This embodiment achieves standardized and precise expression of data flow by constructing structured field-level lineage relationship pairs including source identifiers, target identifiers, and relationship indicators, providing a standard data foundation for efficient storage, fast querying, and complex impact analysis in graph databases.

[0084] In an exemplary embodiment, constructing a target data lineage graph based on the set of field-level lineage pairs includes: merging the set of field-level lineage pairs with a stored set of historical field-level lineage pairs to obtain a merged set of field-level lineage pairs; performing a data verification operation on the merged set of field-level lineage pairs to obtain a target set of field-level lineage pairs, wherein the data verification operation includes cleaning the merged set of field-level lineage pairs and marking conflicting or abnormal lineage pairs in the merged set of field-level lineage pairs; and storing the target set of field-level lineage pairs in a graph database to generate the target data lineage graph, wherein nodes in the target data lineage graph represent data tables and data fields, and edges in the target data lineage graph represent lineage relationships.

[0085] Optionally, in this embodiment, the target data lineage graph refers to a structured knowledge base built based on a graph theory model that reflects the entire chain flow of data assets within an enterprise.

[0086] Nodes represent entities, including tables and fields. They can be uniquely identified by the format "database.table.field". Edges represent relationships, i.e., field-level lineage pairs. They include directionality (directed graph), indicating the flow of data from a source field to a target field. Attributes represent the properties that nodes and edges can have, such as creation time, update time, source system, confidence level, and transformation logic.

[0087] The term "merge" refers to the process of combining newly parsed current field-level lineage pairs with existing historical field-level lineage pairs in the system. This ensures the integrity of the graph, avoids duplicate storage, and preserves historical change records (if versioned). Strategies such as adding / inserting, updating / overwriting, and deleting tags can be employed.

[0088] Optionally, the data verification operation in this embodiment refers to the process of performing quality checks and logical consistency verification on the fused data before writing the data into the graph database.

[0089] Cleaning operations are used to remove duplicate records, correct formatting errors, and standardize field names (such as ensuring case consistency). Specifically, it refers to the deduplication and normalization processes performed on merged data. If the same source-target field pair exists in both historical and current data and their relation indicators are consistent, one copy is retained; otherwise, a conflict is flagged. Normalization ensures that all identifiers (such as database names and table names) follow a unified naming convention, facilitating indexing and querying.

[0090] Conflict / anomaly labeling is used to detect and tag logical contradictions (such as loops, type mismatches, or non-existent source fields). For example, a conflict means that historical data shows field A directly comes from field B, but the new parsing shows that field A comes from field C, and both are valid (logically mutually exclusive). An anomaly points to a non-existent target field, a source field type that is severely incompatible with the target field type and lacks conversion logic, or the formation of a data loop. The lineage of conflicts or anomalies refers to lineage relationships discovered during the fusion or validation process that do not conform to business logic or data specifications.

[0091] This embodiment significantly improves system construction efficiency and real-time performance by incrementally updating by integrating historical and newly generated lineage data; it introduces a data verification mechanism including cleaning and conflict / anomaly marking, which effectively removes redundant noise and identifies logical contradictions, ensuring the accuracy and consistency of the lineage data stored.

[0092] The data query method in this application embodiment is explained below with reference to optional examples. The data lineage graph construction method in this optional example is a precise data lineage field-level parsing method based on a large language model. This method overcomes the limitation of traditional lineage tools that rely solely on data warehouse scheduling logs by constructing a full-link processing flow encompassing multi-source acquisition, intelligent preprocessing, large language model semantic parsing, and graph fusion storage.

[0093] Figure 3 This is a flowchart illustrating the precise parsing method at the data lineage field level based on a large language model, as shown in this optional example. Figure 3 As shown, the process of this precise parsing method at the data lineage field level based on a large language model may include the following steps:

[0094] S301 is a data acquisition system configured and running multiple parallel jobs to obtain raw script content from different data sources, specifically including:

[0095] The data warehouse scheduling log collector connects to ETL scheduling engines (such as Airflow, DolphinScheduler, etc.) and periodically captures the logs of completed SQL scripts through listening or polling to obtain the actual SQL instances and execution contexts during runtime.

[0096] This source code repository collector connects to an enterprise's internal version control system (Git, SVN, etc.). The system supports configuring repository addresses, specific branches, and file extension filtering rules (such as .sql, .py, .java, .xml, .xml, etc.). This collector has code semantic awareness capabilities and can identify SQL snippets in programming languages, for example:

[0097] MyBatis Mapper XML file in a Java project;

[0098] SQLAlchemy ORM statements or native SQL strings in a Python project;

[0099] DataFrame API transformation logic in a Spark project;

[0100] Hard-coded SQL embedded in other application code.

[0101] The script management center's collector connects to the company's unified script management platform and automatically extracts SQL statements called in batch scripts such as Perl and Shell.

[0102] S302, standardize the collected raw scripts to construct an input context suitable for Large Language Model (LLM) understanding, specifically including:

[0103] SQL Extraction and Cleaning: Using regular expressions combined with a parser, pure SQL statements or fragments are precisely extracted from mixed code files, irrelevant program logic code is removed, and basic formatting and cleaning of the SQL are performed.

[0104] Metadata context association: The system automatically identifies database names, table names, and field names involved in the SQL. It queries the metadata management system or connects to the test database to obtain structured metadata for the relevant tables, including: database name, table name, field name, field type, field comments, business meaning, data standard definitions, etc. A mapping relationship is established between SQL statements and metadata to ensure the model has sufficient background knowledge.

[0105] Structured Prompt Construction: The SQL statement to be parsed is combined with the associated metadata information according to a preset template to generate a structured prompt. The prompt includes role definitions, task instructions, SQL content, and metadata JSON / XML data, which serve as input to the LLM.

[0106] S303, the structured Prompt is sent to a dedicated LLM parsing service to perform the following core semantic processing tasks:

[0107] SQL completion and normalization:

[0108] The model recognizes fuzzy expressions in SQL (such as SELECT…) and automatically expands them into a specific list of fields (such as SELECT user_id, user_name) based on the provided metadata.

[0109] The non-standard SQL syntax was restructured and standardized, including keyword case and indentation format.

[0110] Semantic understanding and logical deconstruction: The model deeply analyzes the business logic of SQL, identifying logical units such as table joins (JOIN), field mapping, mathematical calculations (e.g., age+1), conditional filtering (WHERE), and grouping aggregation (GROUP BY). For complex nested queries (Subqueries) or stored procedures, the model uses recursive or iterative strategies to decompose them layer by layer to understand the data flow.

[0111] Field-level lineage generation: The model outputs structured field-level lineage pairs in the format: source identifier - target identifier. The identifier contains complete hierarchical information: database name.schema name.table name.field name.

[0112] For fields that have undergone function calculations or alias conversions, explicitly indicate the conversion logic type (e.g., DIRECT_COPY, CALCULATION, AGGREGATION).

[0113] S304, Post-processing the analysis results and constructing the final data pedigree map, specifically including:

[0114] Relationship Fusion and Consistency Verification: The new lineage relationships resolved by LLM are merged with the historical lineage data already stored in the system. Consistency verification rules are executed: the existence of source / target fields, type compatibility, and logical conflicts (such as loops or duplicate mappings) are checked. Conflicting or abnormal lineage relationships are marked (e.g., "Pending Manual Confirmation," "Low Confidence"), and warning logs are generated.

[0115] Persistent storage to a graph database: Persistently store the cleaned and verified lineage data to a graph database (such as Neo4j).

[0116] Node design: Data tables and data fields are used as nodes, and their attributes include name, type, comments, update time, etc.

[0117] Edge design: Blood relations are treated as directed edges, with attributes including transformation logic, confidence level, source script version, creation time, etc.

[0118] Visualized query and service integration: Provides a lineage query interface, supporting upstream and downstream impact analysis, root cause tracing, and field-level link visualization. Supports integration with data quality management (DQ) and data security anonymization systems, automatically propagating governance rules based on lineage relationships.

[0119] This specific embodiment achieves enterprise-level full-chain lineage coverage. By including source code repositories such as Git in the collection scope, this embodiment breaks through the limitation of traditional lineage tools that only focus on the data warehouse. It can automatically discover and parse the data processing logic scattered in business systems and analysis projects. This makes the constructed lineage graph truly span the entire lifecycle of data, from its generation in business systems to intermediate processing and final application in analysis, resulting in a revolutionary improvement in lineage completeness.

[0120] This significantly improves the accuracy and intelligence of lineage resolution. Leveraging the powerful semantic understanding capabilities of large language models, this implementation can accurately parse non-standard and complex SQL statements, automatically complete missing information, and accurately identify calculation and conversion logic between fields. This greatly reduces parsing errors caused by SQL syntax issues, elevates the accuracy of automated field-level lineage resolution to a practically applicable high level, and significantly reduces manual maintenance workload.

[0121] This invention enhances the proactiveness and foresight of data governance. By continuously monitoring changes in the source code repository, it can identify potential impacts of SQL logic on the data chain at an early stage when developers submit code, enabling left-shifted data governance. This helps to identify design flaws, standardization issues, or potential lineage break risks in advance, ensuring the quality and consistency of data assets from the source.

[0122] A unified, flexible, and scalable lineage analysis platform has been constructed. This embodiment decouples multi-source data collection, intelligent parsing, and graph storage, forming a standardized processing pipeline. The platform boasts excellent scalability, easily adapting to new data source types (such as new version control systems and programming frameworks) and new SQL dialects, providing a solid technical foundation for enterprises to continuously manage data lineage across rapidly evolving technology stacks.

[0123] In summary, this embodiment effectively addresses the core pain points of traditional data lineage construction methods, such as incomplete coverage, poor accuracy, and reliance on manual intervention, by integrating multi-source code acquisition and large language model parsing technologies. It provides strong and reliable technical support for enterprises to conduct refined data asset management, effective data governance, and efficient data value mining.

[0124] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0126] It should be noted that, through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0127] This embodiment also provides an apparatus for constructing a data kinship map, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] Figure 4 This is a structural block diagram of a data pedigree mapping construction device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0129] The first acquisition module 42 is used to execute N acquisition tasks in parallel, and to acquire data processing logic carriers from N heterogeneous data processing source systems to obtain an initial data source file. The data processing logic carrier is used to store information about data processing logic, and N is a natural number greater than 1.

[0130] The first execution module 44 is used to perform a first processing operation on the initial data source file to construct a set of structured prompt words for the initial data source file, wherein the structured prompt words in the set of structured prompt words are used to represent the relationship between a single initial data source file and the associated metadata context;

[0131] The first determining module 46 is used to input the structured prompt word set into a pre-trained large language model, so as to perform a second processing operation on the structured prompt word set through the large language model and output a structured field-level lineage relationship pair set;

[0132] The first construction module 48 is used to construct a target data lineage graph based on the field-level lineage relationship of the set.

[0133] It should be noted that the first acquisition module 42 in this embodiment can be used to execute the above step S202, the first execution module 44 in this embodiment can be used to execute the above step S204, the first determination module 46 in this embodiment can be used to execute the above step S206, and the first construction module 48 in this embodiment can be used to execute the above step S208.

[0134] Through the embodiments provided in this application, by executing N acquisition tasks in parallel, data processing logic carriers storing data processing logic can be efficiently obtained from N heterogeneous data processing source systems, realizing parallel acquisition and aggregation of multi-source data. By constructing a structured set of prompt words representing the contextual relationship between the initial data source file and associated metadata, a large language model can perform deep semantic understanding based on rich contextual information, thereby accurately executing the second processing operation and outputting a structured set of field-level lineage relationship pairs. Finally, based on this set, a more comprehensive and accurate target data lineage relationship graph can be constructed. Therefore, the technical problem of low accuracy in data lineage relationship graphs constructed in related technologies can be solved, achieving the technical effect of improving the accuracy of data lineage relationship graphs.

[0135] In an exemplary embodiment, the first acquisition module 42 is further configured to execute N acquisition tasks in parallel to obtain data processing logic carriers from heterogeneous data processing source systems to obtain an initial data source file: In the case that the data processing source system includes a data warehouse scheduling system, a source code version control system, and a script management platform, three data acquisition tasks are executed in parallel to obtain SQL scripts from the execution logs of the data warehouse scheduling system, source code files including SQL logic from the source code version control system, and script files from the script management platform, respectively. The data warehouse scheduling system stores the data processing logic in the form of structured logs, the source code version control system stores the data processing logic in the form of structured code files, and the script management platform stores the data processing logic in the form of semi-structured script files. The initial data source file is obtained by combining the SQL scripts, the source code files, and the script files.

[0136] In an exemplary embodiment, the first processing operation includes: extracting data processing statements from the initial data source file to obtain a set of data processing statements; associating the structured metadata of each data processing statement in the set of data processing statements to obtain a set of structured metadata; and generating the set of structured prompt words using the set of data processing statements and the set of structured metadata.

[0137] In an exemplary embodiment, the above-described apparatus is further configured to generate the structured prompt word set using the data processing statement set and the structured metadata set in the following manner: constructing a structured prompt word template, wherein the structured prompt word template includes a role definition area, a data context area, and an instruction constraint area; embedding the data processing statement set and the structured metadata set into the data context area to obtain a target data context area; embedding processing rules into the instruction constraint area to obtain a target instruction constraint area, wherein the processing rules include rules for standardizing data processing statements, rules for completing fields, and rules for extracting lineage relationships; assembling the role definition area, the target data context area, and the target instruction constraint area to obtain the structured prompt word set, wherein the role definition area includes task boundaries for activating the knowledge of the large language model and constraining the output format of the large language model.

[0138] In an exemplary embodiment, the second processing operation includes: performing normalization processing on the abnormal structured prompts in the structured prompt set to obtain a target structured prompt set; parsing the business logic of each target structured prompt in the target structured prompt set to obtain a business logic set, wherein the business logic includes at least one of the following: table join relationship, field mapping and calculation, filtering conditions, grouping aggregation; performing semantic understanding on each business logic in the business logic set, and outputting the set of field-level lineage pairs.

[0139] In an exemplary embodiment, the field-level lineage relationship pair set mentioned above includes a source identifier, a target identifier, and a relationship indicator. The source identifier consists of a source database identifier, a source table identifier, and a source field identifier. The target identifier consists of a target database identifier, a target table identifier, and a target field identifier. The relationship indicator is used to indicate the data flow direction. The source identifier points to the target identifier to form a unidirectional data dependency link.

[0140] In an exemplary embodiment, the first construction module 48 is configured to construct a target data lineage graph based on the set of field-level lineage pairs in the following manner: merging the set of field-level lineage pairs with a stored set of historical field-level lineage pairs to obtain a merged set of field-level lineage pairs; performing a data verification operation on the merged set of field-level lineage pairs to obtain a target set of field-level lineage pairs, wherein the data verification operation includes cleaning the merged set of field-level lineage pairs and marking conflicting or abnormal lineage pairs in the merged set of field-level lineage pairs; and storing the target set of field-level lineage pairs in a graph database to generate the target data lineage graph, wherein nodes in the target data lineage graph represent data tables and data fields, and edges in the target data lineage graph represent lineage relationships.

[0141] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0142] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0143] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0144] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0145] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0146] According to another aspect of the embodiments of this application, a computer program product is also provided, which includes a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0147] Figure 5 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 5 As shown, the computer system includes a Central Processing Unit (CPU) 501, which performs various appropriate actions and processes based on programs stored in ROM 502 or loaded into RAM 503 from storage section 508. Random access memory 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0148] The following components are connected to I / O interface 505: input section 506 including keyboard, mouse, etc.; output section 507 including cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 508 including hard disk, etc.; and communication section 509 including network interface card, modem, etc. Communication section 509 performs communication processing via a network such as the Internet. Drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0149] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit 501, it performs various functions defined in the system of this application.

[0150] It should be noted that, Figure 5 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0151] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0152] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing a data kinship map, characterized in that, include: N acquisition tasks are executed in parallel to obtain data processing logic carriers from N heterogeneous data processing source systems to obtain initial data source files. The data processing logic carriers are used to store information about data processing logic, and N is a natural number greater than 1. A first processing operation is performed on the initial data source file to construct a set of structured prompt words for the initial data source file, wherein the structured prompt words in the set of structured prompt words are used to represent the relationship between a single initial data source file and its associated metadata context; The structured prompt word set is input into a pre-trained large language model to perform a second processing operation on the structured prompt word set through the large language model, and outputs a structured set of field-level lineage relationship pairs; Based on the field-level lineage relationships, a lineage relationship graph of the target data is constructed for the set.

2. The method according to claim 1, characterized in that, N data acquisition tasks are executed in parallel to obtain data processing logic carriers from heterogeneous data processing source systems, resulting in initial data source files, including: In the case where the data processing source system includes a data warehouse scheduling system, a source code version control system, and a script management platform, three data acquisition tasks are executed in parallel to obtain structured query language (SQL) scripts from the execution log of the data warehouse scheduling system, source code files including SQL logic from the source code version control system, and script files from the script management platform, respectively. The data warehouse scheduling system stores the data processing logic in the form of structured logs, the source code version control system stores the data processing logic in the form of structured code files, and the script management platform stores the data processing logic in the form of semi-structured script files. The initial data source file is obtained by combining the SQL script, the source code file, and the script file.

3. The method according to claim 1, characterized in that, The first processing operation includes: Extract data processing statements from the initial data source file to obtain a set of data processing statements; By associating the structured metadata of each data processing statement in the data processing statement set with the structured metadata, a structured metadata set is obtained; The set of structured prompt words is generated using the set of data processing statements and the set of structured metadata.

4. The method according to claim 3, characterized in that, The structured prompt word set is generated using the data processing statement set and the structured metadata set, including: Construct a structured prompt word template, wherein the structured prompt word template includes a role definition area, a data context area, and an instruction constraint area; The data processing statement set and the structured metadata set are embedded into the data context region to obtain the target data context region; The processing rules are embedded into the instruction constraint region to obtain the target instruction constraint region. The processing rules include rules for standardizing data processing statements, rules for completing fields, and rules for extracting lineage relationships. The structured prompt word set is obtained by assembling the role definition area, the target data context area, and the target instruction constraint area. The role definition area includes task boundaries, which are used to activate the knowledge of the large language model and constrain the output format of the large language model.

5. The method according to claim 1, characterized in that, The second processing operation includes: Normalization processing is performed on the abnormal structured prompt words in the structured prompt word set to obtain the target structured prompt word set; The business logic of each target structured prompt word in the target structured prompt word set is analyzed to obtain a set of business logic, wherein the business logic includes at least one of the following: table join relationship, field mapping and calculation, filtering conditions, grouping and aggregation; Semantic understanding is performed on each business logic in the set of business logic, and the set of field-level lineage relationship pairs is output.

6. The method according to claim 5, characterized in that, The set of field-level lineage pairs includes source identifiers, target identifiers, and relationship indicators, wherein... The source identifier consists of the source database identifier, the source table identifier, and the source field identifier; The target identifier consists of the target database identifier, the target table identifier, and the target field identifier; The relation indicator is used to indicate the direction of data flow; The source identifier points to the target identifier to form a unidirectional data dependency link.

7. The method according to claim 1, characterized in that, Based on the aforementioned field-level kinship relationships, a target data kinship graph is constructed from the set, including: The set of field-level lineage pairs is merged with the stored set of historical field-level lineage pairs to obtain a merged set of field-level lineage pairs. Perform a data validation operation on the fused field-level lineage relationship pair set to obtain a target field-level lineage relationship pair set. The data validation operation includes cleaning the fused field-level lineage relationship pair set and marking conflicting or abnormal lineage relationship pairs in the fused field-level lineage relationship pair set. The target field-level lineage relationship set is stored in a graph database to generate the target data lineage relationship graph, wherein the nodes in the target data lineage relationship graph are used to represent data tables and data fields, and the edges in the target data lineage relationship graph are used to represent lineage relationships.

8. An apparatus for constructing a data kinship map, characterized in that, include: The first acquisition module is used to execute N acquisition tasks in parallel, and to acquire data processing logic carriers from N heterogeneous data processing source systems to obtain an initial data source file. The data processing logic carrier is used to store information about data processing logic, and N is a natural number greater than 1. A first execution module is configured to perform a first processing operation on the initial data source file to construct a set of structured prompt words for the initial data source file, wherein the structured prompt words in the set of structured prompt words are used to represent the relationship between a single initial data source file and its associated metadata context; The first determining module is used to input the structured prompt word set into a pre-trained large language model, so as to perform a second processing operation on the structured prompt word set through the large language model and output a structured field-level lineage relationship pair set; The first construction module is used to construct a target data lineage graph based on the field-level lineage relationships of the set.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

Images