Metadata blood relationship analysis method and device, storage medium and computer program product

Abstract

The invention discloses a metadata consanguinity analysis method and device, a storage medium and a computer program product, and relates to the technical field of data processing.The method comprises the steps that full-quantity metadata are collected, and the full-quantity metadata comprise structured metadata, semi-structured metadata and unstructured metadata; calling a structured analysis tool to perform structured blood relationship analysis on the structured metadata to obtain a structured blood relationship analysis result; calling a pre-constructed large model to carry out unstructured blood relationship analysis on the semi-structured metadata and the unstructured metadata to obtain an unstructured blood relationship analysis result; and generating a candidate blood relationship pool according to the structured blood relationship analysis result and the unstructured blood relationship analysis result, and performing blood relationship completion on the candidate blood relationship pool to obtain a blood relationship map. And by calling the large model, blood relationship complementation is performed to refine the blood relationship granularity, so that the fine granularity and comprehensive blood relationship analysis of the full-amount metadata is realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to metadata lineage resolution methods, apparatus, storage media, and computer program products. Background Technology

[0002] Currently, metadata lineage analysis primarily relies on traditional parsing tools, whose core logic is based on preset rules and syntax parsers to process structured metadata. However, with the expansion of data scale and complexity, a large amount of unstructured or semi-structured metadata urgently needs lineage analysis. Traditional parsing tools can only cover structured metadata with fixed formats and clear logic, lacking effective processing capabilities for unstructured or complex logical scenarios. Secondly, traditional parsing tools only cover table-level and field-level lineages, resulting in coarse-grained lineage analysis and incomplete or inaccurate lineage relationship resolution.

[0003] Therefore, how to perform fine-grained and comprehensive lineage analysis on all metadata has become a technical problem that this application urgently needs to solve.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a metadata lineage resolution method, apparatus, storage medium, and computer program product, aiming to solve the technical problem of how to perform fine-grained and comprehensive lineage resolution on full metadata.

[0006] To achieve the above objectives, this application proposes a metadata lineage resolution method, the method comprising: Collect full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata; The structured metadata is parsed using a structured parsing tool to obtain the structured lineage parsing result. The pre-built large model is invoked to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; Candidate kinship pools are generated based on the structured kinship analysis results and the unstructured kinship analysis results, and kinship completion is performed on the candidate kinship pools to obtain a kinship relationship map.

[0007] In one embodiment, the step of collecting full metadata includes: Raw full metadata is collected through a pre-built unified data collection interface; The original full metadata is preprocessed, and the integrity of the original full metadata after the data preprocessing is verified to obtain the full metadata.

[0008] In one embodiment, the step of calling a structured parsing tool to perform structured lineage parsing on the structured metadata to obtain the structured lineage parsing result includes: The structured parsing tool is invoked, which includes a script parser, a task parsing tool, and a data model; The script parser performs syntax tree analysis on the structured metadata to extract direct dependencies, and obtains the structured lineage resolution result based on the direct dependencies; or... The task parsing tool reads the task configuration file from the structured metadata, and the structured lineage parsing result is obtained based on the reading result of the task configuration file; or, Based on the data model, the structured metadata is identified, and the structured lineage resolution result is obtained.

[0009] In one embodiment, the step of calling a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result includes: The pre-built large model is invoked to perform natural language semantic understanding on the semi-structured metadata and the unstructured metadata respectively, and the hidden lineage relationship is extracted; The unstructured rules in the semi-structured metadata and the unstructured metadata are decomposed using complex logic to obtain the logical chain reasoning results; Analyze cross-document associations in the semi-structured metadata and the unstructured metadata; The unstructured lineage parsing result is obtained based on the hidden lineage relationship, the logical chain reasoning result, and the cross-document association.

[0010] In one embodiment, the step of generating a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results includes: By fusing the structured kinship analysis results and the unstructured kinship analysis results, an initial candidate kinship pool is obtained; The initial candidate lineage pool is subjected to consistency verification, and the missing links in the initial candidate lineage pool are supplemented to obtain the candidate lineage pool.

[0011] In one embodiment, the step of completing the candidate kinship pool to obtain a kinship map includes: The large model is invoked to perform record-level and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set; Analyze the cross-module dynamic call situation in the first bloodline relationship set, and perform cross-layer dependency completion for the cross-module dynamic call situation to obtain the second bloodline relationship set; Store the second set of blood relations in a graph database in the form of nodes and edges; The metadata management tool is invoked to read database data from the graph database, and a kinship graph is generated based on the database data.

[0012] In one embodiment, the step of calling the large model to perform record-level completion and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set includes: The candidate lineage pool is associated with pre-acquired auxiliary data, which includes log files and business domain partitioning documents; The large model is invoked to identify the record flow description in the log file, and record-level elements are extracted from the record flow description. A record-level lineage link is generated based on the record-level elements, and the candidate lineage pool is completed at the record level based on the record-level lineage link; The large model is invoked to parse the business domain partitioning document, and the table-level lineages in the candidate lineage pool are mapped to domains based on the parsing results of the business domain partitioning document to generate cross-domain lineage links. Based on the cross-domain lineage link, the candidate lineage pool after record-level completion is completed at the service domain level to obtain the first lineage relationship set.

[0013] Furthermore, to achieve the above objectives, this application also proposes a metadata lineage resolution device, which includes: The acquisition module is used to acquire full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata. The structured lineage parsing module is used to call the structured parsing tool to perform structured lineage parsing on the structured metadata and obtain the structured lineage parsing result; The unstructured lineage parsing module is used to call a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; The kinship completion module is used to generate a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results, and to complete the kinship pool to obtain a kinship relationship map.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the metadata lineage resolution method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the metadata lineage resolution method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: Collect all metadata, including structured metadata, semi-structured metadata, and unstructured metadata; use a structured parsing tool to perform structured lineage parsing on the structured metadata to obtain structured lineage parsing results; use a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain unstructured lineage parsing results; generate a candidate lineage pool based on the structured lineage parsing results and the unstructured lineage parsing results, and perform lineage completion on the candidate lineage pool to obtain a lineage relationship graph. First, by collecting all structured and semi / unstructured metadata, a comprehensive data source foundation is provided for lineage resolution. Furthermore, structured metadata with fixed formats is processed by a structured parsing tool, while a large model handles semi / unstructured metadata that relies on natural language understanding. This classification and distribution mechanism collaboratively covers the parsing needs of different types of metadata, avoiding the limitation of traditional tools that can only process structured metadata. Second, lineage completion is performed on the candidate lineage pool, refining the granularity of lineage, thereby achieving fine-grained and comprehensive lineage resolution of all metadata. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the metadata lineage analysis method of this application. Figure 2 This is a flowchart illustrating the third embodiment of the metadata lineage analysis method of this application. Figure 3 This is a flowchart illustrating the fourth embodiment of the metadata lineage analysis method of this application. Figure 4 A simplified flowchart illustrating the metadata lineage resolution method provided in this application; Figure 5 This is a schematic diagram of the module structure of the metadata lineage analysis device in an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the metadata lineage resolution method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: collect full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata; call a structured parsing tool to perform structured lineage parsing on the structured metadata to obtain structured lineage parsing results; call a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain unstructured lineage parsing results; generate a candidate lineage pool based on the structured lineage parsing results and the unstructured lineage parsing results, and perform lineage completion on the candidate lineage pool to obtain a lineage relationship graph.

[0024] In this embodiment, for ease of description, the bloodline analysis system will be used as the execution subject in the following description.

[0025] This application's embodiments take into account that: Currently, metadata lineage analysis primarily relies on traditional parsing tools, whose core logic is based on preset rules and syntax parsers to process structured metadata. However, with the expansion of data scale and increasing complexity, a large amount of unstructured or semi-structured metadata urgently needs lineage analysis. Traditional parsing tools can only cover structured metadata with fixed formats and clear logic, lacking effective processing capabilities for unstructured or complex logical scenarios. Furthermore, traditional parsing tools only cover table-level and field-level lineages, resulting in coarse-grained lineage analysis, leading to incomplete and inaccurate lineage relationship resolution.

[0026] Therefore, this application provides a solution that collects full metadata, including structured metadata, semi-structured metadata, and unstructured metadata; calls a structured parsing tool to perform structured lineage parsing on the structured metadata to obtain structured lineage parsing results; calls a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain unstructured lineage parsing results; generates a candidate lineage pool based on the structured lineage parsing results and the unstructured lineage parsing results, and completes the lineage in the candidate lineage pool to obtain a lineage relationship graph. First, by collecting all structured and semi / unstructured metadata, a comprehensive data source foundation is provided for lineage resolution. Furthermore, structured metadata with fixed formats is processed by a structured parsing tool, while a large model handles semi / unstructured metadata that relies on natural language understanding. This classification and distribution mechanism collaboratively covers the parsing needs of different types of metadata, avoiding the limitation of traditional tools that can only process structured metadata. Second, lineage completion is performed on the candidate lineage pool, refining the granularity of lineage, thereby achieving fine-grained and comprehensive lineage resolution of all metadata.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or bloodline analysis system capable of performing the above functions. The following description uses a bloodline analysis system as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a method for metadata lineage resolution, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the metadata lineage analysis method of this application.

[0029] In this embodiment, the metadata lineage resolution method includes steps S10 to S40: Step S10: Collect full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata; It should be noted that, in this embodiment of the application, full metadata refers to a set of metadata covering the entire lifecycle of all data assets of an enterprise. Full metadata includes structured metadata, semi-structured metadata, and unstructured metadata.

[0030] Structured metadata refers to metadata with a fixed format that follows strict syntax rules, such as database table structures, SQL scripts, and ETL task configurations; semi-structured metadata refers to metadata with some structure but requiring natural language assistance for understanding, such as data transformation rule documents and API comments; unstructured metadata refers to metadata without a fixed format that relies on semantic understanding, such as log files and transformation rules described in free text.

[0031] The purpose of collecting full metadata is to provide a complete and comprehensive data source foundation for subsequent lineage analysis, avoiding problems such as broken or incomplete lineage links due to incomplete metadata collection. By covering all types of metadata, it ensures that subsequent analysis can capture every detail of data flow.

[0032] In one possible implementation, the collection process adopts a hybrid incremental and full collection strategy: for structured metadata, real-time incremental collection is achieved through CDC change data capture, capturing only the changed metadata; for semi / unstructured metadata, a combination of periodic full scans and incremental monitoring is used to ensure full coverage while reducing duplicate collection; at the same time, a new metadata version management module is added to record the historical change trajectory of metadata, making it easy to trace the evolution of lineage.

[0033] Step S20: Call the structured parsing tool to perform structured lineage parsing on the structured metadata to obtain the structured lineage parsing result; It should be noted that, in the embodiments of this application, the structured parsing tool refers to a software component specifically designed to process structured metadata, which has capabilities such as syntax analysis, rule matching, and dependency extraction.

[0034] Structured lineage parsing results refer to the lineage relationship information extracted from structured metadata, such as table-field level dependencies, upstream and downstream relationships between tasks and tables, primary key-foreign key associations, etc.

[0035] The purpose of calling structured parsing tools is to leverage the efficiency and accuracy of traditional parsing tools to quickly extract the basic lineage links from structured metadata, providing reliable basic data for subsequent collaborative parsing with large models, while reducing the computational load of large models.

[0036] In one possible implementation, the structured parsing tool employs a combined cross-validation strategy: for an SQL script, both the ANTLR-based SQL parser and the Calcite SQL parser are invoked simultaneously for parsing; if the results are consistent, they are directly taken as valid results; if they are inconsistent, they are marked as suspicious links, and the original SQL statement is passed to the subsequent large model for secondary validation.

[0037] In addition, the tool supports custom rule extensions, allowing operations and maintenance personnel to add parsing rules based on non-standard SQL syntax within the enterprise (such as custom stored procedure keywords and temporary table naming rules), thereby improving the ability to parse enterprise-specific structured metadata.

[0038] Additionally, it should be noted that the output of the structured parsing tool includes a parsing confidence field. Links with explicit parsing are marked as high confidence, while links with fuzzy parsing are marked as low confidence. This facilitates the subsequent fusion steps by prioritizing high-confidence results.

[0039] Step S30: Call the pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; It should be noted that, in the embodiments of this application, the pre-built large model refers to a large language model that has been fine-tuned after being made up of the metadata corpus in the domain, and has the ability to understand the semantics of multiple types of metadata, make logical inferences, and perform cross-language alignment.

[0040] The purpose of calling a pre-built large model is to leverage the semantic understanding advantages of the large model to solve the problem of parsing semi / unstructured metadata that traditional tools cannot handle, supplement the lineage links not covered by structured parsing, and enhance the integrity of lineage relationships.

[0041] In one possible implementation, the large model employs multimodal input extensions: in addition to text input, it supports image input (such as parsing images containing data flow diagrams) and audio input (such as parsing metadata description audio recorded by operations and maintenance personnel); it also introduces a lineage relationship confidence scoring mechanism, giving a confidence score to each parsed lineage relationship, which facilitates the subsequent fusion steps to select high-confidence results; furthermore, the large model supports cross-language semantic alignment, can identify dependencies between packages in different programming languages, and achieve a unified lineage perspective.

[0042] Additionally, it should be noted that the parsing process of the large model incorporates context awareness, which can decompose unstructured rules containing conditional branches into multi-branch lineage relationships and mark the triggering conditions of each branch.

[0043] Step S40: Generate a candidate bloodline pool based on the structured bloodline analysis results and the unstructured bloodline analysis results, and complete the bloodline pool to obtain a bloodline relationship map.

[0044] It should be noted that, in this embodiment, the candidate lineage pool refers to the set of structured and unstructured lineage resolution results. Lineage completion refers to supplementing and improving missing, incomplete, or cross-level lineage links in the candidate lineage pool. The lineage graph refers to a lineage relationship model presented in graph structure form, including nodes (tables, fields, tasks, business domains, etc.) and edges (dependencies, transformation rules, etc.).

[0045] The purpose of this step is to merge the two types of analysis results, eliminate conflicts and fill in missing links to form a complete and fine-grained lineage map, thus solving the problems of incomplete lineage links and coarse granularity in traditional methods.

[0046] In one possible implementation, lineage completion is aided by a business domain knowledge graph: the system pre-constructs an enterprise business domain knowledge graph (e.g., the user domain contains tables A and C, and the order domain contains tables B and D), and the large model refers to this graph to associate table-level lineages with business domain-level lineages (e.g., user domain → order domain); the fusion step employs conflict detection and resolution rules: when structured and unstructured results conflict, the result with higher confidence is prioritized; if the confidence levels are similar, manual verification is performed in conjunction with business rules; in addition, the graph supports dynamic updates, automatically triggering the parsing process and updating the graph when metadata changes, maintaining the real-time nature of lineage relationships.

[0047] This embodiment provides a metadata lineage parsing method. By collecting all structured metadata and semi / unstructured metadata, it provides a comprehensive data source foundation for lineage parsing. Furthermore, it uses a structured parsing tool to process structured metadata with fixed formats, while a large model processes semi / unstructured metadata that relies on natural language understanding. This classification and diversion mechanism collaboratively covers the parsing needs of different types of metadata, avoiding the limitation of traditional tools that can only process structured metadata. Secondly, it performs lineage completion on the candidate lineage pool, refining the granularity of lineage, thereby achieving fine-grained and comprehensive lineage parsing of all metadata.

[0048] In one feasible implementation, step S10 may include steps S11-S12: Step S11: Collect raw full metadata through a pre-built unified data collection interface; It should be noted that, in this embodiment of the application, the pre-built unified data acquisition interface refers to a standardized interface that integrates the connection capabilities of multiple data sources, is compatible with different access protocols, and can collect various metadata in one stop.

[0049] Raw full metadata refers to the unprocessed collection of metadata covering the entire lifecycle of data assets, including the original descriptions of data generation, flow, storage, and usage.

[0050] The purpose of collecting raw full metadata through a pre-built unified data collection interface is to break down data source barriers, ensure that metadata is comprehensive and without omissions, provide a complete foundation for subsequent lineage analysis, avoid missing / duplication issues caused by scattered collection, and improve collection efficiency.

[0051] In one possible implementation, a dynamic adaptation module can be added to the interface to automatically identify the data source type and select the optimal collection protocol; a collection priority mechanism is also introduced: metadata of core business systems is given high priority, and non-core systems are given low priority to balance efficiency and load.

[0052] Additionally, it should be noted that the interface supports resume from the breakpoint: when data collection is interrupted, the collected position is recorded, and the data collection can continue from the breakpoint after resumption, avoiding duplication or loss.

[0053] Step S12: Perform data preprocessing on the original full metadata and verify the integrity of the original full metadata after data preprocessing to obtain full metadata.

[0054] It should be noted that, in the embodiments of this application, data preprocessing refers to the process of cleaning, transforming and standardizing the original metadata to eliminate noise and inconsistencies; integrity verification refers to checking whether the preprocessed metadata covers the required type / content and whether there are any missing parts; full metadata refers to a high-quality metadata set that meets the parsing requirements after preprocessing and verification.

[0055] The purpose of this step is to improve metadata quality, ensure accurate subsequent parsing, reduce parsing errors, and improve efficiency.

[0056] In one possible implementation, the preprocessing adds an intelligent repair module: automatically repairing format errors; the integrity verification introduces a multi-dimensional rule engine: including rules for missing fields, association rules, and overriding rules, and generates an integrity score, which will trigger an alarm if it is below the threshold.

[0057] Additionally, it should be noted that the verification supports custom rule extensions, allowing users to add personalized rules.

[0058] Based on the first embodiment of this application, a second embodiment of this application is proposed. In the second embodiment of this application, content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter.

[0059] In this embodiment, step S20, which involves calling a structured parsing tool to perform structured lineage parsing on the structured metadata and obtaining the structured lineage parsing result, may include steps S21 to S24: Step S21: Invoke the structured parsing tool, which includes a script parser, a task parsing tool, and a data model; It should be noted that, in the embodiments of this application, the script parser refers to a tool that can perform abstract syntax tree analysis on structured scripts such as SQL, such as an ANTLR-based SQL parser; the task parsing tool refers to a tool used to read ETL task configuration files and extract upstream and downstream relationships, such as the built-in ETL parser of Apache Atlas; and the data model refers to a model that describes the data structure and the relationships between tables, such as a star schema or a snowflake schema.

[0060] The purpose of calling structured metadata parsing tools is to clarify the toolset used for structured metadata parsing, providing support for subsequent accurate and efficient parsing; by integrating multiple tools to cover different types of structured metadata parsing needs, the comprehensiveness of the parsing is ensured.

[0061] Specifically, the structured metadata parsing tools are subdivided into script parsers, task parsing tools, and data models, corresponding to the parsing of three types of structured metadata: SQL scripts, ETL task configurations, and data model relationships, respectively. In one possible implementation, the structured metadata parsing tools include a new intelligent tool scheduling module, which can automatically select the optimal parsing tool based on the type of input structured metadata. For example, it automatically calls the script parser for SQL scripts and the task parsing tool for ETL configuration files.

[0062] Additionally, it should be noted that the structured metadata parsing tool supports plug-in extensions. Operations and maintenance personnel can develop corresponding parsing plugins based on the enterprise's custom structured metadata format to further expand the tool's compatibility.

[0063] Step S22: Perform syntax tree analysis on the structured metadata based on the script parser to extract direct dependencies, and obtain the structured lineage parsing result based on the direct dependencies; Syntax tree analysis refers to the process of converting scripts (such as SQL scripts) in structured metadata into an abstract syntax tree (AST), and extracting dependency information by analyzing the relationships between nodes in the syntax tree.

[0064] Direct dependencies refer to dependencies that are explicitly present in the metadata and have no indirect jumps, such as the direct association between the id field in table A and the aid field in table B.

[0065] The purpose of performing syntax tree analysis on the structured metadata based on the script parser to extract direct dependencies is to extract precise, fine-grained dependencies from the script, obtain field-level lineage links, and improve the granularity of lineage resolution.

[0066] Specifically, the logical structure of the script is broken down through syntax tree analysis to identify the relationship between the data source and the target data. In one possible implementation, the script parser adds a temporary table dependency identification function, which can parse SQL scripts containing WITH...AS temporary table logic and extract the dependency relationships between the temporary table and the source table, and between the temporary table and the target table; it also introduces a dependency conflict detection mechanism, which automatically marks conflicts and generates prompt messages for operation and maintenance personnel to verify when the same target field has multiple source dependencies.

[0067] Step S23: Read the task configuration file in the structured metadata based on the task parsing tool, and obtain the structured lineage parsing result based on the reading result of the task configuration file; It should be noted that, in this embodiment of the application, the task configuration file refers to the configuration document of the ETL task, which includes information such as input table, output table, transformation operator, and scheduling rules; the reading result refers to the upstream and downstream relationships and transformation logic of the task extracted from the task configuration file.

[0068] The purpose of this step is to obtain the lineage at the ETL task level; supplement the task dependencies at the table level; and improve the completeness of the lineage graph.

[0069] Specifically, the input and output information in the task configuration file is read directly to generate the corresponding lineage relationship. In one possible implementation, a function to associate scheduling rules with lineage links is introduced, combining the task scheduling time information with the lineage link to form a time-sequential lineage relationship.

[0070] Step S24: Identify the structured metadata based on the data model to obtain the structured lineage resolution result.

[0071] It should be noted that, in this embodiment of the application, identification refers to matching the table relationships in the structured metadata according to the rules of the data model; the structured metadata refers to the database table structure, the primary key and foreign key relationships between tables, etc.

[0072] The purpose of this step is to extract table-level relationships at the data model level; supplement the relationships at the data model level; and improve the hierarchy of the kinship map.

[0073] Specifically, the dependencies between tables are identified using primary and foreign key relationships and star / snowflake schema structures in the data model. In one possible implementation, the data model adds a cross-model association identification function, which can identify table dependencies between star schemas and snowflake schemas (e.g., the association between the fact table in the star schema and the dimension table in the snowflake schema); it also introduces a model version comparison function to analyze the lineage changes of different versions of the data model (e.g., the order table and user table are directly associated in version 1, while a new order details table is added as an intermediate table in version 2).

[0074] In this embodiment, a structured parsing toolset, including a script parser, task parsing tools, and a data model, is invoked to comprehensively cover the parsing requirements of structured metadata (such as SQL scripts, ETL task configurations, and data model relationships). Specifically, the accuracy of parsing is improved from the fields, tasks, and table structures levels by extracting direct dependencies at the field level based on syntax tree analysis, obtaining upstream and downstream links by reading task configurations, and identifying inter-table relationships through the data model. These structured lineage parsing results supplement the link information in different dimensions, effectively improving the completeness of lineage relationships and laying a solid foundation for subsequent collaboration with large models to form a complete lineage graph.

[0075] Based on the first and / or second embodiments of this application, a third embodiment of this application is proposed. In this third embodiment, content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0076] Based on this, please refer to Figure 2 , Figure 2 This is a schematic diagram of the process of the third embodiment of this application, as shown below. Figure 2 As shown, step S30, which involves calling a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result, may include steps S31 to S34: Step S31: Call the pre-built large model to perform natural language semantic understanding on the semi-structured metadata and the unstructured metadata respectively, and extract the hidden lineage relationship; It should be noted that, in the embodiments of this application, natural language semantic understanding refers to analyzing the natural language content in metadata and extracting semantic information; hidden lineage refers to metadata dependency relationships that are not directly reflected.

[0077] The purpose of this step is to leverage the semantic understanding capabilities of large models to extract lineage relationships from unstructured metadata that traditional tools cannot identify, thereby supplementing the links not covered by structured parsing.

[0078] Specifically, large models are used to analyze natural language content and extract hidden dependencies. In one possible implementation, the large model introduces a contextual association mechanism, combining the preceding and following descriptions in the document to more accurately extract lineage relationships; it can also introduce multimodal input, parsing metadata containing images, such as parsing images containing data flow diagrams to extract lineage relationships.

[0079] Step S32: Perform complex logical decomposition on the unstructured rules in the semi-structured metadata and the unstructured metadata to obtain the logical chain reasoning result; It should be noted that, in the embodiments of this application, unstructured rules refer to rules described in natural language, such as conditional branches and multi-step transformations; complex logic decomposition refers to decomposing unstructured rules into parsable logical chains; and logical chain reasoning results refer to the decomposed logical relationships.

[0080] The purpose of this step is to process the complex logic in unstructured rules, thereby transforming natural language rules into structured lineage relationships.

[0081] Specifically, the large model breaks down rules into conditional branches with a lineage. In one possible implementation, the large model introduces temporal logic decomposition to handle rules with time dependencies. Multi-branch logic decomposition can also be introduced to handle rules with multiple conditions.

[0082] Step S33: Analyze the cross-document associations in the semi-structured metadata and the unstructured metadata; It should be noted that, in the embodiments of this application, cross-document association refers to the association of metadata between different documents.

[0083] The purpose of this step is to supplement cross-document lineage relationships; specifically, the large model performs collaborative analysis on multiple documents to extract cross-document dependencies. In one possible implementation, the large model introduces a document context association mechanism, combining the content of multiple documents for analysis. For example, it combines the conversion rule document of the user table and the annotations of the order table to extract the lineage relationship between them; it can also introduce multilingual document association, analyzing documents in different languages ​​to extract cross-language lineage relationships.

[0084] Additionally, it should be noted that the large model supports document version comparison, analyzing changes in lineage relationships between different document versions to form versioned lineage relationships.

[0085] Step S34: Based on the hidden lineage relationship, the logical chain reasoning result, and the cross-document association, obtain the unstructured lineage parsing result.

[0086] The method integrates hidden lineage relationships, semi-structured parsing results, and cross-document associations. One possible implementation involves introducing a weighting mechanism to weight the parsing results from different sources; for example, a document parsing result might have a weight of 0.7, while an annotation parsing result might have a weight of 0.3. A manual verification step could also be introduced to validate the integrated lineage relationships and ensure accuracy.

[0087] Additionally, it should be noted that the large model supports dynamic adjustment of the fusion strategy, adjusting the weights based on the confidence level of the analytical results.

[0088] In this embodiment, by using a large model to semantically understand unstructured and semi-structured metadata, decompose complex logic, perform cross-document association analysis, and fuse results, the coverage of lineage resolution is effectively expanded. Cross-language semantic alignment is achieved to handle heterogeneous dependencies of multi-language packages and unify the lineage perspective. At the same time, it supplements hidden lineage relationships, complex logical links, and cross-document associations that traditional tools cannot identify, significantly improving the integrity of lineage links, refining the granularity of data management, and providing more comprehensive and accurate unstructured lineage resolution results for the subsequent construction of lineage graphs.

[0089] Based on the above embodiments of this application, a fourth embodiment of this application is proposed. In this fourth embodiment, content that is the same as or similar to that in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0090] Based on this, please refer to Figure 3 , Figure 3 This is a schematic flowchart of the fourth embodiment of this application, as shown below. Figure 3 As shown, step S40, which involves completing the candidate kinship pool to obtain a kinship map, may include steps S41 to S44: Step S41: Call the large model to perform record-level completion and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set; It should be noted that, in the embodiments of this application, record-level completion refers to supplementing the lineage dependency at the level of a single data record (such as the record flow of a specific user ID); business domain-level completion refers to supplementing the cross-domain lineage association at the business domain level (such as the link from the user domain to the order domain); and the first lineage relationship set refers to the complete lineage relationship set after record-level and business domain-level completion.

[0091] The purpose of this step is to leverage the logical reasoning capabilities of large models to supplement the fine-grained (record-level) and high-level (business domain-level) lineage relationships that traditional tools cannot cover, thereby significantly improving the integrity and richness of the lineage chain.

[0092] Specifically, the large model is invoked to perform two types of completion operations respectively. In one possible implementation, the system introduces a dynamic priority completion mechanism to adjust the completion order based on the business importance of the metadata (such as prioritizing the completion of the lineage of core transaction domains).

[0093] Step S42: Analyze the cross-module dynamic call situation in the first bloodline relationship set, and perform cross-layer dependency completion for the cross-module dynamic call situation to obtain the second bloodline relationship set; It should be noted that, in the embodiments of this application, cross-module dynamic calling refers to the dynamic dependencies generated by different functional modules at runtime (such as a Python module calling a function of a Java module); cross-layer dependency completion refers to supplementing the lineage links generated by cross-module dynamic calling; and the second lineage set refers to the lineage set after cross-layer dependency completion.

[0094] The purpose of this step is to complete cross-module dynamic dependency lineages that are difficult for traditional tools to identify, thereby covering the dynamic runtime dependency chain and improving the comprehensiveness of the dependency graph. Specifically, it analyzes dynamic call scenarios and calls the large model to complete cross-layer dependencies. In one possible implementation, the system introduces a dynamic call context tracing function, combining module logs and configuration files to identify triggering conditions (such as calls triggered by specific parameters); cross-language dynamic call completion can also be added to handle dependencies between modules in multiple programming languages.

[0095] Step S43: Store the second set of blood relations into a graph database in the form of nodes and edges; It should be noted that in the embodiments of this application, nodes and edges refer to the basic units for storing lineage in the graph database. For example, nodes represent assets such as tables / fields, and edges represent dependencies; a graph database refers to a database that stores graph structure data.

[0096] The purpose of this step is to store bloodlines in a graphical structure for efficient querying and visualization, thereby achieving structured storage and rapid retrieval of bloodlines.

[0097] Specifically, the second lineage relationship is converted into nodes and edges and stored in a graph database. The system adds attribute extensions to nodes (such as lineage confidence and generation time); adds type markers to edges (such as static / dynamic dependency, field / table level), and incorporates versioned storage to record lineage changes at different points in time. Additionally, it should be noted that the stored procedure removes duplicate nodes and edges to ensure data uniqueness.

[0098] Step S44: Call the metadata management tool to read the database data in the graph database, and generate a bloodline graph based on the database data.

[0099] It should be noted that, in the embodiments of this application, the metadata management tool refers to a tool used to manage and visualize metadata; the database data in the graph database refers to the node and edge data stored in the graph database; and the lineage graph refers to a graphical interface that intuitively displays the lineage links.

[0100] The purpose of this process is to visualize lineage relationships, improving operational and troubleshooting efficiency; and to provide an interactive lineage display, facilitating quick tracing of links for users. Specifically, the tool reads data from a graph database to generate a graph. The tool supports interactive operation; it can also incorporate multi-dimensional filtering and automatically refreshes the graph when the graph database data changes through a real-time update mechanism.

[0101] In this embodiment, multi-level lineage completion at the record level and business domain level is achieved through a large model. Combined with cross-module dynamic invocation to complete cross-layer dependency links, the complete lineage relationship is stored in the graph database in the form of nodes and edges. A visual lineage graph is generated through metadata management tools. This not only expands the lineage resolution coverage and refines the granularity of data management, but also realizes cross-domain association and a unified lineage perspective, significantly improving the efficiency of data problem investigation for operation and maintenance personnel and reducing the time cost of manual traceability verification.

[0102] In one feasible implementation, step S40, which generates a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results, may further include steps A41-A42: Step A41: Combine the structured kinship analysis results and the unstructured kinship analysis results to obtain an initial candidate kinship pool; It should be noted that, in this embodiment, the initial candidate pedigree pool refers to the preliminary pedigree set after fusing the two results. The purpose of this step is to integrate pedigree analysis results from different sources to form a unified initial pool, merging structured and unstructured pedigrees to provide a foundation for subsequent processing.

[0103] In one possible implementation, the system introduces a confidence-weighted fusion mechanism, assigning weights based on source reliability. Pre-fusion conflict detection is also incorporated to pre-identify conflicting lineages. Furthermore, it should be noted that source information is preserved during the fusion process for easy traceability.

[0104] Step A42: Perform consistency verification on the initial candidate lineage pool and supplement the missing links in the initial candidate lineage pool to obtain the candidate lineage pool.

[0105] It should be noted that, in the embodiments of this application, consistency verification refers to checking whether there is a contradiction in the lineage relationship (such as a conflict in the dependency of the same field); missing links refer to lineage dependencies that are not covered in the initial pool; and the candidate lineage pool refers to the final lineage set after verification and completion.

[0106] The purpose of this step is to ensure the accuracy and integrity of the initial pool, so as to obtain a reliable and complete candidate lineage pool.

[0107] In one feasible implementation, step S41, which involves calling the large model to perform record-level and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set, may include steps S411-S415: Step S411: Associate the candidate lineage pool with pre-acquired auxiliary data, the auxiliary data including log files and business domain partitioning documents; It should be noted that, in the embodiments of this application, auxiliary data refers to additional data resources used to supplement or verify lineage links, specifically including log files and business domain partitioning documents; log files refer to text files that record the data flow process (such as migration or conversion of a single record); business domain partitioning documents refer to documents that define the boundaries of business domains and the data tables contained in each business domain.

[0108] The purpose of linking the candidate lineage pool with the pre-acquired auxiliary data is to provide data support for subsequent record-level and business domain-level completion. By linking the candidate lineage pool with the auxiliary data, the completion operation becomes more targeted and accurate, thereby improving the reliability of the completion results.

[0109] Specifically, the association between the candidate lineage pool and the auxiliary data can be achieved by establishing an association mapping between the candidate lineage pool and the auxiliary data.

[0110] Step S412: Call the large model to identify the record flow description in the log file, and extract record-level elements from the record flow description; It should be noted that, in the embodiments of this application, the record flow description refers to the natural language or semi-structured description of the movement or transformation of a single data record in the log file; the record-level elements refer to the core identifier (such as user ID, order number), source table, target table, flow time and other key information of a single data record.

[0111] The purpose of this step is to extract fine-grained record-level lineage information from the logs, making up for the shortcomings of traditional tools that cannot parse record-level links; and to obtain the flow trajectory of a single record, providing a basis for the subsequent generation of record-level lineage links.

[0112] Step S413: Generate a record-level lineage link based on the record-level elements, and perform record-level completion on the candidate lineage pool based on the record-level lineage link; It should be noted that, in the embodiments of this application, record-level lineage refers to the dependency relationship between a single data record and different tables.

[0113] The purpose of this step is to refine the granularity of pedigree analysis to the level of individual records, thereby meeting the needs of accurate data tracing and enabling the pedigree map to cover the entire dimensional link from the table level to the record level.

[0114] Links are generated based on the extracted record-level features and added to the candidate lineage pool. In one possible implementation, the system introduces a record-level conflict detection mechanism. When there are contradictions in the flow descriptions of the same record, the large model will determine the correct link by combining the flow time and operation type (such as insertion / update).

[0115] Step S414: Call the large model to parse the business domain partitioning document, and perform domain mapping on the table-level lineages in the candidate lineage pool according to the parsing result of the business domain partitioning document to generate cross-domain lineage links; It should be noted that, in the embodiments of this application, table-level lineage refers to the dependency relationship between data tables; domain mapping refers to associating a data table with the corresponding business domain; and cross-domain lineage refers to the data dependency relationship between different business domains.

[0116] The purpose of this step is to supplement the cross-domain lineage at the business domain level, and to achieve a high-level association from the table level to the business domain level; to make the lineage graph cover the business domain dimension, so as to facilitate the tracing of cross-domain data links.

[0117] In one possible implementation, the system introduces a dynamic domain mapping mechanism. When the business domain partitioning document is updated, the large model will automatically re-parse and update the domain mapping relationship without manual intervention.

[0118] Step S415: Based on the cross-domain lineage link, perform business domain-level completion on the candidate lineage pool after record-level completion to obtain the first lineage relationship set.

[0119] By integrating record-level and business domain-level links, a multi-dimensional and complete lineage graph is formed, covering the record, table, and business domain levels to meet the traceability needs of different scenarios. Based on the cross-domain links, the candidate pool after record-level completion is further completed at the business domain level to obtain the final set.

[0120] In one possible implementation, the system introduces a hierarchical association verification mechanism to check whether the record-level link and the business domain-level link are consistent, ensuring the consistency of lineage.

[0121] For example, to help understand the implementation flow of the metadata lineage resolution method obtained by combining this embodiment with the above embodiments, please refer to... Figure 4 , Figure 4 A simplified flowchart of a metadata lineage resolution method is provided, specifically: Full metadata collection: Collect all metadata related to data assets through a unified data collection interface, including: structured metadata: database table structure, ETL task configuration, data model, etc.; semi-structured / unstructured metadata: data transformation rule documents, API comments, log files.

[0122] Classification and diversion mechanism: Traditional parsing tools are based on preset rules and syntax parsers to efficiently process structured metadata: Metadata with flexible formats and that depends on natural language understanding (such as documents, comments, and logs) is processed by large models.

[0123] Lineage fusion: Integrating candidate lineages generated by traditional tools with those generated by large models: eliminating conflicts (such as contradictory dependencies), supplementing missing links (such as field-level associations not covered by traditional tools), and forming a complete candidate lineage pool. For example, associating table-level lineages from traditional tools with field-level lineages from large models to construct a hierarchical link of "table-level → field-level".

[0124] Lineage completion: Utilize large models to supplement fine-grained or high-level lineages that traditional tools cannot cover: Combine the flow description of individual records in log files to complete record-level lineage links; divide documents according to business domains and map table-level lineages to cross-domain lineages.

[0125] Lineage visualization: The completed lineage relationships are stored in a graph database in the form of "nodes-edges," where nodes represent data assets and edges represent dependencies. A visual metadata management tool reads the data from the graph database to generate an interactive lineage graph, supporting features such as hierarchical expansion and cross-domain filtering, facilitating rapid data tracing by operations and maintenance personnel.

[0126] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the metadata lineage resolution method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0127] This application also provides a metadata lineage resolution device; please refer to [reference needed]. Figure 5 The metadata lineage resolution device includes: The acquisition module 10 is used to acquire full metadata, which includes structured metadata, semi-structured metadata and unstructured metadata; The structured lineage parsing module 20 is used to call a structured parsing tool to perform structured lineage parsing on the structured metadata and obtain the structured lineage parsing result; The unstructured lineage parsing module 30 is used to call a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; The kinship completion module 40 is used to generate a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results, and to complete the kinship pool to obtain a kinship relationship map.

[0128] The metadata lineage resolution apparatus provided in this application, employing the metadata lineage resolution method in the above embodiments, can solve the technical problem of metadata lineage resolution. Compared with the prior art, the beneficial effects of the metadata lineage resolution apparatus provided in this application are the same as those of the metadata lineage resolution method provided in the above embodiments, and other technical features in the metadata lineage resolution apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0129] This application provides a metadata lineage resolution device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the metadata lineage resolution method in Embodiment 1 above.

[0130] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a metadata lineage resolution device suitable for implementing embodiments of this application. The metadata lineage resolution device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The metadata lineage resolution device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 6As shown, the metadata lineage resolution device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the metadata lineage resolution device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the metadata lineage resolution device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows metadata lineage resolution devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0132] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0133] The metadata lineage resolution device provided in this application, employing the metadata lineage resolution method described in the above embodiments, can solve the technical problem of metadata lineage resolution. Compared with the prior art, the beneficial effects of the metadata lineage resolution device provided in this application are the same as those of the metadata lineage resolution method provided in the above embodiments, and other technical features in this metadata lineage resolution device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0136] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the metadata lineage resolution method in the above embodiments.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in the metadata lineage resolution device; or it may exist independently and not be assembled into the metadata lineage resolution device.

[0139] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the metadata lineage resolution device, the metadata lineage resolution device performs the following actions: collects full metadata, including structured metadata, semi-structured metadata, and unstructured metadata; calls a structured parsing tool to perform structured lineage resolution on the structured metadata, obtaining a structured lineage resolution result; calls a pre-built large model to perform unstructured lineage resolution on the semi-structured metadata and the unstructured metadata, obtaining an unstructured lineage resolution result; generates a candidate lineage pool based on the structured lineage resolution result and the unstructured lineage resolution result, and completes the lineage in the candidate lineage pool to obtain a lineage relationship graph.

[0140] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0143] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described metadata lineage resolution method, thereby solving the technical problem of metadata lineage resolution. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the metadata lineage resolution method provided in the above embodiments, and will not be repeated here.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the metadata lineage resolution method described above.

[0145] The computer program product provided in this application can solve the technical problem of metadata lineage resolution. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the metadata lineage resolution method provided in the above embodiments, and will not be repeated here.

[0146] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for resolving metadata lineage, characterized in that, The metadata lineage resolution method includes: Collect full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata; The structured metadata is parsed using a structured parsing tool to obtain the structured lineage parsing result. The pre-built large model is invoked to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; Candidate kinship pools are generated based on the structured kinship analysis results and the unstructured kinship analysis results, and kinship completion is performed on the candidate kinship pools to obtain a kinship relationship map.

2. The metadata lineage resolution method as described in claim 1, characterized in that, The steps for collecting full metadata include: Raw full metadata is collected through a pre-built unified data collection interface; The original full metadata is preprocessed, and the integrity of the original full metadata after the data preprocessing is verified to obtain the full metadata.

3. The metadata lineage resolution method as described in claim 1, characterized in that, The step of calling a structured parsing tool to perform structured lineage parsing on the structured metadata and obtaining the structured lineage parsing result includes: The structured parsing tool is invoked, which includes a script parser, a task parsing tool, and a data model; The script parser performs syntax tree analysis on the structured metadata to extract direct dependencies, and obtains the structured lineage resolution result based on the direct dependencies; or... The task parsing tool reads the task configuration file from the structured metadata, and the structured lineage parsing result is obtained based on the reading result of the task configuration file; or, Based on the data model, the structured metadata is identified, and the structured lineage resolution result is obtained.

4. The metadata lineage resolution method as described in claim 1, characterized in that, The step of calling a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result includes: The pre-built large model is invoked to perform natural language semantic understanding on the semi-structured metadata and the unstructured metadata respectively, and the hidden lineage relationship is extracted; The unstructured rules in the semi-structured metadata and the unstructured metadata are decomposed using complex logic to obtain the logical chain reasoning results; Analyze cross-document associations in the semi-structured metadata and the unstructured metadata; The unstructured lineage parsing result is obtained based on the hidden lineage relationship, the logical chain reasoning result, and the cross-document association.

5. The metadata lineage resolution method as described in claim 1, characterized in that, The step of generating a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results includes: By fusing the structured kinship analysis results and the unstructured kinship analysis results, an initial candidate kinship pool is obtained; The initial candidate lineage pool is subjected to consistency verification, and the missing links in the initial candidate lineage pool are supplemented to obtain the candidate lineage pool.

6. The metadata lineage resolution method as described in claim 1, characterized in that, The step of completing the candidate kinship pool to obtain a kinship map includes: The large model is invoked to perform record-level and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set; Analyze the cross-module dynamic call situation in the first bloodline relationship set, and perform cross-layer dependency completion for the cross-module dynamic call situation to obtain the second bloodline relationship set; Store the second set of blood relations in a graph database in the form of nodes and edges; The metadata management tool is invoked to read database data from the graph database, and a kinship graph is generated based on the database data.

7. The metadata lineage resolution method as described in claim 6, characterized in that, The step of calling the large model to perform record-level and business domain-level completion on the candidate lineage pool to obtain the first lineage relationship set includes: The candidate lineage pool is associated with pre-acquired auxiliary data, which includes log files and business domain partitioning documents; The large model is invoked to identify the record flow description in the log file, and record-level elements are extracted from the record flow description. A record-level lineage link is generated based on the record-level elements, and the candidate lineage pool is completed at the record level based on the record-level lineage link; The large model is invoked to parse the business domain partitioning document, and the table-level lineages in the candidate lineage pool are mapped to domains based on the parsing results of the business domain partitioning document to generate cross-domain lineage links. Based on the cross-domain lineage link, the candidate lineage pool after record-level completion is completed at the service domain level to obtain the first lineage relationship set.

8. A metadata lineage resolution device, characterized in that, The metadata lineage resolution device includes: The acquisition module is used to acquire full metadata, which includes structured metadata, semi-structured metadata, and unstructured metadata. The structured lineage parsing module is used to call the structured parsing tool to perform structured lineage parsing on the structured metadata and obtain the structured lineage parsing result; The unstructured lineage parsing module is used to call a pre-built large model to perform unstructured lineage parsing on the semi-structured metadata and the unstructured metadata to obtain the unstructured lineage parsing result; The kinship completion module is used to generate a candidate kinship pool based on the structured kinship analysis results and the unstructured kinship analysis results, and to complete the kinship pool to obtain a kinship relationship map.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the metadata lineage resolution method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the metadata lineage resolution method as described in any one of claims 1 to 7.

Images