Data blood relationship generation method and device, electronic equipment and storage medium

Abstract

The invention provides a data blood relationship generation method and device, electronic equipment and a storage medium. Relates to the technical field of data traceability. The fields in the script data are positioned through the keyword template, vector conversion of meta-characters, capital and lowercase and dynamic parameters is realized in combination with a vector library vectorization conversion rule, and the accuracy and suitability of field conversion are improved. Field vectorization, rule matching conversion and mapping relation establishment are completed through the conversion model, the conversion data and the blood relationship are directly output, the generation process of the blood relationship is simplified, and automation and high efficiency of data blood relationship establishment are improved. The blood relationship is defined according to the mapping relationship between the script data and the conversion data, the association before and after data conversion is clearly traced, and the accuracy and traceability of the blood relationship are guaranteed. The conversion rules stored in the vector library can provide reusable rule support for subsequent similar script processing, the expansibility and reusability of the scheme are improved, and the quality and efficiency of cross-database script data blood relationship generation are integrally guaranteed.

Description

Technical Field

[0001] This application relates to the field of data tracing technology, and more specifically, to a method, apparatus, electronic device, and storage medium for generating data lineage. Background Technology

[0002] In the era of big data, digital transformation has become an inevitable path for enterprise development. As a key carrier of enterprise digitalization, the data platform, built upon big data technology, is a core support platform that centrally aggregates, integrates, stores, manages, processes, and shares internal and external data resources. Its core objective is to break down data silos and achieve efficient utilization of data assets.

[0003] With the development of data platforms, data lineage technology has become an important research direction. Data lineage records the entire lifecycle of data, including its origin, processing flow, and dynamic evolution over time. It provides crucial support for the data traceability needs of database systems and is also a core basis for assessing data credibility and quality.

[0004] In existing technologies, most methods rely on SQL (Structured Query Language) scripts to parse data lineage. However, SQL scripts lack sufficient support for data lineage parsing, making it difficult to guarantee parsing accuracy. This often results in parsing failures due to syntax differences, disconnects between conversion and lineage analysis, and low efficiency in manual processing. These issues reduce the accuracy of data lineage parsing and severely restrict effective data lineage tracking. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method, apparatus, electronic device, and storage medium for generating data lineage, which can improve the accuracy of data lineage resolution.

[0006] Firstly, this application provides a method for generating data lineage, comprising:

[0007] Obtain script data and keyword templates corresponding to the database storing the script data; Determine the conversion rules for metacharacters, character case, and dynamic parameters, and vectorize the conversion rules. Store the vectorized conversion rules in a vector library. The script data and the keyword template are input into a preset conversion model, and the conversion model outputs the conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0008] In an optional implementation, the vectorization process of the transformation rule includes: The transformation rules are then converted into structured text data. The structured text data is input into a pre-trained language model, which generates and outputs the original numerical vector. The original numerical vector is numerically processed to obtain the vectorized transformation rules.

[0009] In an optional implementation, storing the vectorized transformation rules in a vector library includes: Establish the encoding and metadata of the conversion rules; The vectorized transformation rules are bound to the encoding and the metadata and stored in the vector library; The meta-information includes at least one of the following: conversion rule type, database to which the conversion rule applies, and original text of the conversion rule; the conversion rule type includes: metacharacters, character case, and dynamic parameters.

[0010] In an optional implementation, determining the fields to be processed in the script data based on the keyword template and vectorizing the fields includes: The keyword template and the script data are parsed separately to obtain keyword information and a list of structured script data fields, respectively. In the list of script data fields, fields other than the keyword information are identified as fields to be processed, and vectorization processing is performed on the fields to be processed according to the vector encoding rules.

[0011] In one optional implementation, the transformation of the vectorized fields based on the transformation rules of the vector library to obtain transformed data includes: The vectorized fields are matched with the conversion rules in the vector library to determine the conversion rule with the highest matching degree with the vectorized fields. The vectorized fields are transformed using the transformation rule with the highest matching degree. The fields after vector transformation are restored to obtain the transformed data.

[0012] In an optional implementation, establishing the mapping relationship between the script data and the transformation data includes: Establish an indirect mapping between the script data, the fields to be processed, and the transformed data; and establish a direct mapping between the fields to be processed before and after the transformation process; The indirect mapping and the direct mapping are integrated to form a mapping relationship between the script data and the transformation data.

[0013] In an optional implementation, after the transformation model outputs the transformation data, the method further includes: Perform syntax validation on the converted data and identify the converted data that fails the syntax validation and its corresponding error information; The converted data that fails the syntax check and the error information are input into the conversion model for correction until the converted data passes the syntax check.

[0014] Secondly, this application provides a data lineage generation apparatus, comprising: A data acquisition unit is used to acquire script data and keyword templates corresponding to the database storing the script data; The vectorization processing unit is used to determine the conversion rules for metacharacters, character case and dynamic parameters, and to perform vectorization processing on the conversion rules, and store the vectorized conversion rules in a vector library; A conversion processing unit is used to input the script data and the keyword template into a preset conversion model, and the conversion model outputs conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0015] Thirdly, this application provides an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the data lineage generation method as described in the first aspect.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data lineage generation method as described in the first aspect.

[0017] The data lineage generation method, apparatus, electronic device, and storage medium provided in this application locate fields in script data using keyword templates and combine vectorized conversion rules from a vector library to achieve vector conversion of metacharacters, case sensitivity, and dynamic parameters, improving the accuracy and adaptability of field conversion. A conversion model completes field vectorization, rule matching conversion, and mapping relationship establishment, directly outputting the converted data and lineage relationships, simplifying the lineage generation process and improving the automation and efficiency of data lineage construction. The lineage is defined by the mapping relationship between script data and converted data, clearly tracing the association before and after data conversion, ensuring the accuracy and traceability of lineage relationships. The conversion rules stored in the vector library can provide reusable rule support for subsequent similar script processing, improving the scalability and reusability of the solution, and overall ensuring the quality and efficiency of cross-database script data lineage generation. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for generating data lineage according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of a data lineage generation device provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of this application more apparent and understandable, specific embodiments of this application are described in detail below with reference to the accompanying drawings. Although some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the accompanying drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0022] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first," "second," etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0024] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] This application provides a method, apparatus, electronic device, and storage medium for generating data lineage. For example... Figure 1 As shown in the embodiment of this application, a method for generating data lineage is provided, with steps S100-S300 as follows: S100: Obtain script data and keyword templates corresponding to the database storing the script data.

[0027] In this step, script data refers to the executable or parsable script text itself, which describes the data processing and data flow logic. Examples include various query, table creation, insert, and update statements in SQL scripts, and the business code in ETL scripts that describes the mapping and transformation logic of data from the source database to the target database and from the source table to the target table.

[0028] In this embodiment, newly added and changed script data in the business code can be monitored and captured.

[0029] Specifically, an execution cycle can be set using a timer (such as Quartz in Java or schedule in Python) to monitor whether there is updated script data in the business code within an execution cycle.

[0030] It's important to note that business logic is typically stored in a database table named `job_sql`. The `job_sql` table has a field called `isUpdate`. Under this `isUpdate` field, a value of 0 indicates no update (unchanged script data), a value of 1 indicates updated (changed script data), and a value of 2 indicates newly added script data. If statements are written in the programming interface that cause changes or additions to the script data in the business logic, the value under `isUpdate` will be updated to either value 1 or value 2.

[0031] This data scraping script captures newly added and changed script data from the business code. Each time, it only filters script data with `isUpdate` set to identifier 1 or 2, instead of scanning all script data. After capturing changed or newly added script data, the identifier under the `isUpdate` field of the captured script data is updated from identifier 1 or 2 to identifier 0. Modifying the identifier of the captured script data switches the script data from "needs to be scraped" to "scrapped". Script data with identifier 0 is excluded from the next filtering range. As the amount of script data increases, this significantly reduces the amount of script data scanning and processing, improving execution speed. It avoids the problem of repeatedly scraping the same batch of script data, thus preventing duplicate records in the database storing transformation data and lineage relationships, as well as business calculation errors caused by duplicate transformation data (such as duplicate data accumulation or duplicate configuration updates).

[0032] In this embodiment, it is necessary to obtain the keyword template corresponding to the database storing script data, specifically including the following: Based on commonly used databases (such as MySQL, Oracle, and Hive), a unique identifier is created for each database, and a dedicated keyword template is configured. A storage table is created that stores the "unique identifier - database type - keyword template". After retrieving a piece of script data, the unique identifier corresponding to the database storing the script data is determined as the target identifier. Using the target identifier as the query condition, an exact match query is performed in the pre-created storage table to obtain the target keyword template corresponding to the target identifier. The target keyword template is then confirmed to be the keyword template corresponding to that database.

[0033] In addition, each unique identifier in the stored data table corresponds to a unique keyword template, and the keyword template contains the database-specific syntax features of the unique identifier (such as INSERT OVERWRITE and LOAD DATA, which are data writing / loading keywords unique to Hive databases, different from MySQL and Oracle databases). Based on the unique syntax features of each database, the uniqueness and matching of the target keyword template obtained by the query can be verified.

[0034] Understandably, a unique identifier for a database is a characteristic value used to uniquely distinguish different databases, and can be customized according to usage requirements (such as English abbreviations or numeric codes). Keyword templates are a set of syntax features specific to a particular database, including keywords, statement structures, field identifiers, operators, etc., unique to that database (such as LIMIT for MySQL and ROWNUM for Oracle).

[0035] In this step, script data is acquired to provide the foundation for subsequent identification and parsing. Keyword templates corresponding to the database to which the script belongs are obtained simultaneously. By using keyword templates, the syntax differences between different databases are avoided, and the effective fields in the script data are identified in a targeted manner to standardize and normalize them, thus avoiding parsing chaos.

[0036] S200: Determine the conversion rules for metacharacters, character case, and dynamic parameters, and vectorize the conversion rules, storing the vectorized conversion rules in a vector library.

[0037] During the data lineage generation process, script data needs to be parsed. This paper analyzes the factors causing parsing errors, such as failed metacharacter parsing, inability to automatically convert case, and failure to recognize dynamic parameters. Metacharacters refer to characters with special meanings in the script program (e.g., ., ...). Parsing failures typically occur because these characters ($, \, and \) are not escaped, or the parsing logic fails to correctly identify their special meanings, causing the script's functionality to deviate from expectations. For example, matching a decimal point in a string (such as the period in "ab") might fail due to the lack of escaped metacharacters. Inability to automatically convert case means the script does not uniformly handle the case of its parameters, causing content with the same semantic meaning to be interpreted as different values ​​depending on the case. For example, a user-defined constant parameter "student" might not be case-sensitive, resulting in "Student" and "STUDENT" being either not recognized or being recognized as another constant parameter. Dynamic parameters refer to parameters with variable values ​​that require dynamic replacement. Failure to recognize these parameters usually stems from the script not adapting to variable parameters or not correctly binding dynamic variables. This embodiment addresses the metacharacters, character case, and dynamic parameters that cause parsing errors by establishing unified conversion rules for processing metacharacters, case, and dynamic parameters, ensuring consistency in conversion and avoiding data corruption caused by inconsistent conversion rules. During the parsing process, metacharacters, character case, and dynamic parameters are parsed according to conversion rules to improve the accuracy of data lineage parsing, thereby greatly improving the accuracy of data lineage.

[0038] The conversion rules are vectorized, transforming discrete, semantic conversion rules into machine-readable vector forms. These vectorized rules are then persistently stored in a vector library, providing a data source for subsequent conversion models to quickly retrieve and match rules. Furthermore, the vector library storage method enables conversion models to quickly match and retrieve corresponding rules (e.g., for case conversion of a specific field, the conversion model can directly retrieve the corresponding vectorized rule from the library and execute it), improving conversion efficiency.

[0039] It is understandable that the conversion rules for metacharacters, character case, and dynamic parameters can be customized according to usage requirements. In this embodiment, the conversion rules for metacharacters, character case, and dynamic parameters are explained through enumeration. For example, among the many metacharacters, a single quote (') is converted to two single quotes (''); the percent sign (%) and underscore (_) are used as wildcards in LIKE queries, and need to be converted to escape characters (ESCAPE); a backslash (\) is converted to a double backslash (\\). When converting character case, keywords are uniformly converted to uppercase, table names or field names retain their original case or are processed according to camelCase rules, and comments and references are not converted. When converting character case, uppercase characters are uniformly converted to lowercase characters, or lowercase characters are uniformly converted to uppercase characters. For example, the character Student and its variants, such as STUdent and stUDENT, are uniformly converted to lowercase Student or uppercase STUDENT. When converting dynamic parameters, identify the constants that should be parameterized and convert them into dynamic parameters. For example, constants within the special symbol #{} need to be converted into dynamic parameters.

[0040] S300: Input the script data and the keyword template into a preset conversion model, and have the conversion model output conversion data and bloodline relationship; The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0041] During the training phase of the conversion model, it's possible to simultaneously learn to handle three types of optimization tasks: metacharacters, character case variations, and dynamic parameters. This can be achieved by designing a comprehensive training dataset containing various script data and their corresponding optimization requirements. A large number of script data samples are collected, including various special characters, case variations, and dynamic parameter usage. This data will be used to train the model. The collected script data samples are preprocessed, including cleaning, formatting, and labeling, to adapt to the model's input requirements. A suitable pre-trained model, such as BERT or GPT, is selected as the foundation. These models have been pre-trained on large-scale text data and possess strong language understanding capabilities. After training, the resulting conversion model will be deployed to a production environment for practical optimization tasks.

[0042] In this step, the conversion model obtains the keyword templates corresponding to the database. Based on the syntactic features and field identifiers in the keyword templates, it accurately extracts the target fields to be converted from the original script data, i.e., the fields to be processed. Then, the identified fields to be processed are converted into vector form, resulting in vectors of the fields to be processed, achieving machine-based representation of the fields. The conversion model retrieves vectorization conversion rules stored in the vector library, matches the "vectors of the fields to be processed" with the "vectorization conversion rules" in the library, and converts the "vectors of the fields to be processed" according to the matched "vectorization conversion rules," ultimately restoring the structured converted data. During the field conversion process, the conversion model automatically establishes a one-to-one mapping relationship between the original script data (source data) and the converted data (target data); and directly confirms this mapping relationship as the data lineage, that is, clearly identifying which field in the source data each field of the target data originates from and what kind of conversion it has undergone.

[0043] In this step, the script data is transformed using a transformation model. No manual intervention in field identification and transformation is required. The transformation model achieves intelligent and standardized processing of script data through matching keyword templates and vectorized transformation rules, improving data processing efficiency and reducing human error. By matching keyword templates corresponding to the database and customized transformation rules, the transformation model can adapt to script data processing in different types of databases and different business scenarios, demonstrating good versatility and adaptability. The transformation model integrates the data transformation process with the lineage relationship construction process. Lineage relationships are generated synchronously with the transformed data, eliminating the need for separate lineage analysis after data transformation, simplifying the process and reducing costs. The lineage relationship established by the transformation model allows tracing the original source of each transformed data field, enabling fine-grained lineage analysis (rather than coarse-grained script-level lineage), meeting core needs such as data traceability and data quality verification.

[0044] In an optional implementation, specific steps for vectorizing the transformation rules are provided, including steps S201 to S203: S201: Convert the transformation rules into structured text data.

[0045] In this step, the transformation rules, whether in the form of natural language, logical expressions, or rule text, are first converted into a standardized text structure that can be understood by the model. This requires breaking down the transformation rules into condition fields, operation fields, target fields, mapping relationship fields, and constraint fields, thus creating semantic units that the model can comprehend. Then, a fixed template (such as JSON, XML, or a unified natural language template) is used to organize the split transformation rules into one or more text segments, ensuring that different transformation rules (metacharacters, character case, and dynamic parameters) can be unified into the same structured text data.

[0046] This step transforms the transformation rules into structured text data that is clear in structure, complete in semantics, and uniform in format. The semantic information of the transformation rules is fully preserved in the structured text data, while the format is more standardized, reducing the model's understanding bias. It can be directly used as input to the pre-trained language model, enabling the pre-trained language model to stably and consistently read and understand the semantics of the transformation rules.

[0047] S202: Input the structured text data into the pre-trained language model, and generate and output the original numerical vector by the pre-trained language model.

[0048] In this step, structured text data is fed into a pre-trained language model, which can be any one of word embeddings, BERT, GPT, or LLaMA.

[0049] The pre-trained language model first segments the structured text data using a tokenizer, then uses a multi-layer attention and feedforward network with a Transformer structure to semantically encode each segment, and finally outputs the original numerical vector through a pooling layer. It should be noted that the original numerical vector is a fixed-dimensional numerical array that does not retain the original text format, only the semantic information.

[0050] This step utilizes a pre-trained language model to map structured text data into a continuous vector space. Vectors are used to quantify the semantics of the rules, transforming the originally uncomputable structured text data into a computable, comparable, and storable numerical representation.

[0051] S203: Perform numerical processing on the original numerical vector to obtain the vectorized transformation rules.

[0052] In this step, the original numerical vector is subjected to numerical processing, such as L2 normalization or mean-variance standardization.

[0053] The original numerical vector is a fixed-dimensional array of numbers, such as: original numerical vector A: [10, 20, 30], original numerical vector B: [3, 4], and original numerical vector C: [0.01, 0.02, 0.03]. The magnitude of the original numerical vector may vary greatly; the magnitudes of original numerical vectors A, B, and C are 37.42, 5, and 0.0374, respectively. The magnitude fluctuates significantly and is relatively unstable. When calculating the similarity between the vectorized field and the conversion rules in the vector library, the magnitude of the numerical value has a greater impact on the similarity than the vector direction of the magnitude. This leads to the inability to correctly calculate the similarity between the field and the conversion rules, and consequently, the inability to correctly match the conversion rules.

[0054] Numerical processing forces the magnitudes of the original numerical vectors to be of the same order of magnitude, eliminating the interference of numerical size on similarity calculation and retaining only vector direction information. This makes the magnitude distribution of the original numerical vectors more stable, facilitating the calculation of the similarity between the vectorized field and the transformation rules in the vector library, and then selecting matching transformation rules from the vector library to transform the field.

[0055] Specifically, L2 normalization (L2 being the Euclidean norm) is used to divide each dimension value of the original numerical vector by the L2 modulus of the original numerical vector, resulting in a normalized vector. It is crucial that the modulus of the normalized vector is fixed at 1. L2 normalization stabilizes the modulus distribution of the original numerical vector, leading to more accurate similarity calculations. This allows for precise selection of matching transformation rules from the vector library, while also improving the computational efficiency of rule selection.

[0056] Let's take the original numerical vector A as an example for explanation: The original numerical vector A has a magnitude of 37.42. The ratio of each dimension (10, 20, and 30) of the original numerical vector A to the magnitude 37.42 is calculated to obtain the normalized vector [0.267, 0.534, 0.801]. It is ensured that the new magnitude corresponding to the normalized vector [0.267, 0.534, 0.801] is approximately equal to 1.

[0057] In an optional implementation, specific steps are provided for storing the vectorized transformation rules in a vector library, including steps S204 and S205: S204: Establish the encoding and meta-information of the conversion rule.

[0058] In this step, a unique code is generated for each vectorized transformation rule, and structured metadata is constructed. The encoding of transformation rules employs standardized encoding rules to assign a globally unique code to each rule. The encoding format can be "rule type abbreviation + auto-incrementing number + timestamp," a random unique string, or a business-defined sequence code, ensuring that the code for each rule in the vector library is unique and allowing for rapid location of individual rules. By assigning a unique code to each transformation rule, precise operations can be performed directly on individual transformation rules without traversing the entire vector library. This solves the problem of difficulty in individually locating transformation rules due to the lack of clear identification in the vector library, and further enables rapid modification or deletion of transformation rules in the vector library.

[0059] The metadata of the transformation rules includes at least one of the following: transformation rule type, applicable database, and original text of the transformation rule; the transformation rule type includes: metacharacters, character case, and dynamic parameters. Furthermore, the structured metadata for each transformation rule must correspond one-to-one with the transformation rule itself. The metadata is ultimately stored in a structured format (such as key-value pairs, JSON, or a dictionary) for easy binding and retrieval.

[0060] Structured metadata associates abstract transformation rules with business attributes (transformation rule type, applicable database, or original text of the transformation rule), making the transformation rule data in the vector library business interpretable, rather than just simple numerical vectors. This provides a data foundation for metadata-based filtering (such as "filtering rules that only apply to MySQL metacharacter types"), improving the efficiency and accuracy of rule retrieval.

[0061] It should be noted that the established transformation rule type needs to be matched and labeled from three categories: metacharacters, character case, and dynamic parameters. The applicable database for the established transformation rule needs to be specified, clearly indicating the database types to which the transformation rule can be applied (e.g., MySQL, Oracle, PostgreSQL, Redis, etc., and can be labeled with a single or multiple categories, such as "MySQL / Oracle"). The original text of the established transformation rule needs to record the natural language description or original definition text of the transformation rule.

[0062] S205: Bind the vectorized conversion rules with the encoding and the meta-information and store them in the vector library.

[0063] In this step, the vectorized transformation rules are used as the main body, the encoding is used as a global identifier field, and the structured meta-information is used as a business attribute field. These are then associated and bound with the vectorized transformation rules to form a complete rule data unit.

[0064] It is understandable that the bound data units must comply with the storage format requirements of the vector library. The typical format is: {unique code: AAA, vector data: [0.12, 0.34, ..., 0.56], meta information: {type: metacharacter, applicable database: MySQL, original text: AAA}}.

[0065] The complete rule data units described above are written to the target vector library (such as Milvus, Chroma, Pinecone, FAISS, etc.) according to its API interface or storage specifications to achieve persistent storage. During the storage process, indexing strategies can be configured for the vector library according to business needs to improve the speed of subsequent similarity retrieval; primary key indexes can be configured for the encoding, and attribute indexes can be configured for key fields of metadata (such as type and applicable database) to improve the speed of accurate retrieval based on encoding and metadata.

[0066] This step achieves centralized management of transformation rules by storing them uniformly in the vector library. Subsequent additions, modifications, and deletions of transformation rules only require operation on individual data units in the vector library, simplifying the rule maintenance process. Furthermore, the bound storage method allows the vector library to simultaneously possess the dual capabilities of "vector semantic retrieval" and "business attribute filtering," perfectly adapting to the actual use cases of transformation rules (such as first narrowing the scope by business attributes, then precisely matching rules based on semantics).

[0067] In an optional implementation, specific steps are provided for determining the fields to be processed in the script data based on the keyword template and vectorizing the fields, including steps S301 and S302: S301: Parse the keyword template and the script data respectively to obtain keyword information and a structured list of script data fields.

[0068] In this step, the fields in the keyword template are parsed to extract standardized keyword information. This keyword information is a set of field names that can be directly used for matching; it represents a range of fields that require no further processing, providing a basis for filtering fields to be processed and avoiding invalid vectorization of the keyword information fields. All fields in the script data are extracted by field dimension, and all extracted fields undergo structured processing, including field deduplication and classification, resulting in a complete, standardized, and matchable list of script data fields.

[0069] This step parses the keyword template and script data, converting the format of the original input data. During script data parsing, field cleaning (duplicate removal and categorization) is performed, eliminating invalid and duplicate fields in advance, reducing the amount of data processing during field matching, and improving overall processing efficiency. Parsing the keyword template and script data ensures that the keyword information and script field list are in a standardized format, allowing subsequent field comparisons to be performed simply through basic set comparisons, making it simple, efficient, and significantly reducing the complexity of the comparison.

[0070] S302: In the script data field list, determine the fields other than the keyword information as fields to be processed, and perform vectorization processing on the fields to be processed according to the vector encoding rules.

[0071] In this step, the keyword information is defined as the exclusion set, and the list of script data fields is defined as the full set. The fields to be processed are filtered using set difference operations. Specifically, fields that exactly match the keyword information are removed from the full set of script data fields; the remaining fields are the fields to be processed. It is important to note that exact character matching is used during the matching process to ensure error-free filtering results. By using set difference filtering, the fields in the script that need to be processed are precisely located, achieving accurate filtering of the fields to be processed. Core keyword fields that do not need processing are excluded, while the fields that require processing in the script are retained. This reduces the amount of data involved in subsequent vector processing and business operations, improving the targeting of data processing and the efficiency of the overall process.

[0072] In this step, a preset vector encoding rule is invoked to uniformly vectorize the selected fields to be processed, that is, to convert character-type field names into numeric vector arrays for subsequent vector retrieval and matching operations. It should be noted that the vector encoding rule in this embodiment is a pre-defined unified rule, which can be based on a general large model embedding interface (such as BERT, Text2Vec, or OpenAI Embedding) or a self-developed encoding algorithm to ensure that the field vectorization rules or vector dimensions are consistent across the same scenario. Through the vector encoding rule, the vector dimensions and format of all fields to be processed are consistent, ensuring seamless vector storage and matching in the vector library, guaranteeing the compatibility and feasibility of the business process.

[0073] In an optional implementation, specific steps are provided for converting vectorized fields based on the conversion rules of the vector library to obtain converted data, including steps S303 to S305: S303: Perform similarity matching between the vectorized field and the conversion rules in the vector library to determine the conversion rule with the highest matching degree with the vectorized field.

[0074] In this step, the similarity between the vectorized fields and the conversion rules in the vector library is calculated one by one, using a cosine similarity algorithm suitable for field matching. The matching result of the cosine similarity algorithm ranges from 0 to 1. The closer to 1, the higher the similarity. Using the cosine similarity algorithm for similarity matching avoids the limitations of precise character matching. Even if the fields and conversion rules are semantically similar but have different characters, it can improve matching efficiency and accuracy, avoid the inefficiency of full library traversal, and ensure the accuracy of rule selection. After determining the matching results of all conversion rules and vectorized fields, they are sorted from high to low matching scores, and the conversion rule with the highest matching score is selected as the target rule. By sorting by similarity, the optimal conversion rule is quickly located, ensuring the accuracy and adaptability of field conversion.

[0075] Furthermore, if the highest matching score among multiple matching results is lower than the preset matching threshold (e.g., 0.7), it is determined that there is no matching rule, the subsequent conversion process is terminated, and a prompt is returned. If multiple rules with the same highest matching score exist, they can be selected according to the preset priority of the encoding or the accuracy of the applicable scenario, or multiple rules can be returned for manual confirmation. By setting a matching threshold, irrelevant rules with low similarity are avoided from being applied to field conversion, preventing erroneous conversion, ensuring the reliability of the conversion, and improving the robustness of the conversion.

[0076] S304: Perform vector transformation on the vectorized fields using the transformation rule with the highest matching degree.

[0077] In this step, the field is transformed into a vector using conversion rules, achieving a vector-level transformation that eliminates the need for frequent character-to-vector conversions, significantly improving processing efficiency. The transformed field retains the core semantic features of the original field, modifying only the feature bits related to the conversion rules. This achieves the conversion effect required by the rules while avoiding the loss of the original field's semantics.

[0078] S305: Perform restoration processing on the fields after vector transformation to obtain the transformed data.

[0079] In this step, the vector encoding rules used when vectorizing the field to be processed are invoked, and the corresponding vector inverse encoding restoration rules are matched. The fields after vector conversion are then restored using these inverse encoding restoration rules, converting the numeric vector array into character field names. In the script data field list, the fields to be processed are replaced with the restored fields, ultimately resulting in character-based converted data that conforms to business specifications. The restoration rules and encoding rules are a one-to-one inverse process. Encoding restoration using the restoration rules ensures that the restored fields maintain consistency with the original fields in terms of character format and semantic expression, without information loss or distortion, guaranteeing the accuracy and integrity of the converted data.

[0080] In an optional implementation, specific steps are provided for establishing the mapping relationship between the script data and the transformation data, including steps S306 and S307: S306: Establish an indirect mapping between the script data, the field to be processed, and the conversion data; and establish a direct mapping between the field to be processed before and after the conversion process.

[0081] In this step, indirect mapping is a cross-level association mapping. It uses the fields to be processed as an intermediate bridge to connect the original script data and the final transformed data, achieving an indirect association between the two. When establishing indirect mapping, a globally unique script identifier is assigned to the original script data as the unique key of the original end of the indirect mapping. The fields to be processed selected in step S302 are extracted as the intermediate association layer; the transformed data output in step S305 is extracted as the target end data, and the transformed data is assigned an association identifier consistent with the script data (ensuring traceability of attribution).

[0082] Direct mapping is a precise one-to-one mapping at the field level. It is the field to be processed itself, and it is the direct association between a single field from its original state to its transformed state. It is the smallest unit of granularity in the mapping relationship. When establishing a direct mapping, the original field to be processed before transformation is extracted from step S302, and the target field (transformed data) after transformation is extracted from step S305, forming a one-to-one corresponding field pair.

[0083] By establishing indirect and direct mappings, different dimensions of tracing and association needs are addressed: Indirect mapping resolves the overall attribution association between the original script data and the final transformed data. Using the fields to be processed as the core bridge, indirect mapping achieves decoupling between script data and transformed data: it eliminates the need for direct hard binding between script data and transformed data, allowing for flexible adjustment of the mapping relationship through intermediate fields, improving the flexibility and maintainability of the mapping. Direct mapping solves the problem of accurate tracing before and after a single field transformation, taking into account both overall hierarchical association and fine-grained field association. Furthermore, direct mapping extracts the core transformation relationships at the field level, which can be reused in other field transformation scenarios with similar scripts, eliminating the need to repeatedly execute the complete vector matching to transformation process, thus improving the reusability and efficiency of field transformation.

[0084] This step enables full data traceability of the transformation process: indirect mapping allows for quick location of the original script data and the final transformed data. Direct mapping allows for precise tracking of fields before and after transformation, improving data traceability and reliability.

[0085] S307: Integrate the indirect mapping and the direct mapping to form a mapping relationship between the script data and the transformation data.

[0086] In this step, based on indirect and direct mappings, two independent mapping types are integrated into a complete and directly usable mapping relationship between script data and transformed data. This ensures that the integrated mapping relationship balances overall association and fine-grained traceability. When integrating indirect and direct mappings, the precise field-level transformation relationships of the direct mappings are embedded into the script-level association structure of the indirect mappings. This supplements the indirect mappings with field-level traceability details through direct mappings. By embedding field-level direct mappings, the integrated mapping relationship possesses both overall attribution and fine-grained traceability capabilities, supporting both global queries of indirect mappings and precise verification of direct mappings.

[0087] In an optional implementation, in the above embodiment of the data lineage generation method, after step S300, step S400 is further included: S400: Perform syntax validation on the converted data and identify the converted data that fails the syntax validation and its corresponding error information; input the converted data that fails the syntax validation and the error information into the conversion model for correction until the converted data passes the syntax validation.

[0088] In this step, syntax validation is first used to accurately capture anomalies in the converted data and extract structured anomaly information. Lightweight parsers are selected for syntax validation (such as sqlparse or PostgreSQL for general scenarios, and libpg_query, MySQL, or mysql-parser for specific scenarios). The converted data generated in step S305 is input into the lightweight parser in batches. The parser performs syntax analysis on each statement in the converted data (verifying whether the statement structure conforms to the syntax specifications of the target database). For statements that fail to be parsed, the parser returns standardized anomaly information, including error code, error type (such as missing semicolon, mismatched parentheses, misspelled keywords), error location (line number or column number), and error description (natural language), and marks the statement as a syntax-level anomaly in the converted data.

[0089] The anomaly information and the original converted data are then input into the conversion model. Based on its own grammatical knowledge and the context of the anomaly information, the conversion model performs targeted corrections on the statements in the anomaly converted data, resulting in corrected statements. The corrected statements replace the original statements, yielding corrected converted data. The corrected converted data undergoes another grammatical check. If it passes the full grammatical check, the corrected converted data is used as the final correct converted data, replacing the original anomaly converted data. If it still fails the check, new structured anomaly information is extracted from the current check result, combined with the original converted data and the anomaly information from the previous check, and re-input into the conversion model for a second correction. This process is repeated until the converted data passes the full grammatical check or reaches the maximum number of iterations. A maximum number of iterations is set (e.g., 3-5 times). If the maximum number of iterations is reached and the data still fails the check, automatic correction terminates, the anomaly data is marked for manual intervention, and all anomaly information and correction records are output to avoid infinite iteration.

[0090] This step, through syntax validation and iterative correction of the conversion model, ensures that the final output converted data conforms to the syntax specifications of the target database and can be executed normally. This provides a high-quality data foundation for accurate generation of data lineage. Moreover, the syntax validation process requires no manual intervention, automatically completing the entire process from validation to correction. Manual intervention is only required for a very small number of extremely abnormal data points, freeing humans from tedious syntax correction work and significantly improving the efficiency of the entire data lineage generation process.

[0091] As described above, the data lineage generation method provided in this application locates fields in script data using keyword templates and combines them with vectorized conversion rules from a vector library to achieve vector conversion of metacharacters, case sensitivity, and dynamic parameters, improving the accuracy and adaptability of field conversion. The conversion model completes field vectorization, rule matching conversion, and mapping relationship establishment, directly outputting the converted data and lineage relationships, simplifying the lineage generation process and improving the automation and efficiency of data lineage construction. The mapping relationship between script data and converted data defines the lineage, clearly tracing the association before and after data conversion, ensuring the accuracy and traceability of the lineage relationships. The conversion rules stored in the vector library can provide reusable rule support for subsequent similar script processing, improving the scalability and reusability of the solution, and overall ensuring the quality and efficiency of cross-database script data lineage generation.

[0092] like Figure 2 As shown in the figure, an embodiment of this application provides a data lineage generation apparatus 200, comprising: The data acquisition unit 210 is used to acquire script data and keyword templates corresponding to the database storing the script data; The vectorization processing unit 220 is used to determine the conversion rules of metacharacters, character case and dynamic parameters, and to perform vectorization processing on the conversion rules, and to store the vectorized conversion rules in a vector library; The conversion processing unit 230 is used to input the script data and the keyword template into a preset conversion model, and the conversion model outputs conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0093] In an optional embodiment, the vectorization processing unit 220 includes: The model processing module is used to convert the transformation rules into structured text data; input the structured text data into a pre-trained language model, which generates and outputs the original numerical vector; and perform numerical processing on the original numerical vector to obtain the vectorized transformation rules.

[0094] In an optional embodiment, the vectorization processing unit 220 further includes: A storage module is used to establish the encoding and meta-information of the conversion rule; bind the vectorized conversion rule with the encoding and the meta-information and store it in the vector library; wherein, the meta-information includes at least one of the following: conversion rule type, database to which the conversion rule applies, and original text of the conversion rule; the conversion rule type includes: metacharacter, character case, and dynamic parameters.

[0095] In an optional embodiment, the conversion processing unit 230 includes: The vectorization module is used to parse the keyword template and the script data respectively to obtain keyword information and a structured list of script data fields; in the list of script data fields, the fields other than the keyword information are identified as fields to be processed, and the fields to be processed are vectorized according to the vector encoding rules.

[0096] In an optional embodiment, the conversion processing unit 230 further includes: The conversion module is used to perform similarity matching between the vectorized fields and the conversion rules in the vector library to determine the conversion rule with the highest matching degree with the vectorized fields; to perform vector conversion processing on the vectorized fields using the conversion rule with the highest matching degree; and to perform restoration processing on the vector-converted fields to obtain the converted data.

[0097] In an optional embodiment, the conversion processing unit 230 further includes: The data lineage module is used to establish an indirect mapping between the script data, the fields to be processed, and the transformed data; and to establish a direct mapping between the fields to be processed before and after the transformation process; and to integrate the indirect mapping and the direct mapping to form a mapping relationship between the script data and the transformed data.

[0098] In an optional embodiment, the data lineage generation apparatus further includes: The verification unit is used to perform syntax verification on the converted data and identify the converted data that fails the syntax verification and its corresponding error information; the converted data that fails the syntax verification and the error information are input into the conversion model for correction until the converted data passes the syntax verification.

[0099] The embodiments of the data lineage generation apparatus provided in this application can be used to execute the processing flow of the data lineage generation method embodiments described above. Its functions will not be repeated here, but can be referred to the detailed description of the above method embodiments.

[0100] like Figure 3As shown in the embodiment of this application, an electronic device 800 includes a memory 820 and a processor 810; the memory 820 is used to store a computer program; the processor 810 is used to implement the model reasoning method as described above when the computer program is executed.

[0101] Alternatively, an electronic device 800 includes a memory 820 and a processor 810 coupled to the memory 820; the memory 820 is configured to store a computer program; and the processor 810 is configured to perform the following operations when the computer program is executed: Obtain script data and keyword templates corresponding to the database storing the script data; Determine the conversion rules for metacharacters, character case, and dynamic parameters, and vectorize the conversion rules. Store the vectorized conversion rules in a vector library. The script data and the keyword template are input into a preset conversion model, and the conversion model outputs the conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0102] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the data lineage generation method described above.

[0103] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations: Obtain script data and keyword templates corresponding to the database storing the script data; Determine the conversion rules for metacharacters, character case, and dynamic parameters, and vectorize the conversion rules. Store the vectorized conversion rules in a vector library. The script data and the keyword template are input into a preset conversion model, and the conversion model outputs the conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data according to the keyword template and to vectorize the fields; to convert the vectorized fields according to the conversion rules of the vector library to obtain converted data; and to establish a mapping relationship between the script data and the converted data and confirm that the mapping relationship is a blood relationship.

[0104] The present invention describes an electronic device 800 that can serve as a server or client for this application, which is an example of a hardware device that can be applied to various aspects of this application. Electronic device 800 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 800 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0105] Electronic device 800 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0106] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0107] Although the above disclosure is provided, the scope of protection of this application is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this application, and all such changes and modifications will fall within the scope of protection of this application.

Claims

1. A method for generating data lineage, characterized in that, include: Obtain script data and keyword templates corresponding to the database storing the script data; Determine the conversion rules for metacharacters, character case, and dynamic parameters, and vectorize the conversion rules. Store the vectorized conversion rules in a vector library. The script data and the keyword template are input into a preset conversion model, and the conversion model outputs the conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data based on the keyword template and to vectorize the fields. Based on the conversion rules of the vector library, the vectorized fields are converted to obtain converted data; and a mapping relationship is established between the script data and the converted data, and the mapping relationship is confirmed to be a blood relationship.

2. The method for generating data lineage according to claim 1, characterized in that, The vectorization process of the transformation rule includes: The transformation rules are then converted into structured text data. The structured text data is input into a pre-trained language model, which generates and outputs the original numerical vector. The original numerical vector is numerically processed to obtain the vectorized transformation rules.

3. The method for generating data lineage according to claim 1, characterized in that, The step of storing the vectorized transformation rules in the vector library includes: Establish the encoding and metadata of the conversion rules; The vectorized transformation rules are bound to the encoding and the metadata and stored in the vector library; The meta-information includes at least one of the following: conversion rule type, database to which the conversion rule applies, and original text of the conversion rule; the conversion rule type includes: metacharacters, character case, and dynamic parameters.

4. The method for generating data lineage according to claim 1, characterized in that, The step of determining the fields to be processed in the script data based on the keyword template and vectorizing the fields includes: The keyword template and the script data are parsed separately to obtain keyword information and a list of structured script data fields, respectively. In the list of script data fields, fields other than the keyword information are identified as fields to be processed, and vectorization processing is performed on the fields to be processed according to the vector encoding rules.

5. The method for generating data lineage according to claim 1, characterized in that, The transformation of vectorized fields based on the transformation rules of the vector library to obtain transformed data includes: The vectorized fields are matched with the conversion rules in the vector library to determine the conversion rule with the highest matching degree with the vectorized fields. The vectorized fields are transformed using the transformation rule with the highest matching degree. The fields after vector transformation are restored to obtain the transformed data.

6. The method for generating data lineage according to claim 1, characterized in that, Establishing the mapping relationship between the script data and the transformation data includes: Establish an indirect mapping between the script data, the fields to be processed, and the transformed data; and establish a direct mapping between the fields to be processed before and after the transformation process; The indirect mapping and the direct mapping are integrated to form a mapping relationship between the script data and the transformation data.

7. The method for generating data lineage according to claim 1, characterized in that, After the transformation model outputs the transformation data, it also includes: Perform syntax validation on the converted data and identify the converted data that fails the syntax validation and its corresponding error information; The converted data that fails the syntax check and the error information are input into the conversion model for correction until the converted data passes the syntax check.

8. A data lineage generation device, characterized in that, include: A data acquisition unit is used to acquire script data and keyword templates corresponding to the database storing the script data; The vectorization processing unit is used to determine the conversion rules for metacharacters, character case and dynamic parameters, and to perform vectorization processing on the conversion rules, and store the vectorized conversion rules in a vector library; A conversion processing unit is used to input the script data and the keyword template into a preset conversion model, and the conversion model outputs conversion data and bloodline relationship. The conversion model is used to determine the fields to be processed in the script data based on the keyword template and to vectorize the fields. Based on the conversion rules of the vector library, the vectorized fields are converted to obtain converted data; and a mapping relationship is established between the script data and the converted data, and the mapping relationship is confirmed to be a blood relationship.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the data lineage generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the data lineage generation method as described in any one of claims 1 to 7.

Images