An operator-level data blood relationship automatic generation method based on a large model

Abstract

The application provides a large model-based operator-level data blood relationship automatic generation method, relates to the field of electric digital data processing, and comprises the following steps: step S1, heterogeneous script preprocessing and semantic normalization; step S2, symbol-enhanced LLM blood inference; step S3, causal logic verification and blood correction; step S4, dynamic adaptation; and step S5, blood intelligent analysis and visualization. The method is suitable for multi-source, multi-format and multi-language data processing scenes, can improve the automation degree, fine capacity and intelligent level of data blood construction, and provides more accurate and efficient technical support for data governance, data security, system migration / reconstruction, data quality control, cost optimization and the like.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a method for automatically generating operator-level data lineage relationships based on a large model. Background Technology

[0002] Data lineage technology is a technical system that tracks the entire lifecycle of data. Its core is to record the entire relationship of data from its source (such as the original database, log file, etc.) to the processing process (such as ETL transformation, SQL calculation, etc.) and then to the end (such as reports, application systems, data products, etc.). At present, data lineage technology is widely used in scenarios such as data governance, data security, system migration / reconstruction, data quality control, and cost optimization.

[0003] An operator is the smallest unit of computation in data processing (such as SELECT, JOIN, WHERE, GROUP BY, etc.); it includes direct / indirect lineage, filtering conditions, aggregation rules, etc. Refined lineage refers to establishing a more granular lineage processing relationship than field dependencies by understanding the processing parameters of fields, enabling a precise, accurate, and comprehensive representation of the data chain.

[0004] Traditional data lineage generation methods include: metadata analysis-based lineage extraction methods, which rely on metadata information recorded in the data platform (such as data table structure, field mapping relationships, ETL process definitions, etc.) to construct a lineage graph by analyzing the dependencies between metadata; SQL parsing-based lineage extraction methods, which perform syntax and semantic analysis on SQL processing scripts to extract data input-output relationships and construct lineage relationships at the field level and even the operator level; and manual annotation or rule configuration methods, which rely on manual annotation or rule templates to specify lineage relationships for complex logical processing or unstructured scripts. Among these, metadata analysis-based lineage extraction methods can only identify table-level or field-level lineages, making it difficult to delve into the operator level, thus limiting accuracy and granularity; SQL parsing-based lineage extraction methods are sensitive to SQL standardization, have weak cross-language and cross-engine processing capabilities, questionable accuracy, high maintenance costs, and difficulty adapting to frequently changing business logic; and manual annotation or rule configuration methods have high labor costs, poor scalability, and are easily affected by subjective factors, leading to inconsistencies or errors in the lineage graph. In other words, existing technologies for pedigree extraction, particularly in the automatic construction of operator-level pedigrees, still suffer from limitations in syntax parsing capabilities, poor compatibility (i.e., existing methods struggle to support mixed scenarios involving multiple data processing languages ​​such as SQL, Python, Spark, and Flink), insufficient versatility and scalability (i.e., existing methods have poor portability when facing constantly changing business logic, complex scripts, and multi-source heterogeneous data systems), inability to effectively understand and extend semantic context (i.e., existing methods lack a deep understanding of business semantics and context, resulting in low recognition accuracy when dealing with complex logic or fuzzy structures), and difficulty in building operator models (i.e., existing methods are prone to extraction failures or inaccuracies during the operator model building process). Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide an automatic generation method for operator-level data lineage relationships based on a large model. This method can perform high-precision parsing and understanding of various data processing language scripts, semantic analysis of structured and semi-structured data processing scripts, adapt to multi-source, multi-format, and multi-language data processing scenarios, improve the automation, refinement, and intelligence of data lineage construction, and provide more accurate and efficient technical support for scenarios such as data governance, data security, system migration / reconstruction, data quality control, and cost optimization.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] An automatic generation method for operator-level data lineage relationships based on a large model, comprising:

[0008] Step S1, Heterogeneous script preprocessing and semantic normalization: First, the input target script language is fragmented and cleaned, then multi-language syntax atomic normalization is performed, and finally multi-modal semantic anchoring and output normalization operator semantic representation library are performed.

[0009] Step S2, Symbolically Enhanced LLM (Large Language Model) Lineage Reasoning: First, construct symbolic logic rule constraints, and then generate candidate operator-level lineages through LLM reasoning;

[0010] Step S3, Causal Logic Verification and Lineage Correction: For candidate operator-level lineages, erroneous lineages are eliminated and missing lineages are supplemented through causal graph and symbolic logic verification.

[0011] Step S4, Dynamic Adaptation: Achieve cross-scene adaptive adaptation through Dynamic Graph Neural Networks (DGNN) to complete cross-scene low-code extension;

[0012] Step S5, Intelligent Bloodline Analysis and Visualization: First, anomaly detection is performed using causal entropy-driven methods, and then visualization is achieved through dynamic force-guided layout, thus completing the dynamic visualization bloodline map.

[0013] Based on further optimization of the above scheme, step S1, which involves sharding and cleaning the input target scripting language (including SQL, Python, Flink / Scala, etc.), specifically involves:

[0014] First, the original script data is split into operator-level fragments based on language feature delimiters. S ={ s 1, s 2,…, s n Each segment contains a single complete operator logic; then, segments with more than 80% pure comment lines, blank lines, and invalid content without a core operator are deleted; finally, for syntactically incomplete segments... s Fuzzy matching algorithm is used M(s) From the standard reference library S ref Matching the optimal standard fragment :

[0015] ;

[0016] In the formula: Representing fragment s and fragment Edit distance between (i.e.) Levenshtein Distance (or distance) is used to measure the syntactic similarity between two script fragments. Indicates the corresponding fragment s or syntactic feature vectors; This represents the cosine similarity between two syntactic feature vectors. These represent the corresponding weight coefficients;

[0017] And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments s Use the effective vocabulary to complete the sentence and obtain the completed paragraph. ;

[0018] After deletion and completion, the final output is a clean and complete operator-level fragment.

[0019] Based on further optimization of the above scheme, step S1, specifically performing multilingual grammar atomic normalization, involves:

[0020] Constructing operator-level syntactic atoms G ={ g 1, g 2,…, g n},in, g i Represents the smallest syntactic atom; G It covers core logic such as field extraction, conditional filtering, aggregation, and nested functions;

[0021] Next, the syntactic atomic mapping values ​​of different languages ​​are unified into a feature space:

[0022] ;

[0023] In the formula: Represents the syntactic atoms in language L; express The set of words included; Words v go through BERT (Bidirectional Encoder Representations from Transformers) model generates language representations; Language type encoding representing syntactic atoms; Indicates type weight; Words v In the fragment s i The number of times it appears in; Excerpt s i The sum of the occurrences of all words in the text; Indicates standard reference library S ref Contains vocabulary vThe number of reference fragments; Indicates the total number of reference segments included; This indicates the core weight.

[0024] Based on further optimization of the above scheme, step S1, specifically performing multimodal semantic anchoring and outputting a normalized operator semantic representation library, involves:

[0025] For operators o The semantic anchoring algorithm through modal fusion integrates syntactic features ,go through BERT Contextual features generated by the model Emb(C(o)) ,go through BERT Metadata features generated by the model Emb(M(o)) The data is fused and output as a unique semantic representation through a three-layer MLP (Multi-Layer Perceptron) model. :

[0026] ;

[0027] in: Derived from multilingual grammar atom normalization ,Right now:

[0028] ;

[0029] In the formula: G(o) Indicates from operator o The set of syntactic atoms extracted from it;

[0030] Operator context features C(o) Extract metadata features from clean, complete operator-level fragments output after sharding and data cleaning. M(o) The clean, complete operator-level fragments output after data fragmentation and cleaning, along with the grammatical atoms extracted through multilingual grammatical atom normalization, are used as indexes.

[0031] Based on further optimization of the above scheme, step S2 specifically includes:

[0032] First, define the symbolic logic rule set for operator-level lineage. R ={ r 1, r 2,…, r k};

[0033] Then, the rule set is transformed into a symbolic logic formula and injected into the LLM's Prompt template:

[0034] ;

[0035] In the formula: Template The prompt template (which includes semantic representation, symbol rules, and output format requirements) is a structured text framework designed for Large Language Models (LLM); Format represents the output format (including source operators, target operators, dependency set types, etc.).

[0036] Then, LLM generates a candidate lineage set using conditional probability distributions:

[0037] ;

[0038] In the formula: o b Indicates bloodline b The corresponding target operator; O Denotes the set of all operators; B c Indicates a set of candidate lineages; b This represents a single bloodline edge (i.e., the line connecting the source operator to the target operator).

[0039] Finally, construct the node set. U c Each node corresponds to the semantic representation and meta-information of the algorithm:

[0040] ;

[0041] In the formula: o id A unique identifier representing an operator; o label Semantic labels representing operators;

[0042] Construct edge set E c Each of its directed edges corresponds to B c A bloodline edge in the middle:

[0043] ;

[0044] In the formula: u s , u t Represent the source node and the target node, respectively; attr b Represents the attributes of an edge;

[0045] Final candidate lineage chart obtained: G c =( U c , E c ).

[0046] Based on further optimization of the above scheme, step S3 specifically includes:

[0047] Candidate lineage chart G c Construct a causal graph based on nodes and edges. G yg = ( U , E ), preserving the semantic representation attributes of operators;

[0048] The Depth-First Search (DFS) algorithm is used to traverse the causal graph and compute node pairs. u i , u j The set of paths (node ​​pairs) u i , u j The line connecting the two sides is called the edge.

[0049] ;

[0050] Path set reachability determination:

[0051] ;

[0052] In the formula: This represents the cosine similarity threshold.

[0053] At the same time, the path length is limited to no more than 3 hops;

[0054] Symbolic logic verification:

[0055] ;

[0056] Traversal rule set R All applicable to u j The rule is only valid if all rules are true. Ver ( u i , u j , r When )=1, the bloodline edge ( u i , u j ) Verification passed;

[0057] To obtain the final, precise bloodline set:

[0058] ;

[0059] In the formula: Indicates for all r ; This indicates a missing bloodline supplement set;

[0060] At the same time, through precise bloodline collection B * Obtain accurate bloodline chart G * =( U , E * ).

[0061] Based on further optimization of the above scheme, in step S3, for bloodline edges that fail verification, the following correction logic is executed:

[0062] when Ver ( u i , u j , r )=0 and Rex ( u i , u j When )=0, delete directly;

[0063] when Rex ( u i , u j )=1 and When necessary, supplement the edge based on rule R and label the dependency type;

[0064] when Ver ( u i , u j , r If )=1 but the dependency type is incorrect, the dependency type is corrected based on the rules.

[0065] Based on further optimization of the above scheme, step S4 specifically includes:

[0066] First, obtain a precise bloodline chart. G * =( U , E * As input to the DGNN, it initializes the DGNN model, preparing it for DGNN training, including:

[0067] Node hidden state initialization: For each operator node u ∈ U Set the initial hidden state to its semantic representation:

[0068] ;

[0069] Graph structure initialization: using a precise kinship map G * =( U , E * Based on the nodes and edges of the ), the graph structure of DGNN is constructed (the dependency relationship of the edges is preserved for the information interaction of neighboring nodes).

[0070] Model parameter initialization: Aggregate weights for each neighborhood. W 1 and W 2. Attention Vector Classification layer weights W 3. Perform initialization;

[0071] Then, the DGNN model is trained to learn the general lineage pattern:

[0072] Perform multiple rounds of layer updates, updating the hidden state of the current node in each round by aggregating information from neighboring nodes:

[0073] ;

[0074] In the formula: Represents a node u exist t +Hidden status after the first update; ReLU This indicates a modified linear unit activation function; N(u) node u The set of neighboring nodes; b h Indicates the bias term;

[0075] This represents the attention weight, used to measure the neighboring nodes. For the current node u Importance:

[0076] ;

[0077] In the formula: LeakyReLU This represents the modified linear unit activation function, which is proportional to... ReLU A small negative slope is added to perform non-linear transformations on the spliced ​​features; Represents a node u In the t Wheel hidden state With neighboring nodes exist t Wheel hidden state splicing;

[0078] By employing the cross-entropy loss function, DGNN learns the differences in lineage features across different scenarios:

[0079] ;

[0080] In the formula: Represents the set of operator nodes in a precise pedigree graph. U The total quantity; Z Represents a set of scene categories; y u,z Represents a node u Corresponding real-world scenario category tags; Softmax Indicates the activation function; Represents a node u After the first t x The final hidden state after the DGNN layer update; b l Indicates the bias term;

[0081] Minimize the loss function using the Adam optimizer Loss When the training value validation set loss is less than 0.001 or the preset number of iterations is reached, the DGNN model training is complete.

[0082] Finally, perform dynamic adaptation for new scenarios:

[0083] For the script of the new scenario, repeat step S1 to obtain the semantic representation of the operators of the new scenario. ;

[0084] Operator semantic representation of new scenarios As the initial hidden state of the nodes, it is input into the trained DGNN model and executed. t x Layer-by-layer update to obtain the adapted hidden state of the node. ;

[0085] Finally, the semantic similarity between the new scene and the reference scene (such as a pre-trained set of scenes) is calculated to obtain the fit coefficient:

[0086] ;

[0087] In the formula: U new This represents the combination of operator nodes in a new scenario. U ref The set of operator nodes representing the reference scenario;

[0088] Preset adaptation coefficient threshold, if adaptation coefficient If the bloodline is greater than the adaptation coefficient threshold, the existing bloodline rules are directly reused to generate a new scene bloodline; otherwise, the DGNN model parameters are adjusted, and the bloodline is generated again after supplementing the training with new scene samples.

[0089] Based on further optimization of the above scheme, in step S5, anomaly detection driven by causal entropy specifically involves:

[0090] ;

[0091] In the formula: Indicates the probability of a node appearing; Represents conditional probability;

[0092] when When this occurs, the corresponding pedigree chart is determined to be an abnormal pedigree chart; among which, Represents each normal lineage chart in the standard lineage bank causal entropy, This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The arithmetic mean; This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The sample standard deviation;

[0093] The visualization of the kinship diagram is achieved through dynamic force-guided layout, specifically as follows:

[0094] ;

[0095] In the formula: F att (u) Represents a node u Attraction F rep (u) Represents a node u The repulsive force received F imp (u) Represents a node u The importance of the force received; Weighting coefficients representing importance;

[0096] ;

[0097] In the formula: Weighting coefficients representing attractiveness; This indicates the nodes in the current layout. u With nodes The Euclidean distance between them; This represents the ideal node distance based on the layout assumption; This represents the weighting coefficient of the repulsive force; Indicates the smoothing coefficient; These represent the damping coefficient and probability coefficient in the PageRank score, respectively. In(u) Represents a node u The set of incoming edge nodes; Indicates the incoming edge node PageRank score; Represents a node The number of outgoing edges; C center The coordinates of the center of the canvas representing the kinship diagram; u pos Represents a node u The current coordinates;

[0098] After each round of layout calculation, the distance (in pixels) that the nodes move from their current position to their new position is calculated. When the displacement of all nodes is less than the preset pixel value, or when the preset maximum number of iterations is reached, the iteration stops and the dynamic layout of the lineage graph is completed.

[0099] The following are the technical effects of the present invention:

[0100] This invention provides clean and complete operator-level materials for language atoms by fragmenting, cleaning, and completing heterogeneous scripts. It resolves the heterogeneity of multilingual syntax through multilingual atom normalization, providing a unified syntactic logic foundation for semantic representation generation. Furthermore, it provides unambiguous and structured input for LLM propulsion through multimodal semantic anchoring and normalized semantic representation, ensuring logical consistency in lineage generation. Through the coordinated preprocessing and semantic normalization of heterogeneous scripts, multilingual and semi-structured scripts are transformed into unified and unambiguous operator semantic representations, thus solving problems such as significant differences in the structure of Abstract Syntax Trees (ASTs) of different languages ​​and the inability of semi-structured scripts to handle them.

[0101] Meanwhile, this invention employs a hybrid reasoning model combining symbolic rules and LLM (Limited Linear Modeling). First, the logic is transformed into executable rules, preventing LLM-generated lineages from contradicting the underlying logic. Then, by combining the LLM model with symbolic rule constraints, automated operator-level candidate lineage graph generation is achieved. The entire process requires no manual annotation, avoiding issues such as insufficient refinement, high labor costs, and questionable accuracy caused by manual intervention. For the candidate operator-level lineages generated by LLM reasoning, causal logic verification and lineage correction are used. Causal graphs and symbolic rules ensure the rationality and logical consistency of lineage causality, further ensuring the accuracy of lineage generation, avoiding result errors, and improving accuracy when handling complex scripts and fuzzy structures. Dynamic graph neural networks are used to achieve cross-scenario adaptive adaptation, breaking through static frameworks and adapting to dynamic scenarios with multiple sources and languages. This allows for adaptive adjustment without reconstructing the core logic when facing new languages ​​and data sources, effectively solving the problems of poor universality and insufficient scalability in existing data lineage generation methods. Finally, based on the accurate lineage graph, causal entropy-driven anomaly detection and dynamic lineage graph generation through dynamic force-oriented layout can not only accurately display lineage, but also monitor anomalies such as redundant dependencies, circular dependencies, and aggregation logic errors in real time. The entire link can be displayed intuitively through dynamic visualization, realizing the automatic, accurate, and cross-scenario generation and application of operator-level data lineage, meeting the needs of actual business scenarios such as data governance, data security, and cost optimization. Attached Figure Description

[0102] Figure 1 This is a flowchart of the method for automatically generating operator-level data lineage relationships in an embodiment of the present invention.

[0103] Figure 2 This is a diagram (a) showing an example of blood relations generated in an embodiment of the present invention.

[0104] Figure 3 This is a blood relationship example diagram (II) generated in an embodiment of the present invention. Detailed Implementation

[0105] The technical solutions in the embodiments of the present invention will be clearly and completely described below. In the following description, specific details such as specific system structures and technologies are presented for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention.

[0106] Example 1:

[0107] An automatic generation method for operator-level data lineage relationships based on a large model, comprising:

[0108] Step S1: Heterogeneous script preprocessing and semantic normalization:

[0109] Step S11: First, perform data sharding and cleaning on the input target scripting language (including SQL, Python, Flink / Scala, etc.):

[0110] First, the original script data is split into operator-level fragments based on language-specific delimiters (e.g., semicolons for SQL, newlines / indentation for Python, semicolons for Scala, etc.). S ={ s 1, s 2,…, s n Each fragment contains a single complete operator logic; then, fragments with more than 80% pure comment lines, blank line fragments, and fragments without core operators (fragments without core operators are script fragments that do not contain core operators, meaning the fragment does not perform key operations such as data extraction, filtering, aggregation, and association, and has no value for generating data lineage; for example, core operators for SQL include: SELECT (field extraction), WHERE (conditional filtering), GROUP BY (grouping), SUM / AVG / COUNT (aggregation), etc.; core operators for Python include: df.select (field extraction), df.filter (conditional filtering), df.agg (aggregation), join (association), etc.; core operators for Flink / Scala include: filter (filtering), sum (aggregation), select (field selection), etc.) are deleted; then, fragments with incomplete syntax are removed. s Fuzzy matching algorithm is used M(s) From the standard reference library S ref Matching the optimal standard fragment (Among them, the standard reference library) S ref The construction rules for the standard script reference library must meet the following requirements: it must cover more than 10 sub-languages ​​in three major categories: SQL, Python, and Flink, with each language categorized by operator type. Each category must contain at least 1000 labeled samples, and the samples must be manually verified to ensure syntactic integrity and semantic clarity.

[0111] ;

[0112] In the formula: Representing fragment s and fragment Edit distance between (i.e.) Levenshtein Distance (or distance) is used to measure the syntactic similarity between two script fragments. Indicates the corresponding fragment s or syntactic feature vectors; This represents the cosine similarity between two syntactic feature vectors. These represent the corresponding weighting coefficients (obtained through experimental calibration and empirical data, where, );

[0113] And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments s Use the effective vocabulary to complete the sentence and obtain the completed paragraph. ;

[0114] After deletion and completion, the final output is a clean and complete operator-level fragment.

[0115] Step S12: Perform multilingual grammar atomic normalization:

[0116] Constructing operator-level syntactic atoms G ={ g 1, g 2,…, g n},in, g i This represents the smallest syntactic atom (e.g., SQL's SELECT field, WHERE condition, GROUP BY aggregation, etc., and Python's df.select, df.filter, df.agg functions, etc.). G It covers core logic such as field extraction, conditional filtering, aggregation, and nested functions;

[0117] Next, the syntactic atomic mapping values ​​of different languages ​​are unified into a feature space:

[0118] ;

[0119] In the formula: Represents the syntactic atoms in language L; express The set of words included; Words v go through BERT (Bidirectional Encoder Representations from Transformers) model generates language representations. BERT The model can be a conventional model in this field, such as Microsoft / Codebert-Base, with a fixed output dimension of 768. The language type encoding of the grammatical atom (e.g., field extraction = 1, conditional filtering = 2, aggregation = 3; for example, if one-hot encoding is used and the output is 8-dimensional, then field extraction = 10000000, conditional filtering = 01000000, aggregation = 00100000, and so on). Indicates the type weight (usually 0.3); Words v In the fragment s i Number of times it appears in (fragment) s i (This refers to clean, complete operator-level fragments obtained through fragmentation and data cleaning). Excerpt s i The sum of the occurrences of all words in the text; Indicates standard reference library S ref Contains vocabulary v The number of reference fragments; Indicates the total number of reference segments included; Represents coreness weight (for core operator terms) For ordinary vocabulary For auxiliary vocabulary ).

[0120] Step S13: Finally, perform multimodal semantic anchoring and output a normalized operator semantic representation library (including operator ID, semantic vector, semantic label, consistency score, etc.):

[0121] For operators o The semantic anchoring algorithm through modal fusion integrates syntactic features ,go through BERT Contextual features generated by the model Emb(C(o)) ,go through BERT Metadata features generated by the model Emb(M(o)) The data is fused and processed using a three-layer MLP (Multi-Layer Perceptron). This embodiment employs a three-layer perceptron, including an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer comprises 768-dimensional syntactic features and 768-dimensional contextual features. C(o)+ 768 Metadata features of dimensions M(o) The first hidden layer is 512-dimensional with ReLU activation; the second hidden layer is 768-dimensional with ReLU activation; the output layer is 768-dimensional with no activation function and bias initialized to 0. The model outputs a unique semantic representation. :

[0122] ;

[0123] in: Derived from multilingual grammar atom normalization ,Right now:

[0124] ;

[0125] In the formula: G(o) Indicates from operator o The set of syntactic atoms extracted from it;

[0126] Operator context features C(o) Extract metadata features from clean, complete operator-level fragments output after sharding and data cleaning. M(o) The clean, complete operator-level fragments output after data fragmentation and cleaning, along with the grammatical atoms extracted through multilingual grammatical atom normalization, are used as indexes.

[0127] Step S2, Symbolically Enhanced LLM (Large Language Model) Lineage Reasoning:

[0128] Step S21: First, construct symbolic logic rule constraints: Define the symbolic logic rule set for operator-level lineage. R ={ r 1, r 2,…, r k},For example: r 1 is the dependency set of aggregation operators (such as SUM, AVG, COUNT, etc.), which is the union of the grouping field (o) and the aggregation field (o); r 2 is the dependency set of the conditional operator (such as WHERE, fikter, etc.), which includes all fields in the condition; r 3 represents the dependency set of nested functions;

[0129] Step S22: Generate candidate operator-level lineages through LLM inference:

[0130] The rule set is transformed into symbolic logic formulas and injected into the Prompt template of the LLM (using a conventional LLM model in this field; for example, choosing the open-source model CodeLlama-70B, with inference parameters: temperature=0.1, top_p=0.9, max_new_tokens=2048):

[0131] ;

[0132] In the formula: Template This indicates a prompt template (which contains semantic representations). Symbol rules R Output format requirements Format The prompt template is a structured text framework designed using a Large Language Model (LLM); Format Indicates the output format (including source operator, target operator, dependency set type, etc.);

[0133] Then, LLM generates a candidate lineage set using conditional probability distributions:

[0134] ;

[0135] In the formula: o b Indicates bloodline b The corresponding target operator; O Denotes the set of all operators; B c Indicates a set of candidate lineages; b This represents a single bloodline edge (i.e., the line connecting the source operator to the target operator).

[0136] Finally, construct the node set. U c Each node corresponds to the semantic representation and meta-information of the algorithm:

[0137] ;

[0138] In the formula: o id A unique identifier representing an operator; o label Semantic labels representing operators;

[0139] Construct edge set E c Each of its directed edges corresponds to B c A bloodline edge in the middle:

[0140] ;

[0141] In the formula: u s , u t Represent the source node and the target node, respectively; attr b This represents the attributes of the edge (i.e., additional information about lineage b).

[0142] Final candidate lineage chart obtained: G c =( U c , E c ).

[0143] Step S3, Causal Logic Verification and Lineage Correction: For candidate operator-level lineages, erroneous lineages are eliminated and missing lineages are supplemented through causal graph and symbolic logic verification.

[0144] Candidate lineage chart G c Construct a causal graph based on nodes and edges. Gyg = ( U , E ), preserving the semantic representation properties of operators (i.e. U = U c , E = E c );

[0145] The Depth-First Search (DFS) algorithm is used to traverse the causal graph and compute node pairs. u i , u j The set of paths (node ​​pairs) u i , u j The line connecting the two sides is called the edge.

[0146] ;

[0147] Path set reachability determination:

[0148] ;

[0149] In the formula: This represents the cosine similarity threshold (typically 0.7).

[0150] At the same time, the path length is limited to no more than 3 hops;

[0151] Symbolic logic verification:

[0152] ;

[0153] For example: rules r for u j It is an aggregation operator. u j The dependency set is the union of the operator nodes corresponding to the grouping field and the operator nodes corresponding to the aggregation field. Therefore, the source node of the current lineage edge... u i belong u j When the dependency set is defined, the rule holds true. Ver ( u i , u j , r )=1;

[0154] Traversal rule set R All applicable to u jThe rule is only valid if all rules are true. Ver ( u i , u j , r When )=1, the bloodline edge ( u i , u j ) Verification passed;

[0155] For bloodline edges that fail verification, execute the following correction logic:

[0156] when Ver ( u i , u j , r )=0 and Rex ( u i , u j When )=0, delete directly;

[0157] when Rex ( u i , u j )=1 and When necessary, supplement the edge based on rule R and label the dependency type;

[0158] when Ver ( u i , u j , r If )=1 but the dependency type is incorrect, the dependency type is corrected based on the rules.

[0159] To obtain the final, precise bloodline set:

[0160] ;

[0161] In the formula: Indicates for all r ; This indicates a missing bloodline supplement set;

[0162] At the same time, through precise bloodline collection B * Obtain accurate bloodline chart G * =( U , E * ).

[0163] Step S4, Dynamic Adaptation: Adaptive adaptation across scenes is achieved using Dynamic Graph Neural Networks (DGNNs), completing low-code extensions across different scenes.

[0164] First, obtain a precise bloodline chart. G * =( U , E * As input to the DGNN, it initializes the DGNN model, preparing it for DGNN training, including:

[0165] Node hidden state initialization: For each operator node u ∈ U Set the initial hidden state to its semantic representation:

[0166] ;

[0167] Graph structure initialization: using a precise kinship map G * =( U , E * Based on the nodes and edges of the ), the graph structure of DGNN is constructed (the dependency relationship of the edges is preserved for the information interaction of neighboring nodes).

[0168] Model parameter initialization: Aggregate weights for each neighborhood. W 1 and W 2. Attention Vector Classification layer weights W 3. Perform initialization (Xavier normal distribution initialization can be used). W 1. W 2. W 3. Initialize using a random normal distribution. );

[0169] Then, the DGNN model is trained to learn the general lineage pattern:

[0170] Perform multiple rounds of layer updates, updating the hidden state of the current node in each round by aggregating information from neighboring nodes:

[0171] ;

[0172] In the formula: Represents a node u exist t +Hidden status after the first update; ReLU This indicates a modified linear unit activation function; N(u) node u The set of neighboring nodes;b h This represents the bias term (used to improve the model's fitting ability, obtained through experimental calibration).

[0173] The attention weight (value between 0 and 1) is used to measure the neighboring nodes. For the current node u Importance:

[0174] ;

[0175] In the formula: LeakyReLU This represents the modified linear unit activation function, which is proportional to... ReLU A small negative slope is added to perform non-linear transformations on the spliced ​​features; Represents a node u In the t Wheel hidden state With neighboring nodes exist t Wheel hidden state splicing;

[0176] By employing the cross-entropy loss function, DGNN learns the differences in lineage features across different scenarios:

[0177] ;

[0178] In the formula: Represents the set of operator nodes in a precise pedigree graph. U The total quantity; Z This represents a collection of scenario categories (such as scenarios using different data processing scripting languages ​​like SQL, Python, and Flink). y u,z Represents a node u Corresponding real-world scenario category tags; Softmax Indicates the activation function; Represents a node u After the first t x The final hidden state after the DGNN layer update; b l This represents the bias term (used to improve the model's fitting ability, obtained through experimental calibration).

[0179] The Adam optimizer (learning rate 1e-4, weight decay 1e-5) is used to minimize the loss function. Loss When the training value validation set loss is less than 0.001 or the preset number of iterations (usually 100 rounds) is reached, the DGNN model training is complete.

[0180] Finally, perform dynamic adaptation for new scenarios (thereby adapting to new languages ​​or new data sources):

[0181] For the script of the new scenario, repeat step S1 to obtain the semantic representation of the operators of the new scenario. ;

[0182] Operator semantic representation of new scenarios As the initial hidden state of the nodes, it is input into the trained DGNN model and executed. t x Layer-by-layer update (usually 3 rounds) to obtain the adapted hidden state of the node. ;

[0183] Finally, the semantic similarity between the new scene and the reference scene (such as a pre-trained set of scenes) is calculated to obtain the fit coefficient:

[0184] ;

[0185] In the formula: U new This represents the combination of operator nodes in a new scenario. U ref The set of operator nodes representing the reference scenario;

[0186] A preset adaptation coefficient threshold (usually 0.6) is set. If the adaptation coefficient... If the bloodline is greater than the adaptation coefficient threshold, the existing bloodline rules are directly reused to generate a new scene bloodline; otherwise, the DGNN model parameters are adjusted (e.g., the learning rate is reduced to 5e-5), and the bloodline is generated again after supplementing the training with new scene samples.

[0187] Step S5, Intelligent Lineage Analysis and Visualization: First, anomaly detection is performed using causal entropy-driven methods (based on the accurate lineage map generated in Step S3 or the new scene lineage map generated in Step S4, where Step S3 outputs the existing scene and Step S4 outputs the new scene). Then, visualization is achieved through dynamic force-guided layout, completing the dynamic visualization of the lineage map.

[0188] Anomaly detection driven by causal entropy specifically involves:

[0189] ;

[0190] In the formula: Indicates the probability of a node appearing (based on statistics of historical normal bloodlines); Represents conditional probability (meta-operator) u i Pointing to target operator u j (normal probability)

[0191] ;

[0192] In the formula: G normol This refers to a standard lineage database (which is typically built in advance with a large amount of historical data and manually verified). Count This represents the total number of times the statistical term appears in the standard lineage database; U normal This represents the set of operator nodes in the standard lineage database;

[0193] when When this occurs, the corresponding pedigree chart is determined to be an abnormal pedigree chart; among which, Represents each normal lineage chart in the standard lineage bank causal entropy, This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The arithmetic mean; This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The sample standard deviation;

[0194] (For the precise lineage map generated in step S3 or the new scene lineage map generated in step S4, where step S3 outputs the existing scene and step S4 outputs the new scene) Visualization is achieved through dynamic force-guided layout, and the dynamic visualized lineage map is completed as follows:

[0195] ;

[0196] In the formula: F att (u) Represents a node u Attraction F rep (u) Represents a node u The repulsive force received F imp (u) Represents a node u The importance of the force received; The weighting coefficient representing importance (usually 0.4);

[0197] ;

[0198] In the formula: The weighting factor for attractiveness (usually 0.8); This indicates the nodes in the current layout. u With nodes The Euclidean distance between them; The ideal node distance represents the layout assumption (such as the average distance that we want to maintain between nodes, which is obtained through experimental calibration). The weighting coefficient representing the repulsive force (usually 0.5); This represents the smoothing coefficient (typically 1e-6). These represent the damping coefficient and probability coefficient (generally) in the PageRank score. ); In (u) Represents a node u The set of incoming edge nodes (i.e., all nodes pointing to the kinship graph) u (operator nodes); Indicates the incoming edge node PageRank score; Represents a node The number of outgoing edges (i.e., in the bloodline diagram) (Number of operator nodes pointed to); C center The coordinates of the center of the canvas representing the bloodline diagram (i.e., the visual center point of the layout). u pos Represents a node u The current coordinates;

[0199] After each round of layout calculation, the distance the nodes move from their current position to their new position is calculated (in pixels). When the displacement of all nodes is less than the preset pixel value (generally the preset pixel value is 0.1, which indicates that the layout area is stable), or when the preset maximum number of iterations is reached (generally the preset maximum number of iterations is 100 rounds to avoid infinite loops), the iteration stops and the dynamic layout of the lineage graph is completed.

[0200] Example 2:

[0201] As another preferred embodiment of the technical solution of the present invention, based on the solution of Embodiment 1 above, in step S5, after anomaly detection is performed by causal entropy driving, automatic anomaly correction is also performed, specifically as follows:

[0202] First, obtain each edge ( u i , u j Contribution to causal entropy:

[0203] ;

[0204] The larger the contribution value, the greater the impact of the corresponding edge on the anomaly;

[0205] Automatic error correction:

[0206] For circular dependency anomalies: delete the edge with the largest contribution value in the circular path, recalculate the causal entropy, and determine... If yes, the correction is complete; otherwise, continue deleting the edge with the largest contribution value, and repeat this process until... ;

[0207] For errors in aggregation logic, the dependency set of the aggregation operator is regenerated based on rule R, and the erroneous edge is directly replaced.

[0208] For redundant dependency anomalies: directly delete the redundant edge with the largest contribution value.

[0209] Example 3:

[0210] As another preferred embodiment of the technical solution of the present invention, based on the above embodiment 1 or embodiment 2, in step S5, after anomaly detection driven by causal entropy and before visualization through dynamic force-guided layout, a hierarchical causal explanation is generated, thereby generating a natural language explanation for each bloodline edge and improving interpretability, specifically:

[0211] First, define an explanation template based on the lineage granularity requirements of the actual processing scenario, including field-level, function-level, and operator-level. For example, at the field level: the target field {target_field} (operator {target_op}) depends on the source field {source_field} (operator {source_op}) because of {rule_desc}; at the function level: the target function {target_func} (operator {target_op}) depends on the output of the source function {source_func} (operator {source_op}) because of {rule_desc}; at the operator level: the target operator {target_op} (type {op_type}) core depends on {source_ops}, which conforms to the rule {rule_desc}.

[0212] Then, for each precise bloodline edge ( u i , u j (Originally obtained from a precise kinship map), extracting its causal path within the causal graph. Path x ( u i , u j and symbolic logic rules Rule ( u i , u j ), and generate "source operators" through LLM. u i→Target operator u j "A natural language explanation of this bloodline dependence:"

[0213] ;

[0214] 100 manually labeled explanatory samples were used to calculate the BLEU score between the automatically generated explanatory samples and the manually labeled samples:

[0215] ;

[0216] In the formula: len (automatically generated explanation) This indicates the length (i.e., word count) of the natural language interpretation text generated by LLM. len (Manual interpretation) Indicates the length (i.e., word count) of the manually annotated standard lineage explanation; This indicates that the matching results of syntaxes 1 to 4 are multiplied (where n=1 represents a single word, n=2 represents two consecutive words, and so on, covering semantic matching of different granularities). Count x (Matching words) This indicates the number of n-grams in the automatically generated explanation that match the manually annotated explanation; Count x (born (As a word) This indicates the total number of n-gram words in the automatically generated explanation; Represents the weight coefficients of n-grams (where, );

[0217] If the BLEU value is less than 0.8, the Prompt template is optimized and regenerated (e.g., adding manually explained examples, supplementing constraints, changing concatenated strings to structured modules, etc.); if the BLEU value is not less than 0.8, the current Prompt template is retained.

Claims

1. A method for automatically generating operator-level data lineage relationships based on a large model, characterized in that: include: Step S1, Heterogeneous Script Preprocessing and Semantic Normalization: First, the input target script language is fragmented and cleaned, specifically as follows: First, the original script data is split into operator-level fragments based on language feature delimiters. S ={ s 1, s 2,…, s n Each segment contains a single complete operator logic; then, segments with more than 80% pure comment lines, blank lines, and invalid content without a core operator are deleted; finally, for syntactically incomplete segments... s Fuzzy matching algorithm is used M(s) From the standard reference library S ref Matching the optimal standard fragment : ； In the formula: Representing fragment s and fragment The edit distance between two script fragments is used to measure the syntactic similarity between them. Indicates the corresponding segment s or syntactic feature vectors; This represents the cosine similarity between two syntactic feature vectors. These represent the corresponding weight coefficients; And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments s Use the effective vocabulary to complete the sentence and obtain the completed paragraph. ; After deletion and completion, the final output is a clean and complete operator-level fragment; Then, multilingual grammar atomic normalization is performed, specifically as follows: Constructing operator-level syntactic atoms G ={ g 1, g 2,…, g n },in, g i Represents the smallest syntactic atom; G It covers the core logic of field extraction, conditional filtering, aggregation, and nested functions; Next, the syntactic atomic mapping values ​​of different languages ​​are unified into a feature space: ； In the formula: Represents the syntactic atoms in language L; express The set of words included; Words v go through BERT Model generates language representations; Language type encoding representing syntactic atoms; Indicates type weight; Words v In the fragment s i The number of times it appears in; Excerpt s i The sum of the occurrences of all words in the text; Indicates standard reference library S ref Contains vocabulary v The number of reference fragments; Indicates the total number of reference segments included; Indicates the coreness weight; Finally, multimodal semantic anchoring is performed, and a normalized operator semantic representation library is output, specifically: For operators o The semantic anchoring algorithm through modal fusion integrates syntactic features ,go through BERT Contextual features generated by the model Emb(C(o)) ,go through BERT Metadata features generated by the model Emb(M(o)) The data is integrated and a unique semantic representation is output through a three-layer MLP model. : ； in: Derived from multilingual grammar atom normalization ,Right now: ； In the formula: G(o) Indicates from operator o The set of syntactic atoms extracted from it; Operator context features C(o) Extract metadata features from clean, complete operator-level fragments output after sharding and data cleaning. M(o) The clean and complete operator-level fragments output after data fragmentation and cleaning, as well as the grammatical atoms extracted by multilingual grammatical atom normalization, are used as indexes. Step S2, Symbolically Enhanced LLM Lineage Reasoning: First, construct symbolic logic rule constraints, and then generate candidate operator-level lineages through LLM reasoning; Step S3, Causal Logic Verification and Lineage Correction: For candidate operator-level lineages, erroneous lineages are eliminated and missing lineages are supplemented through causal graph and symbolic logic verification. Step S4, Dynamic Adaptation: Achieve cross-scene adaptive adaptation through dynamic graph neural networks to complete cross-scene low-code extension; Step S5, Intelligent Bloodline Analysis and Visualization: First, anomaly detection is performed using causal entropy-driven methods, and then visualization is achieved through dynamic force-guided layout, thus completing the dynamic visualization bloodline map.

2. The method for automatically generating operator-level data lineage relationships based on a large model according to claim 1, characterized in that: Step S2 specifically involves: First, define the symbolic logic rule set for operator-level lineage. R ={ r 1, r 2,…, r k }; Then, the rule set is transformed into a symbolic logic formula and injected into the LLM's Prompt template: ； In the formula: Template This indicates the prompt template, which is a structured text framework designed for a large language model; Format indicates the output format; Then, LLM generates a candidate lineage set using conditional probability distributions: ； In the formula: o b Indicates bloodline b The corresponding target operator; O Denotes the set of all operators; B c Indicates a set of candidate lineages; b Indicates a single bloodline edge; Finally, construct the node set. U c Each node corresponds to the semantic representation and meta-information of the algorithm: ； In the formula: o id A unique identifier representing an operator; o label Semantic labels representing operators; Construct edge set E c Each of its directed edges corresponds to B c A bloodline edge in the middle: ； In the formula: u s , u t Represent the source node and the target node, respectively; attr b Represents the attributes of an edge; Final candidate lineage chart obtained: G c =( U c , E c ).

3. The method for automatically generating operator-level data lineage relationships based on a large model according to claim 2, characterized in that: Step S3 specifically involves: Candidate lineage chart G c Construct a causal graph based on nodes and edges. G yg =( U , E ), preserving the semantic representation properties of operators; The causal graph is traversed using a depth-first search algorithm, and node pairs are calculated. u i , u j The set of paths: ； Path set reachability determination: ； In the formula: This represents the cosine similarity threshold. At the same time, the path length is limited to no more than 3 hops; Symbolic logic verification: ； Traversal rule set R All applicable to u j The rule is only valid if all rules are true. Ver ( u i , u j , r When )=1, the bloodline edge ( u i , u j ) Verification passed; Obtain the final, precise bloodline set: ； In the formula: Indicates for all r ; This indicates a missing bloodline supplement set; At the same time, through precise bloodline collection B Obtain accurate bloodline chart G =( U , E ).

4. The method for automatically generating operator-level data lineage relationships based on a large model according to claim 3, characterized in that: Step S4 specifically involves: First, obtain a precise bloodline chart. G =( U , E As input to the DGNN, it initializes the DGNN model, preparing it for DGNN training, including: Node hidden state initialization: For each operator node u ∈ U Set the initial hidden state to its semantic representation: ； Graph structure initialization: using a precise kinship map G =( U , E Based on the nodes and edges of ), the graph structure of DGNN is constructed; Model parameter initialization: Aggregate weights for each neighborhood. W 1 and W 2. Attention Vector Classification layer weights W 3. Perform initialization; Then, the DGNN model is trained to learn the general lineage pattern: Perform multiple rounds of layer updates, updating the hidden state of the current node in each round by aggregating information from neighboring nodes: ； In the formula: Represents a node u exist t +Hidden status after the first update; ReLU This represents the modified linear unit activation function; N (u) node u The set of neighboring nodes; b h Indicates the bias term; This represents the attention weight, used to measure the neighboring nodes. For the current node u Importance: ； In the formula: LeakyReLU This represents the modified linear unit activation function; Represents a node u In the t Wheel hidden state With neighboring nodes exist t Wheel hidden state splicing; By employing the cross-entropy loss function, DGNN learns the differences in lineage features across different scenarios: ； In the formula: Represents the set of operator nodes in a precise pedigree graph. U The total quantity; Z Represents a set of scene categories; y u,z Represents a node u Corresponding real-world scenario category tags; Softmax Indicates the activation function; Represents a node u After the first t x The final hidden state after the DGNN layer update; b l Indicates the bias term; Minimize the loss function using the Adam optimizer Loss When the training value validation set loss is less than 0.001 or the preset number of iterations is reached, the DGNN model training is complete. Finally, perform dynamic adaptation for new scenarios: For the script of the new scenario, repeat step S1 to obtain the semantic representation of the operators of the new scenario. ; Operator semantic representation of new scenarios As the initial hidden state of the nodes, it is input into the trained DGNN model and executed. t x Layer-by-layer update to obtain the adapted hidden state of the node. ; Finally, the semantic similarity between the new scene and the reference scene is calculated to obtain the adaptation coefficient: ； In the formula: U new This represents the combination of operator nodes in a new scenario. U ref The set of operator nodes representing the reference scenario; Preset adaptation coefficient threshold, if adaptation coefficient If the bloodline is greater than the adaptation coefficient threshold, the existing bloodline rules are directly reused to generate a new scene bloodline; otherwise, the DGNN model parameters are adjusted, and the bloodline is generated again after supplementing the training with new scene samples.

5. The method for automatically generating operator-level data lineage relationships based on a large model according to claim 4, characterized in that: In step S5, anomaly detection driven by causal entropy specifically involves: ； In the formula: Indicates the probability of a node appearing; Represents conditional probability; when When this occurs, the corresponding pedigree chart is determined to be an abnormal pedigree chart; among which, Represents each normal lineage chart in the standard lineage bank causal entropy, This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The arithmetic mean; This represents the causal entropy of all normal kinship diagrams in the standard kinship bank. The sample standard deviation; The visualization of the kinship diagram is achieved through dynamic force-guided layout, specifically as follows: ； In the formula: F att (u) Represents a node u Attraction received F rep (u) Represents a node u The repulsive force received F imp (u) Represents a node u The importance of the force received; Weighting coefficients representing the importance of something; ； In the formula: Weighting coefficients representing attractiveness; This indicates the nodes in the current layout. u With nodes The Euclidean distance between them; This represents the ideal node distance based on the layout assumption; This represents the weighting coefficient of the repulsive force; Indicates the smoothing coefficient; These represent the damping coefficient and probability coefficient in the PageRank score, respectively. In(u) Represents a node u The set of incoming edge nodes; Indicates the incoming edge node PageRank score; Represents a node The number of outgoing edges; C center The coordinates of the center of the canvas representing the kinship diagram; u pos Represents a node u The current coordinates; After each round of layout calculation, the distance the nodes move from their current position to their new position is calculated. When the displacement of all nodes is less than the preset pixel value, or when the preset maximum number of iterations is reached, the iteration stops and the dynamic layout of the lineage graph is completed.

Images