Operator-level data blood relationship automatic generation method based on large model

Abstract

The invention provides an operator-level data consanguinity automatic generation method based on a large model, and relates to the field of electric digital data processing, and the method comprises the steps of S1, heterogeneous script preprocessing and semantic normalization, S2, symbol enhanced LLM consanguinity reasoning, S3, causal logic verification and consanguinity correction, S4, dynamic adaptation, and S5, consanguinity intelligent analysis and visualization. The method is suitable for multi-source, multi-format and multi-language data processing scenes, the automation degree, the refinement capability and the intelligent level of data blood relationship construction can be improved, and more accurate and efficient technical support is provided for data governance, data security, system migration / reconstruction, data quality management and control, cost optimization and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, and particularly relates to an operator-level data blood relationship automatic generation method based on a large model. BACKGROUND

[0002] Data lineage technology is a technology system for tracking the whole life cycle of data flow link, and its core is to record the whole link relationship of data from the source (such as original database, log file, etc.), to the processing process (such as ETL conversion, SQL calculation, etc.), and to the terminal (such as report, application system, data product, etc.). At present, data lineage technology is widely used in data governance, data security, system migration / reconstruction, data quality control, cost optimization and other scenarios.

[0003] Operation, that is, the smallest computing unit of data processing (such as SELECT, JOIN, WHERE, GROUP BY, etc.), includes direct / indirect blood relationship, filtering condition, aggregation rule, etc. Fine blood relationship refers to establishing a more fine blood processing relationship than field dependency relationship by understanding the processing range of the field, which can accurately and comprehensively display the data link.

[0004] The traditional data lineage generation method includes: a lineage extraction method based on metadata analysis, which relies on the metadata information recorded in the data platform (such as data table structure, field mapping relationship, ETL process definition, etc.), constructs a lineage graph by analyzing the dependency relationship between the metadata; a lineage extraction method based on SQL parsing, which extracts the data input and output relationship by performing syntax analysis and semantic parsing on the SQL processing script, and realizes the construction of the field-level or even operator-level lineage relationship; a method based on manual annotation or rule configuration, which manually annotates or configures rules to specify the lineage relationship for complex logic processing or unstructured scripts. Among them, the lineage extraction method based on metadata analysis can only identify the table-level or field-level lineage, and it is difficult to go deep into the operator level, and the precision and granularity are limited; the lineage extraction method based on SQL parsing is sensitive to SQL specification, has weak cross-language and cross-engine processing capability, accuracy is questionable, maintenance cost is high, and it is difficult to adapt to frequent changes in business logic; the method based on manual annotation or rule configuration has high labor cost, poor scalability, and is easily affected by subjective factors to cause inconsistent or incorrect lineage graphs. That is, the existing technology still has problems such as limited syntax analysis capability, poor compatibility (that is, the existing method is difficult to support mixed scenarios of multiple data processing languages such as SQL, Python, Spark, Flink, etc.), insufficient universality and scalability (that is, the existing method has poor portability when facing constantly changing business logic, complex scripts and multi-source heterogeneous data systems), cannot effectively understand and extend the semantic context (that is, the existing method lacks deep understanding of business semantics and context, resulting in low recognition accuracy when processing complex logic or ambiguous structure), and operator model establishment is difficult (that is, the existing method is prone to extraction failure or inaccuracy in the operator model establishment process). SUMMARY

[0005] In view of the problems existing in the prior art, the purpose of the present application is to provide an operator-level data lineage relationship automatic generation method based on a large model, which can accurately parse and understand multiple data processing language scripts, analyze the semantics of structured and semi-structured data processing scripts, adapt to multi-source, multi-format and multi-language data processing scenarios, improve the automation, refinement and intelligence of data lineage construction, and provide more accurate and efficient technical support for data governance, data security, system migration / reconstruction, data quality control, cost optimization and other scenarios.

[0006] The purpose of the present application is achieved by the following technical solutions: An operator-level data lineage relationship automatic generation method based on a large model, comprising: Step S1, Heterogeneous script preprocessing and semantic normalization: First, the input target script language is fragmented and cleaned, then multi-language syntax atomic normalization is performed, and finally multi-modal semantic anchoring and output normalization operator semantic representation library are performed. Step S2, Symbolically Enhanced LLM (Large Language Model) Lineage Reasoning: First, construct symbolic logic rule constraints, and then generate candidate operator-level lineages through LLM reasoning; Step S3, Causal Logic Verification and Lineage Correction: For candidate operator-level lineages, erroneous lineages are eliminated and missing lineages are supplemented through causal graph and symbolic logic verification. Step S4, Dynamic Adaptation: Achieve cross-scene adaptive adaptation through Dynamic Graph Neural Networks (DGNN) to complete cross-scene low-code extension; Step S5, Intelligent Bloodline Analysis and Visualization: First, anomaly detection is performed using causal entropy-driven methods, and then visualization is achieved through dynamic force-guided layout, thus completing the dynamic visualization bloodline map.

[0007] Based on further optimization of the above scheme, step S1, which involves sharding and cleaning the input target scripting language (including SQL, Python, Flink / Scala, etc.), specifically involves: First, the original script data is split into operator-level fragments based on language feature delimiters. S ={ s 1, s 2,…, s n Each segment contains a single complete operator logic; then, segments with more than 80% pure comment lines, blank lines, and invalid content without a core operator are deleted; finally, for syntactically incomplete segments... s Fuzzy matching algorithm is used M(s) From the standard reference library S ref Matching the optimal standard fragment : ; In the formula: Representing fragment s and fragment Edit distance between (i.e.) Levenshtein Distance (or distance) is used to measure the syntactic similarity between two script fragments; Indicates the corresponding fragment s or syntactic feature vectors; This represents the cosine similarity between two syntactic feature vectors. These represent the corresponding weight coefficients; And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments s Use the effective vocabulary to complete the sentence and obtain the completed paragraph. ; After deletion and completion, the final output is a clean and complete operator-level fragment.

[0008] Based on further optimization of the above scheme, step S1, specifically performing multilingual grammar atomic normalization, involves: Constructing operator-level syntactic atoms G ={ g 1, g 2,…, g n},in, g i Represents the smallest syntactic atom; G It covers core logic such as field extraction, conditional filtering, aggregation, and nested functions; Next, the syntactic atomic mapping values ​​of different languages ​​are unified into a feature space: ; In the formula: Represents the syntactic atoms in language L; express The set of words included; Words v go through BERT (Bidirectional Encoder Representations from Transformers) model generates language representations; Language type encoding representing syntactic atoms; Indicates type weight; Words v In the fragment s i The number of times it appears in; Excerpt s i The sum of the occurrences of all words in the text; Indicates standard reference library S ref Contains vocabulary v The number of reference fragments; Indicates the total number of reference segments included; This indicates the core weight.

[0009] Based on further optimization of the above scheme, step S1, specifically performing multimodal semantic anchoring and outputting a normalized operator semantic representation library, involves: For operators o, the syntax features , context features BERT generated by the model Emb(C(o)) , metadata features BERT generated by the model Emb(M(o)) are fused, and a unique semantic representation is output by a three-layer MLP (Multi-Layer Perceptron) model : ; Wherein: derived from the in multi-language syntax atom normalization, namely: ; In the formula: G(o) represents the set of syntax atoms extracted from the operator o ; Operator context features C(o) are extracted from clean and complete operator-level fragments output after sharding and data cleaning; metadata features M(o) are extracted by using the clean and complete operator-level fragments output after sharding and data cleaning and the syntax atoms extracted by multi-language syntax atom normalization as indexes.

[0010] Based on the further optimization of the above scheme, the step S2 is specifically: First, define the symbolic logic rule set of the operator-level blood relationship R ={ r 1, r 2,…, r k}; Then, convert the rule set into a symbolic logic formula and inject it into the Prompt template of LLM: ; In the formula: Template represents the prompt template (which contains semantic representation, symbolic rules and output format requirements), and the prompt template is a structured text framework designed for large language model (LLM); Format represents the output format (including source operator, target operator, dependency set type, etc.); After that, LLM generates a candidate blood relationship set through conditional probability distribution: ; In the formula: o b represents the target operator corresponding to the blood relationship b ; O represents the set of all operators; B crepresenting a candidate bloodline set; b representing a single bloodline edge (i.e. a connection line from a source operator to a target operator); Finally, a node set U c is constructed, each node of which corresponds to a semantic representation of an operator + meta information: ; In the formula: o id represents a unique identifier of an operator; o label represents a semantic label of an operator; An edge set E c is constructed, each directed edge of which corresponds to B c a bloodline edge in the candidate bloodline graph: ; In the formula: u s , u t represent a source node and a target node, respectively; attr b represents an attribute of an edge; Finally, a candidate bloodline graph is obtained: G c ( U c , E c ).

[0011] Based on further optimization of the above scheme, the step S3 is specifically: Based on the nodes and edges of the candidate bloodline graph G c , a causal graph G yg = ( U , E ) is constructed, retaining the operator semantic representation attribute; A depth-first search (DFS) algorithm is used to traverse the causal graph and calculate a path set of node pairs ( u i , u j ), the connection line between the node pairs ( u i , u j ) being an edge: ; Path set reachability judgment: ; wherein: denotes the cosine similarity threshold; while limiting the path length to no more than 3 hops; symbolic logic check: ; traverse the rule set R for all applicable u j rules of Ver ( u i , u j , r )=1, the blood relation edge u i , u j passes the check; obtain the final refined blood relation set: ; wherein: denotes that all r ; denotes the missing blood relation supplement set; while passing the refined blood relation set B * obtain the refined blood relation graph G * ( U , E * ).

[0012] Based on further optimization of the above scheme, in step S3, for the blood relation edge that fails the check, the following modification logic is performed: when Ver ( u i , u j , r )=0 and Rex ( u i , u j )=0, it is directly deleted; when Rex ( u i , u j )=1 and , the edge is supplemented based on rule R, and the dependency type is marked; when Ver ( ui , u j , r )=1 but dependent type error, the dependent type is corrected based on the rule.

[0013] Further optimization based on the above scheme, the step S4 is specifically: First, the accurate blood relationship graph G * =( U , E * ) is taken as the input of DGNN, the DGNN model is initialized, and the preparation for DGNN training is made, including: Node hidden state initialization: for each operator node u ∈ U , set the initial hidden state as its semantic representation: ; Graph structure initialization: based on the nodes and edges of the accurate blood relationship graph G * =( U , E * ), the graph structure of DGNN is constructed (the dependency relationship of the edge is reserved for neighborhood node information interaction); Model parameter initialization: initialize the neighborhood aggregation weights W 1 and W 2, attention vector , and classification layer weight W 3 respectively; Then, the DGNN model is trained to learn the general blood relationship pattern: Perform multiple rounds of layer updates, and update the hidden state of the current node through information aggregation of neighborhood nodes in each round: ; In the formula: denotes the hidden state of node u after t +1 round of update; ReLU denotes the rectified linear unit activation function; N(u) The neighborhood node set of node u ; b h denotes the bias term; denotes the attention weight, which is used to measure the importance of neighborhood node to the current node u : ; In the formula: LeakyReLUrepresents a rectified linear unit activation function, which has a small negative slope ReLU for nonlinear transformation of the concatenated features; represents a node u In the first t round hidden state and neighboring nodes In t round hidden state concatenation; Adopting a cross-entropy loss function, let DGNN learn the differences in blood features of different scenes: ; In the formula: represents the total number of operator node sets in the accurate blood graph U ; Z represents a set of scene categories; y u,z represents a node u corresponding to the real scene category label; Softmax represents an activation function; represents a node u after the first t x round DGNN layer update final hidden state; b l represents a bias term; Using the Adam optimizer to minimize the loss function Loss , the loss of the value verification set is less than 0.001 or reaches the preset iteration number, and the DGNN model training is completed; Finally, dynamic adaptation of new scenes is performed: For the script of the new scene, repeat the step S1 process to obtain the semantic representation of the operator of the new scene ; The semantic representation of the operator of the new scene is used as the initial hidden state of the node, input into the trained DGNN model, and executed t x round layer update to obtain the adapted node hidden state ; Finally, the semantic similarity between the new scene and the reference scene (such as the set of already trained scenes) is calculated to obtain the adaptation coefficient: ; In the formula: U new represents the combination of operator nodes of the new scene, U ref represents the set of operator nodes of the reference scene; The preset adaptation coefficient threshold, if the adaptation coefficient If the adaptation coefficient is greater than the threshold value, the existing blood relationship rules are directly reused to generate a new scenario blood relationship; otherwise, the DGNN model parameters are adjusted, and the blood relationship is generated after supplement training with new scenario samples.

[0014] Based on further optimization of the above scheme, the step S5 of performing abnormality detection through causal entropy driving is specifically: ; In the formula: P (X) represents the node occurrence probability; P (X|Y) represents the conditional probability; When , it is determined that the corresponding blood relationship graph is an abnormal blood relationship graph; wherein, H (X) represents the causal entropy of each normal blood relationship graph in the standard blood relationship library, H (X) represents the arithmetic mean of the causal entropies of all normal blood relationship graphs in the standard blood relationship library, H (X) represents the sample standard deviation of the causal entropies of all normal blood relationship graphs in the standard blood relationship library, H (X) represents the sample standard deviation of the causal entropies of all normal blood relationship graphs in the standard blood relationship library, H (X) represents the sample standard deviation of the causal entropies of all normal blood relationship graphs in the standard blood relationship library, The dynamic force-directed layout is used to realize visualization and complete dynamic visualization of the blood relationship graph, specifically: ; In the formula: F att (u) represents the attractive force received by the node u , F rep (u) represents the repulsive force received by the node u , F imp (u) represents the importance force received by the node u , represents the weight coefficient of the importance force; ; In the formula: represents the weight coefficient of the attractive force; represents the Euclidean distance between the node u and the node in the current layout; represents the ideal node distance assumed in the layout; represents the weight coefficient of the repulsive force; represents the smoothing coefficient; represents the damping coefficient and the probability coefficient in the PageRank score, respectively; In(u) represents the in-edge node set of the node u ; PageRank score of the in-edge node out-degree of the node C center canvas center coordinates of the blood relation graph u pos current coordinates of the node u After each round of layout calculation, the movement distance (in pixel value) of the node from the current position to the new position is calculated, and when the displacement amount of all nodes is less than a preset pixel value or a preset maximum iteration number is reached, the iteration is stopped, and the dynamic layout of the blood relation graph is completed.

[0015] The technical effects of the present application are as follows: The present application provides clean and complete operator-level materials for language atoms by sharding, cleaning and completing the heterogeneous script, provides a unified syntax logic base for semantic representation generation by normalizing and resolving the heterogeneous multi-language syntax, and provides unambiguous and structured input for LLM thrust by multi-modal semantic anchoring and normalized semantic representation, thereby ensuring the logical consistency of blood relation generation; the preprocessing and semantic normalization of the heterogeneous script are performed by the cooperation of the three, the multi-language and semi-structured script is converted into unified and unambiguous operator semantic representation, thereby solving the problems of large structural difference of Abstract Syntax Tree (AST) of different languages and inability to process semi-structured script.

[0016] ​​​Meanwhile, the application adopts a hybrid reasoning mode of symbol rule + LLM, first converts the logic into executable rules to avoid LLM from generating blood relations that do not conform to the underlying logic, and then realizes the generation of automatic operator-level candidate blood relation graphs by combining the LLM model with the symbol rule constraint, without manual annotation throughout, avoiding problems such as insufficient refinement, high labor cost and questionable accuracy caused by human involvement. For the candidate operator-level blood relations generated by LLM reasoning, the causality logic verification and blood relation correction are used to ensure the rationality and logical consistency of the blood relations by using the causal graph and the symbol rule, further ensuring the accuracy of the blood relation generation and avoiding result errors, and improving the accuracy when dealing with complex scripts and fuzzy structures. The dynamic graph neural network is used to realize adaptive adaptation across scenes, break through the static framework, adapt to multi-source and multi-language dynamic scenes, and realize adaptive adjustment without reconstructing the core logic when facing new languages and new data sources, effectively solving the problems of poor universality and insufficient extensibility in the prior art data blood relation generation method. Finally, the abnormality detection driven by causal entropy based on the accurate blood relation graph and the dynamic blood relation graph generated by dynamic force-oriented layout can not only accurately display the blood relation, but also monitor the abnormality such as redundant dependency, circular dependency and aggregation logic error in real time, and intuitively display the whole link through dynamic visualization, realizing the automatic, accurate and cross-scene generation and application of operator-level data blood relation, and meeting the needs of actual business scenarios such as data governance, data security and cost optimization. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of an operator-level data blood relation automatic generation method in an embodiment of the application.

[0018] Figure 2 A blood relation instance graph generated in an embodiment of the application (one).

[0019] Figure 3 A blood relation instance graph generated in an embodiment of the application (two). DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the application will be described clearly and completely below. In the following description, specific details such as specific system structures and technologies are given to make the application embodiments thoroughly understood, but the application is not limited thereto.

[0021] Embodiment 1: An operator-level data blood relation automatic generation method based on a large model, comprising: Step S1, heterogeneous script preprocessing and semantic normalization: Step S11, first, the input target script language (including SQL, Python, Flink / Scala, etc.) is subjected to slicing and data cleaning: First, the original script data is split into operator-level fragments based on language-specific delimiters (e.g., semicolons for SQL, newlines / indentation for Python, semicolons for Scala, etc.). S ={ s 1, s 2,…, s n Each fragment contains a single complete operator logic; then, fragments with more than 80% pure comment lines, blank line fragments, and fragments without core operators (fragments without core operators are script fragments that do not contain core operators, meaning the fragment does not perform key operations such as data extraction, filtering, aggregation, and association, and has no value for generating data lineage; for example, core operators for SQL include: SELECT (field extraction), WHERE (conditional filtering), GROUP BY (grouping), SUM / AVG / COUNT (aggregation), etc.; core operators for Python include: df.select (field extraction), df.filter (conditional filtering), df.agg (aggregation), join (association), etc.; core operators for Flink / Scala include: filter (filtering), sum (aggregation), select (field selection), etc.) are deleted; then, fragments with incomplete syntax are removed. s Fuzzy matching algorithm is used M(s) From the standard reference library S ref Matching the optimal standard fragment (Among them, the standard reference library) S ref The construction rules for the standard script reference library must meet the following requirements: it must cover more than 10 sub-languages ​​in three major categories: SQL, Python, and Flink, with each language categorized by operator type. Each category must contain at least 1000 labeled samples, and the samples must be manually verified to ensure syntactic integrity and semantic clarity. ; In the formula: Representing fragment s and fragment Edit distance between (i.e.) Levenshtein Distance (or distance) is used to measure the syntactic similarity between two script fragments; Indicates the corresponding fragment s or syntactic feature vectors; This represents the cosine similarity between two syntactic feature vectors. These represent the corresponding weighting coefficients (obtained through experimental calibration and empirical data, where, ); And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments sUse the effective vocabulary to complete the sentence and obtain the completed paragraph. ; After deletion and completion, the final output is a clean and complete operator-level fragment.

[0022] Step S12: Perform multilingual grammar atomic normalization: Constructing operator-level syntactic atoms G ={ g 1, g 2,…, g n},in, g i This represents the smallest syntactic atom (e.g., SQL's SELECT field, WHERE condition, GROUP BY aggregation, etc., and Python's df.select, df.filter, df.agg functions, etc.). G It covers core logic such as field extraction, conditional filtering, aggregation, and nested functions; Next, the syntactic atomic mapping values ​​of different languages ​​are unified into a feature space: ; In the formula: Represents the syntactic atoms in language L; express The set of words included; Words v go through BERT (Bidirectional Encoder Representations from Transformers) model generates language representations. BERT The model can be any conventional model in this field, such as Microsoft / Codebert-Base, with a fixed output dimension of 768. The language type encoding of the grammatical atom (e.g., field extraction = 1, conditional filtering = 2, aggregation = 3; for example, if one-hot encoding is used and the output is 8-dimensional, then field extraction = 10000000, conditional filtering = 01000000, aggregation = 00100000, and so on). Indicates the type weight (usually 0.3); Words v In the fragment s i Number of times it appears in (fragment) s i (This refers to clean, complete operator-level fragments obtained through fragmentation and data cleaning). Excerpt s i The sum of the occurrences of all words in the text; denotes the standard reference library S ref contains the vocabulary v of the reference fragment; denotes the total number of contained reference fragments; denotes the core degree weight (for core operator vocabulary , for common vocabulary , for auxiliary vocabulary ).

[0023] Step S13, finally, multi-modal semantic anchoring is performed, and a normalized operator semantic representation library (containing operator ID, semantic vector, semantic label, consistency score, etc.) is output: for the operator o , the syntax features , the context features BERT generated by the Emb(C(o)) model, and the metadata features BERT generated by the Emb(M(o)) model are fused through a semantic anchoring algorithm based on modal fusion, and a unique semantic representation is output through a three-layer MLP (Multi-Layer Perceptron) model (in this embodiment, a three-layer perceptron is used, including an input layer, a first hidden layer, a second hidden layer, and an output layer, wherein the input layer includes 768-dimensional syntax features + 768-dimensional context features C(o)+ 768 768-dimensional metadata features M(o) ; the first hidden layer is 512-dimensional with a ReLU activation function; the second hidden layer is 768-dimensional with a ReLU activation function; and the output layer is 768-dimensional without an activation function, with a bias initialized to 0): ; wherein: is derived from the in multi-language syntax atom normalization, i.e.: ; in the formula: G(o) denotes the set of syntax atoms extracted from the operator o ; operator context features C(o) are extracted from clean and complete operator-level fragments output after sharding and data cleaning; and metadata features M(o) are extracted by using clean and complete operator-level fragments output after sharding and data cleaning and syntax atoms extracted through multi-language syntax atom normalization as indexes.

[0024] ​Step S2, symbol-enhanced LLM (Large Language Model) bloodline inference: Step S21, first build symbol logic rule constraints: define the symbol logic rule set of operator-level bloodlines R ={ r 1, r 2,…, r k}, for example: r 1 is the dependency set of aggregation operators (such as SUM, AVG, COUNT, etc.), which is the union set of grouping fields (o) and aggregation fields (o); r 2 is the dependency set of conditional operators (such as WHERE, filter, etc.), which includes all fields in the condition; r 3 is the dependency set of nested functions; Step S22, then generate candidate operator-level bloodlines through LLM inference: Convert the rule set into a symbol logic formula and inject it into the Prompt template of LLM (use a conventional LLM model in the field; for example: select the open-source model CodeLlama-70B, inference parameters: temperature=0.1, top_p=0.9, max_new_tokens=2048): ; In the formula: Template represents the prompt template (which contains semantic representations , symbol rules R , output format requirements Format ), and the prompt template is a structured text framework designed for large language models (LLM); Format represents the output format (including source operators, target operators, dependency set types, etc.); After that, LLM generates a candidate bloodline set through conditional probability distribution: ; In the formula: o b represents the bloodline b corresponding to the target operator; O represents the set of all operators; B c represents the candidate bloodline set; b represents a single bloodline edge (i.e., a connection line from the source operator to the target operator); Finally, build a node set U c , each node of which corresponds to the semantic representation + meta information of an algorithm: ; In the formula: oid a unique identifier of the operator; o label a semantic label of the operator; constructing an edge set E c each directed edge of which corresponds to B c one blood edge in the blood graph: ; wherein: u s , u t respectively represent the source node and the target node; attr b represents the attribute of the edge (i.e. additional information of the blood edge b); finally obtaining a candidate blood graph: G c = U c , E c ).

[0025] Step S3, causal logic verification and blood correction: for the candidate operator-level blood, through the causal graph and symbolic logic verification, the wrong blood is removed and the missing blood is supplemented: based on the nodes and edges of the candidate blood graph G c , a causal graph G yg = ( U , E ) is constructed, and the operator semantic representation attribute (i.e. U = U c , E = E c ) is retained; a depth-first search (DFS) algorithm is used to traverse the causal graph and calculate the path set of the node pair ( u i , u j ) (the connection between the node pair ( u i , u j ) is the edge): ; path set reachability judgment: ; wherein: cosine similarity threshold (usually 0.7); At the same time, the path length is limited to no more than 3 hops; Symbolic logic verification: ; For example: rule r is u j is an aggregation operator, u j The dependency set of is the union of the operator nodes corresponding to the grouping field and the operator nodes corresponding to the aggregation field. When the source node of the current blood edge u i belongs to u j the dependency set of, the rule is true, Ver ( u i , u j , r )=1; Traverse all rules in the rule set R that apply to u j , only when the of all rules Ver ( u i , u j , r )=1, the blood edge u i , u j passes the verification; For blood edges that fail the verification, the following correction logic is executed: When Ver ( u i , u j , r )=0 and Rex ( u i , u j )=0, it is directly deleted; When Rex ( u i , u j )=1 and , based on rule R, the edge is supplemented, and the dependency type is marked; When Ver ( u i , uj , r )=1 but dependent type is wrong, correct the dependent type based on the rule.

[0026] Obtain the final accurate blood relationship set: ; In the formula: represents all r ; represents the missing blood relationship supplement set; At the same time, through the accurate blood relationship set B * Obtain the accurate blood relationship graph G * =( U , E * ).

[0027] Step S4, dynamic adaptation: realize cross-scene adaptive adaptation through Dynamic Graph Neural Networks (DGNN), complete cross-scene low-code expansion: First, the accurate blood relationship graph G * =( U , E * ) is taken as the input of the DGNN, the DGNN model is initialized, and the preparation for DGNN training is made, including: Node hidden state initialization: for each operator node u ∈ U , set the initial hidden state to its semantic representation: ; Graph structure initialization: based on the nodes and edges of the accurate blood relationship graph G * =( U , E * ), the graph structure of the DGNN is constructed (the dependency relationship of the edge is reserved, used for neighborhood node information interaction); Model parameter initialization: initialize the neighborhood aggregation weights W 1 and W 2, attention vectors , and classification layer weights W 3 respectively (Xavier normal distribution initialization W 1, W 2, W 3 can be used, and random normal distribution initialization ) ; Then, the DGNN model is trained to learn the general blood relationship pattern: Multiple rounds of layer updates are performed, and each round updates the hidden state of the current node by aggregating information from neighboring nodes: ; In the formula: represents the node u After the t +1 round of update of the hidden state; ReLU represents the rectified linear unit activation function; N(u) node u The set of neighboring nodes of b h represents the bias term (used to improve the fitting ability of the model, obtained by experimental calibration); represents the attention weight (value between 0 and 1), which is used to measure the importance of the neighboring node to the current node u : ; In the formula: LeakyReLU represents the rectified linear unit activation function, which has a small negative slope compared to ReLU , which is used for nonlinear transformation of the concatenated features; represents the node u in the t round hidden state and the neighboring node in the t round hidden state ; The cross-entropy loss function is used to let the DGNN learn the differences in blood relationship features in different scenarios: ; In the formula: represents the total number of operator node sets in the accurate blood relationship graph U ; Z represents the set of scenario categories (such as different data processing script language scenarios such as SQL, Python, Flink, etc.); y u,z represents the true scenario category label corresponding to the node u ; Softmax represents the activation function; represents the final hidden state of the node u after the t x round DGNN layer update; b l represents the bias term (used to improve the fitting ability of the model, obtained by experimental calibration); The loss function is minimized using the Adam optimizer (learning rate 1e-4, weight decay 1e-5) Loss , the value verification set loss is less than 0.001 or reaches the preset iteration number (usually 100 rounds), and the DGNN model training is completed. Finally, the new scene dynamic adaptation (to adapt to new languages or new data sources) is performed: For the script of the new scene, repeat the process of step S1 to obtain the semantic representation of the operator of the new scene ; The semantic representation of the operator of the new scene is taken as the initial hidden state of the node, and is input into the trained DGNN model to perform t x round layer update (usually 3 rounds) to obtain the adapted node hidden state ; Finally, the semantic similarity between the new scene and the reference scene (such as the already trained scene set) is calculated to obtain the adaptation coefficient: ; In the formula: U new represents the combination of the operator nodes of the new scene, U ref represents the set of operator nodes of the reference scene; The preset adaptation coefficient threshold (usually 0.6) is greater than the adaptation coefficient threshold. If the adaptation coefficient is greater than the adaptation coefficient threshold, the existing blood relationship rules are directly reused to generate the blood relationship of the new scene; otherwise, the DGNN model parameters are adjusted (for example, the learning rate is reduced to 5e-5) and the blood relationship is generated after supplementing training with new scene samples.

[0028] Step S5, intelligent analysis and visualization of blood relationship: first, abnormality detection is performed through causal entropy driving (on the accurate blood relationship graph generated in step S3 or the new scene blood relationship graph generated in step S4, wherein step S3 outputs an existing scene and step S4 outputs a new scene), and then visualization is realized through dynamic force guiding layout to complete the dynamic visualization blood relationship graph: The abnormality detection through causal entropy driving is specifically: ; In the formula: represents the node occurrence probability (based on historical normal blood relationship statistics); represents the conditional probability (meta-operator u i points to the target operator u j normal probability); ; In the formula:G normol represents a standard bloodline library (which can be constructed in advance with a large amount of historical data and verified by humans); Count represents the total number of times the calculation subnode appears in the standard bloodline library; U normal represents a set of calculation subnode nodes in the standard bloodline library; when , it is determined that the corresponding bloodline graph is an abnormal bloodline graph; wherein, represents the causal entropy of each normal bloodline graph in the standard bloodline library, represents the arithmetic mean of the causal entropy of all normal bloodline graphs in the standard bloodline library; represents the sample standard deviation of the causal entropy of all normal bloodline graphs in the standard bloodline library; (For the accurate bloodline graph generated in step S3 or the new scene bloodline graph generated in step S4, wherein step S3 outputs an existing scene and step S4 outputs a new scene) visualization is achieved by dynamic force-directed layout, and the dynamic visual bloodline graph is specifically: ; In the formula: F att (u) represents the attractive force received by the node u , F rep (u) represents the repulsive force received by the node u , F imp (u) represents the importance force received by the node u ; represents the weight coefficient of the importance force (generally 0.4); ; In the formula: represents the weight coefficient of the attractive force (generally 0.8); represents the Euclidean distance between the node u and the node in the current layout; represents the ideal node distance assumed by the layout (such as the average distance that the nodes are expected to maintain, which is obtained by experiment calibration); represents the weight coefficient of the repulsive force (generally 0.5); represents a smoothing coefficient (generally 1e-6); respectively represent the damping coefficient and the probability coefficient in the PageRank score (generally );​​In (u) set of in-edge nodes of the node (i.e. all operator nodes pointing to u in the pedigree graph); u PageRank score of the in-edge node ; number of out-edges of the node (i.e. number of operator nodes pointing from in the pedigree graph); C center canvas center coordinate of the pedigree graph (i.e. visual center point of the layout); u pos current coordinate of the node u ; distance of the node from the current position to the new position after each round of layout calculation (unit: pixel value), when the displacement of all nodes is less than a preset pixel value (generally, the preset pixel value is 0.1, and the displacement less than the preset pixel value indicates that the layout area is stable), or the preset maximum iteration number is reached (generally, the preset maximum iteration number is 100 rounds, to avoid infinite loop), the iteration is stopped, and the dynamic layout of the pedigree graph is completed.

[0029] Embodiment 2: As another preferred embodiment of the technical scheme of the application, on the basis of the scheme in Embodiment 1, after the abnormality detection by the causal entropy driving in step S5, automatic correction of the abnormality is further performed, specifically as follows: First, the contribution value of each edge (i, j) to the causal entropy is obtained: u i , u j ; The greater the contribution value, the greater the influence of the corresponding edge on the abnormality; Automatic correction of the abnormality: For the circular dependency abnormality: the edge with the largest contribution value in the circular path is deleted, the causal entropy is recalculated, and is determined, if yes, the correction is completed, if no, the edge with the largest contribution value is continuously deleted, and the cycle is continued until ; For the aggregation logic error, the dependency set of the aggregation operator is regenerated based on the rule R, and the error edge is directly replaced; For the redundant dependency abnormality: the redundant edge with the largest contribution value is directly deleted.

[0030] Embodiment 3: ​​As another preferred embodiment of the technical solution of the present invention, based on the above embodiment 1 or embodiment 2, in step S5, after anomaly detection driven by causal entropy and before visualization through dynamic force-guided layout, a hierarchical causal explanation is generated, thereby generating a natural language explanation for each bloodline edge and improving interpretability, specifically: First, define an explanation template based on the lineage granularity requirements of the actual processing scenario, including field-level, function-level, and operator-level. For example, at the field level: the target field {target_field} (operator {target_op}) depends on the source field {source_field} (operator {source_op}) because of {rule_desc}; at the function level: the target function {target_func} (operator {target_op}) depends on the output of the source function {source_func} (operator {source_op}) because of {rule_desc}; at the operator level: the target operator {target_op} (type {op_type}) core depends on {source_ops}, which conforms to the rule {rule_desc}. Then, for each precise bloodline edge ( u i , u j (Originated from a precise kinship map), extract its causal path within the causal graph. Path x ( u i , u j and symbolic logic rules Rule ( u i , u j ), and generate "source operators" through LLM. u i →Target operator u j "A natural language explanation of this bloodline dependence:" ; 100 manually labeled explanatory samples were used to calculate the BLEU score between the automatically generated explanatory samples and the manually labeled samples: ; In the formula: len(Autogenerated explanation) This indicates the length (i.e., word count) of the natural language interpretation text generated by LLM. len (Artificial explanation) Indicates the length (i.e., word count) of the manually annotated standard lineage explanation; denotes the product of the matching results of 1-4 grammar (where n=1 represents a single word, n=2 represents two consecutive words, and so on, covering semantic matching of different granularities); Count x (Matched words) denotes the number of n-grams in the automatically generated explanation that match the human-annotated explanation; Count x (Generated words) ​ denotes the total number of n-grams in the automatically generated explanation; denotes the weight coefficient of the n-gram (where, ); If the BLEU value is less than 0.8, the Prompt template is optimized for regeneration (for example: adding human explanation examples, supplementing constraints, changing the spliced string to a structured module, etc.); if the BLEU value is not less than 0.8, the current Prompt template is retained.

Claims

1. A large model-based operator-level data blood relationship automatic generation method, characterized in that: Comprise: Step S1, heterogeneous script preprocessing and semantic normalization: first, the input target script language is divided and data is cleaned, then the multi-language syntax atom is normalized, and finally the multi-modal semantic anchor is output normalized operator semantic representation library; Step S2, symbol enhanced LLM blood relation reasoning: first, the symbol logic rule constraint is constructed, and then the candidate operator level blood relation is generated by LLM reasoning; Step S3, causal logic verification and blood relation correction: for the candidate operator level blood relation, the error blood relation is removed and the missing blood relation is supplemented through causal graph and symbol logic verification; Step S4, dynamic adaptation: through dynamic graph neural network, cross-scene adaptive adaptation is realized, and cross-scene low code expansion is completed; Step S5, blood relation intelligent analysis and visualization: first, abnormality detection is performed through causal entropy driving, and then visualization is realized through dynamic force directed layout, and dynamic visual blood relation graph is completed.

2. The method of claim 1, wherein the method is characterized by: In step S1, the input target script language is divided and data is cleaned, which is: Firstly, the original script data is split into operator-level segments based on language characteristics delimiters S ={ s 1, s 2,…, s n}, each segment contains a single complete operator logic; after that, segments with pure comment lines accounting for more than 80%, empty line segments and invalid content without core operators are deleted; then, for the syntax incomplete segment s , the fuzzy matching algorithm M(s) matches the optimal standard segment S ref from the standard reference library : ； In the formula: represents the edit distance between the segment s and the segment , which is used to measure the syntactic similarity of the two script segments; represents the syntactic feature vector of the corresponding segment s or ; represents the cosine similarity of the two syntactic feature vectors; respectively represents the corresponding weight coefficient; And extract the best matching standard fragment. Syntactic skeleton, filling in missing fragments s Use the effective vocabulary to complete the sentence and obtain the completed paragraph. ; After deletion and completion, the final output is clean and complete operator level segment.

3. The method of claim 2, wherein the method is characterized by: In step S1, the multi-language syntax atom normalization is: Constructing a set of operator-level syntax atoms G ={ g 1, g 2,…, g n}, wherein g i denotes the minimum syntax atom; G Core logic encompassing field extraction, conditional filtering, aggregation, nested functions; Then, the mapping values of syntax atoms of different languages are unified in the feature space: ； wherein: represents a syntactic atom in language L; represents a set of lexemes contained; represents a lexeme v passes through BERT a model to generate a language representation; represents a language type encoding of a syntactic atom; represents a type weight; represents a lexeme v in a segment s i occurs; a segment s i occurs; represents a standard reference library S ref contains the lexeme v ; represents the total number of reference segments contained; represents a coreness weight.

4. The method of claim 3, wherein the method is characterized by: In step S1, the multi-modal semantic anchor is output normalized operator semantic representation library, which is: For operators o The semantic anchoring algorithm through modal fusion integrates syntactic features ,go through BERT Contextual features generated by the model Emb(C(o)) ,go through BERT Metadata features generated by the model Emb(M(o)) The data is integrated and a unique semantic representation is output through a three-layer MLP model. : ； wherein: derived from the normalization of the multilingual grammar atoms i.e.: ； In the formulae: G(o) denotes the set of syntax atoms extracted from the operator o from the operator Operator context features C(o) Extract metadata features from clean, complete operator-level fragments output after sharding and data cleaning. M(o) The clean, complete operator-level fragments output after data fragmentation and cleaning, along with the grammatical atoms extracted through multilingual grammatical atom normalization, are used as indexes.

5. The method of claim 4, wherein the method is characterized by: The step S2 is: First, define the set of symbolic logic rules for operator-level consanguinity R ={ r 1, r 2,…, r k} Then, the rule set is converted into a symbolic logic formula and injected into the Prompt template of LLM: ； In the formula: Template represents a prompt template, the prompt template being a structured text framework designed for a large language model; Format represents an output format; After that, LLM generates a candidate blood relation set through conditional probability distribution: ； wherein: o b denotes a blood relation b a corresponding target operator; O denotes a set of all operators; B c denotes a set of candidate blood relations; b denotes a single blood relation edge; Finally, construct the node set U c Each node corresponds to the semantic representation of the algorithm + meta information: ； In the formulae: o id represents a unique identifier of the operator; o label represents a semantic label of the operator; Constructing edge sets E c each directed edge of which corresponds to B c one blood edge: ； In the formula: u s , u t respectively represent the source node and the target node; attr b represent the attribute of the edge; A candidate genealogy is finally obtained: G c ( U c , E c ).

6. The method of claim 5, wherein the method is characterized by: The step S3 is: Based on the nodes and edges of the candidate pedigree G c Causal graphs are constructed G yg = ( U , E ), preserving operator semantic representation properties; The depth-first search algorithm is used to traverse the causal graph and calculate the path set of node pairs (i, j) u i , u j ) ； Path set reachability judgment: ； In the formula: denotes a cosine similarity threshold value; At the same time, the path length is limited to no more than 3 hops; Symbolic logic verification: ； Traversal ruleset R All rules applicable in u j The rule is only applicable if all rules have Ver ( u i , u j , r )=1 u i , u j Passes the check Obtain the final accurate blood relation set: ； wherein: represents for all r ; represents the missing blood relation supplementary set; At the same time through accurate blood collection B * Obtaining accurate blood graph G * ( U , E * ).

7. The method of claim 6, wherein the method is characterized by: The step S4 is: First, the accurate genealogy graph G * =( U , E * ) is taken as the input of the DGNN, the DGNN model is initialized, and the DGNN training is prepared, including: Node hidden state initialization: For each operator node u ∈ U , set the initial hidden state to its semantic representation: ； Graph structure initialization: with accurate pedigree graph G * =( U , E * ) nodes, edges as the basis, build the graph structure of DGNN; Model parameter initialization: the neighborhood aggregation weights W 1 and W 2, the attention vectors , and the classification layer weights W 3 are initialized, respectively; Then, DGNN model training is performed to learn the general blood relation pattern: Perform multiple rounds of layer update, and update the hidden state of the current node through information aggregation of neighbor nodes in each round: ； wherein: represents a node u In t the hidden state after +1 round update; ReLU represents a rectified linear unit activation function N (u) a node u a set of neighboring nodes of the node b h represents a bias term denotes the attention weight, which measures the importance of the neighborhood nodes to the current node u : ； In the formula: LeakyReLU denotes a rectified linear unit activation function; denotes a node u In the first t round hidden state with neighborhood nodes In t round hidden state concatenation; Cross-entropy loss function is adopted to let DGNN learn the blood relation feature differences of different scenes: ； In the formula: denotes the total number of the set of operator nodes in the accurate pedigree diagram U ; Z denotes the set of scene categories; y u,z denotes the node u corresponding to the real scene category label; Softmax denotes the activation function; denotes the node u after the first t x round of DGNN layer update final hidden state; b l denotes the bias term; Minimizing the loss function with an Adam optimizer Loss The DGNN model training is completed when the loss of the training value verification set is less than 0.001 or a preset iteration number is reached. Finally, new scene dynamic adaptation is performed: For the script of the new scene, repeat step S1 process, obtain the semantic representation of the new scene operator ; Representing operator semantics of new scenarios As the initial hidden state of the node, input the trained DGNN model, execute t x Wheel layer update, get the adapted node hidden state ; Finally, the semantic similarity between the new scene and the reference scene is calculated to obtain the adaptation coefficient: ； In the formula: U new a set of operator nodes representing new scenarios is combined, U ref a set of operator nodes representing reference scenarios; Preset adaptation coefficient threshold, if adaptation coefficient If the bloodline is greater than the adaptation coefficient threshold, the existing bloodline rules are directly reused to generate a new scene bloodline; otherwise, the DGNN model parameters are adjusted, and the bloodline is generated again after supplementing the training with new scene samples.

8. The method of claim 7, wherein the method is characterized by: In step S5, abnormality detection is performed through causal entropy driving, which is: ； In the formula: denotes the probability of node occurrence; denotes the conditional probability; When , it is determined that the corresponding blood relationship graph is an abnormal blood relationship graph; wherein, represents the causal entropy of each normal blood relationship graph in the standard blood relationship library , represents the arithmetic mean of the causal entropies of all normal blood relationship graphs in the standard blood relationship library ; represents the sample standard deviation of the causal entropies of all normal blood relationship graphs in the standard blood relationship library ; Through dynamic force directed layout, visualization is realized, and dynamic visual blood relation graph is completed, which is: ； wherein: F att (u) represents a node u experienced attractive force, F rep (u) represents a node u experienced repulsive force, F imp (u) represents a node u experienced importance force; represents a weight coefficient of the importance force; ； In the formula: represents the weight coefficient of attraction; represents the Euclidean distance between the node u and the node in the current layout; represents the ideal node distance of the layout assumption; represents the weight coefficient of repulsion; represents the smoothing coefficient; respectively represent the damping coefficient and the probability coefficient in the PageRank score; In(u) represents the in-edge node set of the node u ; represents the PageRank score of the in-edge node ; represents the out-edge number of the node ; C center represents the canvas center coordinates of the blood relationship graph; u pos represents the current coordinates of the node u ; After each round of layout calculation, the movement distance of the calculation node from the current position to the new position is calculated. When the displacement of all nodes is less than the preset pixel value, or the preset maximum iteration number is reached, the iteration is stopped, and the blood relation graph dynamic layout is completed.

Images