A full-link visualized traceability method and related products thereof

Abstract

The scheme discloses a full-link visualized tracing method, and steps of the method comprise the following: in response to each prediction request, a weight vector of a feature layer is calculated; based on a pre-constructed operator propagation rule library, a full-link heterogeneous graph is traversed reversely, and an indirect contribution score of each source table field in the full-link heterogeneous graph to a current decision result is obtained. The scheme can accurately identify specific Structured Query Language (SQL) operations that cause abnormal final decision weights, thereby realizing deep tracing at the operator level, and solving the problem that the prior art cannot locate processing logic errors.

Description

Technical Field

[0001] This solution relates to the field of data processing technology. More specifically, it relates to a method for end-to-end visualization and tracing of the source of artificial intelligence (AI) decision-making based on data lineage mapping, and related products. Background Technology

[0002] Currently, to explain the decision-making logic of artificial intelligence (AI) models, the industry mainly uses algorithms such as SHapley Additive exPlanations (SHAP) to calculate the contribution of input features to the prediction results. Meanwhile, to manage big data warehouses, the industry commonly uses data lineage technology to track extract-transform-load (ETL) dependencies between tables.

[0003] Although the two technologies mentioned above exist independently, there is a serious gap in their practical application, which makes it impossible for "explainability" to penetrate into the source of business. Summary of the Invention

[0004] The purpose of this invention is to provide a method for full-link visualization and tracing of AI decision-making based on data lineage mapping and related products, in order to solve the problem that existing technologies cannot locate source data processing logic errors from AI decision results.

[0005] To achieve the above objectives, the following technical solution is adopted: Firstly, this solution provides a full-chain visual tracing method, the steps of which include: Calculate the weight vector of the feature layer in response to each prediction request; Based on a pre-built operator propagation rule base, the entire heterogeneous graph is traversed in reverse to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result.

[0006] In some instances, the step of reversely traversing the entire heterogeneous graph based on a pre-built operator propagation rule base to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the current decision result includes: Starting from the feature node, traverse all upstream nodes in reverse along the entire heterogeneous graph; For any node $N$, the cumulative contribution $Score(N)$ is the weighted sum of the weights of all its downstream neighboring nodes: $Score(N) = \sum_{child \in Children(N)} (Score(child) \times \lambda_{N \to child})$; where $Children(N)$ represents the set of downstream neighboring nodes of node $N$, $child$ is any one of the downstream neighboring nodes, and $\lambda_{N \to child}$ represents the weight allocation coefficient propagated from node $N$ to node $child$; Finally, the indirect contribution score of each source table field in the graph to the decision result is calculated.

[0007] In some instances, the operator propagation rule base includes: Define the weight allocation coefficient $\lambda$ for different operator types: The direct mapping rule states that if node $A$ is directly assigned to feature $F$, then $\lambda_{A \to F} = 1.0$, i.e., $W_A = W_F$. The linear weighting rule states that if the feature $F = w_1 \cdot A + w_2 \cdot B$, then the allocation coefficient is proportional to the linear coefficient: $\lambda_{A \to F} = |w_1| / (|w_1| + |w_2|)$, that is, $W_A = W_F \cdot\lambda_{A \to F}$. Logical gating rules state that if the calculation of feature $F$ depends on the condition WHERE C > threshold (where WHERE is the conditional filtering keyword and threshold indicates setting a threshold), then field $C$ does not directly participate in numerical calculation, but determines whether the data exists; if a fixed gating coefficient $\gamma$ is set, then $W_C = W_F \cdot \gamma$.

[0008] In some instances, the steps for constructing the full-link heterogeneous map include: Genesis analysis is performed on the data to obtain a data subgraph; By fusing data subgraphs and model information, a heterogeneous graph of the entire link is obtained.

[0009] In some instances, the step of performing lineage analysis on the data to obtain a data subgraph includes: The system reads the Structured Query Language (SQL) script from the data warehouse and uses a syntax parser to generate an Abstract Syntax Tree (AST). When traversing the Abstract Syntax Tree (AST), feature operators are extracted by identifying and marking data transformation types to determine key operator nodes; Construct a data subgraph $G_{data}$, where nodes $V_{data}$ are fields or processing tasks, and edges $E_{data}$ carry operator type attributes.

[0010] In some instances, the step of fusing data subgraphs and model information to obtain a full-link heterogeneous graph includes: Read the feature engineering configuration file of the AI ​​model to obtain the model input feature list $F$.

[0011] Match the feature wide table fields at the end of the data subgraph with the input features of the model; By concatenating the model prediction result node $Y$, feature node $F$, and data subgraph $G_{data}$, a full-link heterogeneous graph is generated.

[0012] In some instances, the steps of this method include: Based on energy attenuation, path pruning is performed on the entire heterogeneous graph to obtain the critical paths of the entire heterogeneous graph; and / or, Perform hierarchical swimlane visualization rendering on the critical path of the full-link heterogeneous graph or the full-link heterogeneous graph, and generate a dynamic rendering graph with a visualization interface.

[0013] Secondly, this solution provides a full-chain visual traceability system, characterized in that the system includes: The source tracing calculation module responds to each prediction request and calculates the weight vector of the feature layer; based on the pre-built operator propagation rule library, it traverses the entire heterogeneous graph in reverse to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result. The graph construction module performs lineage analysis on the data to obtain data subgraphs; it then fuses the data subgraphs with model information to obtain a full-link heterogeneous graph; and / or, Rendering Interaction Module: Based on energy attenuation, path pruning is performed on the full-link heterogeneous graph to obtain the critical path of the full-link heterogeneous graph; layered swimlane visualization rendering is performed on the full-link heterogeneous graph or the critical path of the full-link heterogeneous graph to generate a dynamic rendering graph with a visualization interface.

[0014] Thirdly, a full-link visual traceability device includes: a memory, one or more processors; the memory and processors are connected via a communication bus; the processors are configured to execute instructions in the memory; the memory stores instructions for performing the steps of the method as described in any one of claims 1 to 7.

[0015] Fourthly, this solution provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above. The beneficial effects of this invention are as follows: This solution can accurately identify the specific SQL operation that causes the final decision weight to be abnormal, thereby achieving deep tracing at the operator level and solving the problem that existing technologies cannot locate processing logic errors.

[0016] This scheme can quantify the contribution of indirect data. By using a weighting formula based on operator type, fields in intermediate database processes that cannot be directly calculated for SHAP values ​​can also obtain interpretable contribution scores.

[0017] This solution can dynamically focus on key information. Through the energy decay pruning algorithm, the system can automatically filter out the 3-5 core data sources and their processing paths that have the greatest impact on the current AI decision-making results from thousands of data fields, which greatly reduces the cognitive burden of manual screening. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of the end-to-end visual tracing method described in this solution is shown; Figure 2 This diagram illustrates the AST parsing and operator mapping in the end-to-end visual tracing method described in this scheme. Figure 3 This diagram illustrates the principle of the backpropagation algorithm in the end-to-end visual tracing method described in this scheme. Figure 4 This diagram illustrates the end-to-end visual traceability system described in this solution. Figure 5 This diagram illustrates the logical framework of the end-to-end visual traceability system described in this solution. Figure 6 This diagram illustrates the electronic device for end-to-end visual traceability described in this solution. Detailed Implementation

[0020] To make the present invention, its technical solutions, and advantages clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0021] Analysis and research of existing technologies have revealed the following problems: Lack of a weight transfer mechanism from "features" to "source data": Existing interpretation tools can only tell users "feature X affects the result", but cannot quantify "how much field B in source table A contributes to feature X, and thus how much it contributes to the result".

[0022] Integration of heterogeneous graphs across the entire chain is difficult: data lineage graphs (e.g., with SQL tasks and tables as nodes) and model interpretation graphs (e.g., with features and weights as nodes) belong to completely different data structures and lack a unified mapping entity and computational logic.

[0023] Inability to perform in-depth root cause analysis: When AI makes incorrect decisions, business personnel cannot quickly locate which upstream cleaning logic (Filter / Join / Aggregate) caused the abnormal change in feature values ​​through a unified view.

[0024] Therefore, this solution aims to provide a full-link visual tracing method based on data lineage graphs. By constructing heterogeneous graphs and defining weighted backpropagation rules, it achieves end-to-end attribution from AI decision results to raw business data.

[0025] The following section, with reference to the accompanying figures, provides a detailed description of the proposed end-to-end visual tracing method based on data lineage mapping. Specifically, as shown in the figures... Figure 1 As shown, this end-to-end visual tracing method includes: S1. Respond to each prediction request and calculate the weight vector of the feature layer; S2. Based on the pre-built operator propagation rule base, reverse the entire heterogeneous graph to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result.

[0026] In this scheme, step S1 involves the system calculating the feature layer weight vector SHAP values ​​(SHAP values, i.e., the Shapley additive contribution values ​​of each feature) for each prediction request Instance, denoted as $W_{feature}$. Based on the weight vector, the backpropagation algorithm can be executed to calculate the indirect contribution score of each source table field to the current decision result.

[0027] Before executing the backpropagation algorithm, a propagation rules library can be constructed based on different operator types. In some instances, Rule 1 is a direct mapping: If node $A$ is directly assigned to feature node $F$, then $\lambda_{A \to F}=1.0$, that is, $W_A=W_F$.

[0028] Rule 2 is linear weighting: If feature $F = w_1\cdot A + w_2 \cdot B$, then the allocation coefficient is proportional to the linear coefficient: $\lambda_{A \to F} = |w_1| / (|w_1| + |w_2|)$, that is, $W_A = W_F \cdot \lambda_{A \to F}$.

[0029] Rule 3 is a logical gating: If the calculation of feature $F$ depends on the condition WHERE C > threshold, then although field $C$ does not directly participate in numerical calculation, it determines whether the data exists. Setting a fixed gating coefficient $\gamma$ (e.g., 0.3), then $W_C = W_F \cdot \gamma$.

[0030] S2. Based on the pre-built operator propagation rule base, reverse the entire heterogeneous graph to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result.

[0031] Specifically, starting from the feature node, we traverse all upstream nodes in reverse along the entire heterogeneous graph. For the cumulative contribution $Score(N)$ of any node $N$, we pass a weighted sum of weights to all downstream neighboring nodes of that node $N$: $Score(N) = \sum_{child \in Children(N)} Score(child) \times \lambda_{N \tochild})$.

[0032] Finally, the indirect contribution score of each source table field in the full-link heterogeneous graph to the decision result is calculated.

[0033] In some instances, such as Figure 3 As shown, data passes through the input, hidden, and output layers of the algorithm model to generate weights. The gradient of the weights is calculated using a loss function, and then the weights are adjusted using backpropagation and the error signal. Through multiple weight updates and repeated training, the algorithm ultimately outputs the indirect contribution score of each source table field in the graph to the current decision result.

[0034] In this scheme, a full-link heterogeneous graph $G(V, E)$ can be constructed based on an Abstract Syntax Tree (AST). Specifically, this can include the following two steps: 1. Perform lineage analysis on the data to obtain data subgraphs.

[0035] like Figure 2 As shown, the system reads the SQL script from the data warehouse and uses a syntax parser to generate an Abstract Syntax Tree (AST).

[0036] Feature operator extraction: During AST traversal, the data transformation type (Operator Type) is identified and labeled. The system defines three types of key operator nodes: Direct mapping (Direct_Map): For example, SELECT a AS b (that is, select field a and rename it to b, where SELECT is the query keyword and AS is the rename keyword).

[0037] Aggregation: such as SUM(a), AVG(b) (where SUM is the summation function and AVG is the average function).

[0038] Filter: such as WHERE c > 100 (that is, filter out data where field c is greater than 100).

[0039] Construct a data subgraph $G_{data}$, where nodes $V_{data}$ are fields or processing tasks, and edges $E_{data}$ carry operator type attributes.

[0040] 2. Perform entity alignment and graph fusion of data subgraphs and model information to obtain a full-link heterogeneous graph.

[0041] Read the AI ​​model's feature engineering configuration file, Feature Config, and obtain the model's input feature list $F$.

[0042] Establish anchor point connections: Match the "feature wide table field" at the end of the data subgraph with the "input features" of the model.

[0043] Constructing a fusion graph: The model prediction result node $Y$, feature node $F$, and data subgraph $G_{data}$ are concatenated to form a complete heterogeneous graph.

[0044] In this scheme, the number of nodes in the full-link heterogeneous graph is huge. In order to display important path nodes more clearly and conveniently, the path of the graph can be pruned.

[0045] Specifically, path energy calculation: The total energy $E(P)$ of a path $P = (v_1, v_2, ..., v_n)$ is defined as the product of the weights of all edges on the path and the weight of the endpoint feature.

[0046] Top-K pruning (i.e., retaining the top K paths by energy value): Set a threshold $\theta$, for example, 1% of the total weight. The system only retains the set of paths where $E(P) > \theta$. All unretained path nodes are collapsed or hidden during visualization.

[0047] In this solution, the pruning process can be visualized and rendered.

[0048] Specifically, an overall layout strategy can be formulated first. For example, three swimlanes can be set up in parallel: "source data layer", "ETL processing layer", and "model feature layer". Then, the Sugiyama layered layout algorithm can be used to ensure that the connections are free of loops and minimize intersections.

[0049] Visual encoding can be used to render detailed information within the graph. In some instances, node colors can be adjusted. For example, mapping the anomaly level of node data; a deviation of 3 times the standard deviation of the mean ($3 sigma) is displayed as dark red. In other instances, line thickness can be adjusted. For example, linearly mapping the propagation weights ($lambda) calculated using the above scheme.

[0050] Furthermore, the interaction can be designed to facilitate user observation and use. In some instances, when a user clicks on an "aggregate operator" node (such as SUM), the system dynamically expands the upstream branches of that node, displaying the detailed composition of the aggregate value.

[0051] This solution enables deep tracing at the "operator level." It goes beyond simply finding the "source table"; instead, through AST parsing and an operator propagation rule base, it precisely identifies which specific SQL operation caused the abnormal final decision weight—for example, an incorrect filtering condition or an unreasonable normalization operation. This solves the problem of existing technologies being unable to locate errors in processing logic.

[0052] This scheme can quantify the contribution of indirect data. By proposing a weighting formula based on operator type, it enables database intermediate process fields, whose SHAP values ​​cannot be directly calculated, to also have a mathematically interpretable contribution score.

[0053] This solution can dynamically focus on key information. Through the energy decay pruning algorithm, the system can automatically filter out the 3-5 core data sources and their processing paths that have the greatest impact on the current AI decision result from thousands of data fields, greatly reducing the cognitive burden of manual screening.

[0054] This solution can be applied to various scenarios that require rapid and accurate traceability, such as finance and e-commerce.

[0055] The following example further illustrates this solution.

[0056] This example applies to the end-to-end heterogeneous graph tracing of credit risk in the financial sector. It focuses on a bank using an AI model to assess customer credit risk and decide whether to grant a loan. The model's input data comes from multiple data sources, undergoes complex ETL processing, and ultimately yields a credit risk score. To ensure the reliability and interpretability of the model's decisions, the decision results need to be visualized and traced through an end-to-end heterogeneous graph.

[0057] First, using Abstract Syntax Tree (AST), a full-link heterogeneous graph is constructed based on data information from the financial sector.

[0058] 1. Analyze data lineage using script data. For example, a bank's data warehouse might contain the following SQL script: SELECT customer_info.customer_id, transaction_record.transaction_amount, AVG(transaction_record.transaction_amount) OVER (PARTITION BYcustomer_info.customer_id) AS avg_transaction_amount FROM customer_info JOIN transaction_record ON customer_info.customer_id = transaction_record.customer_id WHERE transaction_record.transaction_date > '2023-01-01' The system reads the SQL script (where customer_info is the customer information table, transaction_record is the transaction record table; SELECT, FROM, JOIN, ON, OVER, PARTITION BY, and AS are the query, specifying the source table, joining, specifying the join condition, windowing, partitioning, and renaming keywords in the SQL, respectively), and generates an AST using the syntax parser. During traversal of the AST, AVG is identified as an aggregation operator, and WHERE as a filtering operator.

[0059] Construct a data subgraph Gdata, where node Vdata includes fields such as customer_id (customer ID), transaction_amount (transaction amount), and avg_transaction_amount (average transaction amount), as well as related data processing tasks. Through feature operator extraction, determine the operator type attributes carried by edge Edata, such as AVG aggregation operator and WHERE filtering operator.

[0060] 2. Perform entity alignment and graph fusion of data subgraphs and model information to obtain a full-link heterogeneous graph. The feature engineering configuration file for the AI ​​model specifies input features including avg_transaction_amount. The system matches the "avg_transaction_amount feature wide table field" at the end of the data subgraph with the model's "input features" to establish anchor connections. The model's prediction result nodes (such as credit risk scores), feature nodes, and the data subgraph Gdata are then concatenated to form a complete end-to-end heterogeneous graph.

[0061] Then, based on the operator propagation rule base, weight backpropagation is performed on the graph.

[0062] 1. Construct an operator propagation rule base based on user credit information: Suppose there is a direct mapping relationship in the credit risk scoring model, such as a feature F1 that comes directly from the customer's age field Age, i.e., F1=Age. According to the direct mapping rule, λAge→F1=1.0, i.e., WAge=WF1.

[0063] If the credit risk scoring model has logical gating, such as only considering the impact of a customer's credit history on the score when the number of transactions exceeds 10, and a fixed gating coefficient γ=0.3 is set, if feature F is the credit risk score, which depends on the transaction count field TransactionCount and the credit history field CreditHistory, then WTransactionCount=WF⋅γ.

[0064] 2. Based on the operator propagation rule base, perform a breadth-first search (BFS) reverse traversal. Starting from the credit risk scoring node output by the model, the algorithm traverses backwards along the graph, visiting all upstream nodes. For example, for the `avg_transaction_amount` node, its cumulative contribution score `Score(avgtransactionamount)` is the weighted sum of the input weights from all its downstream neighboring nodes (such as the credit risk scoring node). Finally, the indirect contribution score of each source table field (such as `customer_id`, `transaction_amount`, etc.) in the graph to the credit risk assessment decision is calculated.

[0065] In this example, the heterogeneous graph of the entire link can be optimized. Specifically, Critical path extraction based on energy decay.

[0066] 1. Path Energy Calculation: Assume a path P=(customerid,transactionamount,avgtransactionamount,CreditRiskScore), with edge weights of λ1, λ2, and λ3, and the endpoint feature (credit risk score) weight of WCreditRiskScore. Then the total energy of the path E(P)=λ1×λ2×λ3×WCreditRiskScore.

[0067] 2. Top-K Pruning: Set a threshold θ to 1% of the total weight. The system calculates the energy of all paths and retains only the set of paths where E(P) > θ. For example, if some paths involve some irrelevant notes from customers, their path energy is low, and these path nodes will be collapsed or hidden during visualization.

[0068] Visual rendering based on layered swimlanes.

[0069] 1. Setting the layout strategy.

[0070] Multiple layers can be set up in parallel on the visual interface. These multiple layers can include: The source data layer, for example, displays customer basic information tables, transaction record tables, etc.

[0071] ETL processing layer, for example, displays data cleaning, aggregation, filtering and other operations.

[0072] The model feature layer, for example, displays the features of the input model and the final credit risk score.

[0073] Then, the Sugiyama hierarchical layout algorithm is used to ensure that the connections are free of loops and minimize intersections, so that the entire link map is clearly displayed.

[0074] 2. Design the encoding of the map from a visual perspective.

[0075] For example, consider the design of node colors. If a customer's transaction amount deviates from the mean by three times the standard deviation (3σ), the node will be displayed in dark red to alert the user that the data may be abnormal. Another example is the design of line thickness. The propagation weight λ calculated in the backpropagation of the linear mapping will be used; for instance, if a feature has a significant impact on the credit risk score, its corresponding line will be thicker.

[0076] 3. Design from the perspective of user interaction.

[0077] When a user clicks the "AVG (transaction_amount)" aggregation operator node, the system dynamically expands the upstream branches of that node, displaying the detailed transaction records that make up the average transaction amount, helping the user to gain a deeper understanding of the data composition.

[0078] Based on the aforementioned end-to-end visual traceability method, this solution further provides an end-to-end visual traceability system 301. For example... Figure 4 As shown, this module includes: a graph construction module 302, a source tracing calculation module 303, and a rendering interaction module 304. The graph construction module 302 obtains a data subgraph by performing lineage analysis on the data; it then fuses the data subgraph with model information to obtain a full-link heterogeneous graph. The source tracing calculation module 303 responds to each prediction request, calculating the weight vector of the feature layer; based on a pre-built operator propagation rule library, it traverses the full-link heterogeneous graph in reverse to obtain the indirect contribution score of each source table field in the full-link heterogeneous graph to the current decision result. Finally, the rendering interaction module 304 performs path pruning on the full-link heterogeneous graph based on energy decay to obtain the critical path of the full-link heterogeneous graph; it then performs hierarchical swimlane visualization rendering on the full-link heterogeneous graph or the critical path of the full-link heterogeneous graph to generate a dynamic rendering graph with a visual interface.

[0079] like Figure 5 As shown, SQL scripts or metadata are provided by data sources such as Customer Relationship Management (CRM) databases and log databases. The SQL scripts are processed by the AST parser in the graph construction module 302. Then, the graph construction module 302 performs the process of source table → ETL operator filtering and aggregation → feature node → model node, fusing the data subgraph and model information to obtain a full-link heterogeneous graph.

[0080] Then, using the weight calculation engine in the source tracing calculation module 303, the construction of the propagation rule base and the reverse traversal of the entire heterogeneous graph are executed respectively to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result.

[0081] Finally, the rendering interaction module 304 uses energy attenuation to perform path pruning on the full-link heterogeneous graph to obtain the critical path of the full-link heterogeneous graph; the full-link heterogeneous graph or the critical path of the full-link heterogeneous graph is then rendered in a layered swimlane to generate a dynamic rendering graph for the visualization interface, which is then provided to the front-end user.

[0082] Based on the above-described implementation of the end-to-end visual traceability method, this solution further provides a computer-readable storage medium. This computer-readable storage medium is used to implement the program product of the above-described end-to-end visual traceability method. It can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a device, such as a personal computer. However, the program product of this solution is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0083] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0084] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0085] Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0086] Program code for performing the operations of this solution can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or they can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0087] Based on the above-described implementation method for end-to-end visual traceability, this solution further provides an electronic device. For example... Figure 6 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use of this embodiment.

[0088] like Figure 6 As shown, electronic device 401 is presented in the form of a general-purpose computing device. The components of electronic device 401 may include, but are not limited to: at least one storage unit 402, at least one processing unit 403, a display unit 404, and a bus 405 for connecting different system components.

[0089] The storage unit 402 stores program code, which can be executed by the processing unit 403 to perform the steps of the various exemplary embodiments described in the above-described end-to-end visualization and tracing method. For example, the processing unit 403 can perform actions such as... Figure 1 The steps are shown in the figure.

[0090] Storage unit 402 may include volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0091] Storage unit 402 may also include programs / utilities with program modules, such program modules including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0092] Bus 405 may include a data bus, an address bus, and a control bus.

[0093] Electronic device 401 can also communicate with one or more external devices 407 (e.g., keyboard, pointing device, Bluetooth device, etc.) via input / output (I / O) interface 406. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 401, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, redundant array of independent disks (RAID) systems, tape drives, and data backup storage systems.

[0094] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A method for end-to-end visual traceability, characterized in that, The steps of this method include: Calculate the weight vector of the feature layer in response to each prediction request; Based on a pre-built operator propagation rule base, the entire heterogeneous graph is traversed in reverse to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result.

2. The end-to-end visual traceability method according to claim 1, characterized in that, The steps of reverse traversing the entire heterogeneous graph based on a pre-built operator propagation rule base to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the current decision result include: Starting from the feature node, traverse all upstream nodes in reverse along the entire heterogeneous graph; For any node $N$, the cumulative contribution $Score(N)$ is the weighted sum of the weights of all its downstream neighboring nodes: $Score(N) = \sum_{child \in Children(N)} (Score(child) \times \lambda_{N\to child})$; where $Children(N)$ represents the set of downstream neighboring nodes of node $N$, $child$ is any one of the downstream neighboring nodes, and $\lambda_{N \to child}$ represents the weight allocation coefficient propagated from node $N$ to node $child$; Finally, the indirect contribution score of each source table field in the full-link heterogeneous graph to the decision result is calculated.

3. The end-to-end visual traceability method according to claim 1 or 2, characterized in that, The operator propagation rule base includes: Define the weight allocation coefficient $\lambda$ for different operator types: The direct mapping rule states that if node $A$ is directly assigned to feature $F$, then $\lambda_{A \to F} = 1.0$, i.e., $W_A = W_F$. The linear weighting rule states that if the feature $F = w_1 \cdot A + w_2 \cdot B$, then the allocation coefficient is proportional to the linear coefficient: $\lambda_{A \to F} = |w_1| / (|w_1| + |w_2|)$, that is, $W_A = W_F \cdot \lambda_{A \to F}$. Logical gating rules state that if the calculation of feature $F$ depends on the condition WHERE C > threshold (where WHERE is the conditional filtering keyword and threshold indicates setting a threshold), then field $C$ does not directly participate in numerical calculation, but determines whether the data exists; if a fixed gating coefficient $\gamma$ is set, then $W_C = W_F \cdot \gamma$.

4. The end-to-end visual traceability method according to claim 1, characterized in that, The steps for constructing the full-link heterogeneous graph include: Genesis analysis is performed on the data to obtain a data subgraph; By fusing data subgraphs and model information, a heterogeneous graph of the entire link is obtained.

5. The end-to-end visual traceability method according to claim 4, characterized in that, The steps of performing lineage analysis on the data to obtain a data subgraph include: The system reads the Structured Query Language (SQL) script from the data warehouse and uses a syntax parser to generate an Abstract Syntax Tree (AST). When traversing the Abstract Syntax Tree (AST), feature operators are extracted by identifying and marking data transformation types to determine key operator nodes; Construct a data subgraph $G_{data}$, where nodes $V_{data}$ are fields or processing tasks, and edges $E_{data}$ carry operator type attributes.

6. The end-to-end visual traceability method according to claim 4 or 5, characterized in that, The step of fusing data subgraphs and model information to obtain a full-link heterogeneous graph includes: Read the feature engineering configuration file of the AI ​​model to obtain the model input feature list $F$; Match the feature wide table fields at the end of the data subgraph with the input features of the model; By concatenating the model prediction result node $Y$, feature node $F$, and data subgraph $G_{data}$, a full-link heterogeneous graph is generated.

7. The end-to-end visual traceability method according to claim 1, characterized in that, The steps of this method include: Based on energy attenuation, path pruning is performed on the entire heterogeneous graph to obtain the critical paths of the entire heterogeneous graph; and / or, Perform hierarchical swimlane visualization rendering on the critical path of the full-link heterogeneous map or the full-link heterogeneous map, and generate a dynamic rendering map with a visualization interface.

8. A full-chain visual traceability system, characterized in that, The system includes: The source tracing calculation module responds to each prediction request and calculates the weight vector of the feature layer; based on the pre-built operator propagation rule library, it traverses the entire heterogeneous graph in reverse to obtain the indirect contribution score of each source table field in the entire heterogeneous graph to the decision result. The graph construction module performs lineage analysis on the data to obtain data subgraphs; it then fuses the data subgraphs with model information to obtain a full-link heterogeneous graph; and / or, Rendering Interaction Module: Based on energy attenuation, path pruning is performed on the full-link heterogeneous graph to obtain the critical path of the full-link heterogeneous graph; layered swimlane visualization rendering is performed on the full-link heterogeneous graph or the critical path of the full-link heterogeneous graph to generate a dynamic rendering graph with a visualization interface.

9. A full-chain visual traceability device, characterized in that, include: Memory, one or more processors; The memory and processor are connected via a communication bus; the processor is configured to execute instructions from the memory. The memory stores instructions for performing the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images