Multi-dimensional data blood relationship tracking method and system

Abstract

The invention relates to a multi-dimensional data blood relationship tracking method. The method comprises the following steps: S1, data blood relationship meta-information collection; s2, constructing a multi-dimensional blood relationship; s3, blood relationship analysis; s4, an intelligent early warning mechanism; s5, effect verification is realized; a multi-dimensional data blood relationship tracking system comprises a meta-information acquisition module, a blood relationship construction module, a blood relationship analysis module, an intelligent early warning module and an effect verification module. The method has the advantages of high blood relationship tracking accuracy and high analysis capability.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a multi-dimensional data blood relationship tracking method and system. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, the scale of enterprise data has exploded, and data processing scenarios are becoming increasingly complex. In modern data architecture, data needs to go through multiple links such as collection, cleaning, conversion, calculation, and analysis, involving multiple heterogeneous systems such as data warehouses, data lakes, and stream processing platforms. In the process of data circulation and conversion between these systems, its blood relationship becomes more and more complex, which brings huge challenges to data governance.

[0003] In the existing technology, data lineage tracking mainly has the following problems:

[0004] (1) Tracking dimension limitations: Traditional lineage analysis is mainly limited to static associations at the data table or field level and cannot effectively track the business semantics and multi-dimensional attributes during data flow.

[0005] (2) Insufficient accuracy of bloodline identification: Existing solutions mainly rely on static parsing of ETL jobs and SQL statements, and have limited support for complex data conversion logic and real-time processing scenarios.

[0006] (3) Insufficient real-time performance: Traditional blood relationship analysis mostly uses a batch offline method, and blood relationship updates have a significant time delay, which makes it difficult to meet the needs of real-time data processing scenarios.

[0007] (4) Insufficient intelligence: It lacks the ability to intelligently identify abnormal blood relationships and is unable to automatically discover and warn of potential data quality risks.

[0008] The mainstream lineage analysis solutions on the market (including open source solutions, commercial solutions and self-developed solutions) generally have problems such as limited scalability, obvious performance bottlenecks, and high integration costs, which are difficult to meet the actual needs of modern data governance. Therefore, there is an urgent need for a data lineage relationship tracking solution that can support multi-dimensional, high-precision, and real-time data. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by this patent application is how to provide a multi-dimensional data blood relationship tracking method and system with high blood relationship tracking accuracy and strong analysis capabilities.

[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0011] A multi-dimensional data blood relationship tracking method includes the following steps: S1: data blood relationship metadata collection; S2: multi-dimensional blood relationship construction; S3: blood relationship analysis; S4: intelligent early warning mechanism; S5: implementation effect verification.

[0012] As an optimization, in step S1, when collecting data lineage metadata, the table structure information, data source configuration information, access permission information and scheduling strategy information are obtained through the preset data collection interface; the collected metadata is verified for integrity, consistency, validity and accuracy, and a standardized quality verification report is generated.

[0013] As an optimization, in step S2, when constructing multi-dimensional blood relationships, a blood relationship map is constructed based on the collected metadata, and the nodes in the map include database nodes, data table nodes, field nodes, task nodes, and processing process nodes, and the edges include subordinate relationships, derived relationships, dependent relationships, and transformation relationships; the adjacency matrix is ​​used to represent the blood relationship, and the edge weights are calculated by combining the time decay factor, access frequency factor, data volume factor, and business importance factor, and the importance score of the node is calculated through an iterative algorithm; a multi-level index system including primary key index, type index, and time series index is established to optimize the query and analysis efficiency of blood relationships.

[0014] As an optimization, in step S3, when analyzing blood relationships, first, based on the blood relationship map, a hierarchical influence propagation model is constructed to quantify the influence intensity between nodes and identify key propagation paths; then a third-order feature tensor model is constructed to extract multidimensional characteristics including node features, relationship features and time features, and a tensor decomposition method is used to achieve dimensionality reduction; finally, the support and confidence methods are used to mine frequent patterns, and the lift is calculated to identify the association rules between data; a trend prediction function is constructed based on the autoregressive model and external features to predict the changing trend of data relationships.

[0015] As an optimization, in step S4, when building an intelligent early warning mechanism, first build an anomaly scoring model for topological structure, time series pattern and multi-dimensional attributes to evaluate the degree of abnormality of data nodes and edges; then, based on the anomaly scoring results, comprehensively calculate the risk level of the node, and quantify the impact range by evaluating its correlation with other nodes; finally, generate early warning information, including the unique identifier of the warning event, abnormal node information, risk level, impact range and targeted processing suggestions, and deliver it to the responsible person through the designated distribution channel, and record the warning status and processing process.

[0016] As an optimization, in step S5, when performing effect verification, the accuracy of blood relationship tracking is evaluated using precision, recall and F1 score; the system's performance in response time, throughput and concurrent processing capability is tested to verify its real-time performance; the system's scalability under different data scales and concurrent numbers is evaluated; the stability of the system is verified through long-term operation and abnormal recovery tests; and a performance comparison analysis is conducted with existing methods using relative improvement rate and multi-indicator weighted scoring.

[0017] As an optimization, the time decay factor in the weight calculation measures the decay effect of the relationship over time, the access frequency factor reflects how frequently the data is accessed, the data volume factor represents the transmission volume of data flow, and the business importance factor represents the intensity of business dependence on the data relationship. The sum of the weights of each factor is 1.

[0018] As an optimization, the trend prediction function adopts the form of combining autoregressive terms and external features, and predicts the future change trend of data relationships based on historical data and external influencing factors.

[0019] As an optimization, the generation of the warning information is based on the risk level, abnormal range and specific abnormal type, providing structured information including warning level, impact range and processing suggestions.

[0020] A multi-dimensional data blood relationship tracking system comprises: a meta-information collection module, a blood relationship construction module, a blood relationship analysis module, an intelligent early warning module, and an effect verification module; the meta-information collection module is used to collect table structure information, data source configuration information, access permission information and scheduling strategy information related to data blood relationship, and verify the integrity, consistency, validity and accuracy; the blood relationship construction module is used to build a blood relationship map based on the collected meta-information, calculate edge weights and node importance, and establish primary key indexes, type indexes and time series indexes; the blood relationship analysis module is used to perform hierarchical influence propagation analysis, multi-dimensional feature extraction, association rule mining and trend prediction based on the blood relationship map; the intelligent early warning module is used to evaluate the node risk level and impact range based on the abnormal scoring results, generate early warning information and distribute it through preset channels; the effect verification module is used to verify the actual effect of the experimental evaluation system through accuracy, performance, scalability and stability, and compare and analyze it with the benchmark method.

[0021] The present invention has the following advantages:

[0022] 1. By building a unified graph structure model and a multi-level index system, all-round lineage tracking at the table level, field level, and value level is achieved, technically solving the problem of limited lineage relationship identification dimensions in traditional solutions;

[0023] 2. Based on the improved influence propagation model and feature tensor decomposition method, it can solve the problem of intelligent analysis of blood relationship in complex data conversion scenarios from the mathematical principle;

[0024] 3. Adopting innovative multi-level cache strategy and incremental update mechanism, the technical bottleneck of delayed blood relationship update in traditional solutions is solved from the perspective of architectural design;

[0025] 4. Designed an early warning mechanism based on multi-dimensional anomaly detection, solving the technical defects of the traditional solution in lack of intelligent anomaly identification from the algorithm level;

[0026] 5. A comprehensive effect verification method system is proposed, which solves the problem of lack of systematic verification methods in traditional solutions from the evaluation dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a multi-dimensional data blood relationship tracking method described in the present invention.

[0028] Figure 2 This is a framework diagram of a multi-dimensional data blood relationship tracking system described in the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "upper, lower" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description. Unless otherwise stated, these directional words do not indicate or imply that the devices or components referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present invention; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0030] Example 1

[0031] like Figure 1 As shown, a multi-dimensional data blood relationship tracking method includes the following steps: S1: data blood relationship metadata collection; S2: multi-dimensional blood relationship construction; S3: blood relationship analysis; S4: intelligent early warning mechanism; S5: implementation effect verification.

[0032] In this embodiment, in step S1, when collecting data lineage metadata, table structure information, data source configuration information, access permission information and scheduling strategy information are obtained through a preset data collection interface; the collected metadata is verified for integrity, consistency, validity and accuracy, and a standardized quality verification report is generated.

[0033] Specifically, when collecting table structure information, the table structure information in the database is mapped into a four-tuple T, and each element is defined as follows:

[0034] The table name collection includes the database instance name, which is used to identify the database instance; the schema name, which is used to identify the database schema; the table name, which is used to uniquely identify the table; and the table description, which is used to describe the business purpose of the table.

[0035] The field collection contains the field name, which is used to identify the field; the field type, which is used to define the data type; the field length, which is used to limit the maximum length; the precision, which is used to set the precision of the numeric type; the decimal places, which is used to set the decimal places of the numeric type; whether it is nullable, which is used to identify whether the field allows null values; the default value, which is used to set the field default value; the value range, which is used to limit the field value range; and the field description, which is used to explain the business meaning of the field.

[0036] The primary key collection contains the primary key name, which is used to identify the primary key constraint; it contains fields, which are used to record the list of fields that constitute the primary key; the constraint type, which is used to define the primary key constraint type; and the constraint rules, which are used to set specific constraint rules.

[0037] The index collection contains the index name, which is used to identify the index; the index type, which is used to define the index structure type; the index method, which is used to specify the index implementation method; the index column, which is used to record the fields that constitute the index; whether it is unique, which is used to identify a unique index; whether it is clustered, which is used to identify a clustered index; and the filtering condition, which is used to set the index filtering rules.

[0038] Specifically, when collecting data source configuration information, the data source configuration information is mapped into a four-tuple C, and each element is defined as follows:

[0039] The source type information includes the source type, which is used to identify the data source type; version information, which is used to record the data source version; character set, which is used to define the character encoding used; time zone, which is used to set the data source time zone; and engine type, which is used to specify the storage engine.

[0040] The connection parameters include the host address, which is used to specify the data source location; the port number, which is used to specify the service port; the database name, which is used to specify the target database; the user name and password, which are used for identity authentication; the connection pool configuration, which is used to set the connection pool parameters; and the network configuration, which is used to set the network parameters.

[0041] Access permissions include read permissions, which are used to control data reading; write permissions, which are used to control data writing; execute permissions, which are used to control program execution; authorization rules, which are used to define permission granting rules; effective scope, which is used to limit the scope of permission; and validity period, which is used to set the permission period.

[0042] The scheduling strategy includes the scheduling cycle, which is used to set the collection interval; the task priority, which is used to control the execution order; the concurrency, which is used to limit the number of concurrent tasks; the resource limit, which is used to control resource usage; the timeout setting, which is used to control the execution time; and the retry strategy, which is used to handle failures.

[0043] Specifically, when performing meta-information quality verification, the collected meta-information is subject to multi-dimensional quality verification. The verification dimensions include integrity verification, which is used to ensure that required fields are complete; consistency verification, which is used to ensure that the information is consistent; validity verification, which is used to ensure that the information is within the validity period; and accuracy verification, which is used to ensure that the information is accurate. The system will generate a standardized verification report to record the verification process and results.

[0044] In this embodiment, in step S2, when constructing a multi-dimensional blood relationship, a blood relationship map is constructed based on the collected metadata, and the nodes in the map include database nodes, data table nodes, field nodes, task nodes and processing process nodes, and the edges include subordinate relationships, derived relationships, dependent relationships and conversion relationships; the adjacency matrix is ​​used to represent the blood relationship, and the edge weight is calculated by combining the time decay factor, access frequency factor, data volume factor and business importance factor, and the importance score of the node is calculated through an iterative algorithm; a multi-level index system including primary key index, type index and time series index is established to optimize the query and analysis efficiency of blood relationships.

[0045] Specifically, when creating a blood relationship map, a blood relationship map quadruple structure G is constructed, and its constituent elements are defined as follows:

[0046] Each node in the node set contains a node identifier, which uses a globally unique identifier; node types, including database nodes, data table nodes, field nodes, task nodes, and processing nodes; node status, including active status, inactive status, and deleted status; node attributes, which store attribute information in the form of key-value pairs; creation time, which records the node creation timestamp; update time, which records the last update timestamp; version number, which identifies the node version; owner, which records the node owner; security level, which identifies the node security level; each edge in the edge set contains an edge identifier, which uses a globally unique identifier; source node, which records the starting node of the edge; target node, which records the ending node of the edge; edge type, which includes subordinate relationship, derived relationship, dependent relationship, and transformation relationship; edge weight, which indicates the strength of the relationship, and the value range is 0 to 1; Edge attributes store attribute information in the form of key-value pairs; creation time records edge creation timestamp; update time records last update timestamp; transmission volume records data transmission volume; update frequency records data update frequency; attribute set contains attribute identifier, used to uniquely identify attributes; attribute name, used to name attributes; attribute type, including string type, numeric type, Boolean type, date and time type, array type and object type; attribute constraint, used to limit attribute value; non-empty constraint, identifies whether empty is allowed; unique constraint, identifies whether uniqueness is required; length constraint, limits the length of value; range constraint, limits the range of value; label set contains label identifier, used to uniquely identify labels; label name, used to name labels; label classification, used to group labels; label hierarchy, used to define hierarchical relationships.

[0047] Specifically, when calculating the weight, it is necessary to construct an adjacency matrix and calculate the importance of nodes. When constructing the adjacency matrix, the adjacency matrix A of the graph is generated. The edge weight is calculated using a multi-factor weighted method: the edge weight is equal to the weighted sum of the time decay factor, the access frequency factor, the data volume factor, and the importance factor, where: the time decay factor is used to measure the impact of time on the strength of the relationship; the access frequency factor is used to measure the impact of the data access frequency; the data volume factor is used to measure the impact of the data transmission volume; the importance factor is used to measure the impact of the business importance; the sum of each weight coefficient is 1;

[0048] When calculating node importance, an improved node importance algorithm is used to calculate the node score: the node importance score is related to the importance score of the incoming node and the edge weight; the damping coefficient is used to control the degree of importance propagation attenuation, and its value range is 0 to 1; the iterative calculation process includes initializing all node scores as the inverse of the total number of nodes; iterative calculation is performed according to the formula, and stops when the difference between two iterations is less than the preset threshold or the maximum number of iterations is reached.

[0049] Specifically, index construction includes establishing a node index system, an edge index system, and a time series index system, wherein a multi-level node index structure is constructed, including: the primary key index adopts a B+ tree structure, the index key is the node identifier, and the index value is the complete node information; the type index adopts a hash index, the index key is the node type, and the index value is a list of node identifiers; the attribute index adopts an inverted index, the index key is the attribute key, and the index value is a list of node identifiers and attribute value pairs;

[0050] When establishing the edge index system, a multi-level edge index structure is constructed, including: the relationship index adopts a B+ tree structure, the index key is the edge identifier, and the index value is the complete edge information; the node pair index adopts a combined index, the index key is the source node and target node pair, and the index value is the edge identifier list; the type index adopts a hash index, the index key is the edge type, and the index value is the edge identifier list;

[0051] When establishing a time series index system, a time series index structure is constructed, including: the time index adopts an LSM tree structure, the index key is a timestamp, and the index value is a list of entity identifiers and operation type pairs; the batch update strategy includes setting the cache size, merge strategy, and compression ratio.

[0052] In this embodiment, in step S3, when analyzing blood relationships, first, based on the blood relationship map, a hierarchical influence propagation model is constructed to quantify the influence intensity between nodes and identify key propagation paths; then a third-order feature tensor model is constructed to extract multidimensional characteristics including node features, relationship features and time features, and a tensor decomposition method is used to achieve dimensionality reduction; finally, the support and confidence methods are used to mine frequent patterns, and the lift is calculated to identify the association rules between data; a trend prediction function is constructed based on the autoregressive model and external features to predict the changing trend of data relationships.

[0053] Specifically, the sub-steps of impact propagation analysis are as follows:

[0054] A1: Impact strength calculation

[0055] Calculate the influence strength between nodes based on the multi-level influence propagation model:

[0056]

[0057] Where: I(v t ): node influence strength function, which represents the influence strength value of node v at time t, with a range of [0, 1]; λ: time decay coefficient, which defines the time decay rate of influence strength, with a range of (0, 1); k: propagation level number, which defines the level depth of influence transmission, with a range of [1, n]; w i: edge weight coefficient, defining the influence weight of the propagation path at the i-th layer; n: maximum propagation depth, defining the calculation boundary of the influence propagation; t: time variable, defining the time point for calculating the influence intensity; v: target node identifier, defining the node whose influence intensity is to be calculated;

[0058] Critical Path Identification

[0059] Define the path importance score calculation model:

[0060] PathScore(p)=∑ e∈p w e ·∑ v∈p PR(v)

[0061] Where: PathScore(p): path importance scoring function, defining the importance of propagation path p; p: propagation path set, an ordered sequence of connected edges; e: edge set element, defining a single edge in the path; w e : Edge weight coefficient, defining the influence weight of edge e in the path; v: Node set element, defining a single node in the path; PR(v): Node importance function, defining the PageRank score of node v;

[0062] A2: Multidimensional feature analysis

[0063] Construct a third-order feature tensor model:

[0064] X ijk =f(v i , r j , t k )

[0065] Where: X: third-order feature tensor, dimension is m×n×k; f: feature mapping function, defining the mapping rules of nodes, relationships and time; m: number of nodes, defining the node dimension of feature tensor; n: number of relationship types, defining the relationship dimension of feature tensor; k: number of time windows, defining the time dimension of feature tensor; v i : Node vector, defining the feature representation of the i-th node; r j : Relation vector, defining the characteristic representation of the jth relationship type; t k : Time vector, defining the feature representation of the kth time window.

[0066] Eigendecomposition and Dimensionality Reduction

[0067] Feature extraction based on tensor decomposition method:

[0068]

[0069] Where: X: original feature tensor, representing the third-order tensor to be decomposed; R: decomposition rank, defining the number of dimensions of tensor decomposition; a r : The rth node pattern vector, defining the feature representation of the node dimension; b r : The rth relational pattern vector, defining the feature representation of the relational dimension; c r : The rth time pattern vector, defining the feature representation of the time dimension; Vector outer product operator, defines the outer product operation between vectors.

[0070] A3: Association rule mining

[0071] Frequent pattern recognition, define the support calculation model:

[0072]

[0073] Where: Sup(X): support function, defining the frequency of item set X; T(X): transaction set containing item set X; D: total transaction set; |·|: cardinality operator, defining the number of elements in the set.

[0074] Rule evaluation model, defining confidence and lift calculations:

[0075]

[0076] Among them: Conf(X→Y): confidence function, which defines the reliability of the rule; Lift(X→Y): lift function, which defines the relevance strength of the rule; X: rule antecedent item set; Y: rule consequent item set.

[0077] A4: Analysis of change trends

[0078] Time series feature construction, defining the time series feature vector:

[0079] F t =[f1(t),f2(t),...,f m (t)] T

[0080] Among them: F t : Time series feature vector, defining the feature representation at time t; f i (t): The i-th feature function, including: numerical features: change frequency, access volume, data volume; category features: operation type distribution, access mode distribution; statistical features: mean, variance, skewness, kurtosis.

[0081] Trend prediction model, build time series prediction function:

[0082] y t+h =f(y t ,yt-1 , ..., y t-p+1 , X t )

[0083] The prediction function f is a combination of autoregression and external features:

[0084]

[0085] Where: y t+h : predicted value, defining the predicted result of the next h steps; h: prediction step length, defining the time span of the prediction; p: number of autoregressive terms, defining the length of use of historical observations; q: external feature dimension, defining the number of external features used; α i : Autoregressive coefficient, defining the weight of the i-th historical observation; β j : Feature weight coefficient, defining the weight of the jth external feature; The jth external eigenvector; ∈ t : Random error term, subject to normal distribution with mean 0; y t : Observation vector at time t.

[0086] The prediction accuracy is evaluated using two indicators: root mean square error and mean absolute percentage error:

[0087] Root mean square error calculation:

[0088]

[0089] in:

[0090] RMSE: Root Mean Square Error, defines the overall accuracy of the prediction;

[0091] y i : The actual observation value at the i-th time point;

[0092] The predicted value at the i-th time point;

[0093] n: total number of evaluation samples.

[0094] Mean absolute percentage error calculation:

[0095]

[0096] Where: MAPE: Mean absolute percentage error, defines the relative deviation level of the forecast; y i : The actual observation value at the i-th time point, and y i ; The predicted value at the i-th time point; n: the total number of evaluation samples;

[0097] Evaluation indicator threshold definition:

[0098]

[0099] Where: ∈1: Root mean square error threshold, which defines the upper limit of the acceptable absolute error; ∈2: Mean absolute percentage error threshold, which defines the upper limit of the acceptable relative error.

[0100] In this embodiment, in step S4, when constructing an intelligent early warning mechanism, first, an anomaly scoring model is constructed for topological structure, time series pattern and multidimensional attributes to evaluate the degree of abnormality of data nodes and edges; then, based on the anomaly scoring results, the risk level of the node is comprehensively calculated, and the impact range is quantified by evaluating the degree of association with other nodes; finally, early warning information is generated, including the unique identifier of the warning event, abnormal node information, risk level, impact range and targeted processing suggestions, which are delivered to the responsible person through the designated distribution channel, and the warning status and processing process are recorded.

[0101] Specifically, the detailed steps are as follows:

[0102] B1: Anomaly Detection

[0103] Topology anomaly detection, building an anomaly scoring model for node topology:

[0104] Score topo (v) = α1·D in (v)+α2·D out (v)+α3·C(v)

[0105] Calculation of in-degree anomaly component:

[0106]

[0107] Calculation of out-degree anomaly component:

[0108]

[0109] Calculation of abnormal component of clustering coefficient:

[0110]

[0111] Among them: Score topo (V): topological anomaly score of node v, ranging from [0, 1]; d in (v, t): the in-degree value of node v at time t; d out (v, t): the out-degree value of node v at time t; c(v, t): the clustering coefficient of node v at time t, calculated as c(v) = 2L v / k v (k v -1), where L vis the number of edges actually existing between the neighbors of node v, k v is the degree of node v; ∈: smoothing factor, to prevent the denominator from being 0, usually 0.01; α1, α2, α3: weight coefficients, satisfying the sum of 1;

[0112] Time series pattern anomaly detection, calculating anomaly scores based on time series decomposition:

[0113] Score time (v,t)=β1·T(v,t)+β2·S(v,t)+β3·R(v,t)

[0114] Trend anomaly component calculation:

[0115]

[0116] Seasonal anomaly component calculation:

[0117]

[0118] Calculation of residual abnormal components:

[0119]

[0120] Among them: Score time (v,t): time series anomaly score of node v at time t, with a value range of [0, 1]; trend(v,t): trend value at time t; season(v,t): seasonal value at time t; residual(v,t): residual value at time t; p: length of seasonal cycle; μ r : residual mean; σ r : Residual standard deviation; β1, β2, β3: Weight coefficients, which sum to 1; ∈: Smoothing factor, which prevents the denominator from being 0, usually 0.01.

[0121] Attribute feature anomaly detection, building an anomaly scoring model for multi-dimensional attributes:

[0122]

[0123] Calculation of abnormal component of attribute characteristics:

[0124]

[0125] Among them: Score attr (V): attribute anomaly score of node v, ranging from [0, 1]; x i (v, t): the i-th attribute value of node v at time t; γ i : feature weight coefficient, which sums to 1; n: total number of attribute features;

[0126] B2: Risk Assessment

[0127] Determine the risk level and calculate the risk level based on the multi-dimensional anomaly scores:

[0128] Risk(v)=w1·Score topo (v)+w2·Score time (v)+w3·Score attr (v)

[0129] Risk level classification standards: low risk: Risk(v)<0.3; medium risk: 0.3≤Risk(v)<0.7; high risk: Risk(v)≥0.7; where w1, w2, w3 are dimension weight coefficients, the total of which must be 1.

[0130] Impact scope assessment, calculate the impact propagation scope of risk nodes:

[0131] Impact(v)=∑ u∈N(v) Risk(u)·W(v,u)

[0132] The parameters are defined as follows:

[0133] Impact(v): the influence score of node v; N(v): the set of neighbor nodes of node v; W(v,u): the edge weight from node v to node u; Risk(u): the risk score of neighbor node u.

[0134] B3; Early warning processing

[0135] Warning information generation: warning information is constructed based on risk level and impact scope, including: warning ID: unique identifier of warning event; target node: node information where abnormality occurs; risk level: level determined by Risk (v); impact scope: impact scope based on Impact (v) assessment; warning type: specific category of abnormality; handling suggestions: targeted handling measures;

[0136] Warning distribution strategy, formulate the distribution rules of warning information:

[0137] Warning priority: Determine the processing priority based on the risk level; Distribution channels: Including email, SMS, system notification; Distribution objects: Determine the person in charge based on the scope of impact; Distribution time: Consider the time strategy of business importance; Escalation mechanism: Escalation process for timeout and unprocessed;

[0138] Track the warning status and establish a status management mechanism for the entire life cycle of the warning:

[0139] Warning status: including pending, in process, and processed; status transition: define the transition rules between statuses; processing record: including the person who processed, processing time, and processing measures; processing result: record the processing effect and problem solving status.

[0140] In this embodiment, in step S5, when performing effect verification, the accuracy of blood relationship tracking is evaluated by precision, recall and F1 score; the performance of the system in response time, throughput and concurrent processing capability is tested to verify its real-time performance; the scalability of the system under different data scales and concurrent numbers is evaluated; the stability of the system is verified through long-term operation and abnormal recovery tests; and the performance is compared and analyzed with the existing methods using relative improvement rate and multi-indicator weighted scoring.

[0141] Specifically, the steps include:

[0142] C1: Accuracy Verification:

[0143] Calculation of blood relationship recognition accuracy

[0144] Evaluation is performed using precision, recall, and F1 score:

[0145]

[0146]

[0147] Where: TP: number of correctly identified consanguinity; FP: number of incorrectly identified consanguinity; FN: number of unidentified consanguinity;

[0148] Multi-dimensional tracking coverage verification:

[0149] Calculate dimension coverage metrics:

[0150]

[0151] Where: n: total number of blood relationship dimensions; w i : The weight of the i-th dimension; c i : Coverage of the i-th dimension.

[0152] C2: Performance Verification

[0153] Response time test, calculate the average response time:

[0154]

[0155] Where: m: total number of test requests; The start time of the i-th request; The end time of the i-th request;

[0156] Quantile delay calculation: Latency p =min{t|P(ResponseTime≤t)≥p}, where p is the quantile level;

[0157] System throughput test, calculating the number of requests processed per second:

[0158]

[0159] Concurrent processing capability assessment:

[0160] Concurrency=TPS·ResponseTime

[0161] C3: Scalability Verification

[0162] Data scale scalability test, the growth rate of computing performance as the data scale changes:

[0163]

[0164] Among them: ΔPerformance: change in performance indicators; ΔDataSize: change in data size.

[0165] Concurrency scalability test, calculate the performance decay rate of the system when concurrency increases:

[0166]

[0167] Among them: Performance_n: performance when n concurrent connections are used; Performance_1: performance when single concurrent connection is used; n: number of concurrent connections;

[0168] C4: Stability Verification

[0169] Long-term stability test, calculation of system availability indicators:

[0170] Availability=MTTF / MTTF+MTTR

[0171] Among them: MTTF: mean time between failures; MTTR: ​​mean time to repair;

[0172] Abnormal recovery capability test, calculation of system recovery indicators:

[0173] RecoveryRateRecoveredErrors / TotalErrors

[0174]

[0175] Where: k: total number of errors; The time when the i-th error occurred; The i-th error recovery time;

[0176] C5: Effect comparison

[0177] Compared with the baseline method, the relative improvement rate is used for evaluation:

[0178]

[0179] Among them: Performance new :Performance indicators of new methods; Performance base : Performance indicators of benchmark methods;

[0180] Comprehensive score calculation, based on weighted calculation of multiple indicators:

[0181]

[0182] Where: m: number of evaluation indicators; λ i : The weight of the i-th indicator; metric i : The actual value of the i-th indicator; min i : The minimum value of the i-th indicator; max i : The maximum value of the i-th indicator.

[0183] In this embodiment, the time decay factor in the weight calculation measures the decay effect of the relationship over time, the access frequency factor reflects the frequency of data access, the data volume factor represents the transmission volume of data flow, and the business importance factor represents the intensity of business dependence on the data relationship. The sum of the weights of each factor is 1.

[0184] In this embodiment, the trend prediction function adopts the form of combining autoregressive terms and external features, and predicts the future change trend of data relationships based on historical data and external influencing factors.

[0185] In this embodiment, the generation of the warning information is based on the risk level, abnormal range and specific abnormal type, and provides structured information including the warning level, impact range and processing suggestions.

[0186] Embodiment 2:

[0187] like Figure 2As shown, a multi-dimensional data blood relationship tracking system includes: a meta-information collection module, a blood relationship construction module, a blood relationship analysis module, an intelligent early warning module, and an effect verification module; the meta-information collection module is used to collect table structure information, data source configuration information, access permission information and scheduling strategy information related to data blood relationship, and verify the integrity, consistency, validity and accuracy; the blood relationship construction module is used to build a blood relationship map based on the collected meta-information, calculate edge weights and node importance, and establish primary key indexes, type indexes and time series indexes; the blood relationship analysis module is used to perform hierarchical influence propagation analysis, multi-dimensional feature extraction, association rule mining and trend prediction based on the blood relationship map; the intelligent early warning module is used to evaluate the node risk level and impact range based on the abnormal scoring results, generate early warning information and distribute it through preset channels; the effect verification module is used to verify the actual effect of the experimental evaluation system through accuracy, performance, scalability and stability, and compare and analyze it with the benchmark method.

[0188] Finally, it should be noted that those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-dimensional data blood relationship tracking method, characterized in that: The following steps are involved: S1: Data lineage metadata collection; S2: Multidimensional kinship construction; S3: blood relationship analysis; S4: Intelligent early warning mechanism; S5: Implementation effect verification.

2. According to claim 1, a method for processing multi-model results applied to document review is characterized in that: In step S1, when collecting data lineage metadata, the table structure information, data source configuration information, access permission information and scheduling strategy information are obtained through the preset data collection interface; the collected metadata is verified for integrity, consistency, validity and accuracy, and a standardized quality verification report is generated.

3. The method for processing multi-model results applied to document review according to claim 2 is characterized in that: In step S2, when constructing multi-dimensional blood relationship, a blood relationship map is constructed based on the collected meta-information. The nodes in the map include database nodes, data table nodes, field nodes, task nodes and processing process nodes, and the edges include subordinate relationships, derived relationships, dependent relationships and conversion relationships. Adjacency matrix is ​​used to represent blood relationship, edge weight is calculated by combining time decay factor, access frequency factor, data volume factor and business importance factor, and importance score of node is calculated by iterative algorithm; Establish a multi-level index system including primary key index, type index and time series index to optimize the query and analysis efficiency of blood relationship.

4. The method for processing multi-model results applied to document review according to claim 3 is characterized in that: In step S3, when analyzing blood relationships, firstly, a hierarchical influence propagation model is constructed based on the blood relationship map to quantify the influence intensity between nodes and identify key propagation paths; then a third-order feature tensor model is constructed to extract multidimensional features including node features, relationship features and time features, and a tensor decomposition method is used to achieve dimensionality reduction; finally, frequent patterns are mined using support and confidence methods, and the lift is calculated to identify association rules between data; A trend prediction function is constructed based on the autoregressive model and external features to predict the changing trend of data relationships.

5. The method for processing multi-model results applied to document review according to claim 4 is characterized in that: In step S4, when building an intelligent early warning mechanism, first build an anomaly scoring model for topological structure, time series pattern and multi-dimensional attributes to evaluate the degree of abnormality of data nodes and edges; then, based on the anomaly scoring results, comprehensively calculate the risk level of the node, and quantify the impact range by evaluating its correlation with other nodes; finally, generate early warning information, including the unique identifier of the warning event, abnormal node information, risk level, impact range and targeted processing suggestions, and deliver it to the responsible person through the designated distribution channel, and record the warning status and processing process.

6. The method for processing multi-model results applied to official document review according to claim 5 is characterized in that: In step S5, when performing effect verification, the accuracy of blood relationship tracking is evaluated using precision, recall and F1 score; the performance of the system in terms of response time, throughput and concurrent processing capability is tested to verify its real-time performance; Evaluate the scalability of the system under different data scales and concurrent numbers; verify the stability of the system through long-term operation and abnormal recovery tests; and conduct a performance comparison analysis with existing methods using relative improvement rate and multi-indicator weighted scores.

7. The method for processing multi-model results applied to document review according to claim 6 is characterized in that: The time decay factor in the weight calculation measures the decay effect of the relationship over time, the access frequency factor reflects how frequently the data is accessed, the data volume factor represents the transmission volume of data flow, and the business importance factor represents the intensity of business dependence on the data relationship. The sum of the weights of each factor is 1.

8. The method for processing multi-model results applied to document review according to claim 7 is characterized in that: The trend prediction function adopts the form of combining autoregressive terms and external features, and predicts the future change trend of data relationships based on historical data and external influencing factors.

9. The method for processing multi-model results applied to official document review according to claim 8, characterized in that: The generation of the warning information is based on the risk level, abnormal scope and specific abnormal type, and provides structured information including warning level, impact scope and processing suggestions.

10. A multi-dimensional data blood relationship tracking system, characterized in that: include: Meta-information collection module, blood relationship construction module, blood relationship analysis module, intelligent early warning module, effect verification module; The meta-information collection module is used to collect table structure information, data source configuration information, access permission information and scheduling strategy information related to data lineage, and verify the integrity, consistency, validity and accuracy; The blood relationship construction module is used to construct a blood relationship map based on the collected meta information, calculate edge weights and node importance, and establish a primary key index, a type index, and a time series index; The blood relationship analysis module is used to perform hierarchical influence propagation analysis, multi-dimensional feature extraction, association rule mining and trend prediction based on the blood relationship map; The intelligent early warning module is used to evaluate the node risk level and impact range based on the abnormal scoring results, generate early warning information and distribute it through preset channels; The effect verification module is used to verify the actual effect of the evaluation system through accuracy, performance, scalability and stability verification experiments, and to compare and analyze with the benchmark method.

Images