Big data platform model change influence quantitative evaluation and data verification method

Abstract

The invention relates to the technical field of big data management and data quality control, and discloses a big data platform model change influence quantitative evaluation and data verification method, which comprises the following steps of: firstly, constructing a full-amount blood relationship graph containing dominant and implicit dependencies by combining dynamic job logs and static code abstract syntax tree analysis; performing statistics on change influence indexes, and calculating a total influence score representing a risk level by using a multiple linear regression model; generating a dynamic check threshold based on the score, and selecting a sentinel node according to betweenness centrality; and calculating node data similarity and introducing a distance attenuation factor to obtain a global consistency index so as to judge a verification result. And finally, iteratively correcting the regression model coefficient by using the verification residual error. According to the method, the problem of dependency deficiency of complex links is effectively solved, accurate quantification of alteration risks and adaptive adjustment of verification strategies are realized, and the accuracy and efficiency of data management are improved.

Description

Technical Field

[0001] This invention relates to the field of big data governance and data quality control technology, specifically a method for quantitative assessment and data verification of the impact of big data platform model changes. Background Technology

[0002] In the continuous iteration of big data platforms, data model changes are a frequent and high-risk operation. As data warehouse layered architectures become increasingly complex, modifications to an underlying model often propagate downstream through complex dependency chains, impacting the accuracy of reports from multiple business systems. Therefore, accurately assessing the impact of changes and performing effective data validation are crucial for ensuring platform stability.

[0003] Existing technologies often rely on a single parsing path when constructing data lineage relationships. Some solutions only obtain table-level dependencies by parsing job scheduling logs, failing to deeply identify hard-coded input and output parameters within the code; while other solutions rely solely on static SQL script parsing, making it difficult to verify the dynamic calling logic of tasks during actual runtime. This single-dimensional parsing approach results in breakpoints in the lineage graph, failing to fully cover explicit and implicit dependencies, making it easy to overlook potentially affected nodes when assessing the impact of changes.

[0004] Regarding data verification strategies, existing technologies typically employ static configuration, which pre-sets fixed verification ranges and uniform pass thresholds. This approach lacks the ability to perceive the specific risk level of changes, leading to high-risk changes missing quality issues due to overly broad thresholds, while low-risk changes consume excessive computing resources due to full verification, making it difficult to strike a balance between ensuring data quality and controlling computational costs.

[0005] Furthermore, existing change impact assessment models often rely on expert experience to set fixed indicator weights, lacking the ability to self-correct based on actual operational results. As business logic evolves, the pre-set weighting system often fails to accurately reflect the true impact of each factor under the current system state, leading to discrepancies between risk assessment results and actual data drift, and failing to automatically optimize assessment accuracy with the accumulation of historical data. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for quantitative assessment and data verification of the impact of changes in big data platform models. This method solves the problems of existing technologies, such as the lack of dependency relationships due to a single lineage resolution path, the lack of adaptability in verification strategies leading to an imbalance in resource allocation and quality assurance, and the inability of the assessment model to self-correct based on actual operating results, thus affecting the accuracy of the assessment.

[0007] The first aspect of this invention provides a method for quantitative assessment and data verification of the impact of changes to a big data platform model, the method mainly including the following steps: A two-layer lineage analysis is performed to construct a complete lineage graph. The computer system first reads the job scheduling logs from the big data platform, uses regular expression matching algorithms to extract the dynamic data flow during actual runtime, and establishes the time dependency sequence between nodes based on task start and end timestamps. Simultaneously, a parser is used to parse the source code of the target model and its associated tasks to generate an abstract syntax tree. This syntax tree is traversed to locate data source declarations, table name references in SQL statements, and parameter configurations of data write functions, thereby extracting hard-coded input parameters, output parameters, and external data source connection identifiers. By fusing explicit dependencies verified by dynamic logs with implicit dependencies analyzed by static code, a directed acyclic graph centered on the target model is generated, and the identified independent data entities are instantiated as model nodes or business system nodes in the graph.

[0008] The system statistically analyzes the impact indicators of changes and performs normalization processing. Based on the constructed full-scale lineage graph, it statistically analyzes the multi-dimensional impact indicators caused by changes in the target model. These multi-dimensional impact indicators cover the number of directly related models and systems, the number of indirectly related models and systems, and the number of changes in related fields. To eliminate the influence of different indicator dimensions, the system retrieves the maximum statistical value of each indicator in the historical change records. By calculating the ratio of the current original statistical value to the corresponding historical maximum value, a normalized impact feature vector is generated.

[0009] The total impact score is calculated based on a regression weighting model. A pre-defined multiple linear regression model is used to weight the normalized impact feature vector. Specifically, each component of the feature vector is multiplied by its corresponding regression coefficient and summed to obtain an estimated value of the severity of historical change failures. During this process, the internal weight proportion of various impact indicators is calculated based on the magnitude of the regression coefficients, and a global weighting factor is introduced to aggregate the weighted results of the direct and indirect impact components, ultimately yielding the total impact score characterizing the potential risk level of the change.

[0010] An adaptive data verification strategy is generated. On one hand, a dynamic threshold for data verification is determined based on the calculated total impact score. This threshold is generated through a nonlinear mapping relationship, specifically by using a decay function with the natural constant as the base and the total impact score as the negative exponent to calculate the adjustment term. This ensures that the dynamic threshold monotonically and nonlinearly increases with the total impact score, approaching a value of one, thereby implementing stricter verification standards in high-risk scenarios. On the other hand, sentinel nodes are selected from the entire kinship graph based on betweenness centrality. Specifically, this is achieved by calculating and summing the ratio of the number of shortest paths passing through a downstream node to the total number of shortest paths between all node pairs in the entire graph, thereby quantifying the importance of information flow between nodes and selecting the nodes ranked higher as key verification targets.

[0011] Perform data rationality verification and judgment. Extract the historical data vectors of the target model and sentinel nodes before the change, and the real-time data vectors after the change. These vectors are composed of statistical measures of key numerical fields. Using the cosine similarity algorithm, calculate the data similarity value of each node by taking the quotient of the dot product of the real-time data vector and the historical data vector and the product of their moduli. Further, introduce a distance decay factor to calculate a global consistency index. This involves calculating the natural logarithm of the distance between each sentinel node and the target model, adding one as the denominator, applying distance decay processing to the similarity values, summing them, and then weighting the sum based on the target model's basic weights. Finally, compare this global consistency index with a dynamic threshold to determine whether the verification passes.

[0012] Regression weights are adjusted based on validation residuals. After validation, a closed-loop feedback mechanism is established. The difference between the numerical value and the global consistency index is calculated as the actual data drift, and half the square of the difference between this actual data drift and the total impact score is calculated as the loss function. The partial derivatives of the loss function with respect to each regression coefficient are calculated using the gradient descent algorithm as the gradient values. The regression coefficients of the multiple linear regression model are iteratively adjusted according to a preset learning rate to optimize the accuracy of subsequent evaluations.

[0013] Perform model deployment and graph update. When the verification passes, the system serializes and stores the metadata snapshot and code version of the model before the change, pushes the changed model script and configuration parameters to the production environment to perform the deployment operation, and performs topological correction of nodes and directed edges on the full lineage graph based on the added or removed dependencies brought about by the change.

[0014] A second aspect of the present invention provides a device for quantitative assessment and data verification of the impact of changes in a big data platform model. The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0015] This invention provides a method for quantitatively assessing and validating the impact of changes to a big data platform model. It offers the following advantages: 1. This invention constructs a two-layer full lineage graph containing explicit and implicit dependencies by combining dynamic job log parsing with static code abstract syntax tree analysis. This dual parsing mechanism overcomes the limitations of traditional single log analysis in identifying hard-coded dependencies within the code and single static analysis in confirming the actual data flow at runtime. It effectively solves the problem of missing dependencies in complex data chains, thereby improving the coverage and accuracy of model change impact assessment.

[0016] 2. This invention proposes an adaptive verification strategy based on risk scoring. It utilizes the betweenness centrality algorithm to identify sentinel nodes at critical points in information flow and dynamically adjusts the verification threshold based on the total impact score calculated by a regression model. This mechanism can automatically tighten the judgment criteria in high-risk change scenarios to prevent quality incidents, and avoid the resource waste caused by full verification in low-risk scenarios, achieving an effective balance between computing resource allocation and data quality assurance.

[0017] 3. This invention establishes a closed-loop feedback mechanism based on verification residuals, which can calculate the deviation between the actual data drift and the estimated risk score, and use the gradient descent algorithm to back-correct the weight coefficients of the multiple linear regression model. This enables the evaluation model to have continuous learning capabilities, automatically optimizing the weight configuration of various indicators based on historical data accumulated during system operation, thereby continuously improving the accuracy of risk quantification assessment over time. Attached Figure Description

[0018] Figure 1 This is a system functional module architecture diagram of the present invention; Figure 2 This is a flowchart illustrating the overall method of the present invention; The modules include: 10. Double-layer lineage analysis module; 20. Influence indicator statistics module; 30. Regression weight evaluation module; 40. Adaptive strategy generation module; 50. Data rationality verification module; 60. Weight feedback correction module; and 70. Version and graph management module. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 1The big data platform model change impact quantitative assessment and data rationality verification system provided by the present invention runs on a computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the logic of the following functional modules.

[0021] The system includes: a dual-layer lineage analysis module 10, an influence index statistics module 20, a regression weight evaluation module 30, an adaptive strategy generation module 40, a data rationality verification module 50, a weight feedback correction module 60, and a version and graph management module 70.

[0022] The dual-layer lineage analysis module 10 is configured to perform dynamic log scanning and static code analysis on the target model in the big data platform. The dual-layer lineage analysis module 10 reads job scheduling logs and extracts the data flow and time dependency sequences between model processing nodes. Simultaneously, the dual-layer lineage analysis module 10 parses the interface code abstract syntax tree of the target model, extracting input parameters, output parameters, and data source identifiers. Based on the extracted log and code information, the dual-layer lineage analysis module 10 constructs a directed acyclic lineage graph centered on the target model. This graph includes model nodes, business system nodes, and directed edges with dependency type attributes between nodes.

[0023] The impact indicator statistics module 20, connected to the two-layer lineage analysis module 10, is configured to statistically analyze the scope of impact of changes based on the lineage graph. The impact indicator statistics module 20 traverses the lineage graph, identifying and counting the number of directly related models, directly related systems, indirectly related models, indirectly related systems, and the number of changes in related fields caused by changes in the target model. The impact indicator statistics module 20 obtains the historical maximum values ​​of each indicator, normalizes the current statistical results, and generates a normalized impact feature vector.

[0024] The regression weight assessment module 30, connected to the impact indicator statistics module 20, is configured to calculate the total impact score of the change. Internally, the regression weight assessment module 30 stores a multiple linear regression model and its corresponding regression coefficients, which are trained based on historical change data. The regression weight assessment module 30 inputs the normalized impact feature vector into the multiple linear regression model to calculate the internal weights of various impact indicators. Combining this with a preset global weight factor, the regression weight assessment module 30 aggregates the various weighted indicators to calculate the total impact score representing the potential risk level of the current change.

[0025] The adaptive strategy generation module 40, connected to both the regression weight evaluation module 30 and the two-layer kinship analysis module 10, is configured to dynamically generate data verification configurations. The adaptive strategy generation module 40 calculates a dynamic threshold for data verification based on the total influence score, which monotonically and non-linearly increases with the total influence score. Simultaneously, the adaptive strategy generation module 40 calculates the betweenness centrality of downstream nodes in the kinship graph, selects a predetermined number of nodes with high betweenness centrality ranking as sentinel nodes, and includes these sentinel nodes in the verification scope.

[0026] The data rationality verification module 50, connected to the adaptive strategy generation module 40, is configured to perform data consistency calculation and judgment. The data rationality verification module 50 extracts the historical data vectors of the target model and sentinel nodes before the change and the real-time data vectors after the change. The data rationality verification module 50 calculates the data similarity value of each node using a cosine similarity algorithm. The data rationality verification module 50 introduces a distance decay factor to perform a weighted summation of the similarity values ​​between the target model and the sentinel nodes, obtaining a global consistency index. The data rationality verification module 50 compares the global consistency index with the dynamic threshold generated by the adaptive strategy generation module 40. If the global consistency index is greater than or equal to the dynamic threshold, the verification is considered successful.

[0027] The weight feedback correction module 60, connected to both the data rationality verification module 50 and the regression weight evaluation module 30, is configured to correct the coefficients of the regression model. The weight feedback correction module 60 calculates the difference between the global consistency index and the total impact score, obtaining the residual between the actual data drift and the estimated impact. Based on this residual, the weight feedback correction module 60 constructs a loss function, uses the gradient descent algorithm to calculate the update gradient of the regression coefficients, and updates the regression coefficients stored in the regression weight evaluation module 30.

[0028] The version and graph management module 70, connected to the data validity verification module 50, is configured to perform model publishing and metadata maintenance. When the data validity verification module 50 determines that the verification is successful, the version and graph management module 70 serializes and stores a snapshot of the metadata and code version of the model before the change. The version and graph management module 70 performs the online operation of the changed model and, based on the changed logical dependencies, adds, deletes, and updates nodes and edges in the lineage relationship graph of the two-layer lineage analysis module 10.

[0029] See attached document Figure 2 The method for quantitative assessment of the impact of big data platform model changes and verification of data rationality provided by this invention includes the following steps: S100. Perform two-layer lineage analysis to construct a full lineage graph. The computer system reads the job scheduling logs from the big data platform, extracting runtime data flow and time dependency sequences. Simultaneously, the computer system parses the interface code abstract syntax tree of the target model, extracting input parameters, output parameters, and data source identifiers. Based on the extracted log and code information, the computer system generates a directed acyclic graph centered on the target model, i.e., the lineage graph. The nodes in this lineage graph include model nodes and business system nodes, as well as dependency type attributes and data transmission path attributes.

[0030] S200. Statistical analysis of change impact indicators and normalization processing. Based on the lineage graph generated in step S100, the system iterates through and statistically analyzes five indicators caused by changes in the target model: the number of directly related models, the number of directly related systems, the number of indirectly related models, the number of indirectly related systems, and the number of changes in related fields. The system obtains the maximum value of each indicator in the historical change records, and performs maximum value normalization calculation on the statistical results of the above five indicators to obtain the normalized impact feature vector.

[0031] S300. Calculate the total impact score based on the regression weight model. The system calls a pre-trained multiple linear regression model based on historical change data. The system inputs the normalized impact feature vector obtained in step S200 into the multiple linear regression model to calculate the internal weights of each impact indicator. Subsequently, the system combines the preset global weight factor to aggregate and calculate the weighted indicators to obtain the total impact score representing the potential risk level of this change.

[0032] S400. Generate an adaptive data verification strategy. Based on the total impact score calculated in step S300, the system calculates a dynamic threshold for data verification using a preset nonlinear mapping function. This dynamic threshold increases with the increase of the total impact score. Simultaneously, the system calculates the betweenness centrality index of each downstream node in the kinship graph, sorts the downstream nodes according to their betweenness centrality values, and selects a preset number of top-ranked nodes as sentinel nodes, including these sentinel nodes in the data verification scope.

[0033] S500. Perform data rationality verification and judgment. The system extracts the historical data vectors of the target model and the sentinel nodes selected in step S400 before the change and the real-time data vectors after the change. The system uses the cosine similarity algorithm to calculate the data similarity of each node and introduces a distance decay factor to calculate the global consistency index containing the target model and the sentinel nodes. The system compares the global consistency index with the dynamic threshold generated in step S400. If the global consistency index is greater than or equal to the dynamic threshold, the verification is deemed successful, and the process proceeds to step S700; if the global consistency index is less than the dynamic threshold, the verification is deemed unsuccessful, an alarm is triggered, and the process is terminated.

[0034] S600. Correcting Regression Weights Based on Validation Residuals. After completing the validation in step S500, the system calculates the difference between the global consistency index and the total impact score, obtaining the residual between the actual data drift and the estimated impact. The system constructs a loss function with this residual as the variable and uses the gradient descent algorithm to calculate the update gradient of the regression coefficients, updating the coefficients of the multiple linear regression model used in step S300 for use in the next change assessment.

[0035] S700, Execute model deployment and graph update. After the verification in step S500 is passed, the system serializes and stores the metadata snapshot and code version of the target model before the change. Subsequently, the system performs the online deployment operation of the changed model to make it effective in the production environment. Finally, based on the changed code logic and data flow, the system performs node and edge addition and deletion operations on the lineage graph in step S100 to generate the updated lineage graph.

[0036] One embodiment of the present invention provides a specific implementation method for the automated construction of a two-layer kinship map. This process is executed by the two-layer kinship analysis module 10, aiming to construct a kinship map covering the entire lifecycle of the target model through a combination of dynamic and static methods.

[0037] The dual-layer lineage resolution module 10 first executes the dynamic log parsing process. The dual-layer lineage resolution module 10 connects to the job scheduling center of the big data platform to obtain the full job execution log within a preset time window. This log records the actual execution status, input path parameters, output path parameters, and task start and end timestamps for each data processing task in the platform.

[0038] Based on the acquired job execution logs, the two-layer lineage resolution module 10 uses a pre-defined regular expression matching algorithm to extract data flow information. The two-layer lineage resolution module 10 identifies data read and write operations recorded in the logs, thereby determining the source and target nodes of the data flow. The system stores the extracted dependencies as triples, which include the upstream node identifier, the downstream node identifier, and the specific time the data transmission occurred. By analyzing the task start and end timestamps, the two-layer lineage resolution module 10 establishes a time dependency sequence between nodes to determine the temporal logic of data processing.

[0039] During or after dynamic log parsing, the two-layer lineage parsing module 10 performs static code analysis on the source code of the target model and its associated tasks. The two-layer lineage parsing module 10 reads script files stored in the code repository and uses a parser to convert the source code into an abstract syntax tree.

[0040] During the traversal of the abstract syntax tree, the two-layer lineage resolution module 10 locates key syntax nodes representing data input and output. The system identifies data source declaration statements in the code, table name references in SQL query statements, and parameter configurations of data write functions. The two-layer lineage resolution module 10 extracts hard-coded input parameters, output parameters, and connection identifiers of external data sources from these. This process can identify implicit data dependencies that are not actually executed within the log time window but objectively exist in the code logic.

[0041] Based on the results of the above dynamic and static analysis, the two-layer kinship resolution module 10 performs graph fusion and generation operations. The system initializes an empty directed acyclic graph. ,in, Represents a set of nodes; This represents the set of directed edges.

[0042] The dual-layer lineage analysis module 10 instantiates all identified independent data entities as nodes in the graph. The system then divides these entities into model node sets based on their attributes. With business system node set Model nodes represent data processing models or intermediate tables within the platform, while business system nodes represent external data source systems or data consumption systems.

[0043] The dual-layer bloodline parsing module 10 constructs a set of directed edges based on the extracted dependencies. For each pair of nodes with a data flow relationship... The system creates a path in the map from point to The directed edges. The two-layer lineage resolution module 10 assigns attribute information to each edge, including dependency type and data transmission path. The dependency type is used to identify whether the relationship is an explicit dependency derived from dynamic log verification or an implicit dependency derived from static code analysis; the data transmission path records the specific interface or storage location through which data flows between nodes.

[0044] Finally, the dual-layer bloodline analysis module 10 outputs the target model. This is a complete lineage map centered on the target model. This map fully illustrates the upstream source paths and downstream influence paths of the target model, providing a topological foundation for subsequent impact indicator statistics.

[0045] One embodiment of the present invention provides a specific implementation method for the quantitative statistics and initial assessment of multidimensional impact indicators. This process is jointly executed by the impact indicator statistics module 20 and the regression weight assessment module 30, aiming to transform the potential risks of model changes into measurable numerical scores based on lineage topology.

[0046] The Influence Index Statistics Module 20 performs a breadth-first or depth-first traversal of the kinship graph constructed by the two-layer kinship analysis module 10. Starting from the node, search downstream along the direction of the directed edge to count the five-dimensional indicators caused by the change.

[0047] First, the Influence Indicator Statistics Module 20 counts indicators that directly influence the target model. The number of directly associated models in the computational graph of the Influence Indicator Statistics Module 20, where there are direct edges to the target model and the node type is model node, is denoted as the number of directly associated models. Meanwhile, the number of directly connected systems in the impact indicator statistics module 20 calculation graph that are directly linked to the target model and whose node type is a business system node is recorded as the number of directly associated systems. These two indicators characterize the impact of changes on adjacent downstream nodes.

[0048] Secondly, the Influence Indicator Statistics Module 20 counts indicators at the indirect influence level. This module retrieves all reachable nodes in the graph that originate from the target model and have a path length greater than 1. The module then counts the number of model nodes among these nodes as the number of indirectly associated models. The number of business system nodes is counted as the number of indirectly related systems. These two metrics characterize the scope and depth of change propagation along the data link.

[0049] In addition, the statistical module 20 for influencing indicators counts the number of changes in related fields. The Impact Indicator Statistics Module 20 analyzes the set of output fields involved in changes to the target model and searches for downstream metadata definitions that depend on these output fields in the attributes of the lineage graph. The Impact Indicator Statistics Module 20 calculates the total number of fields in all downstream nodes that require synchronous modification of field definitions or mapping relationships to characterize the maintenance cost at the metadata level.

[0050] After completing the statistics of the original indicators, the Influence Indicator Statistics Module 20 performs data normalization processing. The Influence Indicator Statistics Module 20 accesses the historical change database maintained internally by the system and retrieves the maximum statistical value of all change records for the five indicators within the past preset time period. For any given indicator's original statistical value... and its corresponding historical maximum value The statistical module for influencing indicators uses formulas. Calculate the normalized values. After processing, the influence indicator statistics module 20 generates a normalized influence feature vector containing five components, and transmits it to the regression weight evaluation module 30.

[0051] The regression weight assessment module 30 receives the normalized impact feature vector and performs risk assessment using a pre-built multiple linear regression model. The regression coefficients of this multiple linear regression model are trained based on historical fault severity data. The regression model expression used by the regression weight assessment module 30 is as follows: ; in, Indicates the severity of the fault due to historical changes; to The regression coefficients correspond to the five indicators respectively; This is the random error term. The regression weight evaluation module 30 stores a set of coefficients that have been trained and converged. .

[0052] The regression weight evaluation module 30 calculates the internal weights of various influences based on the stored regression coefficients. For the direct influence components, the regression weight evaluation module 30 calculates the weights. as well as For the indirect impact portion, the regression weight evaluation module 30 calculates the weight denominator. And thus obtain , as well as .

[0053] Finally, the regression weight evaluation module 30 calculates the total impact score of this change by combining the preset global weight factors. Global weighting factors include those that directly influence weighting factors. And indirect influence weighting factors And satisfy The regression weight evaluation module 30 performs the following weighted aggregation calculation: ; Calculated total impact score Limited to the interval [0,1], the higher the value, the greater the potential disruptive risk to downstream business from model changes. The regression weight evaluation module 30 outputs this score to the adaptive strategy generation module 40 for the dynamic generation of subsequent validation strategies.

[0054] One embodiment of the present invention provides a specific implementation method for generating adaptive verification configurations based on impact scores. This process is executed by the adaptive policy generation module 40, which aims to dynamically adjust the stringency and coverage of data verification based on the quantitative assessment results of change risks.

[0055] The adaptive strategy generation module 40 receives the total impact score output by the regression weight evaluation module 30. To ensure that the data validation pass criteria can adapt to the potential risk levels of changes, the adaptive strategy generation module 40 establishes a non-linear mapping relationship between the total impact score and the data similarity validation threshold. The adaptive strategy generation module 40 pre-stores the basic similarity threshold. and adjustment intensity coefficient .

[0056] The adaptive strategy generation module 40 uses an exponential function to calculate the dynamic threshold for data verification in this change. The specific calculation logic follows the formula. In this calculation logic, when the total impact score... When the value approaches 0, the dynamic threshold approaches the basic similarity threshold. This corresponds to the standard verification criteria for low-risk change scenarios. When the total impact score... When it increases, The term increases accordingly, making the dynamic threshold... It approximates the value 1. This monotonically increasing functional relationship ensures that high-risk changes must meet higher data consistency requirements to pass system verification, thereby reducing the probability of data mutations in core business processes.

[0057] While determining the verification threshold, the adaptive policy generation module 40 performs topological analysis based on the lineage graph provided by the two-layer lineage analysis module 10 to determine the set of sentinel nodes. The adaptive policy generation module 40 not only focuses on the direct output of the target model itself, but also on the transmission effect of the data flow throughout the entire link. To this end, the adaptive policy generation module 40 calculates the betweenness centrality index of all nodes downstream of the target model in the graph.

[0058] The adaptive policy generation module 40 calculates betweenness centrality using a traversal algorithm. For any downstream node in the graph The adaptive strategy generation module 40 calculates all node pairs Number of shortest paths between And the nodes passed through in these shortest paths Number of paths The adaptive strategy generation module 40 is based on the formula... Accumulation yields nodes The betweenness centrality value. This value quantifies the importance of a node as an information transmission hub in the entire data network.

[0059] The adaptive strategy generation module 40 sorts all downstream nodes in descending order based on the calculated betweenness centrality values. The adaptive strategy generation module 40 then extracts the top nodes from the sorted list. One node serves as a sentinel node, among which The threshold value is a preset integer. These sentinel nodes constitute the key monitoring set for this change, used to capture data anomalies amplified during long-link propagation. The adaptive policy generation module 40 will use the calculated dynamic threshold value. The data is packaged with the sentinel node set and transmitted to the data validity verification module 50 as the basis for subsequent verification steps.

[0060] One embodiment of the present invention provides a specific implementation of a multi-level data rationality verification method. This process is executed by the data rationality verification module 50, and its core lies in calculating and determining the degree of data consistency by comparing the data vector characteristics before and after the change.

[0061] The data validity verification module 50 first performs data feature extraction. This is done for the target model. and the set of sentinel nodes determined by the adaptive policy generation module 40. For each node in the process, the data validity verification module 50 obtains data snapshots from two states. The data validity verification module 50 reads historical data generated from the last successful run before the change and constructs a historical data vector. Meanwhile, the data rationality verification module 50 reads the real-time data generated during the trial operation after the changes and constructs a real-time data vector. The vectors described above consist of statistics on key numerical fields in the node output table. These statistics include, but are not limited to, the number of rows, the null value rate, and the sum of numerical columns.

[0062] After constructing the data vector, the data validity verification module 50 uses the cosine similarity algorithm to calculate the single-point consistency index. For any node to be verified... The data rationality verification module 50 calculates real-time data vectors. With historical data vectors The cosine value of the included angle in vector space.

[0063] The specific calculation logic follows the formula: ; In the formula, and These represent the magnitudes of the vectors. The calculated values ​​are... The value is located in the range [-1, 1]. The closer the value is to 1, the more similar the data distribution characteristics are before and after the change, and the higher the data stability.

[0064] Subsequently, the data rationality verification module 50 performs aggregate calculation of the global consistency index. The data rationality verification module 50 considers not only the similarity of the target model itself but also the verification results of the sentinel nodes. To reflect the differences in the contribution of different nodes to overall consistency, the data rationality verification module 50 introduces a distance decay mechanism. For sentinel nodes, their hierarchical depth from the target model is... The larger the value, the smaller the impact of its data fluctuations on the weight of the overall indicators.

[0065] Global Consistency Metrics The calculation formula is as follows: ; In the formula, These are the preset base weight coefficients for the target model, used to balance the validation weight of the core model and downstream links. The natural logarithm term in the denominator... This is the distance decay factor, which ensures that minor fluctuations in remote nodes do not excessively interfere with global decision-making.

[0066] Finally, the data validity verification module 50 executes the verification and judgment logic. The data validity verification module 50 receives the dynamic threshold output by the adaptive strategy generation module 40. and the calculated global consistency index Compare the values ​​with this threshold. If The data rationality verification module 50 determines that the data fluctuation in this change is within an acceptable range, generates a verification pass signal, and triggers the version and graph management module 70. If... The data rationality verification module 50 determines that the change has caused abnormal data drift and generates a blocking signal to terminate the release process.

[0067] One embodiment of the present invention provides a specific implementation of dynamic weight correction driven by verification residuals. This process is executed by the weight feedback correction module 60, aiming to utilize the actual data drift results generated by this change to back-calibrate the mathematical model parameters in the regression weight evaluation module 30, thereby achieving closed-loop adaptive optimization of the evaluation system.

[0068] The weight feedback correction module 60 first obtains two key values ​​generated in the previous steps: the estimated total impact score output by the regression weight evaluation module 30. And the global consistency index calculated by the data rationality verification module 50. The weighted feedback correction module 60 calculates the actual data drift. Its numerical value is defined as This drift objectively reflects the true degree of variation in data features after the model change.

[0069] To quantify the deviation between the predicted score and the actual drift, the weighted feedback correction module 60 constructs a loss function. The loss function characterizes the square of the prediction error, and its specific calculation expression is as follows: When the total impact score Compared with actual data drift When the differences are large, the loss function The increase in the value indicates that the current regression model parameters have failed to accurately capture the risk characteristics of this change.

[0070] Based on the constructed loss function, the weight feedback correction module 60 uses the gradient descent algorithm to update the regression coefficients stored in the regression weight evaluation module 30 online. For each coefficient in the multiple linear regression model (For different influencing index dimensions), the weight feedback correction module 60 calculates the partial derivative of the loss function with respect to the coefficient, i.e., the gradient value.

[0071] The weight feedback correction module 60 adjusts the weights based on the preset learning rate. The regression coefficients are iteratively corrected along the opposite direction of the gradient. The correction formula follows... During this process, if the actual drift of a certain change is much greater than the estimated score, and the change occurs on the [number]th [day], [further action will be taken]. If an indicator (such as the number of indirect correlation systems) has a large value, then the regression coefficient corresponding to that indicator is... A positive gain will be obtained.

[0072] After the coefficient update is completed, the weight feedback correction module 60 will set the corrected coefficients. The data is written back to the parameter storage area of ​​the regression weight evaluation module 30. This update operation ensures that the system can call the calibrated parameter model during the next model change evaluation, thereby improving the accuracy of potential risk prediction.

[0073] One embodiment of the present invention provides a specific implementation method for closed-loop management of model versions and map updates. This process is executed by the version and map management module 70, aiming to ensure the traceability of change operations and the timeliness of the lineage map.

[0074] Once the data validity verification module 50 determines that the verification is successful, the version and graph management module 70 initiates the version archiving process. The version and graph management module 70 serializes the target model metadata, code script snapshots, and current runtime parameters before the change. The version and graph management module 70 then writes the serialized data, along with the generated global consistency index report, into the historical version database, forming an immutable change audit record.

[0075] Subsequently, the Version and Graph Management Module 70 performs the model release operation. The Version and Graph Management Module 70 pushes the modified model scripts and configuration parameters to the production environment's job scheduling system. The Version and Graph Management Module 70 updates the system's operation and maintenance log status, updating the target model's running version number to the current modified version identifier, completing the switchover of online logic.

[0076] After the release is completed, the version and graph management module 70 performs a reconstruction and update of the lineage graph. The version and graph management module 70 obtains the new and removed dependencies of the changed model. Based on this change information, the version and graph management module 70 performs topological correction on the global lineage graph maintained in the two-layer lineage resolution module 10.

[0077] The version and graph management module 70 creates new nodes and directed edges in the graph and deletes invalid connections. The reconstructed updated lineage graph is stored as the system baseline, serving as the basic data source for path analysis by the two-layer lineage analysis module 10 during the next model change, thereby maintaining the evaluation system's real-time awareness of the platform's status.

Claims

1. A method for quantitative assessment and data verification of the impact of changes to a big data platform model, characterized in that, Includes the following steps: S100. Perform two-layer lineage analysis to construct a full lineage graph. The computer system reads the job scheduling logs of the big data platform to extract the runtime data flow and parses the interface code abstract syntax tree of the target model to extract implicit data dependencies, generating a directed acyclic graph with the target model as the center. S200. Statistically analyze the impact indicators of changes and implement normalization processing. Based on the full kinship map, statistically analyze the multi-dimensional impact indicators caused by the change of the target model, and generate a normalized impact feature vector by combining the historical maximum values. S300. Calculate the total impact score based on the regression weight model. Utilize a pre-set multiple linear regression model to calculate the internal weights of each multidimensional impact indicator, and combine this with global weight factor aggregation to calculate the total impact score representing the potential risk level of the change. S400. Generate an adaptive data verification strategy, calculate the dynamic threshold for data verification based on the total impact score, and select sentinel nodes from the full kinship graph based on betweenness centrality. S500: Perform data rationality verification and judgment, calculate the data similarity values ​​of the target model and the sentinel node respectively, and introduce a distance decay factor to calculate the global consistency index. By comparing the global consistency index with the dynamic threshold, it is determined whether the verification passes. S600. Based on the verification residual, the regression weights are corrected. After the verification is completed, the residual between the global consistency index and the total impact score is calculated, and the regression coefficients of the multiple linear regression model are updated using the gradient descent algorithm. S700: Execute model release and graph update. When the verification is successful, serialize and store the metadata snapshot before the change, execute the online operation of the changed model, and add, delete and update nodes and edges in the full kinship graph.

2. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The execution of two-layer kinship analysis for constructing a complete kinship map specifically includes: The job scheduling center of the big data platform is connected to obtain the full job operation logs within a preset time window, and the data flow information is extracted using a regular expression matching algorithm. The time dependency sequence between nodes is established based on the task start and end timestamps. The source code of the target model and the task associated with the target model is converted into an abstract syntax tree using a parser. The abstract syntax tree is traversed to locate the data source declaration statement, table name references in the SQL query statement, and parameter configuration of the data writing function. Hard-coded input parameters, output parameters, and connection identifiers of external data sources are extracted. The identified independent data entities are instantiated as nodes in the full lineage graph, and the nodes are divided into model nodes and business system nodes. Directed edges are constructed based on the extracted dependencies. Each edge is assigned a dependency type and a data transmission path attribute. The dependency type identifies whether the dependency is an explicit dependency derived from dynamic log verification or an implicit dependency derived from static code analysis.

3. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The multidimensional impact indicators include: the number of directly related models, the number of directly related systems, the number of indirectly related models, the number of indirectly related systems, and the number of changes in related fields; The normalized influence feature vector is generated by retrieving the maximum statistical value of each multidimensional influence indicator in the historical change record and calculating the ratio of the current original statistical value of each multidimensional influence indicator to the corresponding historical maximum value.

4. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 3, characterized in that, The calculation of the total impact score based on the regression weight model specifically includes: By calling the multiple linear regression model, multiplying each component of the normalized influence feature vector by its corresponding regression coefficient and summing them, an estimated value of the severity of historical change failures is obtained. Based on the regression coefficients, the internal weights of various influences are calculated. For the direct influence, the proportion of the regression coefficients corresponding to the direct correlation model and the direct correlation system in the sum of their coefficients is calculated. For the indirect influence, the proportion of the regression coefficients corresponding to the indirect correlation model, the indirect correlation system, and the changes in the correlation field in the sum of their coefficients is calculated. The total impact score is obtained by performing a weighted aggregation calculation based on the global weight factor, multiplying the weighted sum of the direct impact portion by the direct impact weight factor, multiplying the weighted sum of the indirect impact portion by the indirect impact weight factor, and adding the two results together.

5. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The dynamic threshold for data verification based on the total impact score is specifically calculated based on a nonlinear mapping relationship: Set a basic similarity threshold and an adjustment intensity coefficient, and calculate the attenuation term using an exponential function that uses the natural constant as the base and the total influence score as the negative exponent; The difference between the numerical value and the attenuation term is calculated. The difference is multiplied by the adjustment intensity coefficient and then added to the basic similarity threshold to obtain the dynamic threshold. The dynamic threshold increases monotonically and non-linearly with the increase of the total influence score and approaches the numerical value.

6. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The selection of sentinel nodes from the full kinship graph based on betweenness centrality specifically includes: The betweenness centrality of each node located downstream of the target model in the full kinship graph is calculated by a traversal algorithm. For any downstream node, the total number of shortest paths between all node pairs in the full kinship graph and the number of paths passing through the downstream node in these shortest paths are calculated. The quotient of the latter and the former is calculated, and the quotients of all node pairs are accumulated to obtain the betweenness centrality value of the downstream node. The downstream nodes are sorted in descending order based on the calculated betweenness centrality value. The first preset number of nodes in the sorted list are selected as the sentinel nodes, and the sentinel nodes are included in the data verification scope.

7. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The calculation of the data similarity between the target model and the sentinel node specifically includes: Extract the historical data vectors of the target model and the sentinel node before the change and the real-time data vectors after the change, respectively. The historical data vectors and the real-time data vectors are composed of the statistical values ​​of key numerical fields in the node output table. The cosine similarity algorithm is used to calculate the data similarity value of each node. Specifically, the dot product of the real-time data vector and the historical data vector is calculated, and the quotient of the dot product and the product of the magnitudes of the two vectors is calculated.

8. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 7, characterized in that, The introduction of a distance decay factor to calculate the global consistency index specifically includes: Determine the hierarchical depth of each sentry node from the target model, calculate the natural logarithm of the hierarchical depth and add one, as the denominator of the distance attenuation; Calculate the quotient of the data similarity value of each sentinel node and the corresponding distance decay denominator to obtain the decayed similarity value, and then sum the decayed similarity values ​​of all sentinel nodes. The data similarity values ​​of the target model are weighted using the basic weight coefficients of the target model, and the accumulated result is weighted using the remaining weight coefficients. The two weighted results are then added together to obtain the global consistency index.

9. The method for quantitative assessment and data verification of the impact of big data platform model changes according to claim 1, characterized in that, The regression weights based on the verification residual correction specifically include: The difference between the calculated value and the global consistency index is used as the actual data drift amount. Calculate half the square of the difference between the total impact score and the actual data drift to construct a loss function; The partial derivatives of the loss function with respect to each regression coefficient are calculated as gradient values, and the product of the learning rate and the gradient values ​​is subtracted from the original regression coefficients to iteratively correct the regression coefficients of the multiple linear regression model.

10. A device for quantitative assessment and data verification of the impact of changes to a big data platform model, applied to the method described in any one of claims 1-9, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for quantitative assessment and data verification of the impact of changes in a big data platform model.

Images