Data quality closed-loop management and intelligent evaluation system for ai model iteration

Abstract

This invention discloses a closed-loop data quality management and intelligent evaluation system and method for AI model iteration, relating to the interdisciplinary fields of artificial intelligence and data governance. The method uses causal probes embedded in the model to detect data anomalies in real time and constructs a dynamic causal probability graph to characterize the causal relationship between data quality, internal model perturbations, and performance. It achieves accurate attribution through counterfactual reasoning, utilizes model-based reinforcement learning for simulation and deduction to determine the optimal repair solution, and simultaneously optimizes and adjusts the decision-making strategy based on feedback from the repair effect. This invention realizes a fully intelligent closed loop from data quality perception and attribution diagnosis to accurate repair and system self-learning and self-optimization, providing an automated data quality assurance mechanism for the continuous and stable iteration of AI models.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and data governance, and more specifically, to a data quality closed-loop management and intelligent evaluation system for AI model iteration. Background Technology

[0002] In the process of AI model iteration, data quality directly affects model performance and generalization ability. Traditional data quality probing and evaluation methods mostly rely on manually defined rules or static thresholds, which are difficult to adapt to dynamic and complex data environments, and lack in-depth analysis and quantitative attribution of how data quality problems specifically affect the internal state of the model and its final performance.

[0003] Existing technologies have explored various solutions. For example, Chinese patent CN119938658A discloses a data quality probing method based on a large language model, which uses intelligent agents to automatically identify data types and generate analysis scripts. While this method improves automation, its analysis logic still focuses on rule matching and script execution, failing to establish a quantifiable and interpretable causal model between data quality problems and model performance degradation. This leads to inaccurate attribution and a lack of in-depth basis for remediation decisions. Chinese patent CN120410574A discloses a data quality assessment model based on a multi-task learning framework, evaluating data quality from multiple dimensions. This solution provides multi-dimensional quality scores, but it is essentially an open-loop assessment system. The assessment results lack an automated and intelligent decision-making link with subsequent remediation actions, and the system lacks the ability to self-optimize using remediation feedback.

[0004] In summary, existing technical solutions generally have the following shortcomings: First, data quality assessment and model performance analysis are disconnected, making it difficult to achieve accurate root cause identification; second, a closed loop is formed between quality assessment, problem diagnosis, and remediation actions; and third, the system lacks self-learning and adaptive optimization capabilities based on historical intervention experience. Summary of the Invention

[0005] To overcome the aforementioned deficiencies in the prior art, embodiments of the present invention provide a data quality closed-loop management and intelligent evaluation system for AI model iteration, thereby addressing the problems mentioned in the background section. The present invention aims to solve the aforementioned problems in the prior art by providing a system capable of automatically attributing the impact of data quality on AI model performance, intelligently making decisions and executing corrective actions based on the attribution results, and possessing closed-loop learning capabilities.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data quality closed-loop management and intelligent evaluation system for AI model iteration, comprising: The causal probe module is configured to acquire activation data of a specified layer inside the AI ​​model to be monitored, and to reconstruct the activation data using a lightweight graph neural network whose structure is topologically isomorphic to the specified layer. The data anomaly is determined by comparing the reconstruction error with a preset threshold. The causal probe module is configured to analyze the internal activation patterns of the lightweight graph neural network when an anomaly is detected, and output an output containing a subgraph identifier. and perturbation vector Abnormal signals, among which This is used to identify the most anomaly-affected connected subgraph in the lightweight graph neural network, which maps to a specific set of functional units within the AI ​​model. Used to quantify the vector deviation of the activated data relative to the historical normal pattern; The causal graph modeling and updating module is configured to build and maintain a dynamic causal probability graph. The Includes data quality nodes that characterize the quality status of external data sources. The state is determined by the disturbance vector. The characteristic perturbation node defined by the statistics and model performance nodes that characterize the output performance metrics of the AI ​​model. ; The module is configured to dynamically update the time-series state data of each node using a time-series difference learning algorithm. The weights of the directed edges connecting the nodes are assigned so that the weight values ​​can adaptively reflect the strength of the causal influence of dynamic changes between the nodes. The attribution diagnostic module is configured to respond to the performance node. The state value deviates from its preset normal fluctuation range in the dynamic causal probability diagram. Attribution calculations based on counterfactual reasoning are performed; the calculations include: In the In the middle, the following are sequentially placed The states of each upstream node of a node are simulated to be their respective historical baseline states, while the states of all other nodes, network topology, and weights are fixed, and calculations are performed based on the current graph structure and weight parameters. The counterfactual expected value of a node under each intervention is calculated by... The absolute difference in the current actual observed state value of the node is used to quantify the contribution of each upstream node to the current performance deviation event; The adaptive repair decision module is configured to receive a list of upstream nodes sorted by contribution, and to use the dynamic causal probability graph. The current state is used as an inferable environment model, and a model-based reinforcement learning framework is used for decision-making. The module is configured to be in the The environment simulates the execution of different operations from a predefined set of repair operations. The expected performance gains, required computational resource overhead, and breadth of intervention on the state of nodes in the graph are evaluated by a defined reward function. The overall decision is made to achieve a target repair scheme that balances the benefits and costs. The remediation execution and verification module is configured to safely implement the target remediation scheme in an isolated sandbox testing environment or a low-volume production environment, and to collect actual performance index data of the model in one or more evaluation cycles after implementation; The closed-loop learning module is configured to base its learning on the difference between the actual performance results collected after the repair implementation and the expected results obtained during the decision simulation phase. This generates a unified feedback signal; The module is configured to utilize the feedback signal Simultaneously driving two learning processes: First, update the dynamic causal probability graph. The weights of directed edges directly related to this repair intervention are used to correct the model's understanding of causal effects; Second, the parameters of the reinforcement learning strategy within the adaptive repair decision module are updated to optimize future decision quality.

[0007] Furthermore, the training data of the lightweight graph neural network comes from the activation output of the specified layer of the AI ​​model to be monitored under historical data without quality anomalies. Its training process aims to minimize the reconstruction error of the normal sample set, so that the trained network can encode and reproduce the typical feature distribution pattern of the network layer under healthy data input.

[0008] Furthermore, the causal graph modeling and updating module is configured to dynamically update edge weights using a temporal differential learning algorithm. The temporal differential learning algorithm calculates the predicted value of the node state at the next moment based on the current weight and the node's historical state, compares the predicted value with the actual observed value at the next moment to obtain the temporal differential error, and then traces this error signal back along the time dimension and distributes it to the relevant edges according to the node's historical activity (qualification trace), thereby realizing incremental adjustment of edge weights.

[0009] Furthermore, the predefined set of repair operations includes at least one of the following operations: performing value range cleaning or distribution resampling on the identified problematic data fields; using generation techniques to synthesize samples that meet the requirements for data-scarce fields; and addressing nodes in the AI ​​model to be monitored that are perturbed by specific features. Supervised fine-tuning is performed on a subset of associated local parameters, and adaptable logic units for correcting specific scenario biases are inserted into the inference computation path of the AI ​​model to be monitored.

[0010] Furthermore, the supervised fine-tuning operation on the local parameter subset updates the parameter range based on the feature perturbation node corresponding to the contribution output by the attribution diagnosis module. Associated subgraph identifier Sure; The system uses a pre-stored mapping table to assign the subgraph identifiers. The parameters are parsed into a specific, indexable set in the AI ​​model to be monitored, thereby enabling precise, range-limited model updates.

[0011] The working method of the data quality closed-loop management and intelligent evaluation system for AI model iteration includes the following steps: S1. Obtain activation data of a specified layer within the AI ​​model to be monitored, perform reconstruction analysis using a lightweight graph neural network whose structure is topologically isomorphic to the specified layer, determine whether there are quality anomalies in the current input data by comparing the reconstruction error with a preset threshold, and generate a subgraph identifier when an anomaly is determined. and perturbation vector Abnormal signals; S2. Construct and maintain a dynamic causal probability graph. The Includes data quality nodes Featured perturbation nodes and model performance nodes ,in The state of a node is based on the perturbation vector. Defined by calculated statistical characteristics (such as norm); Based on the time-series state data collected from each node at fixed intervals, a time-series difference learning algorithm is used to dynamically update... The weights of directed edges between nodes are designed to reflect the latest statistical causal relationships. S3, Responding to the performance node The state value continues to deviate from its historical baseline range. Perform counterfactual reasoning to conduct root cause analysis: Sequentially The states of each upstream node of a node are simulated and set to their historical baseline states. The post-intervention state is calculated based on the current graph structure and edge weights. The counterfactual expected value of the node, and by comparing this expected value with The difference between the current actual observations of the nodes is used to calculate the contribution of each upstream node to this performance deviation event; S4. Based on the contribution ranking results, using the... The current state is used as a model of the environment to run a model-based reinforcement learning decision-making process; In the environmental model, the state transitions caused by different operations in the predefined repair operation set are simulated and extrapolated. The long-term value of each operation sequence is evaluated based on a reward function that integrates expected performance gains, estimated execution costs and intervention breadth, and the target repair scheme is selected accordingly. S5. Execute the target repair scheme in an isolated environment or a low-flow experimental group; S6. Based on the difference between the actual performance results observed after the implementation of the plan and the expected results in the decision-making simulation. This generates a unified performance feedback signal, which is then used to synchronously execute two updates: First, update the aforementioned The weights of directed edges directly related to this repair action are used to calibrate the causal cognitive model; Second, update the value function or policy parameters on which the reinforcement learning decision-making process depends to optimize subsequent decisions in similar states.

[0012] Furthermore, the structure of the lightweight graph neural network is isomorphic to the neuron connection topology of the specified layer, and it is trained in a supervised manner using historical activation data of the AI ​​model without quality anomalies. The training objective is to enable the network to recover the normal activation pattern of the input with minimal reconstruction error, thereby learning the baseline feature manifold of the healthy state of the layer.

[0013] Furthermore, in the counterfactual reasoning calculation, for any upstream node... The counterfactual expectation value is calculated through the following process: ,in Indicates the current state of the node. This indicates the historical baseline state of the node. This represents the current edge weight. This formula simulates the process by which the impact propagates along the weighted edges to the target node after "intervention" is performed on a specific node in a causal graph.

[0014] Furthermore, the reward function of the reinforcement learning decision-making process is constructed as follows: In the form of, Represented in the environmental model The expected performance improvement obtained from the simulation This represents the estimated resource cost of performing this operation. This represents the direct impact of the operation. The number of nodes in the middle, To balance the coefficients of intervention breadth, this function structure is designed to guide the search for repair strategies that can achieve better performance improvements with a smaller range of modifications.

[0015] Furthermore, in the step of updating the edge weights, for the node being intervened in this instance... Directly connected and pointing to the performance node The weight adjustment of the edge. Differences between actual performance feedback and decision expectations after implementation and pre-intervention nodes State value The product is directly proportional, that is This mechanism enables causal graphs It can empirically revise the strength of the causal effect represented internally based on feedback from the effects of each real intervention.

[0016] The technical effects and advantages of this invention are as follows: This invention introduces a causal probe module and a dynamic causal probability graph (… This establishes a quantitative analysis foundation for how data quality issues affect model performance. The causal probe captures anomalous perturbations in the model's internal feature layers in real time using a lightweight graph neural network and vectorizes these perturbations. Nodes represent data quality indicators, feature perturbations, and model performance, while edge weights dynamically represent causal strength.

[0017] When performance degrades, the system... Counterfactual reasoning is used to quantify the contribution of a node to the current problem by calculating the expected change in performance indicators after intervening in the state of a specific node to a baseline value. This helps to transform vague quality issues into clear and quantifiable points of responsibility, providing a basis for subsequent precise intervention.

[0018] Based on the aforementioned accurate attribution, this invention employs model-based reinforcement learning for repair decisions. The system combines the attribution results with... The current state is defined as the decision state, and predefined repair operations (such as data cleaning and local fine-tuning) are defined as actions, and are... It functions as an environmental model to simulate and extrapolate action effects.

[0019] By designing a reward function that comprehensively considers expected performance gains, execution costs, and the scope of intervention, and utilizing planning algorithms such as Monte Carlo tree search, the system can evaluate the long-term benefits of different remediation strategies in a virtual environment, thereby autonomously selecting the optimal solution. This approach transforms remediation decision-making from relying on human experience to an automated and intelligent process based on causal models and benefit calculations, helping to improve decision-making efficiency.

[0020] This invention also implements closed-loop learning and optimization. After executing the repair plan, the system compares the actual performance improvement with the expected improvement in the decision-making stage and calculates the prediction error. This error signal is used for dual updates: first, to update... The weights of relevant edges in the model make the causal model more accurate in predicting the "effect of a specific repair action"; secondly, the value function of reinforcement learning is updated. This design allows the system to learn from each intervention, continuously revising its understanding of the causal relationship between data quality and performance, and its corrective decision-making strategies. This helps the system improve the accuracy and autonomy of its governance during long-term model iteration and maintenance. Attached Figure Description

[0021] Figure 1 This is a flowchart of the overall data quality closed-loop management system of the present invention; Figure 2 This is a flowchart of the causal probe and counterfactual attribution of the present invention; Figure 3 This is a flowchart of the repair decision and closed-loop learning branches of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 As attached Figures 1 to 3 The data quality closed-loop management and intelligent evaluation system for AI model iteration, as shown below, has the following specific implementation details: The system operates at a fixed period (For example Run a closed-loop process.

[0024] During the initial deployment of the system or the first online launch of model M, the dynamic causal probability graph... The edge set is empty. The system first enters an initial learning phase (e.g., lasting 24-48 hours), during which the edges of each node are mainly accumulated. , , The state time series data of the causal edge and its weight are calculated based on statistical methods.

[0025] During this period, the attribution diagnosis and automatic repair decision-making functions will be temporarily disabled. Once the number of edges accumulated exceeds a preset threshold and the weights stabilize over a period of time, the system automatically switches to a full-function closed-loop operation mode.

[0026] In each running cycle Internally, the causal probe module processes all online inference requests from model M within the current period and summarizes any detected anomalies. Simultaneously, the system collects various data quality metrics and model performance metrics. Current observation value .

[0027] The causal graph modeling and updating module integrates all information collected in this cycle (including characteristic perturbation signals, data quality indicators, and performance indicators) and updates it. The node state and edge weights. When performance metrics... Below the exponential moving average (EMA) of all its periodic values ​​over the past 24 hours Double time ( For example, it is an adjustable parameter. This triggers the attribution diagnosis module.

[0028] This module is based on the latest Snapshot, perform counterfactual reasoning, calculate the cause The contribution of various data quality issues to the decline The adaptive repair decision module filters based on contribution results. The node, For example, the contribution threshold. (This threshold can be periodically adjusted based on the distribution of historical attribution results). Through simulation, the repair scheme with the highest expected overall benefit is selected from a predefined set of operations. .

[0029] The fix execution and verification module was then implemented in an isolated environment (e.g., 1% of online traffic). In the subsequent evaluation period t', the closed-loop learning module adjusts its performance based on the actual performance changes observed after the fix. Calculate the prediction error And update synchronously with this error signal The edge weights and internal strategies of the decision module are used to complete a full closed-loop iteration.

[0030] The specific implementation methods for each module are as follows: 1. Implementation of the causal probe module This module is designed for each cycle. Internally, it detects anomalies within the model caused by input data quality issues.

[0031] Construction and training of a lightweight graph neural network G: Select an intermediate network layer of model M, denoted as layer M. (For example, the output of the last convolutional layer or an intermediate hidden layer of a Transformer), set the layer The output feature dimension is Build a lightweight graph neural network Its node set Include Each node, and the layer of Each output node corresponds to one other node. edge set The topology and model M in the middle layer The original connection relationship to the next computational layer is completely consistent, and the edge weight is initialized to 1.

[0032] The training data uses historically recognized high-quality data (without known major data problems) as input, and the model M is run with a data acquisition layer. The corresponding activation vectors form the training dataset. (one The matrix, (Number of samples).

[0033] train This enables it to reconstruct the input. Specifically, it allows the samples to be reconstructed. enter This yields the output x'.

[0034] The loss function is defined as (Mean squared error), trained using the Adam optimizer, with an initial learning rate set to With a batch size of 128, training continued until the reconstruction error on the validation set no longer decreased significantly. After training was completed, Encoded layer The activation distribution pattern under normal data has fixed parameters.

[0035] Online monitoring and anomaly detection: In the cycle Within, for each input sample, obtain the layer. activation vector .Will Input trained This yields the reconstructed output x'. The scalar reconstruction error is then calculated. (L2 norm).

[0036] Set an abnormal threshold . It can be initialized based on the 95th percentile value of the reconstruction error distribution calculated on a high-quality validation set during the training phase, and can also be based on the error detected online. The distribution is dynamically calibrated, for example, by recalculating its 95th percentile value weekly to accommodate long-term, slow changes in model M or the data distribution. When When this happens, it is determined that the current input has abnormal data quality.

[0037] Output triples (S, V, c) and cycle summary: Within a cycle, for each request determined to be abnormal, the following triples are generated: Subgraph identifier S: in In the middle, calculate each node Local reconstruction residuals Select the top one with the largest residual. Nodes ( Can be set to 10% of ), and include these nodes in All edges in the graph constitute the critical anomaly subgraph SG.

[0038] generate The specific steps are: (1) Select the Each node index is sorted in ascending order; (2) Concatenate these indices into a string using commas; (3) Calculate the MD5 hash value of the string; (4) Take the first 16 characters of the hash value as a unique identifier. , During system initialization, the structure of model M and its layers have been analyzed. A mapping relationship is established between a specific group of neurons (or feature channels). This mapping relationship is persistently stored in the form of a lookup table, with the key being the subgraph identifier. The value is the corresponding list of model parameter indices.

[0039] Perturbation vector V: Calculation ,in It is the training set The mean vector ( dimension). The vector that deviates from the normal pattern of the current activation is quantified.

[0040] Confidence level c: Calculated .

[0041] At the end of this cycle, this module will generate all triplet pairs. Output to downstream modules.

[0042] 2. Implementation of the Cause-Effect Graph Modeling and Update Module This module is responsible for each cycle. Internally, update the dynamic causal probability graph. .

[0043] Node definition and status acquisition: Data quality node D: Each Each node corresponds to a monitorable data source quality metric. For example: : The missing rate (range [0, 1]) of the "User Age" field in the input request, calculated over a period of time. .

[0044] In the upstream log service, the proportion of records where the delay between the event timestamp and the processing timestamp exceeds 5 seconds (value range [0, 1]), calculated over a period of time. .

[0045] Featured perturbation node F: each Nodes and a unique subgraph identifier Binding. Its state value. In the cycle The internal calculation is as follows: Collect all identifiers as The perturbation vector in the triplet Take the average of the L2 norms of these vectors, that is If no specific [received] within the period The triplet, then Set to 0.

[0046] Model performance node P: for example This indicates that model M in the most recent period Internal performance index observations For example, AUC (range [0, 1]), the evaluation period is .

[0047] Construction and dynamic updates: Edge initialization (one-time): When the system runs for the first time, for any pair of nodes... and ( may be or , may be or We collect their historical state sequences over a relatively long period (e.g., 7 days). We then perform Granger causality tests using time series analysis, setting the maximum lag order to 1. If the p-value of the test Then it is believed that there exists from arrive The causal direction, in Add directed edges Its initial weight Can be set to .

[0048] Weights are dynamically updated (online every period): This process aims to make... The edge weights reflect the latest causal strength in real time, employing the TD (Time Difference) method in temporal difference learning. )algorithm.

[0049] Input: Current period The state of all nodes Edge weights of the previous period and attenuation factor .

[0050] Prediction: For each performance node (e.g.) ),based on The model calculates its expected predicted value. The summation traversal All upstream nodes.

[0051] Error: Obtain the current period Actual observed values Then, the timing difference error is calculated. .

[0052] Qualification trace update: for each upstream node Maintain a qualification record Its update rules are as follows . This is the attenuation factor, with values ​​ranging from [0.8, 0.99], for example... .

[0053] Weight update: based on the formula Update the weights. This is the learning rate, which typically ranges from [0.001, 0.01], for example... (Choose a smaller value to ensure the stability of weight updates).

[0054] Output: Updated edge weights It is used for forecasting and decision-making in the next cycle.

[0055] 3. Implementation of the attribution diagnosis module when When the triggering conditions are met, this module will periodically... The function is called internally.

[0056] Counterfactual reasoning calculation: Obtain the current The complete state, let have One direct upstream node .

[0057] 1. For each Record its current state .

[0058] 2. Determine its baseline state .for Class node, Take the average status over the same period of the past 7 days; for Class node, .

[0059] 3. Key calculations: In Simulated intervention in China. Based on Model calculation In intervention The counterfactual expected value: This formula is based on The linear weighting assumption.

[0060] 4. Calculate the contribution. .

[0061] Output attribution report: All according to Sort in descending order. Set a contribution threshold. (For example This threshold can be adjusted based on the distribution of historical attribution results to filter out weak contributions. Generate a list of reports. .

[0062] 4. Implementation of the Adaptive Repair Decision Module This module is based on Decision-making and repair strategies. The goal of this module is to learn an optimal strategy. To maximize long-term cumulative rewards, the system maintains an action-value function. It indicates the state. Next action After that, the expected cumulative discount reward can be obtained.

[0063] Predefined repair operation set :Include atomic operations .

[0064] Data cleaning. For missing fields, the operation is to fill in the missing values ​​with the historical median of that field.

[0065] Data resampling. To address distribution drift, the operation is as follows: for the current batch of data, calculate the weight of each sample according to the historical reference distribution and resample.

[0066] : Local fine-tuning of the model. The operation is as follows: based on The identifier corresponding to the node The subset of model parameters mapped to it is found by looking up the table. Regarding the isolated data, only for... The parameters in the code perform gradient descent, and the number of fine-tuning rounds is controlled by an early stopping strategy (e.g., terminating when the loss no longer decreases after two consecutive rounds). The learning rate... .

[0067] Inference correction. The operation is as follows: After the final output layer of model M, a learnable scalar bias is added to the predicted scores for a specific user group (such as new users). , It starts at 0 and is updated based on online feedback.

[0068] Model-based reinforcement learning decision making: state Defined as a vector ,in Source .

[0069] action :from Select an operation .

[0070] Environmental Model : that is, the current .

[0071] reward function : .

[0072] (Expected performance improvement): In Chinese simulation action Calculate the state based on the Env model. Execute Post-expected performance value . .

[0073] (Estimated cost): for each Preset fixed cost The unit is a virtual computing unit (e.g., , ).

[0074] :action Directly affected middle Number of nodes.

[0075] : Penalty coefficient, used to balance effect and intervention range, is determined through offline grid search within the range [0.1, 1.0], for example... (This aims to encourage precise intervention and avoid unnecessary global modifications to the model.)

[0076] Decision Process (Monte Carlo Tree Search, MCTS): Input: Current state (As the root node), the environment model reward function operation set .

[0077] Process: Iterative execution of four steps: selection, expansion, simulation, and backtracking. In the "simulation" phase, utilizing... Rapidly extrapolate random action sequences to a certain depth Calculate cumulative discount rewards ,in Discount factor (e.g.) Simulation depth It can be dynamically adjusted based on the available computation time budget for each decision.

[0078] Output: After a preset number of simulations, the sub-action with the most visits under the root node is selected as the target repair solution. .

[0079] 5. Fixed the implementation of the execution and verification module. At the start of the next cycle, strictly implement the protocol in a fully isolated sandbox environment or a dedicated, small-scale (e.g., 1%) online traffic test group. For example, executing At that time, only the data from the experimental group were used to... Make minor adjustments.

[0080] After execution, the experimental group will be continuously monitored throughout one or more subsequent full evaluation cycles. New performance metrics and compared with the baseline value of the experimental group before repair. contrast.

[0081] 6. Implementation of the closed-loop learning module This module starts after the verification cycle ends, enabling the system to evolve itself.

[0082] Effect feedback calculation: Calculate the actual improvement Obtain the forecasts for the decision-making stage. Calculate the prediction error. .

[0083] Update cause-effect graph :Will As an additional temporal differential error signal, it is used to update the edge weights related to this repair action.

[0084] For the operation Nodes that directly intervene , its arrival The edge weights are updated a second time: .in, It is a learning rate specifically designed for closed-loop feedback, and is typically smaller than the online learning rate. ,For example .like The value is below the negative threshold (e.g.) This indicates that the repair action may have significant negative effects, in which case the update magnitude of the edge weights should be reduced (e.g., by reducing the update magnitude). (Temporarily halved).

[0085] Optimize decision-making strategies: As a response to action In state Correcting the execution effect. Updating the action value function: ,in The policy learning rate typically ranges from [0.05, 0.3], for example... .like If this happens, an additional penalty is introduced in the update. This update makes the system more prepared for similar situations in the future. Select action The expected return estimates are more accurate. Significant negative feedback events are marked as high-priority log entries.

[0086] Within the information processing flow of the aforementioned system, the core data flow can be summarized as follows: triples generated by the causal probe module. drive middle Node status update; State and weight are used for performance prediction and attribution diagnosis; attribution reporting. and Together they constitute the decision-making state Actions that result from decision output After execution and verification, the resulting effect deviation Simultaneously feedback to Weight update and decision value function The update. This information flow forms a reinforced loop that connects end to end.

[0087] The above processes together constitute the intelligent evaluation and driving mechanism for data quality, with intelligent evaluation manifested as a dynamic causal probability graph. A multi-dimensional, attributable quality assessment is conducted. This assessment produces a quality score and also quantifies the specific impact of issues in each dimension on model performance through counterfactual reasoning. This quantitative assessment result directly serves as input for remediation decisions, thus making the evaluation process a core decision-making basis driving the management closed loop.

[0088] The closed-loop self-evolution capability stems from the system's ability to continuously compare and predict the quality impact during the evaluation phase. The actual effect after the repair was performed and utilize their differences Simultaneous optimization of the evaluation model (update) Weights) and Decision-Making Strategies (Updates) function).

[0089] Therefore, the accuracy of the evaluation, the precision of the decision-making, and the system's self-learning ability form a mutually reinforcing cycle within this closed loop.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data quality closed-loop management and intelligent evaluation system for AI model iteration, characterized in that, include: The causal probe module is configured to acquire activation data of a specified layer within the AI ​​model to be monitored, and reconstruct the activation data using a lightweight graph neural network whose structure is topologically isomorphic to the specified layer; the causal probe module is configured to output a subgraph identifier based on a comparison between the reconstruction error and a threshold. and perturbation vector Abnormal signals; The causal graph modeling and updating module is configured to build and maintain a dynamic causal probability graph. The Includes data quality nodes Featured perturbation nodes and model performance nodes Among them, characteristic perturbation nodes The state is based on the disturbance vector Confirmed; the module is configured to dynamically update the node's state time-series data. The weights of edges between nodes; The attribution diagnostic module is configured to respond to the performance node. The state is abnormal in the dynamic causal probability graph. Perform counterfactual reasoning calculations on the above; the calculations include: applying the... The state of at least one upstream node of a node is modified to a preset baseline state, and the current edge weight is calculated. The contribution of each upstream node is determined based on the difference between the modified counterfactual expected value and the actual state value. The adaptive repair decision module is configured to use the dynamic causal probability graph based on the contribution level. Make decisions for the reinforcement learning strategy of the environment model and output the target repair scheme selected from the predefined set of repair operations; The repair execution and verification module is configured to execute the target repair scheme in an isolated environment and collect performance feedback data. The closed-loop learning module is configured to update the dynamic causal probability graph based on the difference between the performance feedback data and the decision expectation data. The relevant edge weights in the algorithm and the reinforcement learning strategy in the adaptive repair decision module.

2. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 1, characterized in that, The training data for the lightweight graph neural network comes from the activation output of the specified layer of the AI ​​model to be monitored under historical data without quality anomalies.

3. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 1, characterized in that, The causal graph modeling and updating module is configured to dynamically update edge weights using a temporal difference learning algorithm; the update amount of the temporal difference learning algorithm is jointly determined by the temporal difference error, node state, and learnable qualification trace.

4. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 1, characterized in that, The predefined set of repair operations includes at least one of the following operations: cleaning or resampling specific data fields, generating synthetic samples for specific data fields, updating some parameters in the AI ​​model to be monitored, and inserting correction logic into the inference path of the AI ​​model to be monitored.

5. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 4, characterized in that, The range of updates to certain parameters is determined by the feature perturbation node corresponding to the contribution output by the attribution diagnosis module. Associated subgraph identifier Sure.

6. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to any one of claims 1-5, characterized in that, The system operates by including the following steps: S1. Obtain activation data of a specified layer within the AI ​​model to be monitored, perform reconstruction analysis using a lightweight graph neural network, determine data anomalies based on reconstruction errors, and generate a subgraph identifier. and perturbation vector Abnormal signals; S2. Construct and maintain a dynamic causal probability graph. The Includes data quality nodes Featured perturbation nodes and model performance nodes ,in The state of a node is based on the perturbation vector. Confirmed; dynamically updated based on node status time-series data. The weights of edges between nodes; S3, Responding to the performance node The state is abnormal, in Perform counterfactual reasoning above: modify the stated The state of at least one upstream node of a node is calculated to its baseline state based on the current edge weight. The counterfactual expected value of the node is calculated, and the contribution of each upstream node is calculated based on the difference between the counterfactual expected value and the actual state value. S4. Based on the stated contribution, using the... As an environmental model, a reinforcement learning decision-making process is used to select a target remediation scheme from a predefined set of remediation operations. S5. Execute the target repair plan in an isolated environment; S6. Update the above based on the difference between the actual performance feedback after the implementation of the plan and the expected results. The relevant edge weights and the strategy of the reinforcement learning decision process.

7. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 6, characterized in that, The structure of the lightweight graph neural network is isomorphic to the neuron connection topology of the specified layer, and the AI ​​model is trained using historical activation data without quality anomalies.

8. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 6, characterized in that, In the counterfactual reasoning of the system, for the upstream node The counterfactual expectation value is calculated in the following way: ,in Indicates the node status. Indicates the baseline state. This represents the edge weight.

9. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 6, characterized in that, The reward function of the reinforcement learning decision-making process is determined by the expected performance improvement term, the estimated execution cost term, and the penalty term based on the number of intervention nodes.

10. The data quality closed-loop management and intelligent evaluation system for AI model iteration according to claim 6, characterized in that, In the system update edge weights, for the node being intervened... The difference between the weight adjustment of the relevant edges and the actual performance feedback and decision expectations after the implementation of the scheme. and pre-intervention nodes The state values ​​are positively correlated.

Images