Intelligent cleaning and repairing method for abnormal data of big data system based on ai analysis

Abstract

The AI analysis-based big data system abnormal data intelligent cleaning and repairing method includes five steps: S1: collecting original business data, system running state and downstream task historical feedback signal of a data source end, and generating a multi-dimensional feature fusion signal; S2: based on the fusion signal, an abnormal identification signal containing an abnormal type label and a severity score is generated; S3: the abnormal identification signal is input into a strategy decision intelligent agent, and a cleaning strategy signal containing an optimal cleaning action index and an execution parameter is generated; S4: according to the cleaning strategy, a repairing model is called to process abnormal data; S5: downstream task execution effect feedback signals are collected, and model updating driving signals are formed by aligning the abnormal identification and cleaning strategy signals, and are fed back to the abnormal detection model and the strategy decision intelligent agent. The AI analysis-based big data system abnormal data intelligent cleaning and repairing method can solve the problems that the data cleaning method lacks self-adaptability, cannot cope with complex abnormalities and is difficult to continuously optimize.

Description

Technical Field

[0001] This invention relates to the fields of big data processing and artificial intelligence technology, specifically to a method for intelligent cleaning and repair of abnormal data in big data systems based on AI analysis. Background Technology

[0002] In big data systems, data quality issues directly impact the accuracy of upper-level data analysis, AI model training, and business decision-making. With the diversification of data collection channels, the explosive growth of data scale, and the increasing complexity of data flow, the manifestations of anomalous data have evolved from traditional single-point value anomalies to more complex forms involving temporal context anomalies, multi-field combined logic anomalies, and cross-source data inconsistencies. Existing data cleaning technologies are mainly divided into two categories: rule-based methods and statistical model-based methods. Rule-based methods rely on manually preset thresholds or business rules, such as setting the value range of fields or functional dependencies between fields. These methods are costly to formulate and difficult to maintain, and struggle to cope with dynamic changes in business logic. Statistical model-based methods, such as isolated forests and local outliers, can automatically identify outliers in statistical distributions, but lack the ability to understand business semantics, easily misjudging reasonable fluctuations consistent with business logic as anomalies, leading to a high false positive rate. In recent years, some solutions have begun to incorporate deep learning models for anomaly detection, but the following technical bottlenecks remain: First, anomaly detection and data cleaning / repair are disconnected. Detection results fail to effectively guide the selection of repair strategies, and the repair process often employs a single interpolation method, unable to adaptively adjust repair methods based on anomaly type and business scenario. Second, cleaning strategies are rigid, lacking the ability to perceive the actual effects on downstream tasks and failing to dynamically optimize the cleaning logic based on downstream task performance feedback, leading to a disconnect between the cleaning process and the final application goals of the data. Third, existing methods are mostly static models, making it difficult to adapt to the conceptual drift of data distribution over time in big data systems, resulting in a significant decline in cleaning effectiveness after long-term operation. Fourth, the repair results lack credibility assessment; repair values ​​are output in a black-box format, making it impossible for users to judge their reliability. For high-value scenarios such as finance and healthcare, this lack of interpretability severely restricts the practical application of automated data cleaning technology. Therefore, there is an urgent need for an intelligent data cleaning and repair method with adaptive decision-making capabilities, continuous evolution capabilities, and interpretable results. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an intelligent cleaning and repair method for abnormal data in big data systems based on AI analysis, to solve the problems of existing data cleaning methods lacking adaptability, being unable to cope with complex anomalies, and being difficult to continuously optimize. This invention generates a fusion signal by collecting multi-dimensional data features, outputs an anomaly identification signal through an anomaly detection model, dynamically selects the optimal cleaning strategy based on system load and downstream task sensitivity, calls the repair model to output repair values ​​and confidence scores, transfers low-confidence scores to manual review, automatically adopts high-confidence scores and outputs them downstream, collects feedback on downstream task execution effects to form a model update driving signal, and feeds it back to the anomaly detection model and the strategy decision-making agent to trigger incremental learning, forming a closed-loop self-evolution mechanism.

[0004] This invention provides an AI-based intelligent cleaning and repair method for abnormal data in big data systems, including: S1: Collect raw business data from the data source, system operating status, and historical feedback signals from downstream tasks to generate multi-dimensional feature fusion signals; S2: Based on this fused signal, the data distribution offset and business logic correlation features are extracted through the anomaly detection model to generate an anomaly identification signal containing anomaly type labels and severity scores; S3: Input the anomaly identification signal into the strategy decision-making agent. The agent combines the current system load and the sensitivity of downstream tasks to dynamically optimize within the preset cleaning action space and generate a cleaning strategy signal containing the optimal cleaning action index and execution parameters. S4: Based on the cleaning strategy, call the repair model to process abnormal data. The model outputs repair values ​​and confidence scores simultaneously. If the scores are lower than the preset threshold, they are marked for manual review. Otherwise, the repair values ​​are automatically adopted and output to the downstream. S5: Collect feedback signals of downstream task execution effects, associate and align them with anomaly identification and cleaning strategy signals to form model update driving signals, and send them back to the anomaly detection model and strategy decision-making agent to trigger incremental learning.

[0005] In one embodiment of the present invention, the multi-dimensional feature fusion signal is composed of business logic features, temporal features, and statistical distribution features. The business logic features are extracted by constructing functional dependencies and association rule constraints between data fields to reflect the inherent business semantic consistency within the data. The temporal features are obtained by applying sliding window sampling to the original business data and performing periodic decomposition and trend fitting operations to characterize the evolution of the data over time. The statistical distribution features are obtained by performing kernel density estimation and quantile calculation on the original business data to describe the global distribution pattern and outlier degree of the data, and are acquired synchronously when collecting the original business data from the data source. The system operation status signal includes parameters such as computing resource utilization, storage input / output waiting time, and network transmission latency, which characterize the available resource margin during the data cleaning process. The downstream task historical feedback signal includes the trend of downstream model training loss values ​​and business indicator fluctuation records corresponding to historical cleaning results, which characterize the historical impact of different cleaning strategies on the performance of downstream tasks. These three types of feature signals are input into a multimodal feature extraction network composed of multiple parallel feature extraction branches. Each branch performs nonlinear mapping and dimensional alignment on its corresponding features and then fuses them to output a multi-dimensional feature fusion signal. The calculation formula for the multi-dimensional feature fusion signal is as follows: in, The final output is a multi-dimensional feature fusion signal with a dimension of d (the unified feature dimension after dimension alignment). (∙) The Sigmoid activation function normalizes feature values ​​to [0,1] to adapt to subsequent model inputs; These correspond to business logic characteristics, time sequence characteristics, and statistical distribution characteristics, respectively. The attention weights for the three core features are calculated using a cross-feature attention mechanism, achieving weighted enhancement of important features. This represents the set of features from the two classes other than the i-th class after mapping. S is the nonlinear mapping and dimension alignment operator for the i-th type of feature; S is the system operating state signal; and H is the historical feedback signal of the downstream task. Standardized operators align the dimensions and feature units of S and H; Let S and H be the learnable weight matrices, representing the contribution of the two types of auxiliary signals to the fused features.

[0006] In one embodiment of the present invention, the anomaly detection model uses a lightweight graph neural network to construct a data lineage graph. Data fields are treated as nodes, and dependencies between fields and data flow paths are treated as edges. Based on multi-dimensional feature fusion signals, each node is assigned an initial feature vector. Through the message passing and aggregation mechanism of the graph neural network, the local neighborhood structure features and global topological role features of each data record in the data lineage graph are calculated. Based on these features, anomaly type labels are divided into three categories: isolated point anomalies, context anomalies, and set anomalies. Isolated point anomalies correspond to records where a single field value deviates from its statistical distribution; context anomalies correspond to records where the value is reasonable in a specific temporal or business context but deviates from the overall contextual pattern; and set anomalies correspond to records where a single field value is normal but multiple field combinations violate functional dependencies or business rule constraints. While outputting the anomaly type labels, the anomaly detection model calculates the Mahalanobis distance between the anomaly record and normal records in the feature space and combines this with the prior occurrence frequency of the anomaly category to which the anomaly record belongs to generate an anomaly severity score normalized to the range of zero to one. The anomaly severity scoring algorithm is as follows: in, Score the severity of the abnormal record x under its anomaly category θ; ( Mahalanobis distance represents the feature space distance between abnormal records and normal data of the same category, eliminating linear correlation between features; The feature vector of the abnormal data record to be evaluated. This is a normal data sample set within the same category; Normal sample set under abnormal category θ The mean vector and covariance matrix; The prior frequency of occurrence of the anomaly category θ is obtained from historical anomaly data statistics; The maximum Mahalanobis distance of the normal sample set under the same category is used as a normalization factor to ensure that the score strictly falls within the [0,1] interval. The closer the value is to 1, the more severe the abnormality.

[0007] In one embodiment of the present invention, the policy decision agent is constructed using a deep reinforcement learning network. Its state space is composed of multi-dimensional feature fusion signals, anomaly recognition signals, current system load signals, and downstream task sensitivity signals. The downstream task sensitivity signals include weight parameters of the dependence of downstream tasks on data accuracy, integrity, and real-time performance. The action space of the policy decision agent is a preset cleaning action space, which includes actions such as discarding abnormal data, data imputation based on statistical features, data repair based on generative models, and re-collecting data from the source. The reward function of the policy decision agent is defined as the weighted sum of the downstream task performance improvement gain and the data entropy retention rate after repair, minus the computational resource consumption cost introduced by the cleaning operation. The downstream task performance improvement gain is calculated by inputting the repaired data into the downstream task model and comparing the difference in task performance indicators before and after repair. The data entropy retention rate after repair is determined by calculating the relative entropy distance between the data before and after repair in the probability distribution space. When the policy decision agent optimizes in the action space based on the current state, it is trained using a policy gradient method combined with a priority experience replay mechanism, and outputs a cleaning policy signal containing the optimal cleaning action index and execution parameters.

[0008] In one embodiment of the present invention, the repair model configures differentiated repair network structures for different types of anomaly identification signals. When the anomaly type label is isolated point anomaly, the repair model adopts a generative repair network based on conditional generative adversarial network. This network uses the relevant field features that have not occurred as conditional input, outputs the repair candidate values ​​of the anomaly field through the generator network, and performs adversarial constraints on the consistency between the repair candidate values ​​and the real data distribution through the discriminator network. When the anomaly type label is context anomaly, the repair model adopts a sequence-to-sequence repair network based on a combination of temporal convolutional network and attention mechanism. This network uses the complete record sequence within the temporal window before and after the anomaly record as context encoding, and decodes and generates repair values ​​after weighted aggregation of key information in the temporal context through the attention mechanism. When the anomaly type label is set anomaly, the repair model adopts a relational constraint repair network based on graph neural network. This network uses the features of neighboring nodes that are directly or indirectly related to the anomaly record in the data lineage graph as constraint conditions, and solves the optimal repair value that satisfies all functional dependencies through constraint satisfaction optimization iteration.

[0009] In one embodiment of the present invention, the repair model outputs a repair confidence score while generating repair values. This repair confidence score is a weighted synthesis of a model confidence component and a data confidence component. The model confidence component is obtained by calculating the variance of the repair value distribution through multiple forward propagations of the repair model using the Monte Carlo dropout method, and is used to characterize the uncertainty of the model regarding the current repair result. The data confidence component is obtained by calculating the local density of neighboring normal records in the feature space around the abnormal record and the satisfaction of the abnormal record and the repaired record under business logic constraints, and is used to characterize the reasonableness of the current repair result in terms of local data structure and business semantics. When the repair confidence score is lower than a preset threshold, the system pushes the manual review flag signal, along with the original content of the abnormal record, the abnormal type label and severity score in the abnormal identification signal, and multiple candidate repair values ​​generated by the repair model and their corresponding confidence scores, to the manual review queue. When the repair confidence score reaches or exceeds the preset threshold, the system directly adopts the repair value to generate an automatic adoption repair value signal, and writes the repaired data into the downstream data interface according to the original data format and data lineage.

[0010] In one embodiment of the present invention, after the repaired data is output to the downstream task, a task execution effect feedback signal generated by the downstream task based on the repaired data is collected. This task execution effect feedback signal includes the change in accuracy and recall of the downstream task model on the validation set, as well as the fluctuation of key performance indicators in the downstream business application. The feedback signal is associated and aligned with the anomaly identification signal generated in step S2 and the cleaning strategy signal generated in step S3 according to the data batch identifier and timestamp to form a model update driving signal containing the mapping relationship between the difference in data quality before and after cleaning and the change in downstream task performance. The model update driving signal is used as a supervision signal and input into the anomaly detection model. The network parameters of the anomaly detection model are incrementally updated by calculating the gradient information between the feedback signal and the output of the anomaly detection model. At the same time, the model update driving signal is used as a reward signal and input into the strategy decision agent. The action value evaluation network of the strategy decision agent is updated through temporal differential error, so that the anomaly detection model and the strategy decision agent have the ability to continuously adapt to the dynamic changes in data distribution.

[0011] In one embodiment of the present invention, when incremental learning is triggered, the system synchronously executes a data drift detection operation. This data drift detection operation calculates a drift index by comparing the distribution differences between the current batch of data and historical data in the feature space. When the drift index exceeds a preset threshold, a data drift event is determined to have occurred. After a data drift event occurs, the system temporarily suspends the incremental learning process of the current anomaly detection model and the policy decision agent. First, it triggers an online adaptive adjustment mechanism for the model. This mechanism uses repaired data with confidence scores higher than a preset high threshold in the current batch of data as pseudo-label samples, and combines them with historically retained real labeled samples to construct a temporary training set. This allows for rapid fine-tuning of the anomaly detection model and the policy decision agent. After the fine-tuning is completed and the drift index falls below the preset threshold, the incremental learning process resumes. During the incremental learning process, an elastic weight solidification method is used to apply additional constraints to the key network parameters in the anomaly detection model and the policy decision agent that affect the data drift detection results, in order to prevent catastrophic forgetting during the incremental learning process. The algorithm for calculating the drift index based on distribution differences is as follows: in, This is the data drift index; the larger the value, the greater the degree of drift between the current batch of data and historical data. The weighting coefficient represents the contribution weight of kernel density distribution similarity to the drift index, and is adaptively adjusted according to the business scenario and data type. The KL divergence represents the kernel density distribution of the current batch of data. Kernel density distribution of historical data The similarity is such that a larger KL divergence indicates a greater difference in distribution; P(X) is determined by the Gaussian kernel function K... The kernel density estimation is achieved, where h is the kernel bandwidth; This serves as a reference dataset for drift detection. Here, is the normalization factor for the KL divergence; N and M are the data of the current batch, respectively. Historical data Sample size ∈ , ∈ ; Euclidean distance characterizes the feature space differences between individual samples. The maximum Euclidean distance in the feature space is used to normalize the calculation of this part; The average normalized Euclidean distance between the current batch and historical data represents the overall shift in the feature space.

[0012] In one embodiment of the present invention, in step S3, when the strategy decision-making agent dynamically optimizes the preset cleaning action space, it synchronously acquires the cleaning strategy signals of adjacent nodes in the data lineage graph as a collaborative reference. This collaborative reference is used to trigger a strategy coordination mechanism when the cleaning strategy signal of the current data record conflicts with the cleaning strategy signal of the adjacent node record. This strategy coordination mechanism corrects the cleaning strategy signal of the current data record by constructing a local strategy consistency constraint objective function to minimize the product of the difference in cleaning strategies between adjacent nodes and the strength of data dependency relationship. This ensures that data records with strong dependencies adopt the same or compatible cleaning actions, avoiding the breakage of data lineage relationship or destruction of data consistency due to inconsistent cleaning strategies of adjacent nodes. The corrected cleaning strategy signal, after retaining the original execution parameters and adding a strategy consistency adjustment coefficient, is used as the final cleaning strategy signal input to step S4.

[0013] In one embodiment of the present invention, when collecting original business data from the data source in step S1, a unified semantic alignment mechanism is adopted for structured and unstructured data sources. This semantic alignment mechanism constructs a domain knowledge graph to semantically map data from different data sources that have the same business meaning but differ in field names or data formats, and establishes a semantic association index for cross-source data. In step S4, when the repair model is called to process abnormal data according to the cleaning strategy, if the field containing the abnormal data in the current data source has a semantic association index pointing to a synonym field in another data source, the repair model first extracts the real-time data of the corresponding field from other data sources for cross-source cross-validation. If the cross-source data meets the consistency verification condition, the cross-source data is directly used as the repair value. If the cross-source data does not meet the consistency verification condition or the cross-source data is abnormal at the same time, the repair model generates a repair value based on the confidence weighted fusion of multi-source data, wherein the weighting weight is dynamically calculated based on the historical data quality score and real-time data integrity status of each data source.

[0014] The present invention provides an AI-based intelligent cleaning and repair method for abnormal data in big data systems. The method outputs anomaly identification signals through an anomaly detection model. The strategy decision-making agent dynamically selects the optimal cleaning strategy based on system load and downstream task sensitivity. The repair model is called to output repair values ​​and confidence scores. Low confidence scores are transferred to manual review, while high confidence scores are automatically adopted and output to downstream systems. Feedback on the execution effect of downstream tasks is collected to form a model update driving signal, which is fed back to the anomaly detection model and the strategy decision-making agent to trigger incremental learning, forming a closed-loop self-evolution mechanism. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the method for intelligent cleaning and repair of abnormal data in big data systems based on AI analysis; Figure 2 Flowchart for intelligent cleaning and repair of abnormal data. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0019] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0020] Please see Figure 1-2The image shows the intelligent cleaning and repair method for abnormal data in a big data system based on AI analysis according to the present invention. The method comprises five steps: S1: Collecting original business data from the data source, system operating status, and historical feedback signals from downstream tasks to generate a multi-dimensional feature fusion signal; S2: Based on this fusion signal, extracting data distribution offset and business logic correlation features through an anomaly detection model to generate an anomaly identification signal containing anomaly type labels and severity scores; S3: Inputting the anomaly identification signal into a strategy decision-making agent, which dynamically optimizes within a preset cleaning action space based on the current system load and downstream task sensitivity to generate a cleaning strategy signal containing the optimal cleaning action index and execution parameters; S4: Calling a repair model to process abnormal data according to the cleaning strategy, the model synchronously outputs repair values ​​and confidence scores. If the score is below a preset threshold, it is marked for manual review; otherwise, the repair value is automatically adopted and output to the downstream; S5: Collecting feedback signals of downstream task execution effects, aligning them with the anomaly identification and cleaning strategy signals to form a model update driving signal, which is then fed back to the anomaly detection model and the strategy decision-making agent to trigger incremental learning.

[0021] like Figure 1As shown, the intelligent cleaning and repair method for abnormal data in big data systems based on artificial intelligence analysis proposed in this invention essentially constructs a complete closed-loop system from anomaly perception to strategy decision-making, then to execution repair, and finally to self-evolution based on the execution effect. This method elevates data quality governance from traditional static rule execution to an intelligent process with dynamic adaptability and continuous optimization capabilities through five tightly coupled core steps. In step S1, the system first performs multi-dimensional signal acquisition and fusion. Specifically, the method simultaneously acquires three types of signals from different but complementary sources. The first type is the raw business data signal output from the data source, which is the main object to be cleaned, including structured table data, semi-structured logs, or unstructured text generated by various business systems. The second type is the system operating status signal. This part of the signal does not come directly from the business data itself, but rather characterizes the real-time resource status of the computing environment where data cleaning is performed, including the CPU utilization rate of computing nodes, available memory, input / output waiting time of the storage system, and network latency parameters on the data acquisition and transmission links. The third type is historical feedback signals from downstream tasks. The introduction of this signal distinguishes this method from traditional independent data cleaning tools. It records the performance changes of downstream tasks caused by the input of cleaned data into them during past data cleaning operations, such as machine learning model training or business intelligence report analysis. Specifically, this can be reflected in the trend of the loss function value of the downstream model, the fluctuation of accuracy on the validation set, and the improvement or decline of key business indicators. These three types of signals are simultaneously input into a multimodal feature extraction network. This network is not a simple signal concatenation, but rather configures parallel feature extraction branches for the characteristics of different signal types. Each branch performs targeted nonlinear mapping and dimensional transformation on the input signal. Finally, these aligned feature vectors are deeply fused in the network's fusion layer, outputting a comprehensive multi-dimensional feature fusion signal. The advantage of this fusion signal is that it simultaneously includes the content features of the data itself, the processing capacity features of the current system, and the effectiveness features of the impact of historical data cleaning on downstream tasks, providing a complete input foundation for subsequent anomaly identification.

[0022] Step S2 performs intelligent anomaly detection based on the multi-dimensional feature fusion signal generated in step S1. The calculation formula for the multi-dimensional feature fusion signal is as follows: in, The final output is a multi-dimensional feature fusion signal with a dimension of d (the unified feature dimension after dimension alignment). (∙) The Sigmoid activation function normalizes feature values ​​to [0,1] to adapt to subsequent model inputs; These correspond to business logic characteristics, time sequence characteristics, and statistical distribution characteristics, respectively. The attention weights for the three core features are calculated using a cross-feature attention mechanism, achieving weighted enhancement of important features. This represents the set of features from the two classes other than the i-th class after mapping. S is the nonlinear mapping and dimension alignment operator for the i-th type of feature; S is the system operating state signal; and H is the historical feedback signal of the downstream task. Standardized operators align the dimensions and feature units of S and H; Let S and H be learnable weight matrices representing the contributions of the two auxiliary signals to the fused features. These weight matrices are continuously updated iteratively during model training based on the performance of downstream anomaly detection. If a signal contributes more to anomaly identification, the corresponding weight matrix value increases accordingly, ensuring that information from system operation status and historical feedback from downstream tasks is accurately integrated into the fused features, fully leveraging its auxiliary role in subsequent anomaly detection. The entire formula calculation process is completely synchronized with the forward propagation process of the multimodal feature extraction network. After processing, weighting, and fusing various features, the final output multi-dimensional feature fusion signal fully retains the core information of business semantics, temporal evolution, and statistical distribution, while incorporating auxiliary information such as system resource margin and historical downstream influence. This provides an accurate and comprehensive feature foundation for constructing a data lineage graph and assigning initial feature vectors to nodes in subsequent anomaly detection models. This solves the problems of feature information loss, dimensional interference, and insufficient weights for important features in traditional feature fusion methods, significantly improving the effectiveness and relevance of the fused features.

[0023] The core of this step is a trained anomaly detection model. This model doesn't rely solely on a single statistical outlier detection logic; instead, it extracts two types of deep-level features simultaneously from the fused signal. The first type is data distribution shift features, which are the degree of difference between the statistical distribution of the current batch of data and its historical normal distribution. This shift may manifest as mean drift, variance expansion, or a change in the overall distribution pattern. The second type is business logic association features, which are whether the inherent functional dependencies, association rule constraints, and lineage relationships between data fields have been violated. By jointly analyzing these two types of features, the anomaly detection model can distinguish different types of anomalies. For example, a field's value deviating from its normal distribution range but conforming to business logic fluctuations, or a field's value falling within the normal range but its combination with related fields violating business rules, will be identified by the model as different types of anomalies. Along with outputting anomaly type labels, the model also calculates a normalized anomaly severity score. This score comprehensively considers the distance between the anomalous record and normal records in the high-dimensional feature space, the prior frequency of the anomaly type in historical data, and the degree of impact of the anomalous record on the overall data distribution. In this way, the anomaly identification signal output in step S2 not only tells the system "where the problem is," but also clarifies "what type of problem it is" and "how serious the problem is," providing a precise basis for the differentiated selection of subsequent cleaning strategies. The anomaly severity scoring algorithm is as follows: in, Score the severity of the abnormal record x under its anomaly category θ; ( Mahalanobis distance represents the feature space distance between abnormal records and normal data of the same category, eliminating linear correlation between features; The feature vector of the abnormal data record to be evaluated. This is a normal data sample set within the same category; Normal sample set under abnormal category θ The mean vector and covariance matrix; The prior frequency of occurrence of the anomaly category θ is obtained from historical anomaly data statistics; The maximum Mahalanobis distance of the normal sample set under the same category is used as a normalization factor to ensure that the score strictly falls within the [0,1] interval. The closer the value is to 1, the more severe the abnormality.

[0024] Step S3 introduces a strategy decision-making agent, which is the core component of this method for adaptive data cleaning. This agent receives the anomaly identification signal output from step S2 as input, but its decision-making process is not isolated; it simultaneously considers the real-time load status of the current system and the sensitivity requirements of downstream tasks to data quality. The system load status reflects the available computational resources for cleaning operations. For example, when the system is busy, the agent tends to choose cleaning actions with lower computational overhead. Downstream task sensitivity reflects the different degrees of dependence of different downstream applications on data accuracy, completeness, and real-time performance. For example, data streams used for real-time risk control are sensitive to latency but have relatively relaxed requirements for completeness, while data used for financial statements have extremely high requirements for accuracy and completeness. The strategy decision-making agent internally maintains a preset cleaning action space, which contains various types of cleaning operations, such as directly discarding abnormal records, imputation based on statistical features such as the mean or median, repairing complex fields based on generative artificial intelligence models, or triggering re-collection from the data source. Based on the current input state information, the agent dynamically optimizes within this action space through a reinforcement learning mechanism. The decision-making objective of the intelligent agent is not simply to eliminate anomalies, but to pursue the maximization of comprehensive benefits. These benefits comprehensively consider the improvement of downstream task efficiency after cleaning, the degree to which the repaired data preserves the original information distribution, and the computational resources consumed by the cleaning operation itself. After optimization, the intelligent agent outputs a cleaning strategy signal, which explicitly specifies which cleaning action should be used for the current abnormal record, as well as the specific parameters required for the action, such as the specific algorithm type used during imputation, the model version used during generative repair, or the timeout threshold during resampling.

[0025] Further, step S4 is responsible for executing the cleaning strategy determined in step S3. Based on the action index specified in the cleaning strategy signal, the method calls the corresponding repair model to process the abnormal data. A key innovation is that the repair model, while generating a repair value, does not simply output a result, but simultaneously outputs a repair confidence score. This score reflects the model's confidence in its own repair results. The method then performs triage processing based on the relationship between this score and a preset threshold. If the repair confidence score is lower than the preset threshold, it means the model lacks sufficient confidence in the current repair result. In this case, the system will not blindly adopt the repair result directly, but will generate a signal for manual review, and push the original content of the abnormal record, the type label and severity score in the abnormal identification signal, one or more candidate repair values ​​generated by the repair model and their corresponding confidence scores to a dedicated manual review queue for final decision-making by experienced personnel. Conversely, if the repair confidence score reaches or exceeds the preset threshold, the system considers the repair result to have high reliability, directly generates an automatic adoption repair value signal, and writes the repaired data into the downstream data interface in the original data format for use by downstream tasks. This confidence-based triage mechanism, while ensuring automation efficiency, retains a channel for manual intervention in high-risk scenarios, solving the "hesitation" problem faced by AI black-box models in data governance. Step S5 achieves the closed loop and continuous evolution of the entire method. After outputting the repaired data to downstream tasks in step S4, the method does not stop there, but actively collects feedback signals on the task execution effects generated by downstream tasks based on these repaired data. These feedback signals are specifically manifested as changes in the performance indicators of downstream task models, such as increases or decreases in classification accuracy, fluctuations in recall, or changes in key performance indicators in downstream business applications. The method precisely aligns these feedback signals with the anomaly identification signal generated in step S2 and the cleaning strategy signal generated in step S3 according to data batch identifiers and timestamps. Through the alignment operation, the system can establish a complete causal chain of "what anomaly was targeted, what cleaning strategy was adopted, and what downstream effect was ultimately caused." Based on this, the system generates a model update driving signal. This signal is essentially a training sample with supervision or reward information, which is fed back to the anomaly detection model in step S2 and the policy decision-making agent in step S3, respectively. For anomaly detection models, updating the driving signals enables them to learn from the performance of downstream processes, adjusting their sensitivity and classification criteria for identifying anomalies. This allows anomalies that cause downstream performance to deteriorate to be identified more accurately, while reasonable fluctuations that are misjudged as anomalies can be gradually corrected.For the policy decision-making agent, the update driving signal acts as a reward signal in reinforcement learning, enabling it to evaluate whether the previously selected cleaning strategy was correct, thereby optimizing its policy network and making decisions that are more beneficial to the downstream task effect when faced with similar anomalies in the future. In this way, the entire cleaning system forms a closed loop of "cleaning-use-feedback-optimization", which has the ability to continuously adapt to changes in data distribution and the evolution of business needs, and achieves the self-evolutionary characteristic of "becoming more accurate with use".

[0026] Regarding the composition and extraction method of the multi-dimensional feature fusion signal, this signal is not a simple stacking of raw data, but rather a combination of three features with different semantic dimensions: business logic features, temporal features, and statistical distribution features. Business logic features are extracted by constructing functional dependencies and association rule constraints between data fields. Specifically, the system first scans the metadata at the data source end, identifying linear or nonlinear functional relationships between numerical fields, mutual exclusion or implication relationships between categorical fields, and composite business rules across fields, such as logical consistency constraints between order status and payment amount fields. These relationships are then encoded into feature vectors to reflect the inherent business semantic consistency within the data. Temporal features are obtained by applying sliding window sampling to the original business data and performing periodic decomposition and trend fitting operations. For example, for sensor data or transaction log data with timestamps, the system slides a fixed-length window along the time axis, performs Fourier transform on the data sequence within the window to extract periodic components, and performs multinomial fitting to extract long-term trend components. The periodic and trend components are then concatenated to form a temporal feature vector, used to characterize the evolution of the data over time. Statistical distribution characteristics are obtained by kernel density estimation and quantile calculation of the original business data. Without assuming any data distribution pattern, the system uses kernel density estimation to fit the probability density function of each field and calculates multiple quantiles, including quartiles and percentiles. The key parameters of the density function are combined with the quantile values ​​to form a statistical distribution feature vector, used to describe the global distribution pattern and outlier degree of the data. Simultaneously with the collection of original business data, the system acquires system operation status signals, including computational resource utilization, storage input / output waiting time, and network transmission latency parameters. These parameters characterize the available resource margin during the data cleaning process and are normalized as additional conditional information input to the subsequent strategy decision-making module. Furthermore, historical feedback signals from downstream tasks are also collected synchronously. These signals include the trend of changes in the training loss value of the downstream model corresponding to historical cleaning results and records of fluctuations in business indicators, such as how much the accuracy of the downstream classification model improved or decreased after the previous cleaning operation. This historical feedback data is stored in an experience pool to characterize the historical impact of different cleaning strategies on the performance of downstream tasks. The three types of feature signals mentioned above are input into a multimodal feature extraction network consisting of multiple parallel feature extraction branches. This network uses graph convolutional layers to process business logic features, one-dimensional convolutional layers and long short-term memory networks to process temporal features, and fully connected layers to process statistical distribution features. Each branch performs nonlinear mapping and dimension alignment on the corresponding features. The output feature vectors of all branches are concatenated or weighted and summed in the feature dimension, and finally fused to output a comprehensive multi-dimensional feature fusion signal that includes data content, system state and historical experience.

[0027] Furthermore, regarding the specific implementation of the anomaly detection model, this model employs a lightweight graph neural network to construct a data lineage graph. The construction process of the data lineage graph is as follows: each field or data record in the data source is treated as a node in the graph, and the functional dependencies between fields, foreign key relationships, and the data flow paths in the extraction, transformation, and loading process are treated as edges in the graph. The weights of the edges can be assigned based on the dependency strength or flow frequency. Based on the multi-dimensional feature fusion signal generated in step S1, an initial feature vector is assigned to each node. This vector integrates the business logic features, temporal features, and statistical distribution features of the corresponding field of the node. Then, through the message passing and aggregation mechanism of the graph neural network, each node aggregates the feature information of its neighboring nodes. After multi-layer graph convolution operations, each node obtains an updated feature representation. This representation not only includes the node's own attribute information but also its local neighborhood structure features and global topological role features in the data lineage graph, such as whether a node is located at the central hub position of multiple dependencies. Based on the node features calculated above, the anomaly detection model classifies anomaly type labels into three categories. The first type is isolated anomalies, which correspond to records where the value of a single field deviates significantly from its own statistical distribution. For example, an age field might show a value far exceeding a reasonable range. These anomalies can be identified solely by the distribution of a single field. The second type is contextual anomalies, which correspond to records where the value is reasonable within a specific temporal or business context but deviates from the overall contextual pattern. For example, a sudden traffic spike during a low-load period at night might not be significant in absolute terms, but it is an anomaly relative to its temporal context. The third type is set anomalies, which correspond to records where the values ​​of individual fields are all within the normal range, but the combination of multiple fields violates functional dependencies or business rule constraints. For example, an order status of "shipped" but a tracking number field that is empty, or a combination of an age field and an education field that violates common sense. In addition to outputting anomaly type labels, the anomaly detection model also needs to output an anomaly severity score. The scoring method is as follows: First, the Mahalanobis distance between the abnormal record and the center of the normal record cluster in the feature space is calculated. This distance can eliminate the influence of the difference in the dimensions of different features and take into account the correlation between features. Then, this distance is multiplied by the reciprocal of the prior frequency of the abnormality type in historical data, that is, rare abnormality types are given higher severity weights. Finally, the above product value is normalized to the interval between zero and one through a logistic function to make the score comparable. The higher the score, the more severe the abnormality, and the more priority it needs to be dealt with.

[0028] This invention relates to a deep reinforcement learning decision-making mechanism for a policy decision-making agent. The agent is constructed using a deep reinforcement learning network, specifically a deep Q-network or a near-end policy optimization algorithm. The agent's state space is composed of multi-dimensional feature fusion signals, anomaly identification signals, current system load signals, and downstream task sensitivity signals. The downstream task sensitivity signal is a pre-configured vector containing weight parameters indicating the dependence of downstream tasks on data accuracy, completeness, and real-time performance. For example, for a risk control model, accuracy is weighted highest; for a real-time monitoring dashboard, real-time performance is weighted highest. The agent's action space is a pre-defined cleaning action space, which includes four basic action types: discarding abnormal data, data imputation based on statistical features, data repair based on generative models, and re-collecting data from the source. Discarding is suitable for anomalies of extremely high severity and with excessive repair costs; statistical imputation is suitable for isolated anomalies with relatively simple field distributions; generative repair is suitable for complex scenarios such as contextual anomalies or set anomalies; and re-collecting is suitable for situations where the data source experiences a brief failure and the data can be retransmitted. The agent's reward function is designed as a weighted sum of three terms: the first term is the downstream task performance improvement gain, specifically calculated by inputting the repaired data into the downstream task model and comparing the performance difference of the downstream task when using the data before and after repair, such as the increase in accuracy or the decrease in the loss function; the second term is the information entropy preservation rate of the repaired data, determined by calculating the relative entropy distance between the data before and after repair in the probability distribution space; the smaller the relative entropy distance, the less damage the repair operation caused to the information structure of the original data; the third term is the negative computational resource consumption cost of the cleaning operation, including CPU usage time and peak memory usage. The reward function subtracts the third term from the sum of the first two terms, allowing the agent to actively control computational costs while pursuing downstream performance and data fidelity. During the optimization process, the agent uses a policy gradient method combined with a priority experience replay mechanism for training. The priority experience replay mechanism allows the agent to learn more frequently from historical decision experiences with abnormally high or low reward values, thereby accelerating convergence. Finally, the agent outputs a cleaning strategy signal, which contains the index of the optimal cleaning action and the parameters required to execute the action. For example, when performing statistical imputation, it specifies whether to use the median or the mode, and when performing generative repair, it specifies which version of the generative model to use.

[0029] Specifically, regarding the differentiated network structure of the repair model for different types of anomalies, this invention does not employ a single repair model, but rather dynamically selects differentiated repair networks based on the anomaly type label in the anomaly identification signal. When the anomaly type label is an isolated point anomaly, the repair model employs a generative repair network based on a conditional generative adversarial network. This network consists of a generator and a discriminator trained adversarially. The generator takes relevant field features that have not experienced anomalies as conditional inputs. For example, for an anomaly in the age field of a record, the generator will input other normal fields in the record, such as occupation and income level, as conditions. The generator outputs multiple candidate repair values ​​for the anomaly field through deconvolution or upsampling operations. The discriminator is responsible for distinguishing the repair values ​​output by the generator from the true values ​​in the real data distribution. Through this adversarial training, the generator learns to produce repair values ​​that are indistinguishable from real values, making the repaired record statistically difficult to distinguish from normal records. When the anomaly type label is a contextual anomaly, the repair model employs a sequence-to-sequence repair network based on a combination of temporal convolutional networks and attention mechanisms. This network uses the complete record sequence within the time window before and after the anomaly record as context encoding. For example, for an anomaly record at time point t, the network reads all records within the interval from time point t minus the window size to time point t plus the window size. A temporal convolutional network performs multi-scale convolution operations on this sequence to extract patterns across different time spans. An attention mechanism weights and aggregates features at different positions in the sequence, enabling the network to automatically focus on information from historical or future moments most relevant to the current repair position. The decoder then generates the repair value. This structure is particularly suitable for anomaly repair in sensor data or financial time series. When the anomaly type label is set anomaly, the repair model employs a relational constraint repair network based on a graph neural network. This network uses the features of neighboring nodes in the data lineage graph that are directly or indirectly related to the anomaly record as constraints. These neighboring nodes may correspond to other fields in the same record or fields in other tables linked by foreign keys. The network encodes these constraints as edge constraints in a graph structure, and then iteratively solves for a repair value that simultaneously satisfies all functional dependencies and maintains maximum similarity to the original non-anomaly part through constraint satisfaction optimization. Gradient descent or message passing algorithms are typically used to search for the optimal solution within the feasible region defined by the constraints. Through this differentiated network structure, this invention can employ the most suitable repair model for anomalies of different natures, significantly improving the accuracy and efficiency of the repair process.

[0030] In the incremental learning process of the method of this invention, when the system executes the model update driving signal feedback and triggers the incremental learning of the anomaly detection model and the policy decision agent, it does not simply update the parameters directly, but simultaneously performs a data drift detection operation. The core purpose of this data drift detection operation is to determine whether the distribution characteristics of the current batch of data have changed significantly from the distribution characteristics of the historical training data. This is because once data drift occurs, the original anomaly detection model and policy decision agent may no longer be applicable to the current data environment. Directly performing conventional incremental learning may lead to slow model adaptation to the new data distribution or even erroneous updates. Specifically, the data drift detection operation calculates a quantified drift index by comparing the distribution differences between the current batch of data and historical data in the high-dimensional feature space. This comparison process can use the kernel density estimation method to fit the current batch feature distribution and the historical feature distribution respectively, and then calculate the maximum mean difference or Wasserstein distance between the two distributions as the drift index. Alternatively, a classifier-based detection method can be used, training a binary classifier to distinguish between the current batch of data and historical data, and using the classification accuracy of the classifier as the drift index. When the calculated drift index exceeds a preset threshold, the system determines that a data drift event has occurred. This signifies a substantial change in the data environment, such as altered user behavior patterns, sensor drift, or adjustments to business rules. Following a data drift event, the system adopts a cautious and adaptive handling strategy: it temporarily suspends the regular incremental learning process of the current anomaly detection model and the policy decision-making agent, and instead triggers an online adaptive adjustment mechanism. This online adaptive adjustment mechanism is designed to quickly adapt to the new data distribution without relying on potentially outdated historical experience when data distribution changes abruptly. Instead, it rapidly constructs a temporary training set using the portion of the current batch of data with relatively high confidence in its predictions, quickly fine-tuning the model to adapt it rapidly. Specifically, the mechanism first scans all records in the current batch of data processed in step S4, selecting those repaired data with a repair confidence score higher than a preset high threshold. Because these data have high repair confidence, the system considers them reliable pseudo-label samples, meaning that the anomaly type labels and correct values ​​of these samples have high credibility. Meanwhile, the system retains a portion of genuinely labeled samples from historical storage. These samples have been manually verified or confirmed as correct at the source during historical operations. The pseudo-labeled samples are then mixed with the genuinely labeled samples in a certain proportion to construct a temporary training set. Using this temporary training set, the system rapidly fine-tunes the anomaly detection model and the policy decision-making agent. Fine-tuning uses a small learning rate and a limited number of iterations to enable the model to quickly capture the characteristics of the new data distribution while avoiding overfitting to a small number of pseudo-labeled samples. After fine-tuning is complete, the system performs data drift detection again on the new batch of data to observe whether the drift index falls back below a preset threshold.Once the drift index falls back to an acceptable range, it indicates that the model has successfully adapted to the new data distribution, at which point the system resumes the regular incremental learning process. The algorithm for calculating the drift index based on distribution differences is as follows: in, This is the data drift index; the larger the value, the greater the degree of drift between the current batch of data and historical data. The weighting coefficient represents the contribution weight of kernel density distribution similarity to the drift index, and is adaptively adjusted according to the business scenario and data type. The KL divergence represents the kernel density distribution of the current batch of data. Kernel density distribution of historical data The similarity is such that a larger KL divergence indicates a greater difference in distribution; P(X) is determined by the Gaussian kernel function K... The kernel density estimation is achieved, where h is the kernel bandwidth; This serves as a reference dataset for drift detection. Here, is the normalization factor for the KL divergence; N and M are the data of the current batch, respectively. Historical data Sample size ∈ , ∈ ; Euclidean distance characterizes the feature space differences between individual samples. The maximum Euclidean distance in the feature space is used to normalize the calculation of this part; The average normalized Euclidean distance between the current batch and historical data represents the overall shift in the feature space.

[0031] like Figure 2As shown, the process is a full-link intelligent processing system built on AI-based big data system anomaly data intelligent cleaning and repair technology. It encompasses thirteen core functional modules from top to bottom, with each module building upon the previous one and tightly connected, forming a closed-loop iterative technical logic. This translates core technologies such as multi-dimensional feature fusion, graph neural network anomaly detection, and deep reinforcement learning strategy optimization into concrete execution steps, achieving intelligent processing of anomaly data from collection, detection, repair to continuous model optimization. Data collection and feature fusion are the starting points of the process. This module simultaneously collects raw business data from the data source, system operating status, and historical feedback signals from downstream tasks. It performs unified semantic alignment for structured and unstructured data sources, establishes cross-source semantic association indexes through domain knowledge graphs, and then uses a multi-modal feature extraction network to perform non-linear mapping and dimensional alignment on three core features: business logic, time series, and statistical distribution. Combined with standardized signals from system status and downstream feedback, it finally generates multi-dimensional feature fusion signals, laying a solid feature foundation for subsequent anomaly detection. The anomaly detection and recognition module constructs a data lineage graph using a lightweight graph neural network. Data fields are designated as nodes, and dependencies between fields as edges. Each node is assigned an initial vector of fused features. Local neighborhood and global topological features are extracted through message passing and aggregation mechanisms to accurately classify anomalies into three types: isolated points, context, and sets. Simultaneously, combining Mahalanobis distance and prior frequencies of anomaly categories, an anomaly severity score in the 0-1 interval is calculated, outputting a standardized anomaly recognition signal. The intelligent optimization module for the cleaning strategy employs a strategy decision-making agent constructed using deep reinforcement learning. With fused features, anomaly recognition, system load, and downstream task sensitivity as the state space, it dynamically optimizes the cleaning action space, including discarding, imputation, generative repair, and re-collection. It also acquires the cleaning strategies of adjacent data lineage nodes as collaborative references. If strategy conflicts exist, they are corrected through a local strategy consistency constraint objective function, ultimately generating a cleaning strategy signal with the optimal action index and execution parameters. The cross-source cross-validation module, relying on the previously established cross-source semantic association index, extracts real-time data from synonymous fields in other data sources for fields containing abnormal data. This completes the consistency verification of cross-source data, providing multi-source data support for subsequent repair stages and is a key step in achieving collaborative cross-source data repair. The differentiated anomaly repair module configures a dedicated repair network based on the three types of anomaly labels output by anomaly detection. Isolated point anomalies use a generative repair network with conditional generative adversarial networks; contextual anomalies use a sequence-to-sequence repair network combining temporal convolution and attention; and set anomalies use a relational constraint repair network with graph neural networks. If cross-source cross-validation passes, the cross-source data is directly used as the repair value; otherwise, a repair value is generated based on a weighted fusion of the confidence levels of the multi-source data.

[0032] Furthermore, the repair confidence assessment module synthesizes a repair confidence score through a weighted average of model confidence and data confidence. Model confidence is obtained by calculating the variance of the repair value distribution using the Monte Carlo dropout method, while data confidence is determined by combining data local density and business logic satisfaction. This is the core assessment step for determining the usability of the repair result. Automatic acceptance of repair values ​​is the positive processing branch of the assessment module. It is triggered when the repair confidence score reaches or exceeds a preset threshold. The system directly recognizes the repair result and generates an automatic acceptance signal, requiring no manual intervention and achieving fully automated repair of abnormal data. Manual review is the negative processing branch of the assessment module. It is initiated when the repair confidence score falls below a preset threshold. The system pushes the original abnormal content, abnormal label, severity score, and multiple sets of candidate repair values ​​and their corresponding confidence levels to the manual review queue. Manual reviewers then select the best repair value, ensuring the accuracy of repairing high-risk abnormal data. The downstream data output module, after automatic adoption or manual review, writes the repaired data into the downstream data interface according to the original data format and data lineage. This ensures that downstream tasks can directly call standardized and consistent repaired data, achieving seamless integration between the cleaning and repair process and downstream business. The downstream feedback and signal alignment module collects execution effect feedback signals generated by downstream tasks based on the repaired data, including changes in model accuracy and recall, and fluctuations in key business performance indicators. This signal is then aligned with anomaly identification and cleaning strategy signals according to data batch identifiers and timestamps, forming a model update driving signal that includes the mapping relationship between data quality and downstream performance, providing a core basis for model iteration. The data drift detection module executes synchronously when incremental model learning is triggered. It calculates the kernel density distribution difference using KL divergence, represents the feature space offset using average normalized Euclidean distance, and weights it to obtain a data drift index. Whether the index exceeds a preset threshold determines whether a data drift event has occurred. This is a pre-detection step to ensure the model adapts to dynamic changes in data distribution. The adaptive fine-tuning module is a targeted processing branch for drift detection. It is triggered when the drift index exceeds a threshold. The system temporarily suspends incremental learning, using repaired data with confidence levels above the high threshold as pseudo-label samples. A temporary training set is constructed by combining this data with historical real-label samples. This allows for rapid fine-tuning of the anomaly detection model and the policy decision-making agent until the drift index falls below the threshold. Incremental learning is the core module of the closed-loop process. It is triggered directly when the drift index does not exceed the threshold and resumes after adaptive fine-tuning. The model update driving signal is used as both the supervision and reward signals to update the network parameters of the anomaly detection model and the action value evaluation network of the policy decision-making agent. Simultaneously, a flexible weight solidification method is used to constrain key network parameters to prevent catastrophic forgetting. The updated model is then fed back to the data acquisition and feature fusion module, providing optimized technical support for the next round of data processing and enabling continuous iterative evolution of the model.

[0033] The present invention provides an AI-based intelligent cleaning and repair method for abnormal data in big data systems. The method outputs anomaly identification signals through an anomaly detection model. The strategy decision-making agent dynamically selects the optimal cleaning strategy based on system load and downstream task sensitivity. The repair model is invoked to output repair values ​​and confidence scores. Low confidence scores are transferred to manual review, while high confidence scores are automatically adopted and output to downstream systems. Feedback on the execution effect of downstream tasks is collected to form a model update driving signal, which is fed back to the anomaly detection model and the strategy decision-making agent to trigger incremental learning, forming a closed-loop self-evolution mechanism.

[0034] Therefore, the AI-based intelligent cleaning and repair method for abnormal data in big data systems of the present invention can solve the problems of data cleaning methods lacking adaptability, being unable to cope with complex anomalies, and being difficult to continuously optimize.

[0035] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for intelligent cleaning and repair of abnormal data in big data systems based on AI analysis, characterized in that: include: S1: Collect raw business data from the data source, system operating status, and historical feedback signals from downstream tasks to generate multi-dimensional feature fusion signals; S2: Based on this fused signal, the data distribution offset and business logic correlation features are extracted through the anomaly detection model to generate an anomaly identification signal containing anomaly type labels and severity scores; S3: Input the anomaly identification signal into the strategy decision-making agent. The agent combines the current system load and the sensitivity of downstream tasks to dynamically optimize within the preset cleaning action space and generate a cleaning strategy signal containing the optimal cleaning action index and execution parameters. S4: Based on the cleaning strategy, the repair model is called to process abnormal data. The model outputs repair values ​​and confidence scores synchronously. If the scores are lower than the preset threshold, a three-stage automated processing process is triggered, which includes multi-model cross-verification, data lineage consistency verification, and cross-source data consistency comparison. If the verification meets the standard, the repair value is automatically adopted. If the verification fails, it is marked as pending in-depth verification and enters the system priority processing queue. Otherwise, the repair value is automatically adopted and output to the downstream. S5: Collect feedback signals of downstream task execution effects, associate and align them with anomaly identification and cleaning strategy signals to form model update driving signals, and send them back to the anomaly detection model and strategy decision-making agent to trigger incremental learning.

2. The method according to claim 1, characterized in that, The multi-dimensional feature fusion signal is composed of business logic features, temporal features, and statistical distribution features. Business logic features are extracted by constructing functional dependencies and association rules between data fields to reflect the inherent business semantic consistency within the data. Temporal features are obtained by applying sliding window sampling to the original business data and performing periodic decomposition and trend fitting operations to characterize the evolution of the data over time. Statistical distribution features are obtained by estimating kernel density and calculating quantiles of the original business data to describe the global distribution pattern and outlier degree of the data. The system operating status is simultaneously acquired when collecting original business data from the data source. The system's operational status signals include computational resource utilization, storage input / output waiting time, and network transmission latency parameters, used to characterize the available resource margin during the data cleaning process. Downstream task historical feedback signals include the changing trends of downstream model training loss values ​​and business indicator fluctuation records corresponding to historical cleaning results, used to characterize the historical impact of different cleaning strategies on downstream task performance. These three types of characteristic signals are input into a multimodal feature extraction network composed of multiple parallel feature extraction branches. Each branch performs nonlinear mapping and dimensional alignment on its corresponding features and then fuses them to output a multi-dimensional feature fusion signal. The calculation formula for the multi-dimensional feature fusion signal is as follows: in, The final output is a multi-dimensional feature fusion signal with d dimensions; (∙) is the Sigmoid activation function, which normalizes the feature values ​​to [0,1] to adapt to subsequent model inputs; These correspond to business logic characteristics, time sequence characteristics, and statistical distribution characteristics, respectively. The attention weights for the three core features are calculated using a cross-feature attention mechanism, achieving weighted enhancement of important features. This represents the set of features from the two classes other than the i-th class after mapping. S is the nonlinear mapping and dimension alignment operator for the i-th type of feature; S is the system operating state signal; and H is the historical feedback signal of the downstream task. Standardized operators align the dimensions and feature units of S and H; Let S and H be the learnable weight matrices, representing the contribution of the two types of auxiliary signals to the fused features.

3. The method according to claim 1, characterized in that, The anomaly detection model employs a lightweight graph neural network to construct a data lineage graph. Data fields are treated as nodes, and dependencies between fields and data flow paths are treated as edges. Based on multi-dimensional feature fusion signals, each node is assigned an initial feature vector. Through the message passing and aggregation mechanism of the graph neural network, the local neighborhood structure features and global topological role features of each data record in the data lineage graph are calculated. Based on these features, anomaly type labels are classified into three categories: isolated point anomalies, context anomalies, and set anomalies. Isolated point anomalies correspond to records where a single field value deviates from its statistical distribution; context anomalies correspond to records where the value is reasonable within a specific temporal or business context but deviates from the overall contextual pattern; and set anomalies correspond to records where a single field value is normal but multiple field combinations violate functional dependencies or business rule constraints. While outputting anomaly type labels, the anomaly detection model calculates the Mahalanobis distance between the anomaly record and normal records in the feature space and combines this with the prior occurrence frequency of the anomaly category to which the anomaly record belongs to generate an anomaly severity score normalized to the range of zero to one. The anomaly severity scoring algorithm is as follows: in, Score the severity of the abnormal record x under its anomaly category θ; ( Mahalanobis distance represents the feature space distance between abnormal records and normal data of the same category, eliminating linear correlation between features; The feature vector of the abnormal data record to be evaluated. This is a normal data sample set within the same category; Normal sample set under abnormal category θ The mean vector and covariance matrix; The prior frequency of occurrence of the anomaly category θ is obtained from historical anomaly data statistics; The maximum Mahalanobis distance of the normal sample set under the same category is used as a normalization factor to ensure that the score strictly falls within the [0,1] interval. The closer the value is to 1, the more severe the abnormality.

4. The method according to claim 1, characterized in that, The policy decision-making agent is constructed using a deep reinforcement learning network. Its state space is composed of multi-dimensional feature fusion signals, anomaly recognition signals, current system load signals, and downstream task sensitivity signals. The downstream task sensitivity signals include weight parameters indicating the dependence of downstream tasks on data accuracy, completeness, and real-time performance. The action space of the policy decision-making agent is a preset cleaning action space, which includes actions such as discarding abnormal data, data imputation based on statistical features, data repair based on generative models, and re-collecting data from the source. The reward function of the policy decision-making agent is defined as the weighted sum of the downstream task performance improvement gain and the data entropy retention rate after repair, minus the computational resource consumption cost introduced by the cleaning operation. The downstream task performance improvement gain is calculated by inputting the repaired data into the downstream task model and comparing the difference in task performance indicators before and after repair. The data entropy retention rate after repair is determined by calculating the relative entropy distance between the data before and after repair in the probability distribution space. When the policy decision-making agent optimizes in the action space based on the current state, it uses a policy gradient method combined with a priority experience replay mechanism for training, and outputs a cleaning policy signal containing the optimal cleaning action index and execution parameters.

5. The method according to claim 1, characterized in that, The repair model configures differentiated repair network structures for different types of anomaly identification signals. When the anomaly type label is isolated point anomaly, the repair model adopts a generative repair network based on conditional generative adversarial network. This network uses the relevant field features that have not occurred as conditional input, outputs repair candidate values ​​for the anomaly field through the generator network, and uses the discriminator network to adversarially constrain the consistency between the repair candidate values ​​and the real data distribution. When the anomaly type label is context anomaly, the repair model adopts a sequence-to-sequence repair network based on a combination of temporal convolutional network and attention mechanism. This network uses the complete record sequence within the temporal window before and after the anomaly record as context encoding, and decodes and generates repair values ​​after weighted aggregation of key information in the temporal context through the attention mechanism. When the anomaly type label is set anomaly, the repair model adopts a relational constraint repair network based on graph neural network. This network uses the features of neighboring nodes that are directly or indirectly related to the anomaly record in the data lineage graph as constraint conditions, and solves the optimal repair value that satisfies all functional dependencies through constraint satisfaction optimization iteration.

6. The method according to claim 5, characterized in that, The repair model outputs a repair confidence score while generating repair values. This score is a weighted composite of a model confidence component and a data confidence component. The model confidence component is obtained by calculating the variance of the repair value distribution through multiple forward propagations of the repair model using the Monte Carlo dropout method, and is used to characterize the uncertainty of the model regarding the current repair result. The data confidence component is obtained by calculating the local density of neighboring normal records in the feature space around the abnormal record and the satisfaction of the abnormal record and the repaired record under business logic constraints, and is used to characterize the reasonableness of the current repair result in terms of local data structure and business semantics. When the repair confidence score is lower than a preset threshold, the system pushes the manual review flag signal along with the original content of the abnormal record, the abnormal type label and severity score in the abnormal identification signal, and multiple candidate repair values ​​generated by the repair model and their corresponding confidence scores to the manual review queue. When the repair confidence score reaches or exceeds the preset threshold, the system directly adopts the repair value, generates an automatic adoption repair value signal, and writes the repaired data into the downstream data interface according to the original data format and data lineage.

7. The method according to claim 1, characterized in that, After the repaired data is output to downstream tasks, feedback signals of task execution performance generated by downstream tasks based on the repaired data are collected. These feedback signals include changes in accuracy and recall of the downstream task model on the validation set, as well as fluctuations in key performance indicators in downstream business applications. This feedback signal is then aligned with the anomaly identification signal generated in step S2 and the cleaning strategy signal generated in step S3 according to data batch identifiers and timestamps to form a model update driving signal that includes the mapping relationship between the differences in data quality before and after cleaning and the changes in downstream task performance. This model update driving signal is used as a supervision signal input to the anomaly detection model. The network parameters of the anomaly detection model are incrementally updated by calculating the gradient information between the feedback signal and the output of the anomaly detection model. At the same time, this model update driving signal is used as a reward signal input to the policy decision agent. The action value evaluation network of the policy decision agent is updated through temporal differential error, enabling the anomaly detection model and the policy decision agent to continuously adapt to dynamic changes in data distribution.

8. The method according to claim 7, characterized in that, When incremental learning is triggered, the system simultaneously performs a data drift detection operation. This operation calculates a drift index by comparing the distribution differences between the current batch of data and historical data in the feature space. When the drift index exceeds a preset threshold, a data drift event is determined to have occurred. After a data drift event occurs, the system temporarily suspends the incremental learning process of the current anomaly detection model and the policy decision agent. First, it triggers an online adaptive adjustment mechanism for the model. This mechanism uses repaired data with confidence scores higher than a preset high threshold in the current batch of data as pseudo-label samples, and combines them with historically retained real labeled samples to construct a temporary training set. This allows for rapid fine-tuning of the anomaly detection model and the policy decision agent. Once the fine-tuning is complete and the drift index falls below the preset threshold, the incremental learning process resumes. During incremental learning, an elastic weight solidification method is used to apply additional constraints to key network parameters in the anomaly detection model and the policy decision agent that affect the data drift detection results, in order to prevent catastrophic forgetting during incremental learning. The algorithm for calculating the drift index based on distribution differences is as follows: in, This is the data drift index; the larger the value, the greater the degree of drift between the current batch of data and historical data. The weighting coefficient represents the contribution weight of kernel density distribution similarity to the drift index, and is adaptively adjusted according to the business scenario and data type. The KL divergence represents the kernel density distribution of the current batch of data. Kernel density distribution of historical data The similarity is such that a larger KL divergence indicates a greater difference in distribution; P(X) is determined by the Gaussian kernel function K... The kernel density estimation is achieved, where h is the kernel bandwidth; This serves as a reference dataset for drift detection. Here, is the normalization factor for the KL divergence; N and M are the current batch data, respectively. Historical data Sample size ∈ , ∈ ; Euclidean distance characterizes the feature space differences between individual samples. The maximum Euclidean distance in the feature space is used to normalize the calculation of this part; The average normalized Euclidean distance between the current batch and historical data represents the overall shift in the feature space.

9. The method according to claim 1, characterized in that, In step S3, when the strategy decision-making agent dynamically optimizes the preset cleaning action space, it simultaneously acquires the cleaning strategy signals of adjacent nodes in the data lineage graph as a collaborative reference. This collaborative reference is used to trigger a strategy coordination mechanism when the cleaning strategy signal of the current data record conflicts with the cleaning strategy signal of the adjacent node record. This strategy coordination mechanism corrects the cleaning strategy signal of the current data record by constructing a local strategy consistency constraint objective function to minimize the product of the difference in cleaning strategies between adjacent nodes and the strength of data dependency. This ensures that data records with strong dependencies adopt the same or compatible cleaning actions, avoiding the breakage of data lineage or the destruction of data consistency due to inconsistent cleaning strategies of adjacent nodes. The corrected cleaning strategy signal, after retaining the original execution parameters and adding a strategy consistency adjustment coefficient, is used as the final cleaning strategy signal input to step S4.

10. The method according to claim 1, characterized in that, In step S1, when collecting raw business data from the data source, a unified semantic alignment mechanism is used for both structured and unstructured data sources. This semantic alignment mechanism constructs a domain knowledge graph to semantically map data from different data sources that have the same business meaning but differ in field names or data formats, and establishes a semantic association index for cross-source data. In step S4, when calling the repair model to process abnormal data according to the cleaning strategy, if the field containing the abnormal data in the current data source has a semantic association index pointing to a synonym field in another data source, the repair model will first extract the real-time data of the corresponding field from other data sources for cross-source cross-validation. If the cross-source data meets the consistency verification conditions, the cross-source data will be directly used as the repair value. If the cross-source data does not meet the consistency verification conditions or the cross-source data also contains anomalies, the repair model will generate a repair value based on the confidence weighted fusion of multi-source data. The weighting weight is dynamically calculated based on the historical data quality score and real-time data integrity status of each data source.

Images