A data cleaning method and system for complex logic and long tail exceptions

Abstract

The application discloses a data cleaning method and system for complex logic and long-tail exceptions, comprising: S1, accessing and standardizing processing of multi-source data to generate a standardized data set; S2, performing cross-field logic consistency detection, knowledge graph semantic consistency detection and deep learning anomaly distribution detection on the standardized data set in parallel, and respectively outputting logic deviation degree scores, semantic deviation degree scores and basic anomaly scores; S3, performing long-tail anomaly enhancement identification, and outputting long-tail enhanced anomaly scores; S4, performing confidence weighting fusion on the scores of the three detection dimensions, adaptively adjusting a judgment threshold, and outputting a final anomaly judgment result and a confidence level; and S5, performing automatic repair on the abnormal data and feeding back optimization according to the repair quality. The application effectively solves the problems of cross-field logic conflicts, semantic errors and long-tail exception missed detection in complex data scenarios, and significantly improves the accuracy and intelligent level of data cleaning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data processing technology, specifically to a data cleaning method and system for complex logic and long-tail anomalies. Background Technology

[0002] Existing deep learning-based data cleaning and processing methods primarily rely on data distribution characteristics for anomaly detection and data repair. While effective in handling single-field anomalies or significantly noisy data, they still have significant limitations in complex data scenarios. On one hand, these methods lack the ability to effectively model logical constraints between multiple fields, making it difficult to identify cross-field logical consistency errors. On the other hand, for information errors involving business semantics, such as field type mismatches, inconsistent units, and state conflicts, existing models, lacking semantic understanding capabilities, are prone to misjudgment or missed detection. Furthermore, in real-world data, anomaly samples typically exhibit a long-tail distribution, making deep learning models susceptible to mainstream data patterns and limiting their ability to detect low-frequency anomalies, thus impacting overall data quality. Simultaneously, existing technologies rarely integrate structured knowledge or business rules, resulting in limitations in complex relationship modeling and reasoning, making it difficult to meet the high-precision data cleaning needs of highly complex data scenarios such as finance, e-commerce, and healthcare.

[0003] Therefore, a new data cleaning method is urgently needed to address the aforementioned shortcomings. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a data cleaning method and system for complex logic and long-tail anomalies, thereby solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a data cleaning method for complex logic and long-tail anomalies, comprising the following steps:

[0007] S1: Acquire multi-source data, access and preprocess the multi-source data to obtain a standardized dataset; the standardized dataset consists of multiple data records, each record containing multiple data fields;

[0008] S2: For the standardized dataset, perform cross-field logical consistency detection, semantic consistency detection, and abnormal distribution detection respectively;

[0009] The cross-field logical consistency detection includes: constructing a cross-field logical constraint model that includes time, numerical and conditional dependencies, calculating the logical consistency deviation of each data record, and outputting a logical anomaly type identifier and deviation score;

[0010] The semantic consistency detection includes: introducing a knowledge graph to model data entities and semantic relationships, calculating the semantic consistency deviation of each data record, and outputting a semantic anomaly type identifier and deviation score;

[0011] The anomaly distribution detection includes: using a deep learning model to learn data distribution characteristics, calculating anomaly scores for each data record, and outputting anomaly type identifiers and basic anomaly scores;

[0012] S3: Based on the basic anomaly score obtained in step S2, perform long-tail anomaly enhancement identification, combine resampling weights and anomaly memory units to enhance the score of low-frequency long-tail anomalies, and output the long-tail enhanced anomaly score.

[0013] S4: Normalize the logical deviation score, semantic deviation score and long-tail enhanced anomaly score obtained in step S2 and step S3, and obtain a comprehensive anomaly score based on confidence weighted fusion. Adaptively adjust the judgment threshold and output the final anomaly judgment result and confidence level.

[0014] Preferably, step S1 includes:

[0015] It unifies the access of structured, semi-structured, and unstructured data from different data sources, and completes data aggregation through preset interfaces or data acquisition protocols;

[0016] Perform format parsing and structure conversion on the accessed data, and convert different data formats into a standard data structure;

[0017] Field-level semantic mapping is performed based on predefined data standards and field specifications to map heterogeneous fields to a unified semantic space;

[0018] The data undergoes integrity checks and preprocessing, including missing value identification, preliminary outlier detection, and data type verification. The detected missing and outlier data are then labeled to generate a standardized dataset with a unified format, semantics, and quality identifier.

[0019] Preferably, the cross-field logical consistency detection specifically includes:

[0020] For each data record in the standardized dataset, perform relation parsing on the data fields to identify temporal relationships, numerical balance relationships, and conditional dependencies between fields; construct a set of cross-field logical constraints based on the relationships, and express each constraint as a computable constraint function;

[0021] For each data record in the standardized dataset, the degree of deviation under each constraint function is calculated to determine whether there is a logical consistency anomaly; wherein, by comprehensively evaluating the results of multiple constraints, the data consistency detection result and the corresponding anomaly type identifier are output.

[0022] Preferably, the semantic consistency detection specifically includes:

[0023] Construct a knowledge graph related to the domain to which the data belongs, including entities, attributes, relationships and their semantic types related to the data fields, and perform semantic mapping and reasoning through the graph reasoning mechanism;

[0024] Based on the standardized data records after semantic mapping in step S1, each data record is mapped to a semantic subgraph composed of entities and their relationships in the knowledge graph.

[0025] For the semantic subgraph, the semantic relationships between entities are verified using relational constraint rules and reasoning mechanisms in the knowledge graph.

[0026] Calculate the semantic consistency deviation of each data record based on the reasoning results, and output the corresponding semantic anomaly type identifier and deviation score;

[0027] Construct a semantic consistency deviation function to represent the degree of deviation between data fields and entities and relations in the knowledge graph:

[0028]

[0029] in, : Represents the field vector of a single data record. Field vector With the first in the knowledge graph The mapping result of an entity or relationship. : No. Standardized semantic types or rules for an entity or relationship. : A metric function that represents the similarity and consistency between a data field and entities or relationships in a knowledge graph. : No. The weight coefficient of an entity or relationship. : Data fields and the first The degree of difference of individual unit or numerical constraints : No. Standard values ​​for individual units or numerical constraints : No. Weighting coefficients for unit or numerical constraints, Norm exponent, usually taken as This is used to adjust the sensitivity to anomalies. : Represents the number of semantic constraints associated with entities and relations in the knowledge graph. : Represents quantities related to units or numerical constraints. : Indicates the overall semantic consistency deviation of data records. The larger the value, the higher the degree of semantic inconsistency.

[0030] Preferably, the abnormal distribution detection specifically includes:

[0031] A deep learning model is constructed to characterize the distribution features of data, and the model is trained using standardized data to learn the latent space representation of the data and obtain the reconstruction results of the data records in the latent space.

[0032] The basic anomaly scoring function is calculated based on the latent space representation and reconstruction results. This is used to measure the degree to which data deviates from the normal distribution; the formula is as follows:

[0033]

[0034] in, : Represents a vector of fields for a single input data record. : Encoding function (used to map input data to a latent space representation), Decoding function : The mean vector of the normal data distribution in the latent space. The inverse of the latent space covariance matrix, used to characterize the correlation across different dimensions. Weighted distance with covariance constraints : The squared Euclidean distance between the input data and the reconstructed data. The exponential amplification function is used to enhance sensitivity to large deviation anomalies. Scale adjustment parameters Implicit space structural constraints Weighting coefficients for each part;

[0035] The system determines the basic anomaly score based on a preset threshold to identify abnormal data and outputs the corresponding anomaly degree and type identifier.

[0036] Preferably, the long-tail anomaly enhancement identification includes:

[0037] Long-tail augmentation is performed on any of the abnormal data or labeled abnormal data identified by any detection method in step S2. During the model training phase, the proportion of long-tail abnormal samples is increased through adaptive resampling, and an abnormal memory unit is constructed. During the actual anomaly detection phase, the trained model and the abnormal memory unit are used to perform long-tail augmentation anomaly judgment on new data. An abnormal memory unit is constructed to encode and store the abnormal patterns that appear in the historical detection process, and to match and enhance them in subsequent detection processes. During the model training or inference phase, the abnormal memory unit is introduced to assist in the discrimination of long-tail anomalies, so as to improve the model's sensitivity to low-frequency abnormal patterns. Finally, the augmented anomaly detection results and the corresponding long-tail anomaly labels are output.

[0038] Construct an enhanced anomaly scoring function that integrates resampling weights and anomaly memory matching:

[0039] ,

[0040] Where x represents the field vector of a single input data record. Basic anomaly scoring function, : Modulation top weighting coefficient, : The final anomaly score after long-tail enhancement; the higher the value, the more likely it is to be a long-tail anomaly. : No. The weighting coefficients of each anomalous prototype. Temperature parameters that control similarity decay : The exponential kernel function is used to characterize the similarity to historical anomaly patterns. : A function for estimating the frequency or density of a sample; Latent space representation; : No. An abnormal memory prototype The number of exception prototypes stored in the exception memory unit;

[0041] The resampling weight function, which is related to the frequency of sample occurrence, is usually defined as:

[0042] ,

[0043] :sample The probability of occurrence in the data distribution Smoothing terms prevent the denominator from being zero. : Adjustment index, used to control the degree of long-tail enhancement;

[0044] : Long-tailed distribution modulation function, used to enhance anomaly scoring of samples in low-density regions, is expressed as:

[0045] ,

[0046] in, Smoothing coefficient.

[0047] Preferably, the comprehensive anomaly score obtained in step S4 based on confidence-weighted fusion adopts the following formula:

[0048]

[0049] in, Input data sample, The number of detection dimensions, including logical consistency deviation score, semantic consistency deviation score, and anomaly score after long-tail enhancement. : No. Scoring of each detection dimension The hyperbolic tangent function is used to compress the scoring range and suppress the influence of extreme values. : Scale adjustment parameter, used to adjust the sensitivity of scoring for different detection dimensions. : No. Dynamic confidence weights corresponding to each detection dimension; : Conflict degree function between detection results; Conflict moderating weights : The integrated anomaly score after fusion;

[0050] Defined as:

[0051]

[0052] in, : No. Each detection module in the sample Uncertainty estimation at the location;

[0053] Defined as:

[0054]

[0055] in, : Conflict sensitivity index, j: represents the index of another detection dimension compared with the i-th detection dimension.

[0056] Preferably, the confidence-based weighted decision further includes:

[0057] The overall confidence score of the fusion result is calculated, and the confidence score is measured based on the degree of disagreement among the scores, defined as follows:

[0058]

[0059] in, : The normalized first Each detection module outputs, The deviation sensitivity index is used to control the non-linearity of the consistency measure. : The integrated anomaly score after fusion;

[0060] The anomaly detection threshold is adaptively adjusted based on the aforementioned confidence level.

[0061]

[0062] in, Basic threshold Threshold adjustment coefficient : Overall confidence level of the fusion results;

[0063] The final exception detection function is:

[0064]

[0065] in, : Indicator function, 1 when the condition is true, 0 otherwise;

[0066] Output the final anomaly determination result and the corresponding confidence level .

[0067] Preferably, the method further includes:

[0068] S5: For data identified as abnormal, perform automated data repair, and optimize the adjustable parameters in steps S2 to S4 based on the repair quality; specifically:

[0069] S5-1: Automated Data Repair

[0070] For the data samples identified as abnormal in step S4, perform the following operations:

[0071] S5-1-1: Based on the anomaly type identifiers (including logical anomaly type, semantic anomaly type, and distribution anomaly type) output by each detection step, comprehensively determine the anomaly type category to which the sample belongs. Where C is the number of exception type categories;

[0072] S5-1-2: Based on the exception type, invoke the corresponding repair strategy and construct an exception type-driven adaptive repair mapping function:

[0073]

[0074] in, This represents the original abnormal data. This indicates the data after repair. Indicates sample Belongs to the The probability of class exceptions, Indicates the first The basic repair function corresponding to the exception type. Indicates the first Repair strength adjustment coefficient, The memory-based repair enhancement function is defined as follows:

[0075]

[0076] in, Indicates the first A collection of historical repair patterns for class anomalies. Represents a data feature mapping function. Indicates the first The weight of each memory pattern, This represents the similarity decay parameter. Indicates the number of memory patterns for this type of anomaly;

[0077] S5-1-3: Conduct consistency assessment and quality verification of the data before and after the repair, and construct a repair quality score:

[0078]

[0079] in, Indicates the repair quality score. This represents the Mahalanobis distance weighted by covariance. Indicates the smoothing term. Represents the structural constraint weights. This represents a function that indicates the structural consistency of the repaired data.

[0080] S5-2: Based on the repair results obtained in S5-1, the adjustable parameters in steps S2 to S4 are periodically or in batches optimized and updated, specifically including:

[0081] S5-2-1: Define a feedback-driven loss function by utilizing the deviation between the repair quality score and the detection scores before and after repair:

[0082]

[0083] Among them, among them, Indicates the repair quality score. Indicates the quality sensitivity index. The deviation function between the detection result and the repaired result is defined as follows: , This represents the fusion anomaly scoring function, i.e. , These represent the fusion anomaly scores before and after repair, calculated under the same fusion scoring model. This represents a dataset.

[0084] S5-2-2: Combine all variables learned from training data, statistically calculated from detection results, and feedback-driven loss. Batch gradient descent updates are performed on each adjustable parameter:

[0085]

[0086] in, , These represent the parameters of the model in the t-th and t+1-th iterations, respectively. The learning rate is represented by α and β, respectively; the loss weight coefficients are represented by α and β. This represents the anomaly detection loss function, used to measure the model's bias in identifying logical consistency anomalies, semantic consistency anomalies, and anomaly distribution anomalies. This represents the data repair loss function, used to measure the difference between the repaired data and the target data, as well as the degree to which the repair result violates structural and logical constraints.

[0087] The anomaly detection loss function is defined as follows:

[0088]

[0089] The data repair loss function is defined as follows:

[0090]

[0091] Where D represents the training sample set used for anomaly detection, This represents the anomalous label for sample x, with 1 for anomalous samples and 0 for normal samples. denoted as the fusion anomaly score after normalization in step S4; w₊ and w₋ represent the loss weights of the anomaly samples and normal samples, respectively, where w₊ is adjusted according to the resampling weight of the long-tail samples. This represents the sample set to be repaired. This indicates the data after repair. This indicates the corresponding target data or reference data that has been manually verified and determined by reliable historical records; Denotes the inverse matrix of the covariance matrix. Represents the structural constraint weights. This represents a function that indicates the structural consistency of the repaired data.

[0092] Secondly, the present invention provides a data cleaning system for complex logic and long-tail anomalies, used to perform the above-mentioned data cleaning method, including:

[0093] The multi-source data access and standardization module is used to execute step S1;

[0094] The cross-field logical consistency detection module is used to perform the cross-field logical consistency detection in step S2;

[0095] The semantic consistency detection module is used to perform the semantic consistency detection in step S2;

[0096] An anomaly distribution detection module is used to perform anomaly distribution detection in step S2;

[0097] The long-tail anomaly enhancement and identification module is used to execute step S3;

[0098] The multi-source detection result fusion and confidence determination module is used to execute step S4;

[0099] An automated data repair and feedback optimization module is used to execute step S5.

[0100] The present invention has the following beneficial effects:

[0101] 1. This invention introduces a cross-field logical constraint model and a knowledge graph semantic enhancement mechanism to integrate the temporal consistency, numerical correlation, and conditional dependency relationships between multiple fields with entity semantic relationships. This breaks through the technical bottleneck of traditional methods lacking logical constraints and semantic understanding capabilities. It can accurately identify complex data errors such as cross-field logical conflicts, field mismatches, and semantic inconsistencies, fundamentally solving the problems of high misjudgment rates and difficulty in identifying semantic errors in complex business scenarios in existing technologies, and significantly improving the accuracy and intelligence level of data cleaning.

[0102] 2. This invention constructs an anomaly pattern detection module based on deep learning and combines it with a long-tail anomaly enhancement recognition mechanism. It also introduces a resampling strategy and anomaly memory mechanism, which effectively overcomes the problems of traditional models being heavily dependent on mainstream data distribution and having insufficient ability to recognize low-frequency anomalies. This significantly improves the detection capability of long-tail anomalies and complex anomaly patterns, and enhances the system's generalization performance and detection sensitivity in real data distribution environments.

[0103] 3. This invention establishes a multi-source detection result fusion and confidence assessment mechanism to perform weighted fusion and unified decision-making on the output results of logical constraint detection, semantic analysis, and deep learning models. This avoids the bias and uncertainty caused by a single detection model, thereby improving the stability and reliability of anomaly judgment results and further enhancing the robustness of overall data quality assessment.

[0104] 4. This invention constructs an automated data repair and feedback optimization mechanism to achieve closed-loop linkage between anomaly detection, data repair, and model optimization. It can not only perform adaptive repair according to the anomaly type, but also dynamically adjust model parameters and strategy configuration based on the repair quality, thereby achieving continuous self-evolution optimization of system performance and significantly improving the long-term stability and application adaptability of the system in multi-source heterogeneous and dynamically changing data environments. Attached Figure Description

[0105] Figure 1 This is a flowchart of the method of the present invention;

[0106] Figure 2 This is a schematic diagram illustrating the overall performance evolution of the system according to the present invention;

[0107] Figure 3 This is a schematic diagram illustrating the system iterative optimization effect of the present invention. Detailed Implementation

[0108] To enable those skilled in the art to understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0109] The present invention will now be described in detail with reference to the accompanying drawings:

[0110] Example:

[0111] Please see the appendix Figures 1 to 3 This invention provides a data cleaning method and system for complex logic and long-tail anomalies, wherein the method includes:

[0112] S1: Acquire multi-source data, access and preprocess the multi-source data to obtain a standardized dataset; the standardized dataset consists of multiple data records, each record containing multiple data fields;

[0113] S2: For the standardized dataset, perform cross-field logical consistency detection, semantic consistency detection, and abnormal distribution detection respectively;

[0114] The cross-field logical consistency detection includes: constructing a cross-field logical constraint model that includes time, numerical and conditional dependencies, calculating the logical consistency deviation of each data record, and outputting a logical anomaly type identifier and deviation score;

[0115] The semantic consistency detection includes: introducing a knowledge graph to model data entities and semantic relationships, calculating the semantic consistency deviation of each data record, and outputting a semantic anomaly type identifier and deviation score;

[0116] The anomaly distribution detection includes: using a deep learning model to learn data distribution characteristics, calculating anomaly scores for each data record, and outputting anomaly type identifiers and basic anomaly scores;

[0117] S3: Based on the basic anomaly score obtained in step S2, perform long-tail anomaly enhancement identification, combine resampling weights and anomaly memory units to enhance the score of low-frequency long-tail anomalies, and output the long-tail enhanced anomaly score.

[0118] S4: Normalize the logical deviation score, semantic deviation score and long-tail enhanced anomaly score obtained in step S2 and step S3, and obtain a comprehensive anomaly score based on confidence weighted fusion. Adaptively adjust the judgment threshold and output the final anomaly judgment result and confidence level.

[0119] See appendix Figures 1 to 3 It unifies the access of structured, semi-structured, and unstructured data from different data sources, and completes data aggregation through preset interfaces or data acquisition protocols; it performs format parsing and structure conversion on the accessed data, converting different data formats into a unified standard data structure; it performs field-level semantic mapping based on predefined data standards and field specifications, mapping heterogeneous fields to a unified semantic space; it performs integrity detection and preprocessing on the data, including missing value identification, preliminary outlier detection, and data type verification; and it marks the detected missing and outlier data to generate a standardized dataset with a unified format, unified semantics, and quality identifier.

[0120] Specifically, this implementation method achieves unified expression of heterogeneous data at the format and semantic levels by uniformly accessing and standardizing structured, semi-structured, and unstructured multi-source data, effectively eliminating structural inconsistencies and semantic ambiguities caused by differences in data sources. At the same time, through format parsing, structure conversion, and field-level semantic mapping, it improves the usability and consistency of data. On this basis, combined with integrity detection and preprocessing mechanisms, it identifies and marks missing values, outliers, and type errors in advance, which not only reduces the complexity of subsequent processing but also provides high-quality input for subsequent anomaly detection and data repair, thereby improving the overall efficiency, accuracy, and stability of data cleaning and system processing.

[0121] See appendix Figures 1 to 3The cross-field logical consistency detection specifically involves: parsing the data fields of each data record in the standardized dataset to identify the temporal sequence, numerical balance, and conditional dependencies between fields; constructing a set of cross-field logical constraints based on the relationships and expressing each constraint as a computable constraint function; calculating the deviation of each data record in the standardized dataset under each constraint function to determine whether there is a logical consistency anomaly; and outputting the data consistency detection result and the corresponding anomaly type identifier by comprehensively evaluating the results of multiple constraints.

[0122] Specifically, this implementation method systematically identifies temporal, numerical balance, and conditional dependencies between standardized data fields through relational parsing. Based on this, it constructs a computable set of cross-field logical constraints, formalizing complex business rules into constraint functions and achieving a quantitative expression of data logical consistency. Furthermore, by constructing a unified cross-field consistency deviation function, it performs weighted fusion evaluation of various inequality and equality constraints, accurately characterizing the degree of data deviation under multiple constraints, thereby effectively identifying cross-field logical anomalies that are difficult to detect using traditional methods. At the same time, by comprehensively analyzing the results of multiple constraint detections, this method outputs anomaly type identifiers, which not only improves the accuracy and comprehensiveness of anomaly detection but also enhances the interpretability and stability of the results, providing a reliable basis for subsequent data repair and quality optimization.

[0123]

[0124] in A vector representing the fields of a single data record. : No. Inequality constraint functions, : No. The threshold corresponding to each constraint : Non-linear penalty function : No. Weighting coefficients of each inequality constraint. : No. A set of equality constraint functions, : No. Weighting coefficients of each equality constraint, Norm index , : These represent the number of inequality constraints and the number of equality constraints, respectively. This indicates the degree of cross-field consistency deviation of the data record; the larger the value, the worse the consistency.

[0125] See appendix Figures 1 to 3The semantic consistency detection specifically involves: constructing a knowledge graph related to the domain to which the data belongs, including entities, attributes, relationships, and their semantic types related to the data fields, and performing semantic mapping and reasoning through a graph reasoning mechanism; based on the standardized data records after semantic mapping in step S1, mapping each data record to a semantic subgraph composed of entities and their relationships in the knowledge graph; for the semantic subgraph, using the relationship constraint rules and reasoning mechanism in the knowledge graph, performing consistency verification on the semantic relationships between entities; calculating the semantic consistency deviation of each data record based on the reasoning results, and outputting the corresponding semantic anomaly type identifier and deviation score;

[0126] Construct a semantic consistency deviation function to represent the degree of deviation between data fields and entities and relations in the knowledge graph:

[0127]

[0128] in, A vector representing the fields of a single data record. Field vector With the first in the knowledge graph The mapping result of an entity or relationship. Indicates the first Standardized semantic types or rules for an entity or relationship. : A metric function that represents the similarity and consistency between a data field and entities or relationships in a knowledge graph. Indicates the first The weight coefficient of an entity or relationship. Indicates the relationship between the data field and the first The degree of difference of individual unit or numerical constraints Indicates the first Standard values ​​for individual units or numerical constraints Indicates the first Weighting coefficients for unit or numerical constraints, The norm exponent is usually taken as... This is used to adjust the sensitivity to anomalies. This represents the number of semantic constraints associated with entities and relations in the knowledge graph. Indicates quantities related to units or numerical constraints. This indicates the overall semantic consistency deviation of the data records; the larger the value, the higher the degree of semantic inconsistency.

[0129] Specifically, this implementation constructs a domain-specific knowledge graph to accurately map data fields to the semantic types of related entities, attributes, and relationships, and enhances the semantic verification capability of the data by combining a graph reasoning mechanism. By identifying issues such as field mismatches, inconsistent units, state conflicts, and semantic inconsistencies in the data, the system can efficiently detect semantic anomalies that are difficult to identify using traditional methods. Simultaneously, by constructing a semantic consistency deviation function, the system quantitatively evaluates the degree of deviation between data fields and entities and relationships in the knowledge graph, which not only improves the semantic verification capability of the data but also enhances the accuracy and sensitivity of detection, providing a reliable basis for data quality optimization and subsequent repair.

[0130] See appendix Figures 1 to 3 The anomaly distribution detection specifically involves: constructing a deep learning model to characterize data distribution features, training the model using standardized data to learn the latent space representation of the data, and obtaining the reconstruction results of data records in the latent space; and calculating a basic anomaly scoring function based on the latent space representation and the reconstruction results. This is used to measure the degree to which data deviates from the normal distribution; the formula is as follows:

[0131]

[0132] in, This represents a vector of fields for a single input data record. The encoding function represents the input data that is mapped to the latent space representation. Indicates the decoding function. This represents the mean vector of the normal data distribution in the latent space. The inverse of the latent space covariance matrix, used to characterize the correlation across different dimensions. This represents the weighted distance with covariance constraints. This represents the squared Euclidean distance between the input data and the reconstructed data. This represents the exponential amplification function, used to enhance sensitivity to large deviation anomalies. Indicates the scaling adjustment parameter. Represents implicit space structure constraints. This represents the weight coefficient of each part; it judges the basic anomaly score according to the preset threshold to identify abnormal data and outputs the corresponding anomaly degree and type identifier.

[0133] Specifically, the method utilizes standardized data for training to learn the latent distribution representation of the data. By mapping and reconstructing the latent space of the data to be detected, the model can accurately calculate anomaly scores, measuring the degree to which the data deviates from the normal distribution. Weighted reconstruction error and covariance constraints, combined with exponential amplification functions and latent space structure constraints, enhance the sensitivity to large-deviation anomalies, thereby improving the accuracy and robustness of anomaly detection. This method can effectively identify anomalies in the data and output anomaly severity and type identifiers, providing a reliable basis for subsequent data repair and optimization.

[0134] See appendix Figures 1 to 3 The long-tail anomaly enhancement and identification includes: performing long-tail enhancement processing on the abnormal data or labeled abnormal data identified by any detection module in step S2; increasing the proportion of long-tail anomaly samples through adaptive resampling and constructing an anomaly memory unit during the model training phase; using the trained model and anomaly memory unit to perform long-tail enhanced anomaly judgment on new data during the actual anomaly detection phase; constructing anomaly memory unit to encode and store anomaly patterns that appear in historical detection processes, and matching and enhancing them in subsequent detection processes; introducing the anomaly memory unit to assist in the discrimination of long-tail anomalies during the model training or inference phase, so as to improve the model's sensitivity to low-frequency anomaly patterns; and finally outputting the enhanced anomaly detection results and the corresponding long-tail anomaly identifiers.

[0135] Construct an enhanced anomaly scoring function that integrates resampling weights and anomaly memory matching:

[0136]

[0137] Where x represents the field vector of a single input data record. Basic anomaly scoring function, : Modulation top weighting coefficient, : The final anomaly score after long-tail enhancement; the higher the value, the more likely it is to be a long-tail anomaly. : No. The weighting coefficients of each anomalous prototype. Temperature parameters that control similarity decay : The exponential kernel function is used to characterize the similarity to historical anomaly patterns. : A function for estimating the frequency or density of a sample; Latent space representation; : No. An abnormal memory prototype The number of exception prototypes stored in the exception memory unit. The resampling weight function, which is related to the frequency of sample occurrence, is usually defined as:

[0138]

[0139] in, :sample The probability of occurrence in the data distribution Smoothing terms prevent the denominator from being zero. : Adjustment index, used to control the degree of long-tail enhancement;

[0140] : Long-tailed distribution modulation function, used to enhance anomaly scoring of samples in low-density regions, is expressed as:

[0141]

[0142] in, Smoothing coefficient.

[0143] Specifically, this implementation identifies low-frequency long-tail samples by performing distribution analysis on abnormal data and enhances their weight during training using an adaptive resampling mechanism, effectively alleviating the class imbalance problem. Simultaneously, it constructs an anomaly memory unit to prototype and continuously reinforce historical anomaly patterns, enabling the model to identify potential low-frequency anomaly patterns based on similarity matching in subsequent detections. Furthermore, by fusing resampling weights, anomaly memory matching terms, and a long-tail distribution modulation function, it constructs an enhanced anomaly scoring function, achieving joint enhanced discrimination of low-density region samples and historically similar anomaly samples. This not only significantly improves the model's sensitivity and recognition ability for long-tail anomalies but also enhances the stability, continuity, and interpretability of anomaly detection results, thereby effectively reducing the false negative rate and improving overall anomaly detection performance.

[0144] See appendix Figures 1 to 3 In step S4, the comprehensive anomaly score obtained based on confidence-weighted fusion adopts the following formula;

[0145]

[0146] in, Input data sample, The number of detection dimensions, including logical consistency deviation score, semantic consistency deviation score, and anomaly score after long-tail enhancement. : No. Scoring of each detection dimension The hyperbolic tangent function is used to compress the scoring range and suppress the influence of extreme values. : Scale adjustment parameter, used to adjust the sensitivity of scoring for different detection dimensions. : No. Dynamic confidence weights corresponding to each detection dimension; : Conflict degree function between detection results; Conflict adjustment weights : The integrated anomaly score after fusion;

[0147] Defined as:

[0148]

[0149] in, : No. Each detection module in the sample Uncertainty estimation at the location;

[0150] Defined as:

[0151]

[0152] in, : Conflict sensitivity index, j: represents "the index of another detection dimension compared with the i-th detection dimension".

[0153] The confidence-based weighted decision further includes: calculating the overall confidence of the fusion result, wherein the confidence is measured based on the degree of disagreement among the scores, and is defined as:

[0154]

[0155] in, : The normalized first Each detection module outputs, The deviation sensitivity index is used to control the non-linearity of the consistency measure. : The integrated anomaly score after fusion;

[0156] The anomaly detection threshold is adaptively adjusted based on the aforementioned confidence level.

[0157]

[0158] in, Basic threshold Threshold adjustment coefficient : Overall confidence level of the fusion results;

[0159] The final exception detection function is:

[0160]

[0161] in, : Indicator function, 1 when the condition is true, 0 otherwise;

[0162] Output the final anomaly determination result and the corresponding confidence level .

[0163] Specifically, this implementation normalizes and dynamically weights anomaly scores from different detection modules (such as semantic enhancement detection, deep learning anomaly detection, and long-tail anomaly enhancement recognition) to construct a nonlinear fusion function based on confidence modulation. This integrates the detection results from multiple modules, improving the accuracy and robustness of anomaly detection. The introduction of a confidence assessment model and decision boundary adjustment mechanism not only dynamically adjusts the confidence of the detection modules based on historical data but also adaptively adjusts the judgment threshold, further enhancing the sensitivity and reliability of detection. Ultimately, this fusion method can more accurately identify anomalous data and output confidence levels, effectively reducing false positives and false negatives, and improving the overall performance of the anomaly detection system.

[0164] See appendix Figures 1 to 3 The method further includes:

[0165] S5: For data identified as abnormal, perform automated data repair, and optimize the adjustable parameters in steps S2 to S4 based on the repair quality; specifically:

[0166] S5-1: Automated Data Repair

[0167] For the data samples identified as abnormal in step S4, perform the following operations:

[0168] S5-1-1: Based on the anomaly type identifiers (including logical anomaly type, semantic anomaly type, and distribution anomaly type) output by each detection step, comprehensively determine the anomaly type category to which the sample belongs. Where C is the number of exception type categories;

[0169] S5-1-2: Based on the exception type, invoke the corresponding repair strategy and construct an exception type-driven adaptive repair mapping function:

[0170]

[0171] Where L represents the original abnormal data, The repaired data :sample Belongs to the The probability of class exceptions, : No. The basic repair function corresponding to the exception type. : No. Repair strength adjustment coefficient, The memory-based repair and enhancement function is defined as follows:

[0172]

[0173] in, : No. A collection of historical repair patterns for class anomalies. Data feature mapping function : No. The weight of each memory pattern, Similarity decay parameter The number of memory patterns for this type of anomaly;

[0174] S5-1-3: Conduct consistency assessment and quality verification of the data before and after the repair, and generate a repair effect score;

[0175]

[0176] in, : Fix quality score, Mahalanobis distance with covariance weighting Smoothing term Structural constraint weights : Function for ensuring the structural consistency of the repaired data.

[0177] S5-2: Based on the repair results obtained in S5-1, the adjustable parameters in steps S2 to S4 are periodically or in batches optimized and updated, specifically including:

[0178] S5-2-1: Define a feedback-driven loss function by utilizing the deviation between the repair quality score and the detection scores before and after repair:

[0179]

[0180] in, : Fix quality score, Quality sensitivity index The deviation function between the detection result and the repaired result is defined as follows: , Let represent the fusion anomaly scoring function, and D represent the dataset.

[0181] S5-2-2: Combine all variables learned from training data, statistically calculated from detection results, and feedback-driven loss. Batch gradient descent updates are performed on each adjustable parameter:

[0182]

[0183] in, , These represent the parameters of the model in the t-th and t+1-th iterations, respectively. The learning rate is represented by α and β, respectively; the loss weight coefficients are represented by α and β. This represents the anomaly detection loss function, used to measure the model's bias in identifying logical consistency anomalies, semantic consistency anomalies, and anomaly distribution anomalies. This represents the data repair loss function, used to measure the difference between the repaired data and the target data, as well as the degree to which the repair result violates structural and logical constraints.

[0184] The anomaly detection loss function is defined as follows:

[0185]

[0186] The data repair loss function is defined as follows:

[0187]

[0188] in, Let y(x) represent the training sample set used for anomaly detection, and let y(x) represent the anomaly label of sample x, where anomalous samples are assigned a value of 1 and normal samples are assigned a value of 0. denoted as the fusion anomaly score after normalization in step S4; w₊ and w₋ represent the loss weights of the anomaly samples and normal samples, respectively, where w₊ is adjusted according to the resampling weight of the long-tail samples. Let x̂ represent the sample set to be repaired, x̂ represent the repaired data, and x* represent the corresponding target data or reference data determined by manual verification and reliable historical records; Σ⁻¹ represents the inverse of the covariance matrix. Represents the structural constraint weights. This represents a function that indicates the structural consistency of the repaired data.

[0189] To further illustrate the effectiveness of the method of this invention, a multi-source business dataset consisting of e-commerce orders, payments, logistics, product records, and user behavior records was used for verification. The dataset includes relational table data, JSON format log data, and text-based field data; it includes anomaly types such as time sequence conflicts, amount or quantity balance conflicts, conditional dependency conflicts, semantic inconsistencies in fields or units, and low-frequency long-tail anomalies. The experimental environment consisted of a Windows 11 64-bit operating system, an Intel Core i7-12700K processor, 32 GB of RAM, and an NVIDIA RTX 3080 graphics card (10 GB of VRAM). The software environment included Python 3.10, PyTorch 2.0, and CUDA 11.8. Five-fold cross-validation was used, with precision, recall, F1-score, and AUC as evaluation metrics, and each module was progressively stacked for comparative experiments. The experimental results are shown in Table 1. Figure 2 , Figure 3 As shown:

[0190] Table 1

[0191]

[0192] Experimental results show that the baseline model (using only simple rules) has an F1-score of only 0.67. After introducing cross-field logical constraints, the F1-score improves to 0.75, an increase of 11.9%, indicating that logical constraints can effectively identify cross-field consistency errors. Further introducing knowledge graph semantic enhancement improves the F1-score to 0.80, significantly increasing the detection rate of semantically related anomalies. Adding deep learning anomaly detection further enhances the F1-score to 0.87, significantly improving the detection capability for distribution anomalies. Adding a long-tail anomaly enhancement mechanism further improves the F1-score to 0.90, reducing the false negative rate of long-tail anomalies by approximately 73%. Through multi-source detection result fusion, the F1-score further improves to 0.92. Finally, after confidence assessment and feedback optimization, the overall system F1-score reaches 0.94, an improvement of 40.3% compared to the baseline model, with an AUC of 0.97. The experiments verify the effectiveness and synergistic enhancement effect of each module in this invention.

[0193] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data cleaning method for complex logic and long-tail anomalies, characterized in that, Includes the following steps: S1: Acquire multi-source data, access and preprocess the multi-source data to obtain a standardized dataset; the standardized dataset consists of multiple data records, each record containing multiple data fields; S2: For the standardized dataset, perform cross-field logical consistency detection, semantic consistency detection, and abnormal distribution detection respectively; The three detection methods are independent of each other and are executed in parallel. The cross-field logical consistency detection includes: constructing a cross-field logical constraint model that includes time, numerical and conditional dependencies, calculating the logical consistency deviation of each data record, and outputting a logical anomaly type identifier and deviation score; The semantic consistency detection includes: introducing a knowledge graph to model data entities and semantic relationships, calculating the semantic consistency deviation of each data record, and outputting a semantic anomaly type identifier and deviation score; The anomaly distribution detection includes: using a deep learning model to learn data distribution characteristics, calculating a basic anomaly score for each data record, and outputting an anomaly distribution type identifier and a basic anomaly score; S3: Based on the basic anomaly score output by the anomaly distribution detection in step S2, perform long-tail anomaly enhancement identification, combine resampling weights and anomaly memory units to enhance the score of low-frequency long-tail anomalies, and output the long-tail enhanced anomaly score. S4: The logical deviation score, semantic deviation score, and long-tail enhanced anomaly score obtained in step S2 are used as scores for three detection dimensions. The scores are normalized and weighted based on confidence to obtain a comprehensive anomaly score. The judgment threshold is adaptively adjusted, and the final anomaly judgment result and confidence level are output. S5: For the data identified as abnormal in step S4, perform automated data repair and optimize the adjustable parameters in steps S2 to S4 based on the repair quality.

2. The data cleaning method according to claim 1, characterized in that, Step S1 includes: It unifies the access of structured, semi-structured, and unstructured data from different data sources, and completes data aggregation through preset interfaces or data acquisition protocols; Perform format parsing and structure conversion on the accessed data, and convert different data formats into a standard data structure; Field-level semantic mapping is performed based on predefined data standards and field specifications to map heterogeneous fields to a unified semantic space; The data undergoes integrity checks and preprocessing, including missing value identification, preliminary outlier detection, and data type verification. The detected missing and outlier data are then labeled to generate a standardized dataset with a unified format, semantics, and quality identifier.

3. The data cleaning method according to claim 1, characterized in that, The cross-field logical consistency check specifically refers to: For each data record in the standardized dataset, perform relation parsing on the data fields to identify the temporal sequence relationship, numerical balance relationship, and conditional dependency relationship between the fields; Based on the aforementioned relationship, a set of cross-field logical constraints is constructed, and each constraint is expressed as a computable constraint function; The unified cross-field consistency deviation function is constructed as follows: in A vector representing the fields of a single data record. : No. Inequality constraint functions, : No. The threshold corresponding to each constraint Non-linear penalty function : No. Weighting coefficients of each inequality constraint. : No. A set of equality constraint functions, : No. Weighting coefficients of each equality constraint, Norm index , : These represent the number of inequality constraints and equality constraints, respectively; For each data record in the standardized dataset, the degree of deviation under each constraint function is calculated to determine whether there is a logical consistency anomaly; wherein, by comprehensively evaluating the results of multiple constraints, the data consistency detection result and the corresponding anomaly type identifier are output.

4. The data cleaning method according to claim 1, characterized in that, The semantic consistency detection specifically refers to: Construct a knowledge graph related to the domain to which the data belongs, including entities, attributes, relationships and their semantic types related to the data fields, and perform semantic mapping and reasoning through the graph reasoning mechanism; Based on the standardized data records after semantic mapping in step S1, each data record is mapped to a semantic subgraph consisting of entities and their relationships in the knowledge graph. For the semantic subgraph, the semantic relationships between entities are verified using relational constraint rules and reasoning mechanisms in the knowledge graph. Construct a semantic consistency deviation function: in, A vector representing the fields of a single data record. Represents field vector With the first in the knowledge graph The mapping result of an entity or relationship. Indicates the first Standardized semantic types or rules for an entity or relationship. Represents the metric function. Indicates the first The weight coefficient of an entity or relationship. Represents field vector With the The degree of difference of individual unit or numerical constraints Indicates the first Standard values ​​for individual units or numerical constraints Indicates the first Weighting coefficients for unit or numerical constraints, The norm index is used to adjust for sensitivity to anomalies. This represents the number of semantic constraints associated with entities and relations in the knowledge graph. Indicates quantities related to units or numerical constraints. This indicates the overall semantic consistency deviation of the data records; the larger the value, the higher the degree of semantic inconsistency. The deviation score is used to identify whether there are problems such as field mismatch, inconsistent units, conflicting states, and semantic inconsistencies in the data, and the corresponding semantic anomaly type identifier and deviation score are output.

5. The data cleaning method according to claim 1, characterized in that, The abnormal distribution detection specifically refers to: A deep learning model is constructed to characterize the distribution features of data, and the model is trained using standardized data to learn the latent space representation of the data and obtain the reconstruction results of the data records in the latent space. The following basic anomaly scoring function is used: in, This represents a vector of fields for a single input data record. Represents the encoding function. Indicates the decoding function. This represents the mean vector of the normal data distribution in the latent space. This represents the inverse of the latent space covariance matrix, used to characterize the correlation across different dimensions. This represents the weighted distance with covariance constraints. This represents the squared Euclidean distance between the input data and the reconstructed data. Represents the exponential amplification function. Indicates the scaling adjustment parameter. Represents implicit space structure constraints. Indicates the weighting coefficient; The basic anomaly score is determined based on a preset threshold to identify abnormal data and output the corresponding distribution anomaly type identifier and basic anomaly score.

6. The data cleaning method according to claim 1, characterized in that, Step S3 specifically includes: Obtain the basic anomaly score output by the anomaly distribution detection in step S2. and the corresponding data samples; Distribution analysis is performed on samples whose basic anomaly scores exceed a preset threshold to identify low-frequency long-tailed anomaly samples. During the model training phase, adaptive resampling is used to increase the proportion of long-tailed anomalous samples in the training data, and anomaly memory units are constructed to encode and store anomalous patterns that appear in the historical detection process. During the model inference phase, the aforementioned anomaly memory unit is introduced to assist in the discrimination of long-tail anomalies, thereby improving the model's sensitivity to low-frequency anomaly patterns. Construct an enhanced anomaly scoring function that integrates resampling weights and anomaly memory matching: Where x represents the field vector of a single input data record, This represents the basic anomaly scoring function. Indicates the modulation top weighting coefficient. Indicates the first The weighting coefficients of each anomalous prototype. This represents the temperature parameter that controls the decay of similarity. This represents a function that estimates the frequency or density of a sample. Represented in the latent space; Indicates the first An abnormal memory prototype Indicates the number of abnormal prototypes stored in the abnormal memory unit; The resampling weight function is defined as follows: in, Indicates sample The probability of occurrence in the data distribution For smoothing terms, To adjust the index; The long-tailed distribution modulation function is defined as follows: in, For smoothing coefficients; Output the enhanced anomaly detection results and the corresponding long-tail anomaly identifiers.

7. The data cleaning method according to claim 1, characterized in that, In step S4, the comprehensive anomaly score obtained based on confidence-weighted fusion is calculated using the following formula: in, This represents the input data sample. This indicates the number of detection dimensions, including logical consistency deviation score, semantic consistency deviation score, and anomaly score after long-tail augmentation. Indicates the first Scoring of each detection dimension Hyperbolic tangent function Indicates the scaling adjustment parameter. Indicates the first Dynamic confidence weights corresponding to each detection dimension; : Conflict degree function between detection results; Conflict moderating weights : The integrated anomaly score after fusion; Defined as: in, Indicates the first Each detection dimension in the sample Uncertainty estimation at the location; Defined as: in, Indicates the conflict sensitivity index. Indicates the relationship with the first Another detection dimension index that compares the two detection dimensions; Subsequently, the overall confidence level of the fusion result is calculated and defined as: in, Represents the normalized i-th Scoring of each detection dimension Indicates the bias sensitivity index. This represents the overall anomaly score after fusion; The anomaly detection threshold is adaptively adjusted based on the aforementioned confidence level. in, Indicates the basic threshold. This represents the threshold adjustment coefficient; The final exception detection function is: in, This indicates an indicator function, which is 1 when the condition is true and 0 otherwise. Output the final anomaly determination result and the corresponding confidence level .

8. The data cleaning method according to claim 1, characterized in that, Step S5 specifically includes: S5-1: Automated Data Repair For the data samples identified as abnormal in step S4, perform the following operations: S5-1-1: Based on the anomaly type identifiers output by each detection step, comprehensively determine the anomaly type category to which the sample belongs. ,in Number of exception type categories; S5-1-2: Based on the exception type, invoke the corresponding repair strategy and construct an exception type-driven adaptive repair mapping function: in, This represents the original abnormal data. This indicates the data after repair. Indicates sample Belongs to the The probability of class exceptions, Indicates the first The basic repair function corresponding to the exception type. Indicates the first Repair strength adjustment coefficient, The memory-based repair enhancement function is defined as follows: in, Indicates the first A collection of historical repair patterns for class anomalies. Represents a data feature mapping function. Indicates the first The weight of each memory pattern, This represents the similarity decay parameter. Indicates the number of memory patterns for this type of anomaly; S5-1-3: Conduct consistency assessment and quality verification of the data before and after the repair, and construct a repair quality score: in, Indicates the repair quality score. This represents the Mahalanobis distance weighted by covariance. Indicates the smoothing term. Represents the structural constraint weights. This represents a function that indicates the structural consistency of the repaired data.

9. The data cleaning method according to claim 8, characterized in that, Step S5 also includes: S5-2: Closed-loop feedback optimization Based on the repair results, the adjustable parameters in steps S2 to S4 are periodically or in batches optimized and updated; By utilizing the discrepancy between the repair quality score and the detection scores before and after repair, a feedback-driven loss function is defined: in, Indicates the repair quality score. Indicates the quality sensitivity index. The deviation function between the detection result and the repaired result is defined as follows: , This represents the fusion anomaly scoring function. This represents a dataset. Batch gradient descent updates are performed on each adjustable parameter: in, , These represent the parameters of the model in the t-th and t+1-th iterations, respectively. Indicates the learning rate; , Indicates the loss weighting coefficient; This represents the anomaly detection loss function, used to measure the model's bias in identifying logical consistency anomalies, semantic consistency anomalies, and anomaly distribution anomalies. This represents the data repair loss function, used to measure the difference between the repaired data and the target data, as well as the degree to which the repair result violates structural and logical constraints. The anomaly detection loss function is defined as follows: The data repair loss function is defined as follows: in, This represents the training sample set used for anomaly detection. This represents the anomalous label for sample x, with 1 for anomalous samples and 0 for normal samples. This represents the fusion anomaly score after normalization in step S4; , represents the loss weights for abnormal samples and normal samples, respectively, where w₊ is adjusted according to the resampling weights of long-tail samples; This represents the sample set to be repaired. This indicates the data after repair. This indicates the corresponding target data or reference data that has been manually verified and determined based on reliable historical records; Denotes the inverse of the covariance matrix. Represents the structural constraint weights. This represents a function that indicates the structural consistency of the repaired data.

10. A data cleaning system for complex logic and long-tail anomalies, characterized in that, A method for performing the data cleaning method according to any one of claims 1-9 includes: The multi-source data access and standardization module is used to execute step S1; The cross-field logical consistency detection module is used to perform the cross-field logical consistency detection in step S2; The semantic consistency detection module is used to perform the semantic consistency detection in step S2; An anomaly distribution detection module is used to perform anomaly distribution detection in step S2; The long-tail anomaly enhancement and identification module is used to execute step S3; The multi-source detection result fusion and confidence determination module is used to execute step S4; An automated data repair and feedback optimization module is used to execute step S5.

Images