A service data cleaning method based on data duplication relationship

Abstract

This invention discloses a business data cleaning method based on data duplication relationships, belonging to the field of business data cleaning technology. The method includes step one: selecting the 20 most recent consecutive batches of data confirmed to be of normal quality from historical data archives, calculating the clustered duplication rate for each field in each batch, and establishing a baseline for the normal duplication rate of each field; combining the comprehensive anomaly score within the horizontal batch with the vertical cross-batch drift identification through two-dimensional cross-validation, marking the top 10% of the scores and those marked as drift-affected records as double-confirmed anomalies for direct cleaning, and marking those with scores contradicting the drift identification conclusions as single-dimensional anomalies entering the manual verification queue. This two-dimensional cross-validation significantly reduces the risk of erroneous or missed cleaning caused by misjudgment in a single dimension, ensuring cleaning accuracy while minimizing reliance on manual review.

Description

Technical Field

[0001] This invention belongs to the field of business data cleaning technology, specifically a business data cleaning method based on data duplication relationships. Background Technology

[0002] Business data cleaning is a fundamental step in data governance. In scenarios such as e-commerce orders, medical records, financial transactions, and government archives, data tables are usually imported periodically in batches, with each batch containing thousands to hundreds of thousands of records, and each record covering multiple fields. However, existing technologies, when dealing with duplicate anomalies in batch business data, typically rely mainly on single-field duplication rate thresholds or rule bases for independent field-by-field judgment, generally ignoring the overall structural changes in the co-occurrence relationships between fields. Batch error entry often does not occur in isolation in a single field, but rather manifests as a synchronous abnormal expansion or breakage of relationships between multiple fields, which cannot be captured by single-field duplication rates alone. Furthermore, existing technologies lack the ability to perceive the continuous drift of duplicate anomalies over time, leading to the gradual pollution problem of some fields' duplication rates slowly accumulating and increasing across multiple batches being overlooked until data quality deteriorates significantly. In addition, existing technologies often employ coarse-grained methods such as batch deletion or full field clearing after identifying abnormal records, failing to distinguish which fields and values ​​in the records truly belong to the anomalies caused by excessive duplication, and which fields' information remains valid, resulting in unnecessary loss of valid business data. Existing technologies also lack a cross-validation mechanism between horizontal batch-level anomaly scoring and vertical cross-batch drift identification; single-dimensional judgments are prone to misjudgment or omission due to local fluctuations. Therefore, a business data cleaning method based on data duplication relationships is proposed. Summary of the Invention

[0003] The purpose of this invention is to provide a business data cleaning method based on data duplication relationships to solve the problems mentioned in the background art.

[0004] A business data cleaning method based on data duplication relationships includes: Step 1: Select the 20 most recent consecutive batches of data that have been confirmed to be of normal quality from the historical data archive, calculate the clustered repetition for each field in each batch, and establish the normal repetition baseline for each field; Step 2: Used to calculate the co-occurrence correlation strength between any two fields from the data table to be cleaned and the original data of 20 historical normal batches, determine whether the correlation strength deviates from the historical normal level, and output the field pair correlation matrix; Step 3: Based on the field-to-field association matrix, the abnormal relationships between fields are mapped into a graph structure to identify the sets of mutually related abnormal fields, i.e., duplicate abnormal clusters, and to locate the core abnormal field in each abnormal cluster that is at the center of abnormal propagation. Step 4: Using the normalized value of the abnormal correlation degree plus 1 as the weight of each field, calculate the field duplication abnormality score and correlation abnormality score for each record, and then sum them up to obtain the comprehensive abnormality score. Step 5: Calculate the batch repetition rate for each field in 20 historical normal batches to form a time series. Obtain the comprehensive drift intensity score by multiplying the linear trend slope by the trend consistency score, and identify significant drift fields and drift warning fields. Step 6: Used to locate the drift start batch by comparing the mean difference between segments in the drift significant field through traversal segmentation positions, and to identify excess duplicate records within the drift batch by setting the boundary at twice the normal occurrence frequency benchmark for each value; Step 7: The drift is attributed to three types: default value type, system switching type, and progressive contamination type, and the corresponding cleaning strategy is matched. At the same time, a two-dimensional cross-validation is performed on the comprehensive anomaly score and the drift identification result. The double-confirmed anomaly records are directly cleaned, and the single-dimensional anomaly records are manually checked.

[0005] As a further aspect of this invention, the specific method for establishing the normal repeatability baseline for each field is as follows: To determine the central tendency of a field in a given batch, we first calculate the central tendency of a field in each batch by summing the total number of records corresponding to the top 20% of values ​​and dividing this sum by the total number of records for that field in the current batch. Then, we calculate the central tendency of a field in each of the 20 batches, resulting in a numerical sequence of length 20. We then calculate the arithmetic mean and standard deviation of this sequence. The mean plus twice the standard deviation is used as the upper limit of the normal range, and the mean minus twice the standard deviation is used as the lower limit. If the lower limit is less than 0, it is set to 0. This forms the baseline range for the normal repetition of the field. Finally, we record the type of each field, categorized as enumeration, numbered, free text, and numeric.

[0006] As a further aspect of the present invention: the specific method for calculating the co-occurrence correlation strength between any two fields is as follows: For any two fields, A and B, in the data table to be cleaned, enumerate all distinct values ​​in field A. For each value, find all records with that value in the data to be cleaned, and count the number of distinct values ​​of field B appearing in these records. This count is called the one-to-many count for that value of field A. Sort all the one-to-many counts of field A in ascending order. If the number of values ​​is odd, take the value in the middle position; if it is even, take the average of the two middle values. This gives the current median one-to-many count of field A for field B. Calculate the median one-to-many count of field A for field B in the same way for 20 historical normal batches to obtain 20 historical data. The historical value is calculated by averaging 20 historical values ​​as the historical baseline median one-to-many count for this field pair. The deviation ratio is obtained by dividing the current median one-to-many count by the historical baseline median one-to-many count. If the historical baseline median one-to-many count is 0, it means that this field pair has never had a co-occurrence association in the historical batch, and it is considered as 0.5 in the calculation to avoid the case of dividing by 0. 0.5 is the smallest non-zero substitute value less than 1 to reflect that any co-occurrence at present constitutes an inflationary tendency. A deviation ratio greater than 3 marks the field pair as an association inflation anomaly, a deviation ratio less than 0.33 marks it as an association contraction anomaly, and a deviation ratio between 0.33 and 3 is marked as normal.

[0007] As a further aspect of the present invention, the specific method for identifying the set of interrelated abnormal fields, i.e., the duplicate abnormal clusters, is as follows: An undirected graph is constructed with each field in the data table as a node. For all field pairs marked as having abnormal association expansion or contraction in the field pair association matrix, an edge is connected between the corresponding two nodes. The edge attributes record the anomaly type and deviation ratio. Nodes that do not form any abnormal association with any field are isolated nodes and do not participate in subsequent anomaly cluster analysis. Starting from any unvisited node, all nodes directly connected to that node are checked sequentially, and then nodes directly connected to these nodes but not yet visited are checked. This process is continued until no new nodes are available for access. All nodes involved in this traversal are grouped into the same connected subgraph as a cluster. Repeat the above process for the remaining unvisited nodes until all nodes are visited. Each connected subgraph corresponds to a repeating anomaly cluster. For each node in the repeating anomaly cluster, count the number of abnormal edges connected to it and record it as the abnormal correlation degree of that field. The node with the highest abnormal correlation degree is marked as the core anomaly field. If multiple nodes have the same highest abnormal correlation degree, further calculate the degree of deviation of the concentration of repetition for each node. The degree of deviation of the concentration of repetition is calculated by dividing the absolute value of the difference between the current concentration of repetition of the field and the baseline mean of the field in step one by the baseline standard deviation. Take the node with the largest deviation degree as the core anomaly field.

[0008] As a further aspect of the present invention, the specific method for calculating the comprehensive anomaly score for each record is as follows: For a field belonging to a certain duplicate anomaly cluster, increment the field's anomaly correlation score by 1. Sum the anomaly correlation scores of all fields within the cluster, increment them by 1, and then divide the sum by the incremented anomaly correlation score of each field to obtain the normalized weight of that field. The sum of the weights of all fields within the cluster equals 1. Assign a uniform minimum weight (1 divided by the total number of fields in the data table) to isolated fields that do not belong to any duplicate anomaly cluster. For each field value in each record, count the actual occurrences of that value in the data table to be cleaned. Multiply the upper limit of the normal range for that field in the field baseline table by the total number of records in the current batch to obtain the expected maximum occurrence count. If the actual occurrence count exceeds the expected maximum occurrence count, the field's duplicate anomaly score is equal to the actual occurrence count divided by the expected maximum occurrence count minus 1; otherwise, the field's duplicate anomaly score is 0. For records belonging to a certain duplicate anomaly cluster, take the core data from that record... The current value of the abnormal field is used to count the number of different values ​​of other fields in the cluster that appear simultaneously with the core field value in the data to be cleaned. The average of the number of different values ​​of all other fields in the cluster is taken as the actual association breadth. The expected value of the historical baseline is the average of the one-to-many counts of the core abnormal field to other fields in the cluster in the historical baseline of the field-to-association matrix. If the expected value of the historical baseline is 0, it is regarded as 0.5 and included in the calculation. The association abnormality score is equal to the actual association breadth divided by the expected value of the historical baseline and then subtracted by 1. If the actual association breadth does not exceed the expected value of the historical baseline, the association abnormality score is 0. The association abnormality score of records that do not belong to any abnormal cluster is directly set to 0. For each record, the duplicate abnormality scores of each field are multiplied by the corresponding normalized weights and summed. Then, the association abnormality score is multiplied by the normalized weight of the core abnormal field to obtain the comprehensive abnormality score.

[0009] As a further aspect of the present invention, the specific method for analyzing the continuous trend of repeatability changes in 20 historical normal batches is as follows: To calculate the batch repetition rate of a field in a batch of data, count the total number of records where a certain field appears more than 1. Divide this count by the total number of records for that field in the batch. Calculate this rate for each of the 20 historical batches to form a repetition rate sequence of length 20. The average batch number is fixed at 1 to 20 (the sum of these 20 integers divided by 20 equals 10.5). Calculate the average repetition rate of the 20 batches. For each batch, subtract 10.5 from the batch number, multiply by the batch repetition rate minus the average repetition rate, and sum all the products to obtain the cumulative sum in the numerator. Then, subtract 10.5 from the batch number, square the result, and sum all 20 squares to obtain the cumulative sum in the denominator. The slope of the linear trend is set to 665, and the cumulative sum of the numerator is divided by 665. The overall trend direction is determined by the sign of the slope. Each element from the 2nd to the 20th in the sequence is compared with the previous element. If it is greater than the previous element, it is recorded as an increase; if it is less than the previous element, it is recorded as a decrease; if it is equal, it is recorded as flat. Flat values ​​are not included in the statistics of effective adjacent pairs. The number of adjacent pairs with the same trend direction as the overall trend is divided by the total number of effective adjacent pairs participating in the comparison to obtain the trend consistency score. The absolute value of the slope is multiplied by the trend consistency score to obtain the comprehensive drift intensity score. Fields with a score greater than 0.005 are marked as drift significant fields, and fields with a score greater than 0.001 but less than 0.005 are marked as drift warning fields.

[0010] As a further aspect of the present invention, the specific method for locating the drift initiation batch and identifying excessive duplicate records is as follows: Starting from the third position, attempt to divide the 20 batches of repetition rate sequence into a first half and a second half at each position. For each division method, calculate the average of all repetition rate values ​​in the first half and the average of all repetition rate values ​​in the second half, and record the absolute value of the difference between the two, which is the inter-segment mean difference. After traversing all possible division positions, ensure that each segment contains at least 2 data points. Find the division position that maximizes the inter-segment mean difference, and mark the batch number corresponding to this position as the drift start batch. The average repetition rate of all batches before the drift start batch is recorded as the baseline repetition rate before drift, and the average repetition rate of all batches from the drift start batch onwards is recorded as the actual repetition rate after drift. The absolute value of the difference between the actual repetition rate after drift and the baseline repetition rate before drift is recorded as the drift amplitude. The quasi-duplicate rate is the average of the duplicate rates of the drift-initiated batch and all subsequent batches, which is recorded as the actual duplicate rate after the drift. For fields that experience upward drift, the number of times each value of the field appears in each batch of data before the drift is counted. The average of the occurrences in each batch before the drift is taken and rounded down to obtain the normal occurrence benchmark for that value. If a value has never appeared in the historical batches before the drift, the median of the average occurrences of all values ​​in each batch before the drift is used as the alternative benchmark. In the data of the drift-initiated batch and subsequent batches, if the actual occurrence of that value exceeds twice the normal occurrence benchmark, all corresponding records of that value in that batch are arranged from latest to earliest according to the entry timestamp. The records with the difference between the actual occurrence and the normal occurrence benchmark are marked as excess duplicate records generated by the drift.

[0011] As a further aspect of the present invention, the specific method for attributing drift to the three types of business causes is as follows: First, determine if it is a concentrated default value type drift. In the data from the starting batch of the drift and subsequent batches, count the number of records corresponding to the single value that appears most frequently for this field, and divide the number of records by the total number of records in the drift batch. If this ratio exceeds 50%, it is determined to be a concentrated default value type drift. If it does not meet the condition, determine if it is a system switch type drift. Count all different values ​​of this field that appeared in all batches before the drift to form the value set before the drift, and all different values ​​that appeared in all batches after the drift to form the value set after the drift. Calculate the number of identical values ​​in the two sets and divide it by the total number of different values ​​after merging the two sets to obtain the overlap rate. If the overlap rate is less than 10%, it is determined to be a system switch type drift. If neither of the above two conditions is met, it is determined to be a gradual contamination type drift.

[0012] As a further aspect of the present invention, the specific method for matching differentiated cleaning strategies to the three drift types is as follows: Centralized default value drift modifies the field value of excessive duplicate records corresponding to the frequently occurring values ​​in the drift batch to null and adds a "to be supplemented" mark instead of directly deleting the entire record. Simultaneously, a data supplementation notification is generated, including the drift field name, abnormal value content, number of records involved, and suggested supplementation deadline. System switchover drift does not perform any deletion or field modification operations on the relevant records. Two independent baseline records are established for this field, one before the switchover and one after, with the drift start batch as the dividing point. The drift start batch is marked as the baseline switchover node. Gradual contamination drift... The process involves cleaning each field value in the drift batch by retaining the earliest recorded record and deleting subsequent duplicate records that exceed the normal occurrence frequency benchmark. The earliest recorded record is determined by its entry timestamp. Simultaneously, an upstream data source investigation notification is generated. The notification field includes the drift field name, the drift start batch time, the drift amplitude value (the absolute value of the actual repetition rate after drift minus the benchmark repetition rate before drift), and the suspected problematic data source identifier extracted from the data source identifier field that marks the source of the records in this batch in the data table to be cleaned. If the data source identifier field does not exist in the data table, this item is marked as empty.

[0013] As a further aspect of the present invention, the specific method for performing two-dimensional cross-validation is as follows: Records whose overall anomaly score ranks in the top 10% of all records and are also marked as drift-affected records in step six are marked as double-confirmed anomaly records and will be directly cleaned according to the corresponding drift type's cleaning strategy, with the highest cleaning priority. Records whose overall anomaly score ranks between the top 10% and top 30% but are not marked as drift-affected records in step six, or records marked as drift-affected records in step six but whose overall anomaly score ranks in the bottom 50% of all records, are marked as single-dimensional anomaly records and will enter the manual verification queue. Records that are not identified as anomalies in steps four and six will remain in normal status and will not be cleaned.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention establishes a personalized redundancy baseline for each field from 20 consecutive normal batches in history, so that the judgment criteria can automatically adapt to the differences in business characteristics of different fields. The province field naturally obtains a higher normal baseline because the value is fixed and limited, thus avoiding the problem of false alarms caused by overly sensitive fixed thresholds for some fields and false alarms caused by overly lenient thresholds for other fields. (2) In this invention, by calculating the median one-to-many count of the number of different values ​​of field B corresponding to the value of field A, and comparing it with the historical baseline to obtain the deviation ratio, it is possible to discover structural anomalies caused by batch erroneous data entry. By automatically identifying the interconnected abnormal field set through breadth-first traversal, the core abnormal field with the highest abnormal correlation in each abnormal cluster is located as the source of abnormal propagation, providing a basis for weight allocation for subsequent record-level scoring. (3) In this invention, the record-level comprehensive scoring no longer simply marks all records that exceed the threshold of a certain field as abnormal. Instead, it comprehensively considers the degree of repetition of the record in multiple field dimensions and the depth of participation in abnormal correlation. By normalizing the weight based on the degree of abnormal correlation, the core abnormal field is given a greater weight in the scoring. Records that are truly involved in structural anomalies are given a higher score to facilitate priority cleaning. Records that only slightly exceed the baseline due to normal fluctuations in a few fields are given a lower score to avoid false negatives. By calculating the linear trend slope and trend consistency score of the repetition rate sequence of each field in the historical 20 batches, the two are multiplied to obtain the comprehensive drift intensity score, which can identify the drift phenomenon where the repetition rate gradually increases or decreases over time. (4) In this invention, the drift start batch is located by trying all possible segmentation positions to select the position with the largest difference in the mean repetition rate between the front and back segments as the jump node, and the average number of occurrences of each batch before the drift is used as the normal benchmark. Only the part of the drift batch that exceeds twice the benchmark is marked as an excessive repetition record. By combining the comprehensive anomaly score within the horizontal batch with the vertical cross-batch drift identification through two-dimensional cross-validation, the top 10% of the scores and the records marked as drift-affected records are marked as double-confirmed anomalies and directly cleaned. The scores that contradict the drift identification conclusions are marked as single-dimensional anomalies and enter the manual verification queue. By verifying each other through two dimensions, the risk of incorrect cleaning or missed cleaning caused by misjudgment of a single dimension is greatly reduced. While ensuring the accuracy of cleaning, the reliance on manual review is minimized. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method framework structure of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: Please refer to Figure 1 This application provides a business data cleaning method based on data duplication relationships, including: Step 1: Select the 20 most recent consecutive batches of data that have been confirmed to be of normal quality from the historical data archive, calculate the clustered repetition for each batch and each field, and establish a normal repetition baseline for each field; Here, "confirmed as normal quality" refers to batch data that has been manually reviewed or verified through historical business processes and confirmed to have no batch errors in data entry; "concentration repetition" refers to the proportion of records covered by a few frequently occurring values ​​in this field, reflecting the degree of concentration of values ​​in this field; "normal repetition baseline" refers to the reasonable fluctuation range of the concentration repetition of this field under normal business conditions. The method for calculating the clustered repetition is as follows: count all different values ​​of a certain field in the current batch and the number of times each value appears, sort all different values ​​from highest to lowest according to the number of occurrences, take the sum of the total number of records corresponding to the top 20% of values, divide it by the total number of records of that field in the current batch, and obtain a ratio between 0 and 1, which is the clustered repetition of that field in the current batch.

[0018] After calculating the central repetition of a certain field in 20 batches, a numerical sequence of length 20 is obtained. The mean and standard deviation of the sequence are calculated. The mean plus twice the standard deviation is used as the upper limit of the normal interval, and the mean minus twice the standard deviation is used as the lower limit of the normal interval. If the lower limit is less than 0, it is taken as 0. This constitutes the normal repetition baseline interval of the field.

[0019] At the same time, the type of each field is recorded, which is divided into enumeration type (the value is fixed to a limited number of options, such as province and status code), number type (the value is theoretically unique or rarely repeated, such as order number and customer number), free text type (the value is relatively scattered, such as remarks and address description), and numeric type (the value is a continuous number, such as amount and quantity). The field type is used as a reference for the weight rationality verification in step four.

[0020] The most recent 20 consecutive batches of data confirmed to be of normal quality are selected from the historical archive. For each field in each batch, the concentration repetition is calculated, which is the proportion of records covered by the top 20% of high-frequency values. The mean and standard deviation of the concentration repetition of the 20 batches are calculated. The mean plus or minus 2 times the standard deviation constitutes the normal repetition baseline interval for that field. At the same time, the type of each field is recorded, and an independent benchmark interval based on its own historical fluctuation range is established for each field. This makes the anomaly judgment criteria match the actual business rules of the field. Different field types (enumerated, numbered, free text, numerical) have significant differences in concentration repetition under normal conditions, and a uniform threshold cannot be adapted. However, the historical statistical benchmark naturally absorbs this difference. The upper and lower limits of the baseline interval are derived from historical data rather than manually set, reducing the reliance on human experience. The field type record provides a basis for the weight rationality verification in step four, realizing the substantial impact of the data in step one on the strategies of subsequent steps.

[0021] Step 2: Calculate the co-occurrence correlation strength between any two fields from the data table to be cleaned and the original data of 20 historical normal batches, determine whether the correlation strength deviates from the historical normal level, and output the field pair correlation matrix; Here, the data table to be cleaned refers to the target data table that needs to undergo quality inspection and cleaning. It is organized in batches, with each batch containing several business records, and each record consisting of multiple fields. The field structure of the data table to be cleaned is consistent with the field structure of the historical archive batches, that is, the field names, number of fields, and field meanings are the same, ensuring that each field in the data table to be cleaned can be compared one by one with the corresponding fields in the historical batches. The data table to be cleaned enters the processing flow of this method in a single batch, that is, each time this method is executed, one batch to be cleaned is processed, and after processing is completed, the next batch can be repeated. The data table to be cleaned must contain an entry timestamp field.

[0022] The value co-occurrence association strength here refers to the number of different values ​​of field B corresponding to a certain value of field A in the same record, reflecting the tightness of the one-to-many relationship between the two fields; an abnormally high value co-occurrence association strength indicates that some values ​​of field A are incorrectly reused for values ​​of field B that should not be associated with it, which is usually a structural feature of batch erroneous data entry.

[0023] For any two fields, A and B, in the data table to be cleaned, calculate the median one-to-many count as follows: Enumerate all distinct values ​​in field A. For each value, find all records with that value in the data to be cleaned, and count the number of distinct values ​​of field B appearing in these records. This count is the one-to-many count for that value of field A. Sort all the one-to-many counts of field A in ascending order. If the number of values ​​is odd, take the value in the middle; if it is even, take the average of the two middle values. This gives the current median one-to-many count of field A for field B. The median is used here instead of the mean to exclude the interference of a few extreme values ​​on the overall correlation level. For example, if a certain abnormal customer ID corresponds to hundreds of contact numbers, taking the average would significantly inflate the overall level and lead to distorted judgments.

[0024] In 20 historical normal batches, the median one-to-many count of field A to field B was calculated in the same way to obtain 20 historical values. The average of the 20 historical values ​​was taken as the historical baseline median one-to-many count of the field pair.

[0025] The deviation ratio is obtained by dividing the current median one-to-many count by the historical baseline median one-to-many count. If the historical baseline median one-to-many count is 0, it is treated as 0.5 in the calculation to avoid division by zero. A deviation ratio greater than 3 marks the field pair as having an abnormal association expansion, indicating that the value of field A corresponds to a far greater number of different values ​​of field B in the current data than is normal. A deviation ratio less than 0.33 marks the field pair as having an abnormal association contraction, indicating that there is an abnormal break between fields that originally had an association relationship. A deviation ratio between 0.33 and 3 is marked as normal.

[0026] To improve computational efficiency, an inverted index is pre-built for each field. An inverted index is a list of record row numbers corresponding to each different value of the field. When calculating the correlation strength between fields, the co-occurring records can be quickly located by querying the inverted index, avoiding pairwise comparisons of all records. When the number of records in the data table reaches tens of thousands, the inverted index can improve computational efficiency by tens of times.

[0027] For any two fields in the data table to be cleaned, calculate the median one-to-many count (i.e., the median number of different values ​​of field B corresponding to each value of field A in the same record). Compare this to the historical baseline calculated using the same criteria for 20 normal batches to obtain the deviation ratio. A deviation ratio greater than 3 is marked as an association expansion anomaly, and a deviation ratio less than 0.33 is marked as an association contraction anomaly. Output the field pair association matrix. At the same time, pre-build an inverted index for each field to improve calculation efficiency. Through the overall structural analysis of the association strength between field values ​​co-occurrence, capture structural batch anomalies that cannot be detected by single field repetition rate. These anomalies often manifest as synchronous expansion of multi-field associations, which is a typical feature of batch erroneous data entry. Use the median instead of the mean to eliminate the interference of extreme values, making the measurement of association strength more robust. Use the historical baseline ratio instead of a fixed absolute threshold to make the judgment standard adaptable to the actual degree of association of different field pairs. The inverted index improves the field pair calculation efficiency by tens of times in scenarios with more than 10,000 data entries, ensuring the engineering usability of the method.

[0028] Step 3: Based on the field-to-field association matrix, map the abnormal relationships between fields into a graph structure, identify the sets of mutually related abnormal fields, i.e., duplicate abnormal clusters, and locate the core abnormal field in each abnormal cluster that is located at the center of abnormal propagation. Here, a duplicate anomaly cluster refers to a set of fields that are interconnected through abnormal relationships, representing a local structural duplicate anomaly area in the data table; the core anomaly field refers to the field within the anomaly cluster that has the most abnormal relationships with other fields, and is usually the source field of the abnormal duplicate relationship.

[0029] An undirected graph is constructed with each field in the data table as a node. For all field pairs in the field pair association matrix marked as association expansion anomalies or association contraction anomalies, an edge is connected between the corresponding two nodes. The edge attributes record the anomaly type and deviation ratio. Nodes that do not form anomaly associations with any field are considered isolated nodes and are not included in subsequent anomaly cluster analysis.

[0030] Perform connectivity analysis on the graph: Starting from any unvisited node, perform a breadth-first traversal to check all nodes directly connected to that node. Breadth-first traversal means starting from the starting node, first adding all nodes directly connected to the starting node to the queue to be visited, then taking out nodes from the queue in a first-in-first-out order and checking their directly connected but unvisited nodes to add them to the queue. Repeat this process until the queue is empty, and continue expanding until there are no new nodes to visit. All nodes involved in this traversal are grouped into the same connected subgraph. Repeat the above process for the remaining unvisited nodes until all nodes have been visited. Each connected subgraph corresponds to a duplicate anomalous cluster.

[0031] For each node within a cluster of duplicate anomalies, count the number of anomalous edges connected to it, and record this as the anomalous correlation degree of that field. The node with the highest anomalous correlation degree is marked as the core anomaly field. If multiple nodes have the same highest anomalous correlation degree, further calculate the degree of deviation of the clustered duplication degree of each node. The calculation method is to divide the absolute value of the difference between the current clustered duplication degree of the field and the baseline mean of the field in step one by the baseline standard deviation, and take the node with the largest deviation degree as the core anomaly field. If they are still tied, take the field whose name is earlier in lexicographical order.

[0032] An undirected graph is constructed using each field in the data table as a node and the outlier field pairs identified in step two as edges. Breadth-first search is used to perform connectivity analysis, grouping related outlier fields into the same repeating outlier cluster. Within each cluster, the core outlier field is located based on the highest outlier correlation (or the largest deviation in cluster repetition if they are tied). This integrates isolated field pairs into structured outlier clusters, enabling subsequent analysis to understand the local structure of data anomalies at the cluster level, rather than relying on scattered field pairs. The location of the core outlier field provides a basis for weight allocation in step four, allowing fields with high outlier correlation to receive higher weights in the overall score, thus realizing the substantial impact of the output of step three on the strategy of step four. The computational cost of breadth-first search connectivity analysis is linearly related to the number of edges, making it suitable for data tables with tens to hundreds of fields.

[0033] Step 4: Based on the full records of the data table to be cleaned, the field baseline table, and the list of duplicate anomaly clusters, calculate a comprehensive anomaly score for each record to quantify the degree to which the record participates in the abnormal duplication relationship, and provide a sorting basis for subsequent hierarchical cleaning; For a field belonging to a duplicate anomaly cluster, increment its anomaly correlation score by 1. This increment prevents the weight from dropping to zero when the anomaly correlation score is zero, ensuring each field has at least a minimum weight. Summate the anomaly correlation scores of all fields within the cluster by incrementing them by 1, then divide the sum by the incremented anomaly correlation score of each field to obtain the normalized weight of that field. The sum of the weights of all fields within the cluster equals 1. For isolated fields that do not belong to any duplicate anomaly cluster, assign a uniform minimum weight, which is 1 divided by the total number of fields in the data table.

[0034] For each field of each record, count the actual number of times the value appears in the data table to be cleaned; multiply the upper limit of the normal range of the field in the field baseline table of step one by the total number of records in the current batch to obtain the expected maximum number of occurrences; if the actual number of occurrences exceeds the expected maximum number of occurrences, the field duplication anomaly score is equal to the actual number of occurrences divided by the expected maximum number of occurrences and then minus 1; if the actual number of occurrences does not exceed the expected maximum number of occurrences, the field duplication anomaly score is 0.

[0035] For records belonging to a certain duplicate anomaly cluster, the current value of the core anomaly field in that record is taken. In the data to be cleaned, the number of different values ​​of other fields within the cluster that appear simultaneously with the core field value (i.e., appear in the same record) is counted. The average of the number of different values ​​of all other fields within the cluster is taken as the actual association breadth. The expected historical baseline value is the average of the one-to-many counts of the core anomaly field against the historical baseline values ​​of other fields within the cluster in the association matrix from step two. If the expected historical baseline value is 0, it is treated as 0.5 in the calculation to avoid division by zero. The association anomaly score equals the actual association breadth divided by the expected historical baseline value minus 1. If the actual association breadth does not exceed the expected historical baseline value, the association anomaly score is 0. Records not belonging to any anomaly cluster have an association anomaly score of 0.

[0036] For each record, the duplicate anomaly score for each field is multiplied by its corresponding normalized weight, and the sum is then added to the correlation anomaly score multiplied by the normalized weight of the core anomaly field to obtain a comprehensive anomaly score. A higher score indicates a more severe duplicate anomaly relationship and a greater likelihood that the record may require cleaning.

[0037] The normalized value of the anomaly correlation score plus 1 is used as the weight of each field. For each field of each record, a field repetition anomaly score is calculated (the number of actual occurrences exceeding the expected maximum number of occurrences minus 1). For records belonging to anomaly clusters, an additional correlation anomaly score is calculated (the ratio of the actual correlation breadth to the expected value of the historical baseline minus 1). The two parts are weighted and summed to obtain a comprehensive anomaly score. The comprehensive score integrates information from two dimensions: field-level repetition degree and inter-field correlation breadth. It can more accurately reflect the depth of the record's participation in the anomaly relationship than the ranking of the repetition frequency of a single field. The normalized weight plus 1 ensures that each field has at least a minimum weight, preventing a field from being completely ignored when the anomaly correlation score is 0. Isolated fields that do not belong to any anomaly cluster are assigned a uniform minimum weight, preserving the ability to detect anomalies in isolated fields. The scoring results provide a quantitative basis for the cleaning priority ranking and two-dimensional cross-validation in step seven.

[0038] Step 5: Based on the original data of 20 consecutive batches of historical archives, the list of duplicate anomaly clusters, and the record-level anomaly scoring table, detect from the time dimension whether the duplication rate of each field has a continuous changing trend in multiple historical batches; The mean comprehensive anomaly score here refers to the average comprehensive anomaly score of all records within the anomaly cluster to which this field is the core anomaly field. The design logic of the priority order is: fields that have been identified as highly suspicious in the horizontal batch analysis are more likely to have drift problems in the vertical cross-batch analysis. Prioritizing detection makes the two types of analysis dimensions mutually corroborate each other.

[0039] The batch repetition rate is calculated by dividing the total number of records corresponding to values ​​of a certain field that appear more than 1 in a batch of data by the total number of records for that field in the same batch. The resulting ratio, between 0 and 1, is the batch repetition rate for that field in that batch. This calculation is performed sequentially for 20 historical batches, forming a repetition rate sequence of length 20. These sequences are arranged chronologically from earliest to latest, with the first element corresponding to the earliest batch and the 20th element corresponding to the latest batch.

[0040] The average batch number is fixed as the sum of 20 integers from 1 to 20 divided by 20, which equals 10.5. For each batch, subtract 10.5 from the batch number, multiply by the batch repetition rate minus the average repetition rate, and sum all the products of the 20 batches to obtain the cumulative sum in the numerator. For each batch, subtract 10.5 from the batch number, square the result, and sum all the squares of the 20 batches to obtain the cumulative sum in the denominator, which is fixed at 665. The slope of the linear trend is equal to the cumulative sum in the numerator divided by 665. A positive slope of the linear trend represents an upward trend, and a negative slope represents a downward trend.

[0041] The average batch number is fixed as the sum of 20 integers from 1 to 20 divided by 20, which equals 10.5. The average repetition rate of the 20 batches is calculated. For each batch, subtract 10.5 from the batch number, multiply by the batch repetition rate minus the average repetition rate, and sum all the products of the 20 batches to obtain the cumulative sum in the numerator. For each batch, subtract 10.5 from the batch number, square the result, and sum all 20 squares to obtain the cumulative sum in the denominator, which is fixed at 665. The slope of the linear trend is equal to the cumulative sum in the numerator divided by 665. A positive slope represents an upward trend, and a negative slope represents a downward trend. The larger the absolute value of the slope, the steeper the trend.

[0042] The overall trend direction is determined by the sign of the slope. Each element from the 2nd to the 20th in the sequence is compared with the previous element. If it is greater than the previous element, it is recorded as an increase; if it is less than the previous element, it is recorded as a decrease; if they are equal, they are recorded as flat and are not included in the statistics. The number of adjacent pairs with the same trend direction is counted and divided by the total number of valid adjacent pairs participating in the comparison (flat pairs are not included in the numerator or denominator) to obtain the trend consistency score. The closer the score is to 1, the more stable the trend is, and the less likely it is to be a false impression caused by local random fluctuations.

[0043] Multiply the absolute value of the slope by the trend consistency score to obtain the comprehensive drift intensity score. Fields with scores exceeding 0.005 are marked as significant drift fields and proceed to step six for drift initiation batch location; fields with scores exceeding 0.001 but not reaching 0.005 are marked as drift warning fields and recorded in the monitoring list for priority review during subsequent batch data imports. The business meaning of the threshold 0.005 is as follows: if the repetition rate of a certain field increases monotonically at a rate of 0.005 per batch within 20 batches with high directional consistency, then the cumulative change in the repetition rate within 20 batches is approximately 0.10, i.e., from 10% to 20%. This magnitude is sufficient to cause a perceptible deviation between the actual data quality of the field and the baseline, which is a drift level worthy of attention.

[0044] For each of the 20 historical normal batches, the batch repetition rate of each field is calculated, forming a time series of length 20. The trend slope is calculated using least squares linear regression (cumulative sum of numerator divided by a fixed denominator of 665), and then the trend consistency score is calculated (adjacent comparisons with the same overall trend direction). The absolute value of the slope is multiplied by the trend consistency score to obtain a comprehensive drift intensity score. Fields exceeding 0.005 are marked as significantly drifted, and fields exceeding 0.001 but not reaching 0.005 are marked as drift warning fields. The trend detection in the time dimension compensates for the gradual accumulation problem that cannot be detected by the horizontal batch analysis in step four. The two dimensions complement each other to form a three-dimensional detection system. The trend consistency score corrects the slope, eliminating false slopes caused by abnormal fluctuations in individual batches, ensuring that fields marked as significantly drifted have continuous and stable directional changes rather than occasional disturbances. The 0.005 threshold corresponds to approximately 10% of the cumulative change within the 20 batches, which corresponds to the business-perceived deviation magnitude, balancing sensitivity and practicality.

[0045] Step 6: Based on the list of significant drift fields and their 20 batch repetition rate sequences, as well as the original data records of the significant drift batches, accurately locate the batches where the overall repetition rate of each significant drift field changes, and identify which specific records in the drift starting batch and subsequent batches are excess duplicate records caused by the drift, rather than treating all duplicate records in the drift batch as abnormal. Starting from the 3rd position, attempt to divide the 20 batches of repetition rate sequence into a first half and a second half at each position. For each division method, calculate the average of all repetition rate values ​​in the first half and the average of all repetition rate values ​​in the second half, and record the absolute value of the difference between the two, which is the inter-segment mean difference. After traversing all possible division positions, i.e. from the 3rd to the 18th, ensure that each segment contains at least 2 data points. Find the division position that maximizes the inter-segment mean difference, and mark the batch number corresponding to this position as the drift start batch. The average of the repetition rates of all batches before the drift start batch is recorded as the baseline repetition rate before drift, and the average of the repetition rates of all batches after the drift start batch is recorded as the actual repetition rate after drift. The absolute value of the difference between the two is recorded as the drift amplitude.

[0046] For fields exhibiting upward drift, the frequency of each value in each batch of data prior to the drift is counted. The average of these frequencies, rounded down, is used as the baseline for the normal frequency of that value. If a value has never appeared in any historical batches prior to the drift, the median of the average frequency of all values ​​in each batch prior to the drift is used as a substitute baseline to avoid incomparability due to zero historical occurrences of new values. In the batches following the drift start, if the actual frequency of a value exceeds twice the baseline for normal frequency, all corresponding records for that value in that batch are sorted from latest to earliest by entry timestamp. The records where the difference between the actual frequency and the baseline for normal frequency is subtracted are marked as excess duplicate records generated by the drift. For fields exhibiting downward drift, new isolated value records appearing in batches after the drift that break the existing correlation with values ​​before the drift (i.e., records that never appeared before the drift and appear only once in batches after the drift) are marked as abnormally discrete records generated by the drift.

[0047] For fields showing significant drift, the mean difference between segments is compared by traversing all possible split positions (positions 3 to 18, ensuring that each segment contains at least two data points). The position with the largest mean difference between segments is used as the starting batch for drift. In the starting batch and subsequent batches, records exceeding twice the normal occurrence frequency benchmark are sorted by entry timestamp from latest to earliest and marked as excess duplicate records, rather than treating all duplicate records in the drift batch as abnormal. The split point search method does not require a preset drift time; the jump position is determined by the data's own characteristics, adapting to real-world scenarios where the drift occurrence time is unknown. The fine-grained identification method using twice the normal occurrence frequency benchmark distinguishes the excess portion from the reasonable duplicate portion, preserving reasonable duplicate records from accidental deletion. For new values ​​that have never appeared before drift, the median is used instead of the benchmark, avoiding the incomparability problem caused by zero historical occurrence frequency. The sorting logic of timestamps from latest to earliest is consistent with the cleaning principle of retaining the earliest entry and deleting subsequent duplicates, ensuring that the cleaning operation matches the actual business.

[0048] Step 7: Based on the drift impact record details table, record-level anomaly scoring table, and field-to-field correlation matrix, drift is attributed to three business cause types and matched with corresponding cleaning strategies. At the same time, the horizontal anomaly scoring and vertical drift identification results are cross-validated in two dimensions. Records that are confirmed to be abnormal in both dimensions are given higher cleaning priority, and records with contradictory conclusions in the two dimensions are guided to manual review to avoid incorrect cleaning or missed cleaning caused by misjudgment in a single dimension. First, determine if it's a concentrated default value type drift. In the data from the starting batch of the drift and subsequent batches, count the number of records corresponding to the single value that appears most frequently for this field, divided by the total number of records in the drift batch. If this ratio exceeds 50%, it's considered a concentrated default value type drift. This type of drift is usually caused by upstream system configuration errors or the use of fixed default values ​​during batch import. If this condition is not met, determine if it's a system switch type drift. Count all different values ​​that appeared in all batches before the drift to form the pre-drift value set, and all different values ​​that appeared in all batches after the drift to form the post-drift value set. Calculate the number of identical values ​​in the two sets and divide it by the total number of different values ​​after merging the two sets to obtain the overlap rate. If the overlap rate is less than 10%, it's considered a system switch type drift. This type of drift usually corresponds to business system upgrades or changes in data sources, which is a reasonable business change. If neither of the above two conditions is met, it's considered a gradual contamination type drift. This is usually caused by persistent defects in the upstream data collection process, leading to a slow and synchronous increase in the repetition rate of multiple values.

[0049] Centralized default value drift modifies the field value of the records with concentrated duplicate values ​​in the drift batch to null and adds a "to be supplemented" mark, instead of directly deleting the entire record (because information in other fields of the record may still be valid, and direct deletion would cause unnecessary loss of valid business data). Simultaneously, a data supplementation notification is generated, with fields including the drift field name, the content of the abnormal value, the number of records involved, and the suggested supplementation deadline. System switchover drift does not perform any deletion or field modification operations on the relevant records. Instead, it establishes two independent baseline records for the field, one before the switchover and one after, using the drift start batch as the dividing point. This is designated as a baseline switching node for use during baseline update in step eight. For progressive contamination drift, the values ​​of each field in the drift batch are cleaned by retaining the earliest recorded record and deleting subsequent duplicate records that exceed the normal occurrence frequency baseline. The earliest recorded record is determined by its timestamp. Simultaneously, an upstream data source investigation notification is generated. The notification field includes the drift field name, drift start batch time, drift amplitude value (the absolute value of the actual repetition rate after drift minus the baseline repetition rate before drift), and a suspected problematic data source identifier extracted from the data source identifier field marking the source of this batch of records in the data table to be cleaned. If the data source identifier field does not exist in the data table, this field is marked as empty.

[0050] Two-dimensional cross-validation is handled according to the following rules: records whose comprehensive anomaly score ranks in the top 10% of all records and are also marked as drift-affected records in step six are marked as double-confirmed anomaly records and are directly cleaned according to the corresponding drift type's cleaning strategy, with the highest cleaning priority; records whose comprehensive anomaly score ranks between the top 10% and top 30% but are not marked as drift-affected records in step six, or records marked as drift-affected records in step six but whose comprehensive anomaly score ranks in the bottom 50%, are marked as single-dimensional anomaly records and enter the manual verification queue; records that are not identified as anomalies in both steps four and six remain in normal status and are not cleaned.

[0051] First, based on two conditions—a majority of records containing default values ​​exceeding 50% and an overlap rate of less than 10% between the value sets before and after the drift—drift was categorized into three types: default value type, system switch type, and gradual pollution type. Three differentiated cleaning strategies were then applied: setting field values ​​to null and adding a "to be supplemented" mark, establishing dual baselines before and after the switch without deleting, and retaining the earliest entry while deleting excessive duplicates. Next, a two-dimensional cross-validation was performed based on two conditions: whether the record ranked in the top 10% of the comprehensive anomaly score and whether it was marked as a record affected by drift. Records with double confirmation were directly cleaned, records with single-dimensional anomalies were entered into a manual verification queue, and records without confirmation in either dimension remained in normal status. By attributing three types of drift causes, the problem of using a uniform cleaning strategy for all drift types is avoided. In particular, system switching drift is identified as a reasonable business change rather than an anomaly, preventing the major risk of accidentally deleting legitimate data. Setting the field value of the centralized default value type to null instead of deleting the entire record preserves the business value of other valid fields in the record. The two-dimensional cross-validation mutually corroborates the conclusions of the two independent analysis dimensions of horizontal scoring and vertical drift. This not only improves the cleaning confidence of double-confirmed abnormal records, but also avoids incorrect cleaning or missed cleaning caused by misjudgment of a single dimension by guiding manual verification through single-dimensional anomalies, forming a complete risk classification and handling closed loop.

[0052] By establishing a centralized redundancy baseline range for each field based on its own historical fluctuations, the problem of misjudgment caused by using a uniform fixed threshold for fields with different business attributes in existing technologies is solved. The normal redundancy of enumerated fields and numbered fields naturally differs significantly, and a uniform standard will inevitably overlook some aspects, while historical statistical benchmarks naturally absorb this difference. Secondly, through structural analysis of the co-occurrence correlation strength between field values ​​and identification of redundancy anomaly cluster maps, the problem of synchronous expansion of multi-field correlations that cannot be detected by single-field redundancy rate methods is captured. By detecting cross-batch time trend drift, the detection perspective is expanded from horizontal comparison within a single batch to vertical evolution between multiple batches, providing an effective early warning for the gradual pollution problem of slowly accumulating redundancy rates, avoiding the long-term accumulation of data quality problems before they are discovered. Through the attribution classification of three types of drift causes, legitimate data changes due to system switching are not misjudged as abnormal, centralized default value type only sets the problematic field to null without deleting the entire record, and gradual pollution type retains the earliest entry and deletes the excess part according to the timestamp, and the handling strategy closely corresponds to the actual business. Finally, the dual-dimensional cross-validation mechanism of horizontal comprehensive anomaly scoring and vertical drift identification allows the conclusions of the two independent analysis dimensions to corroborate each other. The double-confirmed record cleaning has high confidence, while the single-dimensional anomaly records guide manual verification, avoiding incorrect cleaning or missed cleaning caused by misjudgment in a single dimension.

[0053] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0054] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A business data cleaning method based on data duplication relationships, characterized in that, include: Step 1: Select the 20 most recent consecutive batches of data that have been confirmed to be of normal quality from the historical data archive, calculate the clustered repetition for each field in each batch, and establish the normal repetition baseline for each field; Step 2: Used to calculate the co-occurrence correlation strength between any two fields from the data table to be cleaned and the original data of 20 historical normal batches, determine whether the correlation strength deviates from the historical normal level, and output the field pair correlation matrix; Step 3: Based on the field-to-field association matrix, the abnormal relationships between fields are mapped into a graph structure to identify the sets of mutually related abnormal fields, i.e., duplicate abnormal clusters, and to locate the core abnormal field in each abnormal cluster that is at the center of abnormal propagation. Step 4: Using the normalized value of the abnormal correlation degree plus 1 as the weight of each field, calculate the field duplication abnormality score and correlation abnormality score for each record, and then sum them up to obtain the comprehensive abnormality score. Step 5: Calculate the batch repetition rate for each field in 20 historical normal batches to form a time series. Obtain the comprehensive drift intensity score by multiplying the linear trend slope by the trend consistency score, and identify significant drift fields and drift warning fields. Step 6: Used to locate the drift start batch by comparing the mean difference between segments in the drift significant field through traversal segmentation positions, and to identify excess duplicate records within the drift batch by setting the boundary at twice the normal occurrence frequency benchmark for each value; Step 7: The drift is attributed to three types: default value type, system switching type, and progressive contamination type, and the corresponding cleaning strategy is matched. At the same time, a two-dimensional cross-validation is performed on the comprehensive anomaly score and the drift identification result. The double-confirmed anomaly records are directly cleaned, and the single-dimensional anomaly records are manually checked.

2. The business data cleaning method based on data duplication relationships according to claim 1, characterized in that, The specific method for establishing the normal repeatability baseline for each field is as follows: To determine the central tendency of a field in a given batch, we first calculate the central tendency of a field in each batch by summing the total number of records corresponding to the top 20% of values ​​and dividing this sum by the total number of records for that field in the current batch. Then, we calculate the central tendency of a field in each of the 20 batches, resulting in a numerical sequence of length 20. We then calculate the arithmetic mean and standard deviation of this sequence. The mean plus twice the standard deviation is used as the upper limit of the normal range, and the mean minus twice the standard deviation is used as the lower limit. If the lower limit is less than 0, it is set to 0. This forms the baseline range for the normal repetition of the field. Finally, we record the type of each field, categorized as enumeration, numbered, free text, and numeric.

3. The business data cleaning method based on data duplication relationships according to claim 2, characterized in that, The specific method for calculating the co-occurrence correlation strength between any two fields is as follows: For any two fields, A and B, in the data table to be cleaned, enumerate all distinct values ​​in field A. For each value, find all records with that value in the data to be cleaned, and count the number of distinct values ​​of field B appearing in these records. This count is called the one-to-many count for that value of field A. Sort all the one-to-many counts of field A in ascending order. If the number of values ​​is odd, take the value in the middle position; if it is even, take the average of the two middle values. This gives the current median one-to-many count of field A for field B. Calculate the median one-to-many count of field A for field B in the same way for 20 historical normal batches to obtain 20 historical data. The historical value is calculated by averaging 20 historical values ​​as the historical baseline median one-to-many count for this field pair. The deviation ratio is obtained by dividing the current median one-to-many count by the historical baseline median one-to-many count. If the historical baseline median one-to-many count is 0, it means that this field pair has never had a co-occurrence association in the historical batch, and it is considered as 0.5 in the calculation to avoid the case of dividing by 0. 0.5 is the smallest non-zero substitute value less than 1 to reflect that any co-occurrence at present constitutes an inflationary tendency. A deviation ratio greater than 3 marks the field pair as an association inflation anomaly, a deviation ratio less than 0.33 marks it as an association contraction anomaly, and a deviation ratio between 0.33 and 3 is marked as normal.

4. The business data cleaning method based on data duplication relationships according to claim 3, characterized in that, The specific method for identifying interrelated sets of abnormal fields, i.e., duplicate abnormal clusters, is as follows: An undirected graph is constructed with each field in the data table as a node. For all field pairs marked as having abnormal association expansion or contraction in the field pair association matrix, an edge is connected between the corresponding two nodes. The edge attributes record the anomaly type and deviation ratio. Nodes that do not form any abnormal association with any field are isolated nodes and do not participate in subsequent anomaly cluster analysis. Starting from any unvisited node, all nodes directly connected to that node are checked sequentially, and then nodes directly connected to these nodes but not yet visited are checked. This process is continued until no new nodes are available for access. All nodes involved in this traversal are grouped into the same connected subgraph as a cluster. Repeat the above process for the remaining unvisited nodes until all nodes are visited. Each connected subgraph corresponds to a repeating anomaly cluster. For each node in the repeating anomaly cluster, count the number of abnormal edges connected to it and record it as the abnormal correlation degree of that field. The node with the highest abnormal correlation degree is marked as the core anomaly field. If multiple nodes have the same highest abnormal correlation degree, further calculate the degree of deviation of the concentration of repetition for each node. The degree of deviation of the concentration of repetition is calculated by dividing the absolute value of the difference between the current concentration of repetition of the field and the baseline mean of the field in step one by the baseline standard deviation. Take the node with the largest deviation degree as the core anomaly field.

5. The business data cleaning method based on data duplication relationships according to claim 4, characterized in that, The specific method for calculating the comprehensive anomaly score for each record is as follows: For a field belonging to a certain duplicate anomaly cluster, increment the anomaly correlation degree of that field by 1. Sum the anomaly correlation degrees of all fields within the cluster by 1. Then, increment the anomaly correlation degree of each field by 1 and divide by the sum to obtain the normalized weight of that field. The sum of the weights of all fields within the cluster equals 1. Assign a uniform minimum weight (1 divided by the total number of fields in the data table) to isolated fields that do not belong to any duplicate anomaly cluster. For each field value in each record, count the actual occurrences of that value in the data table to be cleaned. Multiply the upper limit of the normal range of that field in the field baseline table by the total number of records in the current batch to obtain the expected maximum occurrence count. If the actual occurrence count exceeds the expected maximum occurrence count, the field duplicate anomaly score is equal to the actual occurrence count divided by the expected maximum occurrence count and then minus 1; otherwise, the field duplicate anomaly score is 0. For records belonging to a certain duplicate anomaly cluster, the current value of the core anomaly field in that record is taken. In the data to be cleaned, the number of different values ​​of other fields within the cluster that appear simultaneously with the value of the core field is counted. The average of the number of different values ​​of all other fields within the cluster is taken as the actual association breadth. The expected value of the historical baseline is the average of the one-to-many counts of the core anomaly field to other fields within the cluster in the historical baseline of the field-to-association matrix. If the expected value of the historical baseline is 0, it is considered as 0.5 and included in the calculation. The association anomaly score is equal to the actual association breadth divided by the expected value of the historical baseline and then subtracted by 1. If the actual association breadth does not exceed the expected value of the historical baseline, the association anomaly score is 0. The association anomaly score of records that do not belong to any anomaly cluster is directly set to 0. For each record, the duplicate anomaly score of each field is multiplied by the corresponding normalized weight and summed. Then, the association anomaly score is multiplied by the normalized weight of the core anomaly field to obtain the comprehensive anomaly score.

6. The business data cleaning method based on data duplication relationships according to claim 5, characterized in that, The specific method for analyzing the continuous trend of repeatability in 20 historical normal batches is as follows: To calculate the batch repetition rate of a field in a batch of data, count the total number of records where a certain field appears more than 1. Divide this count by the total number of records for that field in the batch. Calculate this rate for each of the 20 historical batches to form a repetition rate sequence of length 20. The average batch number is fixed at 1 to 20 (the sum of these 20 integers divided by 20 equals 10.5). Calculate the average repetition rate of the 20 batches. For each batch, subtract 10.5 from the batch number, multiply by the batch repetition rate minus the average repetition rate, and sum all the products to obtain the cumulative sum in the numerator. Then, subtract 10.5 from the batch number, square the result, and sum all 20 squares to obtain the cumulative sum in the denominator. The slope of the linear trend is set to 665, and the cumulative sum of the numerator is divided by 665. The overall trend direction is determined by the sign of the slope. Each element from the 2nd to the 20th in the sequence is compared with the previous element. If it is greater than the previous element, it is recorded as an increase; if it is less than the previous element, it is recorded as a decrease; if it is equal, it is recorded as flat. Flat values ​​are not included in the statistics of effective adjacent pairs. The number of adjacent pairs with the same trend direction as the overall trend is divided by the total number of effective adjacent pairs participating in the comparison to obtain the trend consistency score. The absolute value of the slope is multiplied by the trend consistency score to obtain the comprehensive drift intensity score. Fields with a score greater than 0.005 are marked as drift significant fields, and fields with a score greater than 0.001 but less than 0.005 are marked as drift warning fields.

7. The business data cleaning method based on data duplication relationships according to claim 6, characterized in that, The specific method for locating the starting batch of drift and identifying excessive duplicate records is as follows: Starting from the third position, attempt to divide the 20 batches of repetition rate sequence into a first half and a second half at each position. For each division method, calculate the average of all repetition rate values ​​in the first half and the average of all repetition rate values ​​in the second half, and record the absolute value of the difference between the two, which is the inter-segment mean difference. After traversing all possible division positions, ensure that each segment contains at least 2 data points. Find the division position that maximizes the inter-segment mean difference, and mark the batch number corresponding to this position as the drift start batch. The average repetition rate of all batches before the drift start batch is recorded as the baseline repetition rate before drift, and the average repetition rate of all batches from the drift start batch onwards is recorded as the actual repetition rate after drift. The absolute value of the difference between the actual repetition rate after drift and the baseline repetition rate before drift is recorded as the drift amplitude. The quasi-duplicate rate is the average of the duplicate rates of the drift-initiated batch and all subsequent batches, which is recorded as the actual duplicate rate after the drift. For fields that experience upward drift, the number of times each value of the field appears in each batch of data before the drift is counted. The average of the occurrences in each batch before the drift is taken and rounded down to obtain the normal occurrence benchmark for that value. If a value has never appeared in the historical batches before the drift, the median of the average occurrences of all values ​​in each batch before the drift is used as the alternative benchmark. In the data of the drift-initiated batch and subsequent batches, if the actual occurrence of that value exceeds twice the normal occurrence benchmark, all corresponding records of that value in that batch are arranged from latest to earliest according to the entry timestamp. The records with the difference between the actual occurrence and the normal occurrence benchmark are marked as excess duplicate records generated by the drift.

8. The business data cleaning method based on data duplication relationships according to claim 7, characterized in that, The specific way to attribute drift to the three types of business causes is as follows: First, determine if it is a concentrated default value type drift. In the data from the starting batch of the drift and subsequent batches, count the number of records corresponding to the single value that appears most frequently for this field, and divide the number of records by the total number of records in the drift batch. If this ratio exceeds 50%, it is determined to be a concentrated default value type drift. If it does not meet the condition, determine if it is a system switch type drift. Count all different values ​​of this field that appeared in all batches before the drift to form the value set before the drift, and all different values ​​that appeared in all batches after the drift to form the value set after the drift. Calculate the number of identical values ​​in the two sets and divide it by the total number of different values ​​after merging the two sets to obtain the overlap rate. If the overlap rate is less than 10%, it is determined to be a system switch type drift. If neither of the above two conditions is met, it is determined to be a gradual contamination type drift.

9. A business data cleaning method based on data duplication relationships according to claim 8, characterized in that, The specific method for matching differentiated cleaning strategies to the three drift types is as follows: Centralized default value drift modifies the field value of the excess duplicate records corresponding to the concentrated values ​​in the drift batch to null and adds a pending record mark instead of directly deleting the entire record. At the same time, a data supplementation notification is generated. The notification fields include the drift field name, the abnormal value content, the number of records involved, and the suggested supplementation deadline. For system switching drift, no deletion or field modification operations are performed on the relevant records. Two independent baseline records are established for the field, one before the switchover baseline and one after the switchover baseline, with the drift start batch as the dividing point. The drift start batch is marked as the baseline switching node. The progressive contamination drift process cleans each field value in the drift batch by retaining the earliest recorded record and deleting subsequent duplicate records that exceed the normal occurrence frequency baseline. The earliest recorded record is determined by its entry timestamp. Simultaneously, an upstream data source investigation notification is generated. The notification field includes the drift field name, the drift start batch time, the drift amplitude value (the absolute value of the actual repetition rate after drift minus the baseline repetition rate before drift), and the suspected problematic data source identifier extracted from the data source identifier field that marks the source of the records in this batch from the data table to be cleaned. If the data table does not have a data source identifier field, this item is marked as empty.

10. A business data cleaning method based on data duplication relationships according to claim 9, characterized in that, The specific method for performing two-dimensional cross-validation is as follows: Records whose overall anomaly score ranks in the top 10% of all records and are also marked as drift-affected records in step six are marked as double-confirmed anomaly records and will be directly cleaned according to the corresponding drift type's cleaning strategy, with the highest cleaning priority. Records whose overall anomaly score ranks between the top 10% and top 30% but are not marked as drift-affected records in step six, or records marked as drift-affected records in step six but whose overall anomaly score ranks in the bottom 50% of all records, are marked as single-dimensional anomaly records and will enter the manual verification queue. Records that are not identified as anomalies in steps four and six will remain in normal status and will not be cleaned.

Images