An AI-based data asset quality evaluation and automatic repair method and system

Abstract

The application discloses an AI-based data asset quality evaluation and automatic repair method and system, which comprises the following steps: receiving a target data asset to be processed and inputting the target data asset into a multi-dimensional joint quality scoring engine to generate a comprehensive quality evaluation report containing an integrity defect atlas, a distribution anomaly heat map and a semantic conflict link; based on the comprehensive quality evaluation report, a strategy generator combining rules and learning is used to analyze repair requirements and constraint conditions contained in the defect atlas, the anomaly heat map and the conflict link, and dynamically generate a repair operation instruction set and a priority sequence for different types of defects; the target data asset, the repair operation instruction set and the priority sequence are input into a multi-expert collaborative repair execution module, defects in the target data asset are repaired according to the priority sequence, and a repaired data asset is output. By using the embodiment of the application, comprehensive and accurate data quality evaluation can be realized, and the automation level and accuracy of data repair are improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular to an AI-based method and system for data asset quality assessment and automatic repair. Background Technology

[0002] With the rapid development of big data technology, data assets have become a crucial component of enterprises' core competitiveness. However, data quality issues (such as missing values, abnormal distributions, and semantic inconsistencies) severely impact data credibility and application value. Traditional data quality assessment methods typically rely on manual rules or single-dimensional statistical analysis, making it difficult to comprehensively cover complex scenarios such as data integrity issues, the rationality of numerical distributions, and cross-field semantic relationships. Furthermore, existing remediation technologies often employ fixed rules or simple imputation strategies, lacking the ability to collaboratively analyze and dynamically remediate multi-dimensional quality defects, resulting in limited remediation effectiveness and insufficient adaptability. In recent years, artificial intelligence technology has demonstrated enormous potential in the field of data governance, but existing solutions still suffer from problems such as single assessment dimensions, rigid remediation strategies, and low levels of automation, failing to meet enterprises' needs for efficient and accurate data governance. Summary of the Invention

[0003] The purpose of this invention is to provide an AI-based method and system for data asset quality assessment and automatic repair, in order to overcome the shortcomings of existing technologies, achieve comprehensive and accurate data quality assessment, and improve the automation level and accuracy of data repair.

[0004] One embodiment of this application provides an AI-based method for data asset quality assessment and automatic repair, the method comprising:

[0005] The system receives the target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine. This engine performs data integrity verification, numerical distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes an integrity defect map, a distribution anomaly heatmap, and semantic conflict links.

[0006] Based on the comprehensive quality assessment report, a strategy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects.

[0007] The target data asset, the repair operation instruction set, and the priority sequence are input into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0008] Optionally, the process of receiving the target data assets to be processed and inputting them into a multi-dimensional joint quality scoring engine, which performs data integrity verification, numerical distribution rationality analysis, and cross-field semantic dependency verification in parallel, generates a comprehensive quality assessment report including an integrity defect map, a distribution anomaly heatmap, and semantic conflict links, including:

[0009] The system receives the target data assets to be processed, parses their data structure, uses an adaptive sliding window filtering algorithm to remove high-frequency noise, and extracts the data generation time, source identifier, and field dependency relationship to build a metadata index table. The system then outputs the preprocessed data assets and the metadata index table.

[0010] Based on the metadata index table, integrity verification is performed in parallel by multiple threads. The binary matrix marking method is used to record the location of missing values ​​in each field. The integrity score is calculated by combining the importance weight of the fields. At the same time, the hash check is used to detect duplicate records and mark redundant locations, generating an integrity defect matrix containing the distribution of missing values, the coordinates of duplicate items, and the integrity score.

[0011] Using the integrity defect matrix as a filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of the effective samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heatmap to generate a heatmap of numerical distribution anomalies.

[0012] A cross-field semantic network is constructed using preprocessed data assets. The semantic similarity between fields is calculated using a knowledge graph-enhanced BERT model. When the similarity is lower than the similarity threshold, it is marked as a semantic conflict. The propagation path of the conflict is traced using a time-series backtracking method, generating a semantic conflict link that includes conflict nodes, association strength, and propagation path.

[0013] The integrity defect matrix is ​​converted into a visual defect map. Combined with a numerical distribution anomaly heatmap and semantic conflict links, quality scores for each dimension and defect impact range assessment are added to generate a comprehensive quality assessment report.

[0014] Optionally, based on the comprehensive quality assessment report, a rule-based and learning-integrated strategy generator is used to analyze the repair requirements and constraints contained in the defect map, anomaly heatmap, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects, including:

[0015] The comprehensive quality assessment report is analyzed to extract the coordinates of missing fields and duplicate item identifiers from the integrity defect map, the high abnormal value ranges from the distribution anomaly heatmap, and the conflict field pairs from the semantic conflict link. These are then organized into a defect-location-severity mapping table in the defect type-location-severity triplet format.

[0016] Based on the defect-location-severity mapping table and combined with the business scenario of the target data assets, the repair constraints, including the value range of numeric fields, the format specifications of character fields, and the business logic dependencies across fields, are extracted to generate a set of repair constraints.

[0017] The defect-location-severity mapping table and the set of repair constraints are input into the policy generator that integrates rules and learning. The rule module generates basic repair instructions, the learning module generates optimized repair instructions, and the basic repair instructions and optimized repair instructions are merged to obtain the initial set of repair operation instructions.

[0018] Conflict detection is performed on the initial repair operation instruction set. A greedy algorithm is used to retain instructions with a priority higher than the preset level. At the same time, the repair cost of each instruction is calculated, and invalid instructions with a cost exceeding the cost threshold are filtered out to generate a candidate repair operation instruction set.

[0019] Based on the severity of the defect, the urgency of repair, and the operational relevance of each instruction in the candidate repair operation instruction set, the priority weight is calculated using the analytic hierarchy process (AHP). The priority sequence is generated by sorting the instructions by weight and then binding the sequence with the candidate instruction set to output the repair operation instruction set and the priority sequence.

[0020] Optionally, the target data asset, the repair operation instruction set, and the priority sequence are input into a multi-expert collaborative repair execution module. This module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset, including:

[0021] The repair operation instruction set and priority sequence are parsed, and the tasks are assigned to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type. The task index and priority sort of each model are recorded, and a model-task allocation table is generated.

[0022] Using a numerical interpolation expert model, the location of the missing value to be processed is obtained according to the model-task allocation table. The spatiotemporal attention interpolation algorithm is used to calculate the interpolated value. At the same time, cross-validation is used to ensure that the interpolated value conforms to the distribution law, and the interpolated data numerical fragment is output.

[0023] Using a semantic reconstruction expert model, the system receives field data corresponding to semantic conflict links, calls a knowledge graph-enhanced pre-trained language model to generate candidate field values ​​that conform to semantic logic, filters candidate values ​​that do not conform to business terms through a domain dictionary, selects the one with the highest confidence as the reconstruction result, and outputs semantically corrected data fragments.

[0024] Using a rule correction expert model, for the numerical range marked by the distribution anomaly heatmap, the abnormal values ​​are adjusted according to the threshold range in the business rule base. The gradient descent method is used to fine-tune the values ​​to minimize the distribution deviation, and the rule-corrected data fragment is output.

[0025] By integrating the imputed data numerical fragments, semantically corrected data fragments, and rule-corrected data fragments according to priority sequence, a complete dataset is obtained, which serves as the repaired data asset.

[0026] Another embodiment of this application provides an AI-based data asset quality assessment and automatic repair system, the system comprising:

[0027] The evaluation module receives the target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine. This engine performs data integrity verification, numerical distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality evaluation report that includes an integrity defect map, a distribution anomaly heatmap, and semantic conflict links.

[0028] The parsing module is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict link based on the comprehensive quality assessment report and using a strategy generator that integrates rules and learning, and dynamically generate repair operation instruction sets and priority sequences for different types of defects.

[0029] The repair module is used to input the target data asset, the repair operation instruction set, and the priority sequence into the multi-expert collaborative repair execution module. This module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0030] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.

[0031] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.

[0032] Compared with existing technologies, this invention provides an AI-based data asset quality assessment and automatic repair method. It receives the target data asset to be processed and inputs it into a multi-dimensional joint quality scoring engine to generate a comprehensive quality assessment report including an integrity defect map, a distribution anomaly heatmap, and semantic conflict links. Based on the comprehensive quality assessment report, a rule-based and learning-integrated strategy generator analyzes the repair needs and constraints contained in the defect map, anomaly heatmap, and conflict links, dynamically generating repair operation instruction sets and priority sequences for different types of defects. The target data asset, repair operation instruction sets, and priority sequences are input into a multi-expert collaborative repair execution module, which repairs the defects in the target data asset according to the priority sequence, outputting the repaired data asset. This enables comprehensive and accurate data quality assessment, improving the automation level and accuracy of data repair. Attached Figure Description

[0033] Figure 1 A hardware structure block diagram of a computer terminal for an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention;

[0034] Figure 2 A flowchart illustrating an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of an AI-based data asset quality assessment and automatic repair system provided in an embodiment of the present invention. Detailed Implementation

[0036] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0037] This invention first provides an AI-based method for data asset quality assessment and automatic repair, which can be applied to electronic devices, such as computer terminals, specifically ordinary computers.

[0038] The following detailed explanation uses a computer terminal as an example. Figure 1 This is a hardware structure block diagram of a computer terminal for an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0039] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any AI-based data asset quality assessment and automatic repair method.

[0040] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0041] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any AI-based data asset quality assessment and automatic repair method.

[0042] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0043] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0044] See Figure 2 The present invention provides an AI-based method for data asset quality assessment and automatic repair, which may include the following steps:

[0045] S201: Receive the target data assets to be processed and input them into the multi-dimensional joint quality scoring engine. The engine performs data integrity verification, numerical distribution rationality analysis and cross-field semantic dependency verification in parallel, and generates a comprehensive quality assessment report including integrity defect map, distribution anomaly heat map and semantic conflict link.

[0046] Specifically, it can receive target data assets to be processed, parse their data structure, use an adaptive sliding window filtering algorithm to remove high-frequency noise, and extract the data generation time, source identifier and field dependency relationship to build a metadata index table, and output the preprocessed data assets and metadata index table.

[0047] The target data assets to be processed can be structured or semi-structured data sets such as transaction data from e-commerce platforms and user credit data from financial systems. Taking e-commerce transaction data as an example, it contains fields such as order ID, user ID, product ID, order time, payment amount, and shipping address. The order ID is a string, the payment amount is a numeric value, and the order time is a timestamp. When receiving data, the system first reads the raw data file (such as CSV or JSON format) through a data interface and parses the file header, recognizing format identifiers such as field separators (such as commas and colons) and quotation marks (such as double quotes) to ensure correct matching of fields and values.

[0048] During the data structure parsing process, the system automatically identifies the data type of each field: fields consisting solely of numbers and containing decimal points (such as payment amount "99.99") are classified as floating-point; fields composed of numbers and letters with a fixed length (such as order ID "ORD20250725001") are classified as string; and fields conforming to the format "YYYY-MM-DD HH:MM:SS" (such as order time "2025-07-25 10:30:00") are classified as datetime. Simultaneously, the system records the length range of each field (such as user ID being 10 digits) and whether null values ​​are allowed, providing foundational information for subsequent quality assessment.

[0049] The adaptive sliding window filtering algorithm is used to remove high-frequency noise from data. This noise may originate from sensor errors during data acquisition or interference during transmission (such as occasional outliers containing letters like "100.0a" in payment amounts). The algorithm's window size is not fixed but dynamically adjusted based on the degree of data fluctuation: when data values ​​are continuously stable (e.g., payment amounts fluctuate slightly between 90-110 yuan), the window size is set to 5 (containing the current data and two records before and after it); when data changes abruptly (e.g., a payment amount suddenly jumps to 10,000 yuan), the window automatically expands to 10, smoothing out fluctuations with more historical data. During filtering, the algorithm calculates the median of the data within the window as the filtered result for the current value. For example, if the data within the window is [98, 102, 99, 101, 100], and the median is 100, then 100 replaces the original value 99 at the center of the window, thus weakening the impact of isolated outliers.

[0050] When extracting metadata, the generation time refers to the timestamp of the first time the data was recorded, such as "2025-07-25 10:30:00" for order data, accurate to the second to ensure the accuracy of time series analysis; the source identifier is used to distinguish the data collection channel, such as "APP", "web page", "third-party interface", to facilitate tracing the source of data quality problems; field dependency refers to the logical relationship between different fields, such as "payment amount" depending on "product unit price" and "purchase quantity" (payment amount = product unit price × purchase quantity), and the "province" field in "delivery address" depending on the "city" field (e.g., if the city is "Shanghai", the province must be "Shanghai").

[0051] The metadata index table is a structured collection of this information. Its core fields include: field name (e.g., "Payment Amount"), data type (e.g., "Floating-point"), generation time range (e.g., "2025-07-01 to 2025-07-25"), source identifier list (e.g., "APP version, Web version"), dependent fields (e.g., "Product unit price, Purchase quantity"), and data length limit (e.g., "Payment amount retains a maximum of 2 decimal places"). Taking the "Payment Amount" field as an example, its record in the index table is: field name "Payment Amount", data type "Floating-point", generation time range "2025-07-01 00:00:00 to 2025-07-25 23:59:59", source identifier "APP version", dependent fields "Product unit price, Purchase quantity", and length limit "≤10 integer digits + 2 decimal digits".

[0052] The preprocessed data assets are datasets that have undergone noise removal. For example, the original payment amount field might contain values ​​such as "100.5", "99.8a", and "105.0". "99.8a" is high-frequency noise (a non-numeric character introduced due to input error). The adaptive sliding window filtering algorithm identifies the deviation of this value from adjacent normal data (100.5, 105.0) and corrects it to "100.5" by replacing it with the median within the window (when the window size is 3, the median is 100.5). In the final output preprocessed data, this field contains only valid values. A metadata index table is output synchronously with the preprocessed data assets, providing basic metadata support for subsequent quality assessment.

[0053] Based on the metadata index table, integrity verification is performed in parallel by multiple threads. The binary matrix marking method is used to record the location of missing values ​​in each field. The integrity score is calculated by combining the importance weight of the fields. At the same time, the hash check is used to detect duplicate records and mark redundant locations, generating an integrity defect matrix containing the distribution of missing values, the coordinates of duplicate items, and the integrity score.

[0054] The metadata index table provides basic information on field structure and dependencies for integrity verification, and multi-threaded parallel execution can improve verification efficiency. Taking e-commerce transaction data containing 10 fields and 1000 records as an example, the system will start threads matching the number of fields (e.g., 10 threads). Each thread is responsible for the integrity verification of one field, and the threads exchange progress information through shared memory to avoid redundant calculations.

[0055] Binary matrix notation is the core method for recording the location of missing values. The matrix has the same number of rows as the number of data records and the same number of columns as the number of fields. Matrix elements are either 0 or 1, where 1 indicates a missing value and 0 indicates a normal value. For example, if the "Payment Amount" field is missing in record 3 and the "Shipping Address" field is missing in record 5, then the elements in the binary matrix corresponding to the "Payment Amount" column in row 3 will be 1, the elements corresponding to the "Shipping Address" column in row 5 will be 1, and the remaining positions will be 0. This notation method visually displays the distribution of missing values ​​in the dataset; a higher percentage of 1s in a column (field) indicates a more severe shortage of values ​​in that field.

[0056] Field importance weights are set according to business scenarios, reflecting the degree of impact of fields on data availability. In e-commerce transaction data, "Order ID" and "Payment Amount" are core fields (weight 0.8), and their absence will invalidate the order; the "Remarks" field is a non-core field (weight 0.2), and its absence has a smaller impact on business. The total weight value is 1, determined by a combination of business expert scoring and historical data utilization statistics. The formula for calculating the completeness score is: Completeness Score = Σ(1 - Field Missing Rate) × Field Weight, where the field missing rate = number of missing field values ​​ / total number of records. For example, if the "Order ID" missing rate is 0.01 (10 missing records), the "Payment Amount" missing rate is 0.02 (20 missing records), and the other fields are not missing, then the score = (1-0.01)×0.8 + (1-0.02)×0.2 + (1-0)×(1-0.8-0.2) = 0.792 + 0.196 = 0.988 (i.e., 98.8 points).

[0057] Hash verification is used to detect duplicate records. Each record is converted into a unique hash value (such as an MD5 value). The calculation method is to concatenate all field values ​​in the record into a string in sequence, and then generate a 32-bit character using a hash algorithm. For example, if two records have exactly the same field values ​​(order ID "ORD001", user ID "USR001", etc.), their hash value will both be "e10adc3949ba59abbe56e057f20f883e". The system will mark these two records as duplicates, and the redundant position will be the index of the later record (e.g., if the 12th record is a duplicate of the 5th record, then the 12th record will be marked as redundant).

[0058] The integrity defect matrix is ​​a structured result that integrates the above information. It contains three parts: the distribution of missing values ​​(a visual transformation of the binary matrix, such as expressing the missing rate of each column as a percentage), the coordinates of duplicate items (such as "(5,12)" indicating that the 5th and 12th records are duplicates), and the integrity score (such as 98.8 points). Taking e-commerce data as an example, the missing rate of the "Payment Amount" column in the matrix is ​​2%, the missing rate of the "Shipping Address" column is 5%, the coordinates of duplicate items are "(8,25)" and "(30,45)", and the integrity score is 96.5 points, which intuitively reflects the quality defects of the data in the integrity dimension.

[0059] Using the integrity defect matrix as a filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of the effective samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heatmap to generate a heatmap of numerical distribution anomalies.

[0060] In the integrity defect matrix, valid samples refer to records that are not marked as missing or duplicates. For example, from 1000 original records, after filtering and removing 50 records with missing values ​​and 20 duplicate records, 930 records remain as valid samples. Numerical fields include "payment amount," "purchase quantity," and "unit price," etc. The reasonableness of the distribution of these fields directly affects the data analysis results (e.g., abnormally high payment amounts may lead to distorted sales statistics).

[0061] Kernel density estimation is used to describe the actual distribution characteristics of numerical fields. Taking the "payment amount" field as an example, the algorithm assigns a Gaussian kernel function to each valid sample value (the kernel function bandwidth is automatically adjusted according to the sample size; the larger the sample size, the smaller the bandwidth, such as setting the bandwidth to 5 for 930 samples). By superimposing all kernel functions, a smooth probability density curve is obtained. For example, if the payment amount is concentrated in the range of 50-200 yuan, the kernel density curve will peak at 100 yuan, indicating that this amount is the most common transaction value.

[0062] Gaussian mixture model fitting is used to construct a theoretical distribution. The model assumes that the data follows a superposition of multiple Gaussian distributions, and estimates the mean, variance, and weight of each Gaussian component using the expectation-maximization algorithm. For example, the theoretical distribution of "payment amount" consists of three Gaussian components: component 1 (mean 80, variance 20, weight 0.3) corresponds to low-priced goods, component 2 (mean 150, variance 30, weight 0.5) corresponds to mid-range goods, and component 3 (mean 300, variance 50, weight 0.2) corresponds to high-end goods. The fitted theoretical distribution should match the actual business scenario, such as mid-range goods having the highest transaction volume.

[0063] The Kullback-Leibler (KL) divergence is used to quantify the difference between the actual and theoretical distributions. The formula is KL(P||Q) = ΣP(x) log(P(x) / Q(x)), where P is the probability density of the actual distribution and Q is the probability density of the theoretical distribution. A larger divergence value indicates a more significant distribution difference (a more severe anomaly). For example, the actual distribution of "payment amount" shows a large number of samples above 500 yuan, while the probability of this interval is extremely low in the theoretical distribution. The calculated KL divergence is 0.8 (divergence values ​​are typically between 0 and 2), indicating the presence of a distribution anomaly.

[0064] The heatmap for numerical distribution anomalies visually displays the degree of anomaly through color depth. The horizontal axis represents numerical fields (such as "payment amount" and "purchase quantity"), and the vertical axis represents numerical ranges (such as payment amount 0-100 yuan, 100-200 yuan, etc.). The color gradually changes from green (dispersion 0-0.3, normal) to yellow (0.3-0.6, slight anomaly) and then to red (above 0.6, severe anomaly). For example, the dispersion value of "payment amount" in the 500-1000 yuan range is 0.9, which corresponds to a dark red position on the heatmap, clearly marking the fields and numerical ranges with anomalies.

[0065] A cross-field semantic network is constructed using preprocessed data assets. The semantic similarity between fields is calculated using a knowledge graph-enhanced BERT model. When the similarity is lower than the similarity threshold, it is marked as a semantic conflict. The propagation path of the conflict is traced using a time-series backtracking method, generating a semantic conflict link that includes conflict nodes, association strength, and propagation path.

[0066] A cross-field semantic network is a topological structure that describes the semantic relationships between fields. Nodes are data fields, and edges represent the semantic relationships between fields (such as "containment" and "dependency"). Taking e-commerce data as an example, the "shipping address" node is connected to the "province" and "city" nodes through the "containment" relationship, and the "order time" and "payment time" nodes are connected through the "chronological order" relationship. The weight of the edge represents the strength of the association (between 0 and 1, such as the association strength of "order time - payment time" being 0.9, because payment must occur after ordering).

[0067] A knowledge graph-enhanced BERT model is used to calculate semantic similarity. This model integrates a domain-specific knowledge graph (such as an "address-administrative division" graph or a "transaction-process" graph in the e-commerce domain) into a pre-trained BERT model, associating field values ​​with entities in the knowledge graph through entity links. For example, when calculating the semantic similarity between the "city" field value "Beijing" and the "province" field value "Hebei," the model considers the fact in the knowledge graph that "Beijing is a municipality directly under the central government, not Hebei Province," outputting a similarity of 0.2 (normally it should be ≥0.8), thus determining a semantic conflict.

[0068] The similarity threshold is set according to business logic, and the threshold varies for different field pairs. For example, the threshold for the "city-province" field pair is 0.8 (strict matching of administrative divisions is required), while the threshold for the "product name-product category" field pair is 0.6 (allowing some fuzzy matching). When the calculated similarity is lower than the threshold, it is marked as a semantic conflict. For example, if "city" is "Shanghai" and "province" is "Jiangsu", the similarity is 0.1, which is lower than 0.8, and is marked as a conflict.

[0069] The time-series backtracking method is used to trace the conflict propagation path. Taking the conflict of "payment time earlier than order time" as an example, the system starts from the conflict record (100th record) and traces back the time sequence nodes of data processing: during the original data entry, the "payment time" was mistakenly entered as "2025-07-25 09:00", while the order time was "2025-07-25 10:00" (first step error); the time sequence anomaly was not detected during data verification (second step omission); the conflict was retained during final storage (third step propagation). The propagation path record is "entry error → verification omission → storage retention", with an association strength of 0.9 (strong conflict, affecting transaction validity).

[0070] The semantic conflict link integrates the above information, including conflicting field pairs (such as "payment time - order time"), association strength (0.9), propagation path (entry → validation → storage), and conflict record index (100th record). For example, a link is described as: "Conflicting field pair: city (Shanghai) - province (Jiangsu), similarity 0.1 < 0.8, association strength 0.8, propagation path: user input → format validation not intercepted → entry into database, involved records: 58th, 72nd records."

[0071] The integrity defect matrix is ​​converted into a visual defect map. Combined with a numerical distribution anomaly heatmap and semantic conflict links, quality scores for each dimension and defect impact range assessment are added to generate a comprehensive quality assessment report.

[0072] The integrity defect matrix visualization defect map displays missing and duplicate defects through a node-edge structure. In the map, each node represents a field, and the node size is positively correlated with the missing rate of that field (e.g., the "Shipping Address" node has a 5% missing rate, which is larger than the "Payment Amount" node with a 2% missing rate). The color of the node indicates the impact of duplicate records (e.g., a red node indicates that the record containing that field has a high duplication rate). The dashed lines of the edges indicate the association between fields caused by duplicate records (e.g., the edge between "Order ID" and "User ID" is a dashed line because two duplicate records involve both fields).

[0073] The comprehensive quality assessment report integrates defect maps, distribution anomaly heatmaps, and semantic conflict links, and adds scores and impact assessments for each dimension. Quality scores for each dimension include: completeness score (e.g., 96.5 points), distribution rationality score (e.g., 88 points, converted based on the KL divergence mean), and semantic consistency score (e.g., 92 points, converted based on the proportion of conflicting fields). The total score is calculated using a weighted average (weights: completeness 0.4, distribution 0.3, semantics 0.3), e.g., (96.5×0.4+88×0.3+92×0.3)=92.2 points.

[0074] The defect impact assessment describes the impact of the defect on the business. For example, an abnormal distribution of the "Payment Amount" field (more high values) may lead to an overestimation of sales revenue, and semantic conflicts in the "City-Province" field may cause logistics and delivery errors. The report will indicate the impact level (severe / moderate / minor) and the recommended handling priority (e.g., semantic conflicts should be handled first).

[0075] The final comprehensive quality assessment report is presented in a combination of text and graphics. For example: "The integrity defect map shows a 5% missing rate for the 'delivery address' field (medium-sized nodes), with 2 sets of duplicate records (connected by red dashed lines); in the distribution anomaly heatmap, the 'payment amount' range of 500-1000 yuan is dark red (severe anomaly); the semantic conflict link contains 3 pairs of conflicting fields, among which the 'city-province' conflict significantly impacts logistics (severe). Scores for each dimension: integrity 96.5, distribution 88, semantics 92, total score 92.2. It is recommended to prioritize the repair of semantic conflicts and high-value anomalies."

[0076] S202, Based on the comprehensive quality assessment report, using a rule-based and learning-integrated strategy generator, analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects.

[0077] Specifically, it can parse the comprehensive quality assessment report, extract the coordinates of missing fields and duplicate item identifiers in the integrity defect map, the high abnormal value range in the distribution anomaly heatmap, and the conflict field pairs in the semantic conflict link, organize them in the defect type-location-severity triplet format, and generate a defect-location-severity mapping table.

[0078] A comprehensive quality assessment report is a concentrated presentation of data quality defects, and the analysis process requires precise extraction of key defect information from each dimension. Taking a comprehensive quality assessment report of e-commerce transaction data as an example, the report includes a completeness defect map (such as missing "payment amount" field, duplicate records 8 and 15), a heatmap of abnormal numerical distribution (such as the KL divergence of "purchase quantity" in the 100-200 range being 0.8, which is a high anomaly), semantic conflict links (such as the semantic similarity of "city = Guangzhou" and "province = Hunan" being 0.2 < 0.8), and scores for each dimension (completeness 85 points, distribution rationality 78 points, semantic consistency 80 points).

[0079] When extracting the coordinates of missing fields, it is necessary to combine the visualization information of the integrity defect map to determine the name of the missing field and the corresponding record index. For example, if the missing value of the "Delivery Address" field in the map is marked as a red node, and its associated record indexes are records 5, 12, and 20, then the coordinates of the missing field record are ("Delivery Address", [5, 12, 20]). For duplicate entries, the index pairs of duplicate records must be clearly identified. For example, if records 8 and 15, and records 30 and 45 in the map are marked as duplicate links, then the duplicate entries are identified as ([8, 15], [30, 45]).

[0080] Extracting high-anomaly ranges from heatmaps of distribution anomalies requires considering the color depth and KL divergence threshold of the heatmap. Typically, ranges with a KL divergence ≥ 0.6 are defined as high anomalies. For example, in the heatmap for the "Purchase Quantity" field, the 100-200 range is dark red, corresponding to a KL divergence of 0.8 ≥ 0.6, thus it is extracted as a high-anomaly range ("Purchase Quantity", [100, 200]). In the "Unit Price" field, the 5000-10000 range has a KL divergence of 0.5 < 0.6 and is not included in the high-anomaly category.

[0081] Extracting conflicting field pairs from semantic conflict links requires focusing on field combinations with similarity below a threshold within the link. For example, if the similarity between "City = Shenzhen" and "Province = Guangxi Zhuang Autonomous Region" is 0.1 < 0.8, and the similarity between "Order Status = Paid" and "Payment Time = Empty" is 0.3 < 0.7 (threshold 0.7), then the conflicting field pair records are ("City - Province", "Order Status - Payment Time"), and associated with the corresponding record indexes (records 25 and 33).

[0082] The format for the defect type-location-severity triplet must be standardized. Defect types are categorized into four types: "Missing", "Duplicate", "Distribution Anomaly", and "Semantic Conflict". Location must be precise down to the field and record index, such as ("Payment Amount", [5,10]). Severity is scored on a scale of 1-5 (5 being the most severe), based on the impact of the defect on business operations: missing core fields (such as "Order ID") are worth 5 points, while missing non-core fields (such as "Remarks") are worth 2 points; duplicate records affect statistical accuracy and are worth 3 points; distribution anomalies affect trend analysis (such as abnormally high values ​​for "Payment Amount") and are worth 4 points; semantic conflicts lead to logical errors (such as "City-Province" mismatch) and are worth 5 points.

[0083] For example, a set of triples might be: ("Missing", ("Payment Amount", [5,10]), 4 points), ("Duplicate", ([8,15]), 3 points), ("Distribution Anomaly", ("Purchase Quantity", [100,200]), 4 points), ("Semantic Conflict", ("City-Province", [25,33]), 5 points). These triples are sorted by field to generate a defect-location-severity mapping table. Each record in the table corresponds to one triple, clearly displaying the specific information of various defects and providing precise target location for subsequent remediation.

[0084] Based on the defect-location-severity mapping table and combined with the business scenario of the target data assets, the repair constraints, including the value range of numeric fields, the format specifications of character fields, and the business logic dependencies across fields, are extracted to generate a set of repair constraints.

[0085] The defect-location-severity mapping table clearly identifies the defects that need to be fixed, while the fix constraints limit the boundaries of the fix operations, ensuring that the fixed data conforms to the logical rules of the business scenario. The business scenario determines the specific content of the constraints. For example, the constraints in e-commerce transaction scenarios differ significantly from those in financial lending scenarios: e-commerce scenarios focus more on the timing and reasonableness of transaction processes and amounts, while financial scenarios emphasize the compliance of user information and the correlation of risk indicators.

[0086] Value range constraints for numeric fields are the most basic constraint type, used to limit the legal range of values ​​and prevent repair operations from introducing values ​​beyond the business logic. In e-commerce scenarios, "Purchase Quantity" must be a positive integer (value range ≥ 1), as it is impossible to purchase 0 or negative items; "Payment Amount" must be ≥ 0 and ≤ 100,000 yuan (assuming the platform's single order limit is 100,000 yuan), negative values ​​or values ​​exceeding the limit are considered abnormal; "Discount Rate" must be between 0 and 1 (0 represents no discount, 1 represents the original price), exceeding this range does not conform to the discount logic. For example, if a repair operation inserts a missing value of "Purchase Quantity" as -2, it violates the value range constraint and must be prohibited.

[0087] Formatting constraints for character fields ensure consistent formatting for text fields, facilitating data storage and retrieval. In e-commerce scenarios, the format for "Order ID" is "ORD + 8 digits" (e.g., "ORD20250725"), requiring the prefix "ORD" followed by 8 uppercase digits. "Mobile Number" must be 11 digits (e.g., "13800138000") and cannot contain letters or symbols. "Shipping Address" must include province, city, district, and detailed address (e.g., "No. 1, Science and Technology Park Road, Nanshan District, Shenzhen, Guangdong Province"). Missing hierarchical information (e.g., only "No. 1, Science and Technology Park Road") does not conform to the specifications. These specifications are typically based on industry standards or internal enterprise data governance rules; character values ​​generated during repair operations must strictly match the format.

[0088] Cross-field business logic dependency constraints are a more complex type of constraint, involving relationships between multiple fields, ensuring that the corrected data is logically consistent. Typical logical dependencies in e-commerce scenarios include: "Payment time" must be later than "Order placement time" (time-sequence dependency), because payment cannot occur before order placement; "Payment amount" = "Unit price" × "Quantity" × (1 - "Discount rate") (calculation dependency), all three must satisfy this formula; when "Order status" is "Paid," "Payment time" cannot be missing (state dependency), otherwise the state and time would contradict each other. For example, if the missing value of "Payment time" is set to be earlier than "Order placement time" when correcting it, it violates the time-sequence dependency constraint and needs to be corrected to a reasonable value later than the order placement time.

[0089] Extracting constraints for improvement requires combining expert knowledge of the business scenario. This involves analyzing key nodes and rule documents within the business process to transform implicit rules into explicit constraints. For example, rules such as "order ID format" and "maximum payment amount" can be extracted from an e-commerce platform's "Order Data Specification Manual," and dependencies such as "payment time is later than order placement time" can be derived from the transaction process. During extraction, the priority of constraints must be considered: core constraints (such as "payment amount ≥ 0") must not be violated, while secondary constraints (such as "shipping address includes street address") can be more flexible but should be satisfied as much as possible.

[0090] The generated set of repair constraints is a collection of these constraints, categorized by field type and dependency. For example, the constraint set for an e-commerce scenario includes: numeric constraints ("Purchase quantity ≥ 1", "Payment amount ≥ 0 and ≤ 100000", "Discount rate ∈ [0,1]"), character constraints ("Order ID format is 'ORD+8 digits'", "Mobile number is 11 digits"), and cross-field constraints ("Payment time > Order time", "Payment amount = Product unit price × Purchase quantity × (1 - Discount rate)", "Payment status → Payment time is not empty"). Each constraint must clearly define the applicable fields, constraint content, and the handling method after violation (such as refusing repair or issuing a warning), providing rigid rules for the generation of subsequent repair instructions.

[0091] The defect-location-severity mapping table and the set of repair constraints are input into the policy generator that integrates rules and learning. The rule module generates basic repair instructions, the learning module generates optimized repair instructions, and the basic repair instructions and optimized repair instructions are merged to obtain the initial set of repair operation instructions.

[0092] The rule-based and learning-integrated strategy generator is the core module for generating remediation instructions. Its rule module ensures the fundamental nature and compliance of the remediation, while the learning module optimizes the remediation effect based on historical data. The integration of the two achieves the remediation goal of "minimum guarantee + improvement". The generator takes a defect-location-severity mapping table (remediation target) and a set of remediation constraints (remediation boundary) as input, and outputs initial remediation operation instructions covering various types of defects.

[0093] The rules module generates basic repair instructions based on a pre-defined repair rule library. Each rule in the library corresponds one-to-one with a defect type, ensuring that each defect has a clear handling solution. Rule formulation follows the principle of "simplicity and effectiveness," prioritizing industry-standard repair methods. For example, for "missing value" defects, the rule library stipulates that core numeric fields (such as "payment amount") are imputed using the median of non-missing values ​​in the same field (because the median is highly resistant to extreme values); non-core numeric fields (such as "discount rate") are imputed using the mean; and character fields (such as "shipping address") are imputed using the pattern (taking the most frequently occurring value). For "duplicate item" defects, the rule is "keep the first record and delete the remaining duplicate records," because the first record is usually the original entry and is more likely to be accurate.

[0094] To address the "distribution anomaly" defect, the rule module formulates strategies based on the anomaly type: if it is an extremely high value (e.g., "payment amount" = 1 million yuan, far exceeding the upper limit of 100,000 yuan), it is truncated to the maximum value (100,000 yuan); if it is an outlier low value (e.g., "purchase quantity" = 0.5, which should be an integer), it is rounded up to 1; if it is an interval anomaly (e.g., "purchase quantity" is concentrated in the range of 100-200), it is redistributed according to the normal distribution probability (e.g., adjusting 150 to 50 to make the distribution smoother). To address the "semantic conflict" defect, the rules prioritize the "main field corrects subordinate field" strategy. For example, when there is a conflict between "city" and "province", "city" is used to correct "province" (because cities are more specific, such as "Guangzhou City" corresponding to "Guangdong Province", "Hunan Province" is corrected to "Guangdong Province").

[0095] For example, the basic instruction generated by the rules module for a missing value triple ("Missing", ("Payment Amount", [5,10]), 4 points) is: "For the 'Payment Amount' field of records 5 and 10, use the median of the non-missing values ​​of this field (assuming the median is 299 yuan) for imputation. The imputation value must be ≥0 and ≤100000 yuan". The instruction for a duplicate triple ("Duplicate", ([8,15]), 3 points) is: "Keep record 8 and delete record 15". The instruction for a semantic conflict triple ("Semantic Conflict", ("City-Province", [25,33]), 5 points) is: "Based on the value of the 'City' field, correct the 'Province' of records 25 and 33 to the corresponding province (e.g., 'Guangzhou City' → 'Guangdong Province')".

[0096] The learning module uses a machine learning model to learn better repair strategies from historical repair data and generate optimized repair instructions. The core of the learning module is a trained repair effect prediction model. This model uses "defect features - repair method - repair effect" as training data (e.g., historically, KNN imputation of missing "payment amount" values ​​has outperformed median imputation). It takes the current defect's features (such as missing rate, field type, and distribution characteristics) as input and outputs the repair method with the best predicted effect. For example, when the missing rate of the "payment amount" field is less than 10%, the model predicts that KNN imputation (based on the amount of similar orders) has a smaller error (MAE=5 yuan), which is better than median imputation (MAE=15 yuan), thus generating an optimized KNN imputation instruction.

[0097] The training data for the learning module comes from the enterprise's data remediation logs, containing 10,000 remediation records from the past 6 months. Each record includes the defect type, field name, missing rate, historical remediation methods, and post-remediation accuracy metrics (such as the error between imputed and actual values). The model employs the Random Forest algorithm (due to its ability to handle mixed-type features and its resistance to overfitting). Features include defect severity, field importance, and data distribution characteristics, with the target variable being remediation accuracy (such as the reciprocal of the error). After training, the model's accuracy on the test set must be ≥85% to ensure the effectiveness of the predicted remediation methods.

[0098] For example, the learning module, for missing values ​​in "Payment Amount" (missing rate 8%, field importance score 4), predicts that KNN imputation (taking the average amount of the 5 most similar orders) is more effective, generating the optimization instruction: "For the 'Payment Amount' field of records 5 and 10, use KNN imputation (k=5), referencing the amounts of similar orders in 'Unit Price' and 'Quantity', the imputed value must satisfy Payment Amount = Unit Price × Quantity × (1 - Discount Rate)". For abnormal distribution of "Quantity" (100-200 range), the model predicts that cluster correction (assigning abnormal range values ​​to nearby normal cluster centers) is better than simple truncation, generating the instruction: "For records with 'Quantity' in the 100-200 range, correct the values ​​to the corresponding cluster centers (e.g., 150→50) based on the K-means clustering (k=3) results".

[0099] When merging basic repair instructions and optimized repair instructions, potential conflicts must be handled: If the optimized instruction and the basic instruction address the same defect and do not conflict (e.g., both are interpolation but use different methods), the optimized instruction is retained (because it is more effective); if the optimized instruction violates the repair constraints (e.g., the interpolated value exceeds the range), the optimized instruction is discarded, and the basic instruction is retained; if the two address different defects, both are retained. For example, if both the basic instruction (median interpolation) and the optimized instruction (KNN interpolation) for missing "payment amount" meet the constraints, the optimized instruction is retained; if the interpolated value of the optimized instruction is -50 yuan (violating the ≥0 constraint), the median interpolation of the basic instruction (299 yuan) is retained.

[0100] The merged initial repair operation instruction set contains all valid instructions. Each instruction must specify the operation object (field, record index), operation type (imputation, deletion, correction, etc.), operation parameters (such as the k value of KNN = 5), and constraint check requirements. For example, the initial instruction set may include: "KNN imput 'payment amount' [5,10], k=5, constraint ≥0", "delete duplicate records

[15] ", "correct 'province' [25,33] to 'Guangdong Province'", "cluster correction 'purchase quantity' [100-200], k=3", covering all extracted defects and laying the foundation for subsequent conflict detection and screening.

[0101] Conflict detection is performed on the initial repair operation instruction set. A greedy algorithm is used to retain instructions with a priority higher than the preset level. At the same time, the repair cost of each instruction is calculated, and invalid instructions with a cost exceeding the cost threshold are filtered out to generate a candidate repair operation instruction set.

[0102] The initial repair operation instruction set may contain instruction conflicts and invalid instructions. Conflict detection and cost filtering are key steps to ensure the feasibility of repair. A conflict refers to two or more instructions operating on the same record in the same field, and the operation results are incompatible (e.g., one instruction interpolates the "payment amount" to 299 yuan, while another instruction corrects it to 399 yuan). Invalid instructions refer to instructions that are too costly to repair (e.g., the repair benefit is lower than the cost) or cannot be executed (e.g., the interpolated value always violates the constraint).

[0103] Conflict detection is achieved by comparing the operands (fields + record indexes) of the instructions. If two instructions have exactly the same operand, they are considered to be in conflict. For example, instruction A: "Correct the 25th record of 'Province' to 'Guangdong Province'" and instruction B: "Correct the 25th record of 'Province' to 'Guangxi Zhuang Autonomous Region'", both operate on ("Province", 25), but the results are different, so they are considered to be in conflict. The scope of conflict detection covers the entire initial instruction set. All instruction pairs are traversed through a double loop to ensure no omissions. The detection frequency is consistent with the instruction set generation frequency (e.g., once every 100 records).

[0104] Greedy algorithms are used to resolve conflicts. Their core idea is to "prioritize keeping instructions with higher priority," where priority is determined by the severity of the defect (the higher the severity score, the higher the priority). The default priority level is typically set to 3 points (out of 1-5). Instructions with a score higher than this level (4-5 points) are considered high priority and should be retained; those with a score lower than or equal to 3 points are considered low priority and can be replaced. For example, in a conflict between instruction A (severity score 5, semantic conflict) and instruction B (severity score 3, abnormal distribution), instruction A is retained because it has a higher priority, and instruction B is discarded.

[0105] If conflicting instructions have the same priority (e.g., both scored 5), the "operation relevance" is used to determine the order of priority: if the instructions target upstream and downstream logic of the same field (e.g., "payment amount" and "product unit price"), the instruction targeting the core field (e.g., "payment amount") is retained; otherwise, one is randomly selected and retained (in practice, this needs to be determined in conjunction with business experts). For example, if two 5-point instructions respectively modify the same record for "payment amount" and "order ID", the instruction for "payment amount" is retained (because it has a greater impact on transaction validity).

[0106] The calculation of repair costs comprehensively considers time cost, accuracy loss, and business impact. The higher the cost, the less worthwhile it is to execute the instruction. Time cost refers to the computing resources required to execute the instruction (e.g., the time cost of KNN interpolation is higher than that of median interpolation because similarity needs to be calculated), and the unit is "milliseconds per record" (e.g., KNN interpolation = 10ms / record, median interpolation = 2ms / record); accuracy loss refers to the expected error between the repaired value and the true value (e.g., the accuracy loss of cluster correction is higher than that of truncation because it may deviate from the original data), represented by 0-1 (0 for no loss, 1 for complete error); business impact refers to the interference of the repair operation on downstream business (e.g., deleting duplicate records may affect statistical reports, business impact = 0.3).

[0107] The cost calculation formula is: Repair cost = (Time cost × 0.2) + (Accuracy loss × 0.5) + (Business impact × 0.3), with weights reflecting the importance of each factor (accuracy loss has the highest weight). For example, the time cost of KNN interpolation of "Payment Amount" [5,10] is 10ms / item × 2items = 20ms (converted to 0.2), accuracy loss is 0.1 (small error), and business impact is 0.1 (interpolation has little impact on downstream processes). Therefore, the cost is 0.2 × 0.2 + 0.1 × 0.5 + 0.1 × 0.3 = 0.04 + 0.05 + 0.03 = 0.12.

[0108] The cost threshold is set based on the maximum cost acceptable to the business, typically 0.5 (range 0-1). Instructions exceeding this value are considered invalid. For example, an instruction to fix an abnormal distribution of "purchase quantity" requires calling a complex deep learning model (time cost = 0.8), accuracy loss = 0.4 (the corrected value may deviate from reality), business impact = 0.6 (affects inventory statistics), and cost = 0.8 × 0.2 + 0.4 × 0.5 + 0.6 × 0.3 = 0.16 + 0.2 + 0.18 = 0.54 > 0.5, therefore it is determined to be an invalid instruction and filtered out.

[0109] After conflict detection and cost filtering, a candidate repair operation instruction set is generated. All instructions in this set meet the following requirements: no conflict, priority meets the standard (higher than the preset level), and cost ≤ threshold. For example, the candidate instruction set may retain: "KNN interpolate 'payment amount' [5,10]", "delete duplicate records

[15] ", and "correct 'province' [25,33]" (all without conflict, with costs of 0.12, 0.05, and 0.1 respectively, and priority of 4-5 points), filtering out conflict instruction B and high-cost distribution anomaly repair instructions to ensure that the instructions executed subsequently are feasible and efficient.

[0110] Based on the severity of the defect, the urgency of repair, and the operational relevance of each instruction in the candidate repair operation instruction set, the priority weight is calculated using the analytic hierarchy process (AHP). The priority sequence is generated by sorting the instructions by weight and then binding the sequence with the candidate instruction set to output the repair operation instruction set and the priority sequence.

[0111] The candidate repair operation instruction set determines the repair operations to be performed, but the execution order affects the repair efficiency and effectiveness: prioritizing the repair of defects with high severity and urgency can reduce business losses; prioritizing closely related instructions (such as repairing "payment amount" before repairing "discount rate") can avoid duplicate operations. The Analytic Hierarchy Process (AHP) ensures reasonable sorting by quantifying the impact of each factor and scientifically calculating priority weights.

[0112] The first step in the Analytic Hierarchy Process (AHP) is to construct a hierarchical structure. The target layer is "determining repair priorities." The criteria layer contains three factors: defect severity (impact on data quality), repair urgency (losses due to delayed repair), and operational relevance (dependencies with other instructions). The solution layer consists of each instruction in the candidate repair operation instruction set. For example, the weights of the three factors in the criteria layer need to be determined through pairwise comparisons to reflect their importance in prioritization.

[0113] The weights of the criterion layer are determined using a 1-9 scale: 1 indicates that both factors are equally important, 3 indicates that the former is slightly more important than the latter, 5 indicates that it is significantly important, 7 indicates that it is strongly important, and 9 indicates that it is extremely important; conversely, the reciprocal is used. In the e-commerce scenario, "defect severity" (out of 5) is slightly more important than "repair urgency" (because severe defects have a more far-reaching impact), with a scale of 3; "repair urgency" is significantly more important than "operational relevance" (delayed repair may lead to order timeouts), with a scale of 5; therefore, the scale for "severity" and "relevance" is 3 × 5 = 15 (taking 9, since 9 is the highest). By constructing a judgment matrix and calculating the eigenvectors, the weights of the criterion layer are obtained: severity 0.5, urgency 0.3, and relevance 0.2 (the sum of the weights is 1).

[0114] For each instruction at the solution layer, a score must be assigned under each criterion. The severity score directly uses the rating from the triplet (1-5 points), e.g., instruction A (semantic conflict) gets 5 points, and instruction B (duplicate) gets 3 points. The urgency score is set according to the defect type: semantic conflicts (causing logical errors) and missing core fields (e.g., "payment amount") have the highest urgency (5 points); distribution anomalies (affecting trends but not fatal) are next (3 points); duplicates (only redundancy) have the lowest urgency (2 points). The operational relevance score measures the number of dependencies between instructions: if instruction C needs to be executed after instruction A (e.g., "discount rate" depends on "payment amount"), then A's relevance score increases by 1, while C's relevance score does not increase; instructions with no dependencies get 0 points, and the more dependencies, the higher the score (maximum 5 points).

[0115] For example, there are 4 instructions in the candidate instruction set:

[0116] Instruction 1: Correct the "City-Province" conflict (Severity 5 points, Urgency 5 points, 1 instance of association → Score 1);

[0117] Instruction 2: KNN imputation of "payment amount" (severity 4 points, urgency 5 points, association 2 times → score 2);

[0118] Instruction 3: Delete duplicate records (Severity 3 points, Urgency 2 points, Relevance 0 → Score 0);

[0119] Instruction 4: Clustering to correct “Purchase Quantity” (Severity 4 points, Urgency 3 points, 1 association → Score 1).

[0120] For each instruction, calculate the weighted sum of its score under each criterion and the criterion weight, i.e., priority weight = (severity score / 5) × 0.5 + (urgency score / 5) × 0.3 + (relevance score / 5) × 0.2 (scores are normalized to 0-1).

[0121] Instruction 1: (5 / 5)×0.5 + (5 / 5)×0.3 + (1 / 5)×0.2 = 0.5 + 0.3 + 0.04 = 0.84;

[0122] Instruction 2: (4 / 5)×0.5 + (5 / 5)×0.3 + (2 / 5)×0.2 = 0.4 + 0.3 + 0.08 = 0.78;

[0123] Instruction 3: (3 / 5) × 0.5 + (2 / 5) × 0.3 + 0 × 0.2 = 0.3 + 0.12 + 0 = 0.42;

[0124] Instruction 4: (4 / 5)×0.5 + (3 / 5)×0.3 + (1 / 5)×0.2 = 0.4 + 0.18 + 0.04 = 0.62.

[0125] Sort by priority weight from highest to lowest, resulting in the following priority sequence: Instruction 1 (0.84) → Instruction 2 (0.78) → Instruction 4 (0.62) → Instruction 3 (0.42). This sequence ensures that the most serious and urgent semantic conflicts are fixed first, followed by missing core fields, and finally, less impactful duplicates and distribution anomalies are addressed.

[0126] The priority sequence is bound to the candidate instruction set, and the repair operation instruction set and priority sequence are output. In the bound result, each instruction has a corresponding priority ranking, for example: 1. Correct "City-Province" [25,33]; 2. KNN imputation of "Payment Amount" [5,10]; 3. Clustering correction of "Purchase Quantity" [100-200]; 4. Delete duplicate records

[15] . This result clarifies the order of repair, provides a clear operation guide for the multi-expert collaborative repair execution module, and ensures that the repair process is efficient and orderly.

[0127] In addition, the priority sequence needs to be validated for rationality: check for obvious logical reversals (such as fixing dependencies before fixing dependents), and if so, adjust them manually (e.g., if instruction C depends on instruction A, and A comes after C, the order needs to be swapped). After the validation passes, the final set of repair operation instructions and priority sequence will serve as input to the multi-expert collaborative repair execution module, initiating the actual repair process.

[0128] S203, the target data asset, the repair operation instruction set, and the priority sequence are input into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0129] Specifically, it can parse the repair operation instruction set and priority sequence, assign tasks to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type, record the pending task index and priority sort of each model, and generate a model-task allocation table.

[0130] The repair operation instruction set and priority sequence serve as a guide for multi-expert collaborative repair. The parsing process requires transforming abstract instructions into specific, executable tasks. The repair operation instruction set includes instruction type (such as "numerical interpolation", "semantic reconstruction", "rule correction"), operation object (field name, record index), and operation parameters (such as the k value of the interpolation algorithm, the threshold for rule correction). The priority sequence is ordered by weight (such as [semantic reconstruction instruction 1, numerical interpolation instruction 2, rule correction instruction 3]), clearly defining the execution order.

[0131] During parsing, the type label for each instruction is first identified: instructions containing keywords such as "missing value" or "imputation" are classified as "numerical imputation"; those containing keywords such as "semantic conflict" or "field pair" are classified as "semantic reconstruction"; and those containing keywords such as "distribution anomaly" or "range adjustment" are classified as "rule correction". For example, the instruction "Perform KNN imputation (k=5) on the 'payment amount' field of records 5 and 10" is labeled as "numerical imputation"; the instruction "Correct the 'city-province' semantic conflict in records 25 and 33" is labeled as "semantic reconstruction".

[0132] The multi-expert collaborative repair execution module comprises three expert models, each with clearly defined functional boundaries: the numerical imputation expert model specifically handles missing value repair, supporting algorithms such as KNN and spatiotemporal attention interpolation; the semantic reconstruction expert model focuses on semantic conflict correction, relying on knowledge graphs and pre-trained language models; and the rule correction expert model is responsible for distribution anomaly adjustment, based on a business rule base and optimization algorithms. Task allocation must strictly match the instruction type with the model function to avoid cross-model allocation (e.g., semantic conflict instructions cannot be assigned to the numerical imputation model).

[0133] When recording the index of tasks to be processed, the record position corresponding to each instruction must be clearly defined. For example, the numerical imputation instruction involves the 5th and 10th records of the "Payment Amount" field, and the task index is recorded as ("Payment Amount", [5,10]); the semantic reconstruction instruction involves the 25th and 33rd records of the "City-Province" field pair, and the task index is recorded as ("City-Province", [25,33]). Priority sorting directly follows the weight order of the priority sequence. For example, if the semantic reconstruction instruction 1 has a weight of 0.84 and the numerical imputation instruction 2 has a weight of 0.78 in the priority sequence, then the semantic reconstruction task has a higher priority than the numerical imputation task.

[0134] The model-task allocation table is a structured result integrating the above information, containing five columns: model name, task type, task index, priority weight, and operation parameters. Taking e-commerce data repair as an example, the allocation table might look like this: The numerical imputation expert model corresponds to the "numerical imputation" task, with a task index ("payment amount", [5,10]), priority 0.78, and parameters "k=5, spatiotemporal attention window=3"; the semantic reconstruction expert model corresponds to the "semantic reconstruction" task, with a task index ("city-province", [25,33]), priority 0.84, and parameters "knowledge graph = e-commerce geographic graph, pre-trained model = BERT-base"; and the rule correction expert model corresponds to the "rule correction" task, with a task index ("purchase quantity", [100-200]), priority 0.62, and parameters "threshold range = 1-50, gradient learning rate = 0.01".

[0135] After the allocation table is generated, a validity check is required: This checks whether the task index is within the valid record range (e.g., the record index cannot exceed 1000 records), whether the parameters meet the model requirements (e.g., the k value must be a positive integer), and whether the priority is consistent with the sequence. If a task index is incorrect (e.g., the 1001st record for "Payment Amount"), it is marked as an invalid task and the process returns to the policy generator to regenerate instructions, ensuring that all tasks in the allocation table are executable.

[0136] Using a numerical interpolation expert model, the location of the missing value to be processed is obtained according to the model-task allocation table. The spatiotemporal attention interpolation algorithm is used to calculate the interpolated value. At the same time, cross-validation is used to ensure that the interpolated value conforms to the distribution law, and the interpolated data numerical fragment is output.

[0137] The core of the numerical imputation expert model is to accurately fill in missing values ​​while ensuring that the imputed values ​​conform to the inherent distribution patterns of the data. The model first extracts the locations of the missing values ​​to be processed from the model-task assignment table. For example, ("Payment Amount", [5,10]) indicates that the "Payment Amount" field in records 5 and 10 is missing and needs to be imputed. This location information must be aligned with the indexes of the original data assets to avoid positioning errors (such as confusing row indexes with column indexes).

[0138] Spatiotemporal attention interpolation is an advanced method for handling missing temporal data. It considers both "spatial similarity" (the association with other records in the same field) and "temporal correlation" (the trend between adjacent time points) and dynamically allocates the weights of the two through an attention mechanism. Taking the missing "payment amount" in e-commerce as an example, spatial similarity refers to records similar to the missing record in fields such as "unit price," "quantity," and "user level" (e.g., records with the same user level of VIP and a purchase quantity of 2). Temporal correlation refers to the trend of the "payment amount" changes in the three records before and after the missing record (e.g., the third record is 199 yuan, the fourth record is 299 yuan, and the sixth record is 399 yuan, showing an upward trend).

[0139] The attention mechanism assigns weights based on similarity scores: for the spatial dimension, cosine similarity is used to calculate the feature vector distance between the record to be interpolated and other records (e.g., the similarity between record 5 and record 7 is 0.9, weight 0.6); for the temporal dimension, an exponential decay function is used to calculate the time difference weight (e.g., the time difference between record 5 and record 4 is 1 minute, weight 0.3; the time difference with record 6 is 1 minute, weight 0.1). The interpolated value is a weighted sum. For example, the interpolated value for record 5, "Payment Amount", is: (Record 7, 399 yuan × 0.6) + (Record 4, 299 yuan × 0.3) + (Record 6, 399 yuan × 0.1) = 399 × 0.6 + 299 × 0.3 + 399 × 0.1 = 239.4 + 89.7 + 39.9 = 369 yuan.

[0140] Cross-validation is used to verify the reasonableness of the imputed value. A 5-fold cross-validation is employed: the non-missing "payment amount" data is randomly divided into 5 groups, each group serving as the validation set in turn (assuming group 3 contains 100 records). Missing records are artificially created for each record in the validation set, and the spatiotemporal attention interpolation algorithm is used to impute them. The mean absolute error (MAE) between the imputed value and the true value is calculated. If MAE ≤ a preset threshold (e.g., 20 yuan), the algorithm is considered stable; if MAE = 15 yuan < 20 yuan, the imputed value is deemed valid. For example, for the imputed value of 369 yuan for record 5, cross-validation reveals that the MAE of similar imputed values ​​is 12 yuan, which meets the requirements.

[0141] In addition, it is necessary to check whether the imputed value conforms to the distribution pattern: calculate the difference between the imputed value and the mean and standard deviation of the original non-missing value (e.g., the original mean is 300 yuan, the standard deviation is 100 yuan; the imputed value is 369 yuan, which is within the range of mean ± 1 standard deviation (200-400 yuan)), and does not violate the repair constraints (e.g., "payment amount ≥ 0" "payment amount = unit price of goods × quantity purchased × (1 - discount rate)", 369 yuan = 199 yuan × 2 × (1-0.1) = 358.2 yuan, and the error is within 5%, which is considered reasonable).

[0142] The imputed data fragments are local datasets containing the imputation results. For example, the "Payment Amount" field in records 5 and 10 changes from missing to 369 yuan and 299 yuan respectively, while other fields remain unchanged. The fragments need to be labeled with an imputation identifier (e.g., "imputation_spatiotemporal attention") for easy tracking later. The output fragments will be integrated with other repaired fragments to form complete repaired data.

[0143] Using a semantic reconstruction expert model, the system receives field data corresponding to semantic conflict links, calls a knowledge graph-enhanced pre-trained language model to generate candidate field values ​​that conform to semantic logic, filters candidate values ​​that do not conform to business terms through a domain dictionary, selects the one with the highest confidence as the reconstruction result, and outputs semantically corrected data fragments.

[0144] The semantic reconstruction expert model focuses on resolving semantic conflicts across fields. Its input consists of field data corresponding to the semantic conflict chain, such as conflict records (records 25 and 33) between "city = Guangzhou" and "province = Hunan". The "province" field value needs to be reconstructed to match "city". The core of the model is to combine knowledge graphs and pre-trained language models to generate candidate values ​​that conform to semantic logic.

[0145] The knowledge graph-enhanced pre-trained language model is based on BERT and integrates domain knowledge graphs (such as e-commerce geographic knowledge graphs, which contain "city-province" mappings: Guangzhou → Guangdong Province, Changsha → Hunan Province, etc.). The model's input is a text sequence of conflicting field pairs (such as "city: Guangzhou, province: ?"). Through the knowledge graph entity linking layer, "Guangzhou" is mapped to the "Guangzhou" entity in the graph, activating its associated "province = Guangdong Province" attribute. Then, the BERT model generates candidate province values.

[0146] When generating candidate field values, the model predicts the missing correct values ​​using a masked language model (MLM) mechanism. For example, given the input text "The city is Guangzhou, and the province is [MASK]", the model outputs candidate values ​​and confidence scores: "Guangdong Province" (confidence score 0.95), "Guangxi Zhuang Autonomous Region" (0.03), and "Hunan Province" (0.02). The confidence score is calculated by the model's softmax layer and reflects the reliability of the prediction; a higher value indicates a better fit to the semantic logic.

[0147] The domain dictionary is used to filter candidate values ​​that do not conform to business terminology. The e-commerce geography domain dictionary contains the standard names of all provinces and cities in China (e.g., "Guangdong Province" and "Guangzhou City" are legal terms, while "Guang Province" and "Guang City" are illegal terms). For example, if the model generates the candidate value "Guang Province", the domain dictionary detects that the term does not exist and filters it directly; it retains legal terms such as "Guangdong Province" and "Guangxi Zhuang Autonomous Region".

[0148] The candidate value with the highest confidence is selected as the reconstruction result. For example, in record 25, "Guangdong Province" has the highest confidence of 0.95, so the "Province" field is reconstructed from "Hunan Province" to "Guangdong Province". In record 33, "City = Shenzhen" conflicts with "Province = Guangxi Zhuang Autonomous Region", so the model generates the candidate value "Guangdong Province" (confidence of 0.98), and the "Province" is reconstructed to "Guangdong Province".

[0149] After reconstruction, semantic similarity needs to be verified: the semantic similarity of "city-province" is recalculated using a BERT model enhanced with a knowledge graph. The original conflicting pair had a similarity of 0.2, and after reconstruction, the similarity of "Guangzhou City - Guangdong Province" is 0.95, which is greater than or equal to the threshold of 0.8, thus resolving the conflict. Simultaneously, it is checked whether the reconstructed values ​​comply with business rules (e.g., "province" must be the full name of the provincial-level administrative region, and "Guangdong Province" conforms to the specifications).

[0150] Semantic correction data fragments are local datasets containing reconstruction results. For example, the "Province" field in records 25 and 33 might be changed from "Hunan Province" and "Guangxi Zhuang Autonomous Region" to "Guangdong Province," and marked with a reconstruction identifier (e.g., "Reconstruction_Knowledge Graph + BERT"). Fragments must retain the correspondence between the original and reconstructed values ​​(e.g., "Original: Hunan Province → Reconstruction: Guangdong Province") for auditing and backtracking. The output fragments will participate in subsequent integration steps to ensure that semantic conflicts are completely resolved.

[0151] Using a rule correction expert model, for the numerical range marked by the distribution anomaly heatmap, the abnormal values ​​are adjusted according to the threshold range in the business rule base. The gradient descent method is used to fine-tune the values ​​to minimize the distribution deviation, and the rule-corrected data fragment is output.

[0152] The rule correction expert model is used to handle distribution anomalies of numerical fields. Its input is the high anomaly interval marked by the distribution anomaly heatmap. For example, the KL divergence of "purchase quantity" in the 100-200 interval is 0.8 (high anomaly). The values ​​in this interval need to be adjusted to conform to the normal distribution (e.g., the 1-50 interval is the main distribution, with a KL divergence of 0.1).

[0153] The business rule base is the foundation for rule correction, containing the legal threshold range for numerical fields, distribution characteristic parameters (such as mean and standard deviation), and business logic constraints. For "purchase quantity," the rule base defines: the legal threshold is 1-100 (the quantity in a single order usually does not exceed 100); the normal distribution has a mean of 5 and a standard deviation of 3 (most orders purchase 1-10 items); and the business logic constraints are "purchase quantity = integer" and "purchase quantity × unit price ≤ payment amount."

[0154] First, adjust the outlier values ​​based on the threshold range: truncate values ​​in the 100-200 range to the upper threshold of 100. For example, adjust record 45, "Purchase Quantity = 150", to 100, and record 60, "Purchase Quantity = 200", to 100. However, simple truncation may lead to a new concentration of values ​​(too many values ​​near 100), requiring further fine-tuning.

[0155] Gradient descent is used to minimize distribution bias. The objective function is the KL divergence between the actual distribution and the theoretical normal distribution, i.e., min KL(P||Q), where P is the adjusted actual distribution and Q is the theoretical normal distribution (mean 5, standard deviation 3). The parameters of gradient descent include the learning rate (0.01, controlling the adjustment magnitude) and the number of iterations (100 times, ensuring convergence).

[0156] For example, for a value initially adjusted to 100, calculate its contribution to the KL divergence (100 deviates significantly from the mean of 5, contributing 0.3), and the gradient is positive (the value needs to be reduced to decrease the deviation). Each iteration adjusts the value by the learning rate multiplied by the gradient, which is 0.01 multiplied by 0.3, gradually bringing 100 closer to 5. However, this must be done in accordance with business rules; the adjusted value must be an integer and ≥1. After 100 iterations, 100 may be adjusted to 50 (still within the legal range and closer to the normal distribution).

[0157] For record 45, "Purchase Quantity = 150 → Initial Adjustment to 100 → Gradient Descent Fine-tuning to 50", at this point, 50 is in the range of 1-100, an integer, and the deviation of 50 from the mean of 5 is less than 100. The KL divergence decreases from 0.8 to 0.2 (the acceptable threshold of ≤0.3). Simultaneously, the business logic is checked: 50 × unit price (199 yuan) = 9950 yuan ≤ payment amount (10000 yuan), which meets the constraints.

[0158] The rule-corrected data segment is a local dataset containing the adjustment results. For example, the "Purchase Quantity" of records 45 and 60 changes from 150 and 200 to 50 and 60, while other fields remain unchanged. The segment needs to be labeled with the adjustment method (e.g., "rule truncation + gradient descent") and the distribution parameters before and after adjustment (e.g., KL divergence 0.8 before adjustment, 0.2 after adjustment) to demonstrate that the distribution anomaly has been improved. The output segment will be integrated with other repair segments to complete the quality repair of the distribution dimension.

[0159] By integrating the imputed data numerical fragments, semantically corrected data fragments, and rule-corrected data fragments according to priority sequence, a complete dataset is obtained, which serves as the repaired data asset.

[0160] The imputed data numerical fragments, semantically corrected data fragments, and rule-corrected data fragments are local repair results. The integration process must follow a priority sequence to ensure the temporal rationality of the repair and data consistency, ultimately generating a complete repaired data asset.

[0161] The priority sequence determines the order of integration: high-priority fragments should cover the original data first to avoid low-priority repairs covering high-priority results. For example, if the priority sequence is [semantic correction fragment (weight 0.84), numerical imputation fragment (0.78), rule correction fragment (0.62)], then the semantic correction fragment will be integrated first, followed by the numerical imputation fragment, and finally the rule correction fragment.

[0162] When integrating, overlapping records between fragments need to be handled: if different fragments involve different fields of the same record, they are directly merged (e.g., for record 25, the semantic correction fragment adjusts "province", while the numerical imputation fragment does not involve this record, so both results are retained); if they involve the same field, only the results of the higher priority fragment are retained (e.g., assuming record 5 is involved by both numerical imputation and rule correction, the imputation result is retained because numerical imputation has a higher priority).

[0163] Data consistency verification is crucial for integration, ensuring cross-field logical consistency: Check if "Payment Amount = Unit Price × Quantity × (1 - Discount Rate)" is true. For example, after interpolation in record 5, "Payment Amount = 369 yuan", "Unit Price = 199 yuan", "Quantity = 2", and "Discount Rate = 0.1" are true. 199 × 2 × 0.9 = 358.2 yuan. The error between 369 yuan and 358.2 yuan is ≤5% (within the business allowable range), considered consistent. Check if "Payment Time > Order Time". Records after semantic correction all meet this condition. Check if "Quantity" is an integer within the range of 1-100. Values ​​after rule correction all meet this condition.

[0164] Integrity checks ensured that all marked defects were fixed: Comparing the defect-location-severity mapping table, missing values ​​[5,10] were imputed, duplicates [8,15] were removed, the distribution anomaly range [100-200] was adjusted, semantically conflicting field pairs [25,33] were reconstructed, and no defects remained unfixed. Simultaneously, the repair coverage rate (number of fixed defects / total number of defects) must be ≥95%, and this repair coverage rate is 100%, meeting the requirements.

[0165] The repaired data asset is a complete dataset containing all repair results. For example, in 1000 records, 50 missing values ​​were imputed, 20 duplicates were removed, 30 outliers were corrected, and 40 semantic conflicts were reconstructed. All fields conform to business rules and distribution patterns. The dataset needs to be accompanied by repair metadata: repair time (e.g., "2025-07-25 14:30:00"), repair method (e.g., "spatiotemporal attention imputation" or "knowledge graph + BERT reconstruction"), and quality scores before and after repair (e.g., completeness score increased from 85 to 98, and semantic consistency score increased from 80 to 96).

[0166] The final output of repaired data assets needs to be standardized in format (e.g., CSV format, with commas as field separators and strings enclosed in double quotes) to facilitate direct use by downstream systems (such as data analysis platforms and business systems). Simultaneously, a repair report should be generated, recording a comparison of key indicators before and after the repair, providing a basis for continuous improvement of data quality.

[0167] As can be seen, the system receives the target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine, generating a comprehensive quality assessment report that includes an integrity defect map, a distribution anomaly heatmap, and semantic conflict links. Based on the comprehensive quality assessment report, a policy generator that integrates rules and learning is used to analyze the repair needs and constraints contained in the defect map, anomaly heatmap, and conflict links, dynamically generating repair operation instruction sets and priority sequences for different types of defects. The target data assets, repair operation instruction sets, and priority sequences are then input into a multi-expert collaborative repair execution module, which repairs the defects in the target data assets according to the priority sequence and outputs the repaired data assets. This enables a comprehensive and accurate data quality assessment, improving the automation level and accuracy of data repair.

[0168] Another embodiment of the present invention provides an AI-based data asset quality assessment and automatic repair system, see [link to relevant documentation]. Figure 3 The system may include:

[0169] The evaluation module 301 is used to receive the target data assets to be processed and input them into the multi-dimensional joint quality scoring engine. The engine performs data integrity verification, numerical distribution rationality analysis and cross-field semantic dependency verification in parallel, and generates a comprehensive quality evaluation report including integrity defect map, distribution anomaly heat map and semantic conflict link.

[0170] The parsing module 302 is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict link based on the comprehensive quality assessment report and using a rule and learning fusion strategy generator, and dynamically generate repair operation instruction sets and priority sequences for different types of defects.

[0171] Repair module 303 is used to input the target data asset, the repair operation instruction set and the priority sequence into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0172] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0173] Specifically, in this embodiment, the storage medium can be configured to store a computer program for performing the following steps:

[0174] S201, at the communication transmitting end, converts the input large data set into a quantized data set;

[0175] S202, extract chaotic features from the quantized data set to generate an initial symmetric key;

[0176] S203, the initial symmetric key is enhanced by a deep chaotic transformation to obtain an enhanced symmetric key;

[0177] S204, perform an encryption operation on the quantized data set using the enhanced symmetric key to obtain an encrypted data set;

[0178] S205, the encrypted data set is encoded into a quantum bit carrier and transmitted to the communication receiving end through quantum entanglement, so that the communication receiving end can decrypt and calculate the data integrity index to verify the integrity and consistency of the data.

[0179] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0180] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0181] Specifically, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0182] S201, at the communication transmitting end, converts the input large data set into a quantized data set;

[0183] S202, extract chaotic features from the quantized data set to generate an initial symmetric key;

[0184] S203, the initial symmetric key is enhanced by a deep chaotic transformation to obtain an enhanced symmetric key;

[0185] S204, perform an encryption operation on the quantized data set using the enhanced symmetric key to obtain an encrypted data set;

[0186] S205, the encrypted data set is encoded into a quantum bit carrier and transmitted to the communication receiving end through quantum entanglement, so that the communication receiving end can decrypt and calculate the data integrity index to verify the integrity and consistency of the data.

[0187] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A data asset quality assessment and automatic repair method based on AI, characterized in that, The method includes: The system receives target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine. This engine performs data integrity verification, numerical distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes an integrity defect map, a distribution anomaly heatmap, and semantic conflict links. Specifically, it receives target data assets to be processed, parses their data structure, uses an adaptive sliding window filtering algorithm to remove high-frequency noise, and extracts the data's generation time, source identifier, and field dependencies to construct a metadata index table, outputting the preprocessed data assets and the metadata index table. Based on the metadata index table, it performs integrity verification in parallel through multi-threading. Specifically, it uses a binary matrix marking method to record the location of missing values ​​in each field, calculates the integrity score by combining the field importance weight, and detects duplicate records and marks redundant locations through hash verification, generating an integrity defect matrix that includes the distribution of missing values, the coordinates of duplicate items, and the integrity score. Using the integrity defect matrix as a filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual and theoretical distributions is calculated, and the divergence values ​​are mapped to the color depth of a heatmap to generate a heatmap of numerical distribution anomalies. A cross-field semantic network is constructed using preprocessed data assets. The semantic similarity between fields is calculated using a BERT model enhanced with a knowledge graph. When the similarity is lower than the similarity threshold, it is marked as a semantic conflict. The propagation path of the conflict is tracked using a time-series backtracking method to generate a semantic conflict link containing conflict nodes, association strength, and propagation path. The integrity defect matrix is ​​converted into a visualized defect map. Combined with the heatmap of numerical distribution anomalies and the semantic conflict link, quality scores for each dimension and defect impact range assessment are added to generate a comprehensive quality assessment report. Based on the comprehensive quality assessment report, a strategy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects. The target data asset, the repair operation instruction set, and the priority sequence are input into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

2. The method according to claim 1, characterized in that, Based on the comprehensive quality assessment report, a rule-based and learning-integrated strategy generator is used to analyze the repair requirements and constraints contained in the defect map, anomaly heatmap, and conflict links. This dynamically generates repair operation instruction sets and priority sequences for different types of defects, including: The comprehensive quality assessment report is analyzed to extract the coordinates of missing fields and duplicate item identifiers from the integrity defect map, the high abnormal value ranges from the distribution anomaly heatmap, and the conflict field pairs from the semantic conflict link. These are then organized into a defect-location-severity mapping table in the defect type-location-severity triplet format. Based on the defect-location-severity mapping table and combined with the business scenario of the target data assets, the repair constraints, including the value range of numeric fields, the format specifications of character fields, and the business logic dependencies across fields, are extracted to generate a set of repair constraints. The defect-location-severity mapping table and the set of repair constraints are input into the policy generator that integrates rules and learning. The rule module generates basic repair instructions, the learning module generates optimized repair instructions, and the basic repair instructions and optimized repair instructions are merged to obtain the initial set of repair operation instructions. Conflict detection is performed on the initial repair operation instruction set. A greedy algorithm is used to retain instructions with a priority higher than the preset level. At the same time, the repair cost of each instruction is calculated, and invalid instructions with a cost exceeding the cost threshold are filtered out to generate a candidate repair operation instruction set. Based on the severity of the defect, the urgency of repair, and the operational relevance of each instruction in the candidate repair operation instruction set, the priority weight is calculated using the analytic hierarchy process (AHP). The priority sequence is generated by sorting the instructions by weight and then binding the sequence with the candidate instruction set to output the repair operation instruction set and the priority sequence.

3. The method according to claim 2, characterized in that, The target data asset, the repair operation instruction set, and the priority sequence are input into the multi-expert collaborative repair execution module. This module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset, including: The repair operation instruction set and priority sequence are parsed, and the tasks are assigned to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type. The task index and priority sort of each model are recorded, and a model-task allocation table is generated. Using a numerical interpolation expert model, the location of the missing value to be processed is obtained according to the model-task allocation table. The spatiotemporal attention interpolation algorithm is used to calculate the interpolated value. At the same time, cross-validation is used to ensure that the interpolated value conforms to the distribution law, and the interpolated data numerical fragment is output. Using a semantic reconstruction expert model, the system receives field data corresponding to semantic conflict links, calls a knowledge graph-enhanced pre-trained language model to generate candidate field values ​​that conform to semantic logic, filters candidate values ​​that do not conform to business terms through a domain dictionary, selects the one with the highest confidence as the reconstruction result, and outputs semantically corrected data fragments. Using a rule correction expert model, for the numerical range marked by the distribution anomaly heatmap, the abnormal values ​​are adjusted according to the threshold range in the business rule base. The gradient descent method is used to fine-tune the values ​​to minimize the distribution deviation, and the rule-corrected data fragment is output. By integrating the imputed data numerical fragments, semantically corrected data fragments, and rule-corrected data fragments according to priority sequence, a complete dataset is obtained, which serves as the repaired data asset.

4. An AI-based data asset quality assessment and automatic repair system, characterized in that, The system includes: The evaluation module receives the target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine. This engine performs data integrity verification, numerical distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality evaluation report that includes an integrity defect map, a distribution anomaly heatmap, and semantic conflict links. Specifically, it receives the target data assets to be processed, parses their data structure, uses an adaptive sliding window filtering algorithm to remove high-frequency noise, and extracts the data's generation time, source identifier, and field dependencies to construct a metadata index table, outputting the preprocessed data assets and the metadata index table. Based on the metadata index table, it performs integrity verification in parallel through multi-threading. Specifically, it uses a binary matrix marking method to record the location of missing values ​​in each field, calculates the integrity score by combining the field importance weight, and detects duplicate records and marks redundant locations through hash verification, generating an integrity defect matrix that includes the distribution of missing values, the coordinates of duplicate items, and the integrity score. Using the integrity defect matrix as a filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual and theoretical distributions is calculated, and the divergence values ​​are mapped to the color depth of a heatmap to generate a heatmap of numerical distribution anomalies. A cross-field semantic network is constructed using preprocessed data assets. The semantic similarity between fields is calculated using a BERT model enhanced with a knowledge graph. When the similarity is lower than the similarity threshold, it is marked as a semantic conflict. The propagation path of the conflict is tracked using a time-series backtracking method to generate a semantic conflict link containing conflict nodes, association strength, and propagation path. The integrity defect matrix is ​​converted into a visualized defect map. Combined with the heatmap of numerical distribution anomalies and the semantic conflict link, quality scores for each dimension and defect impact range assessment are added to generate a comprehensive quality assessment report. The parsing module is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map and conflict link based on the comprehensive quality assessment report and using a strategy generator that integrates rules and learning, and dynamically generate repair operation instruction sets and priority sequences for different types of defects. The repair module is used to input the target data asset, the repair operation instruction set, and the priority sequence into the multi-expert collaborative repair execution module. This module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

5. The system according to claim 4, characterized in that, The parsing module is specifically used for: The comprehensive quality assessment report is analyzed to extract the coordinates of missing fields and duplicate item identifiers from the integrity defect map, the high abnormal value ranges from the distribution anomaly heatmap, and the conflict field pairs from the semantic conflict link. These are then organized into a defect-location-severity mapping table in the defect type-location-severity triplet format. Based on the defect-location-severity mapping table and combined with the business scenario of the target data assets, the repair constraints, including the value range of numeric fields, the format specifications of character fields, and the business logic dependencies across fields, are extracted to generate a set of repair constraints. The defect-location-severity mapping table and the set of repair constraints are input into the policy generator that integrates rules and learning. The rule module generates basic repair instructions, the learning module generates optimized repair instructions, and the basic repair instructions and optimized repair instructions are merged to obtain the initial set of repair operation instructions. Conflict detection is performed on the initial repair operation instruction set. A greedy algorithm is used to retain instructions with a priority higher than the preset level. At the same time, the repair cost of each instruction is calculated, and invalid instructions with a cost exceeding the cost threshold are filtered out to generate a candidate repair operation instruction set. Based on the severity of the defect, the urgency of repair, and the operational relevance of each instruction in the candidate repair operation instruction set, the priority weight is calculated using the analytic hierarchy process (AHP). The priority sequence is generated by sorting the instructions by weight and then binding the sequence with the candidate instruction set to output the repair operation instruction set and the priority sequence.

6. The system according to claim 5, characterized in that, The repair module is specifically used for: The repair operation instruction set and priority sequence are parsed, and the tasks are assigned to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type. The task index and priority sort of each model are recorded, and a model-task allocation table is generated. Using a numerical interpolation expert model, the location of the missing value to be processed is obtained according to the model-task allocation table. The spatiotemporal attention interpolation algorithm is used to calculate the interpolated value. At the same time, cross-validation is used to ensure that the interpolated value conforms to the distribution law, and the interpolated data numerical fragment is output. Using a semantic reconstruction expert model, the system receives field data corresponding to semantic conflict links, calls a knowledge graph-enhanced pre-trained language model to generate candidate field values ​​that conform to semantic logic, filters candidate values ​​that do not conform to business terms through a domain dictionary, selects the one with the highest confidence as the reconstruction result, and outputs semantically corrected data fragments. Using a rule correction expert model, for the numerical range marked by the distribution anomaly heatmap, the abnormal values ​​are adjusted according to the threshold range in the business rule base. The gradient descent method is used to fine-tune the values ​​to minimize the distribution deviation, and the rule-corrected data fragment is output. By integrating the imputed data numerical fragments, semantically corrected data fragments, and rule-corrected data fragments according to priority sequence, a complete dataset is obtained, which serves as the repaired data asset.

7. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1-3 when it is run.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1-3.

Images