Data asset quality evaluation and automatic repair method and system based on AI

Abstract

The invention discloses an AI-based data asset quality assessment and automatic repair method and system, and the method comprises the steps: receiving a to-be-processed target data asset, inputting the target data asset into a multi-dimensional joint quality scoring engine, and generating a comprehensive quality assessment report containing an integrity defect map, a distribution abnormality thermodynamic diagram and a semantic conflict link; based on the comprehensive quality evaluation report, utilizing a rule and learning fused strategy generator to analyze repair requirements and constraint conditions contained in a defect map, an abnormal thermodynamic diagram and a conflict link, and dynamically generating repair operation instruction sets and priority sequences for different types of defects; and inputting the target data assets, the repair operation instruction set and the priority sequence into a multi-expert collaborative repair execution module, repairing defects in the target data assets according to the priority sequence, and outputting the repaired data assets. According to the embodiment of the invention, comprehensive and accurate data quality evaluation can be realized, and the automation level and accuracy of data restoration are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an AI-based data asset quality assessment and automatic repair method and system. Background Art

[0002] With the rapid development of big data technology, data assets have become an essential component of a company's core competitiveness. However, data quality issues (such as missing values, abnormal distributions, and semantic inconsistencies) seriously affect the credibility and application value of data. Traditional data quality assessment methods typically rely on manual rules or single-dimensional statistical analysis, which makes it difficult to fully cover complex scenarios such as data integrity issues, the rationality of numerical distributions, and cross-field semantic associations. In addition, existing repair technologies often use fixed rules or simple interpolation strategies, lacking the ability to collaboratively analyze and dynamically repair multi-dimensional quality defects, resulting in limited repair effects and insufficient adaptability. In recent years, artificial intelligence technology has shown great potential in the field of data governance, but existing solutions still have problems such as a single assessment dimension, rigid repair strategies, and a low degree of automation, which cannot meet the needs of enterprises for efficient and accurate data governance. Summary of the Invention

[0003] The purpose of this invention is to provide an AI-based data asset quality assessment and automatic repair method and system to address the deficiencies in the existing technology, enable comprehensive and accurate data quality assessment, and improve the automation level and accuracy of data repair.

[0004] One embodiment of the present application provides an AI-based data asset quality assessment and automatic repair method, the method comprising: Receive the target data assets to be processed and input them into the multi-dimensional joint quality scoring engine, which performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes integrity defect maps, distribution anomaly heat maps, and semantic conflict links; Based on the comprehensive quality assessment report, a policy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects. The target data asset, the repair operation instruction set and the priority sequence are input into a multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0005] Optionally, the target data assets to be processed are received and input into a multi-dimensional joint quality scoring engine, which performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel to generate a comprehensive quality assessment report containing an integrity defect map, a distribution anomaly heat map, and semantic conflict links, including: Receive the target data assets to be processed, analyze their data structure, use an adaptive sliding window filtering algorithm to remove high-frequency noise, extract the data generation time, source identifier, and field dependencies to build a metadata index table, and output the preprocessed data assets and metadata index table; Based on the metadata index table, the integrity check is performed in parallel through multiple threads. A binary matrix notation method is used to record the missing value location of each field. The integrity score is calculated based on the field importance weight. A hash check is used to detect duplicate records and mark redundant locations. This generates an integrity defect matrix containing the missing value distribution, duplicate item coordinates, and integrity scores. Using the integrity defect matrix as the filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heat map to generate a heat map of numerical distribution anomalies. A cross-field semantic network is constructed using pre-processed data assets. The semantic similarity between fields is calculated using the knowledge graph-enhanced BERT model. When the similarity falls below the similarity threshold, it is marked as a semantic conflict. A time-series backtracking method is used to trace the propagation path of the conflict, generating a semantic conflict link that includes the conflict node, association strength, and propagation path. The integrity defect matrix is ​​converted into a visual defect map, combined with the numerical distribution anomaly heat map and semantic conflict links, and the quality scores of each dimension and the defect impact range assessment are added to generate a comprehensive quality assessment report.

[0006] Optionally, based on the comprehensive quality assessment report, a policy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects, including: Parse the comprehensive quality assessment report, extract missing field coordinates and duplicate identifiers from the integrity defect map, high anomaly value intervals from the distribution anomaly heat map, and conflicting field pairs from the semantic conflict link, organize them into a defect type-location-severity triple format, and generate a defect-location-severity mapping table. Based on the defect-location-severity mapping table and the business scenarios of the target data assets, we extract the repair constraints, including the value range of numeric fields, the format specifications of character fields, and cross-field business logic dependencies, to generate a repair constraint set. The defect-location-severity mapping table and the repair constraint set are input into the rule and learning fusion strategy generator, where the rule module generates basic repair instructions, and the learning module generates optimized repair instructions. The basic repair instructions and the optimized repair instructions are combined to obtain the initial repair operation instruction set; Conflict detection is performed on the initial repair operation instruction set. Instructions with a priority higher than a preset level are retained using a greedy algorithm. The repair cost of each instruction is calculated, and invalid instructions with a cost exceeding a cost threshold are filtered out to generate a candidate repair operation instruction set. According to the defect severity, repair urgency and operation relevance corresponding to each instruction in the candidate repair operation instruction set, the hierarchical analysis method is used to calculate the priority weight, and the priority sequence is generated by sorting the weights. The sequence is bound to the candidate instruction set to output the repair operation instruction set and the priority sequence.

[0007] Optionally, the target data asset, the repair operation instruction set, and the priority sequence are input into a multi-expert collaborative repair execution module. The module calls a corresponding numerical interpolation expert model, semantic reconstruction expert model, or rule correction expert model according to the instruction set, repairs defects in the target data asset according to the priority sequence, and outputs a repaired data asset, including: Parse the repair operation instruction set and priority sequence, assign tasks to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type, record the pending task index and priority ranking of each model, and generate a model-task allocation table; Using the numerical interpolation expert model, the missing value locations to be processed are obtained according to the model-task allocation table. The interpolation values ​​are calculated using the spatiotemporal attention interpolation algorithm. Cross-validation is performed to ensure that the interpolation values ​​conform to the distribution law. The interpolated data value fragments are output. Using the semantic reconstruction expert model, we receive the field data corresponding to the semantic conflict link, call the pre-trained language model enhanced by the knowledge graph to generate candidate field values ​​that conform to the semantic logic, filter the candidate values ​​that do not conform to the business terms through the domain dictionary, select the one with the highest confidence as the reconstruction result, and output the semantically corrected data fragment; Using the rule-corrected expert model, we adjust the abnormal values ​​based on the threshold range in the business rule library for the numerical interval marked by the distribution anomaly heat map, use the gradient descent method to fine-tune the values ​​to minimize the distribution deviation, and output the rule-corrected data fragments. The interpolated data numerical fragments, semantically corrected data fragments and rule-corrected data fragments are integrated in priority sequence to obtain a complete data set as the repaired data asset.

[0008] Another embodiment of the present application provides an AI-based data asset quality assessment and automatic repair system, the system comprising: The evaluation module receives the target data assets to be processed and feeds them into a multi-dimensional joint quality scoring engine. The engine performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes an integrity defect map, distribution anomaly heat map, and semantic conflict links. An analysis module is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links based on the comprehensive quality assessment report and utilize a policy generator that integrates rules and learning to dynamically generate repair operation instruction sets and priority sequences for different types of defects; The repair module is used to input the target data asset, the repair operation instruction set and the priority sequence into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0009] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when run.

[0010] Yet another embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0011] Compared with the existing technology, the present invention provides an AI-based data asset quality assessment and automatic repair method, which receives the target data assets to be processed and inputs them into a multi-dimensional joint quality scoring engine to generate a comprehensive quality assessment report including an integrity defect map, a distribution anomaly heat map and a semantic conflict link; based on the comprehensive quality assessment report, a strategy generator that integrates rules and learning is used to parse the repair requirements and constraints contained in the defect map, anomaly heat map and conflict link, and dynamically generate repair operation instruction sets and priority sequences for different types of defects; the target data assets, repair operation instruction sets and priority sequences are input into a multi-expert collaborative repair execution module, the defects in the target data assets are repaired according to the priority sequence, and the repaired data assets are output, thereby achieving comprehensive and accurate data quality assessment and improving the automation level and accuracy of data repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A hardware block diagram of a computer terminal for an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention; Figure 2A flowchart of an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an AI-based data asset quality assessment and automatic repair system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0014] The embodiment of the present invention first provides an AI-based data asset quality assessment and automatic repair method, which can be applied to electronic devices such as computer terminals, specifically ordinary computers.

[0015] The following describes it in detail by taking running on a computer terminal as an example. Figure 1 The hardware structure block diagram of a computer terminal for an AI-based data asset quality assessment and automatic repair method provided in an embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0016] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any one of the AI-based data asset quality assessment and automatic repair methods.

[0017] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0018] The internal memory provides an environment for the operation of computer programs in non-volatile storage media. When the computer program is executed by the processor, the processor can execute any AI-based data asset quality assessment and automatic repair method.

[0019] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0020] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0021] See also Figure 2 , an embodiment of the present invention provides an AI-based data asset quality assessment and automatic repair method, which may include the following steps: S201: Receive the target data asset to be processed and input it into a multi-dimensional joint quality scoring engine. The engine concurrently performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification to generate a comprehensive quality assessment report including an integrity defect map, distribution anomaly heat map, and semantic conflict links. Specifically, it can receive the target data assets to be processed, parse their data structure, use an adaptive sliding window filtering algorithm to remove high-frequency noise, extract the data generation time, source identifier and field dependency to build a metadata index table, and output the pre-processed data assets and metadata index table; The target data assets to be processed can be structured or semi-structured data sets, such as transaction data from e-commerce platforms and user credit records from financial systems. For example, e-commerce transaction data contains fields such as order ID, user ID, product ID, order time, payment amount, and delivery address. The order ID is a string, the payment amount is a numeric value, and the order time is a timestamp. When receiving data, the system first reads the raw data file (e.g., CSV or JSON format) through a data interface and parses the file header to identify format identifiers such as field delimiters (e.g., commas, colons) and quotation marks (e.g., double quotes) to ensure that fields and values ​​are correctly matched.

[0022] During data structure parsing, the system automatically identifies the data type of each field: Fields containing pure numbers and decimal points (such as the payment amount "99.99") are classified as floating-point; fields consisting of numbers and letters and having a fixed length (such as the order ID "ORD20250725001") are classified as strings; and fields that conform to the "YYYY-MM-DD HH:MM:SS" format (such as the order time "2025-07-25 10:30:00") are classified as datetime. The system also records attributes such as the length range (for example, the user ID is 10 digits long) and whether null values ​​are allowed, providing basic information for subsequent quality assessments.

[0023] An adaptive sliding window filtering algorithm is used to remove high-frequency noise from data. This noise may arise from sensor errors during data acquisition or interference during transmission (e.g., occasional outliers containing letters such as "100.0a" in payment amounts). The algorithm's window size is not fixed but dynamically adjusted based on data volatility. When data values ​​are consistently stable (e.g., payment amounts fluctuate slightly between 90 and 110 yuan), the window size is set to 5 (including the current data and two records before and after). When data values ​​fluctuate suddenly (e.g., a payment amount suddenly jumps to 10,000 yuan), the window size is automatically expanded to 10 to smooth out the fluctuations using more historical data. During the filtering process, the algorithm calculates the median of the data within the window as the filtered value for the current value. For example, if the data within the window is [98, 102, 99, 101, 100] and the median is 100, the original value 99 at the center of the window is replaced with 100, thereby reducing the impact of isolated outliers.

[0024] When extracting metadata information, the generation time refers to the timestamp when the data was first recorded, such as "2025-07-25 10:30:00" for order data, accurate to the second to ensure the accuracy of time series analysis; source identifiers are used to distinguish data collection channels, such as "app side", "web side", and "third-party interface", to facilitate tracing the source of data quality issues; field dependencies refer to the logical associations between different fields, such as "payment amount" depends on "product unit price" and "purchase quantity" (payment amount = product unit price × purchase quantity), and the "province" field in the "delivery address" depends on the "city" field (for example, if the city is "Shanghai", the province must be "Shanghai").

[0025] The metadata index table is a structured collection of this information. Its core fields include: field name (such as "Payment Amount"), data type (such as "float"), generation time range (such as "2025-07-01 to 2025-07-25"), source identifier list (such as "app, website"), dependent fields (such as "product unit price, purchase quantity"), and data length limit (such as "payment amount must have a maximum of 2 decimal places"). Taking the "Payment Amount" field as an example, its record in the index table is: field name "Payment Amount", data type "float", generation time range "2025-07-01 00:00:00 to 2025-07-25 23:59:59", source identifier "app", dependent fields "product unit price, purchase quantity", and length limit "≤10 integer + 2 decimal places".

[0026] The preprocessed data asset is a data set that has been noise-removed. For example, the original payment amount field may contain values ​​such as "100.5," "99.8a," and "105.0." "99.8a" is high-frequency noise (a non-numeric character introduced by an input error). The adaptive sliding window filtering algorithm identifies the deviation between this value and the adjacent normal data (100.5, 105.0) and corrects it to "100.5" through median replacement within the window (when the window size is 3, the median is 100.5). The final preprocessed data output contains valid values ​​for this field. A metadata index table is output simultaneously with the preprocessed data asset, providing basic metadata support for subsequent quality assessment.

[0027] Based on the metadata index table, the integrity check is performed in parallel through multiple threads. A binary matrix notation method is used to record the missing value location of each field. The integrity score is calculated based on the field importance weight. A hash check is used to detect duplicate records and mark redundant locations. This generates an integrity defect matrix containing the missing value distribution, duplicate item coordinates, and integrity scores. The metadata index table provides basic information about field structure and dependencies for integrity verification. Multi-threaded parallel execution improves verification efficiency. For example, for e-commerce transaction data containing 10 fields and 1,000 records, the system will launch a number of threads matching the number of fields (e.g., 10 threads), each responsible for integrity verification of a single field. Threads exchange progress information via shared memory to avoid duplicate computations.

[0028] The binary matrix notation method is the core approach for recording missing value locations. This matrix has a number of rows equal to the number of data records, and a number of columns equal to the number of fields. Matrix elements are either 0 or 1, with 1 indicating a missing value at the corresponding location and 0 indicating a normal value. For example, if the "Payment Amount" field in the third record is missing, and the "Shipping Address" field in the fifth record is missing, then the element in the third row of the binary matrix corresponding to the "Payment Amount" column will be 1, the element in the fifth row corresponding to the "Shipping Address" column will be 1, and all other positions will be 0. This notation method can intuitively demonstrate the distribution of missing values ​​in a dataset. For example, a higher proportion of 1s in a column (field) indicates a more severe missing value in that field.

[0029] Field importance weights are set based on the business scenario, reflecting the degree of impact a field has on data availability. In e-commerce transaction data, "Order ID" and "Payment Amount" are core fields (weighted 0.8); missing fields will invalidate the order. The "Notes" field is non-core (weighted 0.2); missing fields have minimal business impact. The weights sum to 1 and are determined by combining business expert ratings and historical data utilization statistics. The completeness score is calculated as follows: Completeness Score = Σ(1 - Field Missing Rate) × Field Weight, where Field Missing Rate = Number of Missing Values ​​in the Field / Total Number of Records. For example, if the missing rate for "Order ID" is 0.01 (10 missing records), the missing rate for "Payment Amount" is 0.02 (20 missing records), and all other fields are intact, the score = (1 - 0.01) × 0.8 + (1 - 0.02) × 0.2 + (1 - 0) × (1 - 0.8 - 0.2) = 0.792 + 0.196 = 0.988 (98.8 points).

[0030] Hash checksums are used to detect duplicate records. Each record is converted into a unique hash value (such as an MD5 value). This is calculated by sequentially concatenating all field values ​​in the record into a string and then applying a hash algorithm to the resulting 32-bit character string. For example, if two records have identical field values ​​(order ID "ORD001" or user ID "USR001"), and their hash values ​​are both "e10adc3949ba59abbe56e057f20f883e"), the system will mark these two records as duplicates, with the redundant position being the index of the later record (for example, if record 12 is a duplicate of record 5, record 12 will be marked as redundant).

[0031] The integrity defect matrix is ​​a structured result of integrating the above information. It consists of three components: missing value distribution (a visual transformation of the binary matrix, such as expressing the missing value rate for each column as a percentage), duplicate item coordinates (e.g., "(5,12)" indicates that the 5th and 12th records are duplicates), and a completeness score (e.g., 98.8). For e-commerce data, the missing value rate for the "Payment Amount" column in the matrix is ​​2%, the missing value rate for the "Shipping Address" column is 5%, and the duplicate item coordinates are "(8,25)" and "(30,45)." The completeness score is 96.5, which directly reflects the quality defects of the data in the integrity dimension.

[0032] Using the integrity defect matrix as the filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heat map to generate a heat map of numerical distribution anomalies. Valid samples in the integrity defect matrix are records that are not marked as missing or duplicate. For example, out of 1,000 original records, after filtering, 50 records with missing values ​​and 20 duplicate records are removed, leaving 930 valid samples. Numerical fields include "payment amount," "purchase quantity," and "unit price." The rationality of the distribution of these fields directly affects data analysis results (for example, abnormally high payment amounts may distort sales statistics).

[0033] Kernel density estimation is used to describe the actual distribution characteristics of numeric fields. Taking the "Payment Amount" field as an example, the algorithm assigns a Gaussian kernel function to each valid sample value (the kernel function bandwidth automatically adjusts based on the sample size; larger samples result in smaller bandwidths; for example, a bandwidth of 5 for 930 samples is used). By superimposing all kernel functions, a smooth probability density curve is generated. For example, if payment amounts are concentrated in the 50-200 yuan range, the kernel density curve will peak at 100 yuan, indicating that this amount is the most common transaction value.

[0034] Gaussian mixture model fitting is used to construct a theoretical distribution. The model assumes that the data follows the superposition of multiple Gaussian distributions and uses the expectation-maximization algorithm to estimate the mean, variance, and weight of each Gaussian component. For example, the theoretical distribution of "payment amount" consists of three Gaussian components: component 1 (mean 80, variance 20, weight 0.3) corresponds to low-priced goods, component 2 (mean 150, variance 30, weight 0.5) corresponds to mid-range goods, and component 3 (mean 300, variance 50, weight 0.2) corresponds to high-end goods. The fitted theoretical distribution should match the actual business scenario, for example, mid-range goods account for the highest proportion of transactions.

[0035] KL divergence (Kullback-Leibler divergence) quantifies the difference between the actual distribution and the theoretical distribution. It is calculated as KL (P||Q) = ΣP (x) log (P (x) / Q (x)), where P is the probability density of the actual distribution and Q is the probability density of the theoretical distribution. A larger divergence value indicates a more significant distribution difference (a more severe anomaly). For example, the actual distribution of "payment amount" has a high number of samples above 500 yuan, while the theoretical distribution has a very low probability in this range. The calculated KL divergence is 0.8 (divergence values ​​are typically between 0 and 2), indicating a distribution anomaly.

[0036] The Numeric Distribution Anomaly Heatmap visually displays the degree of anomaly through color depth. The horizontal axis represents numeric fields (such as "Payment Amount" and "Purchase Quantity"), while the vertical axis represents numeric ranges (such as payment amounts of 0-100 yuan, 100-200 yuan, etc.). The colors gradually change from green (divergence 0-0.3, normal), to yellow (0.3-0.6, mild anomaly), to red (above 0.6, severe anomaly). For example, the "Payment Amount" range of 500-1000 yuan has a divergence of 0.9, corresponding to a dark red color in the heatmap, clearly marking the field and value range with anomalies.

[0037] A cross-field semantic network is constructed using pre-processed data assets. The semantic similarity between fields is calculated using the knowledge graph-enhanced BERT model. When the similarity falls below the similarity threshold, it is marked as a semantic conflict. A time-series backtracking method is used to trace the propagation path of the conflict, generating a semantic conflict link that includes the conflict node, association strength, and propagation path. A cross-field semantic network is a topological structure that describes the semantic associations between fields. Nodes represent data fields, and edges represent semantic relationships between fields (such as "include" and "depends on"). Taking e-commerce data as an example, the "Shipping Address" node is connected to the "Province" and "City" nodes through a "include" relationship, while the "Order Time" and "Payment Time" nodes are connected through a "Sequential" relationship. Edge weights represent the strength of the association (ranging from 0 to 1; for example, the "Order Time - Payment Time" association has a strength of 0.9, as payment must occur after the order is placed).

[0038] The knowledge graph-enhanced BERT model is used to calculate semantic similarity. This model integrates pre-trained BERT with domain knowledge graphs (such as the "address-administrative division" graph and the "transaction-process" graph in the e-commerce sector). Entity linking links field values ​​with entities in the knowledge graph. For example, when calculating the semantic similarity between the "city" field value "Beijing" and the "province" field value "Hebei Province," the model, taking into account the fact in the knowledge graph that "Beijing is a municipality directly under the Central Government, not Hebei Province," outputs a similarity of 0.2 (normally ≥ 0.8), indicating a semantic conflict.

[0039] The similarity threshold is set based on business logic and varies for different field pairs. For example, the threshold for the "City - Province" field pair is 0.8 (strict matching of administrative divisions is required), while the threshold for the "Product Name - Product Category" field pair is 0.6 (allowing for a certain degree of fuzzy matching). When the calculated similarity falls below the threshold, a semantic conflict is flagged. For example, if "City" is "Shanghai" and "Province" is "Jiangsu Province," the similarity is 0.1, which is lower than 0.8, and is therefore flagged as a conflict.

[0040] The time series backtracking method is used to track the conflict propagation path. For example, in the case of a "payment time earlier than order time" conflict, the system starts with the conflict record (item 100) and traces back the data processing time series: During the original data entry, the "payment time" was mistakenly entered as "2025-07-25 09:00," while the order time was "2025-07-25 10:00" (a first-step error); no time series anomalies were detected during data verification (a second-step omission); and the conflict was retained during final storage (a third-step propagation). The propagation path is recorded as "entry error → verification omission → storage retention," with a correlation strength of 0.9 (a strong conflict, affecting transaction validity).

[0041] The semantic conflict link integrates the above information, including the conflicting field pair (e.g., "Payment Time - Order Time"), the correlation strength (0.9), the propagation path (input → verification → storage), and the conflicting record index (item 100). For example, a link description might be: "Conflicting field pair: City (Shanghai) - Province (Jiangsu), similarity 0.1 < 0.8, correlation strength 0.8, propagation path: user input → format verification not blocked → storage, records involved: items 58 and 72."

[0042] The integrity defect matrix is ​​converted into a visual defect map, combined with the numerical distribution anomaly heat map and semantic conflict links, and the quality scores of each dimension and the defect impact range assessment are added to generate a comprehensive quality assessment report.

[0043] The visual defect map of the integrity defect matrix uses a node-edge structure to illustrate missing and duplication defects. In the map, each node represents a field, and the size of the node is positively correlated with the missing rate of that field (for example, the "Shipping Address" node, with a 5% missing rate, is larger than the "Payment Amount" node, with a 2% missing rate). The color of the node indicates the impact of duplicate records (for example, a red node indicates a high duplication rate in records containing that field). Dashed lines on the edges indicate associations between fields due to duplicate records (for example, the dashed line between "Order ID" and "User ID" is because two duplicate records contain both fields).

[0044] The comprehensive quality assessment report integrates defect maps, distribution anomaly heatmaps, and semantic conflict links, and adds scores and impact assessments for each dimension. The quality scores for each dimension include: completeness score (e.g., 96.5), distribution rationality score (e.g., 88, based on the mean KL divergence), and semantic consistency score (e.g., 92, based on the proportion of conflicting field pairs). The total score is calculated using a weighted average (weights: completeness 0.4, distribution 0.3, semantics 0.3), for example (96.5 × 0.4 + 88 × 0.3 + 92 × 0.3) = 92.2.

[0045] The defect impact assessment describes the business impact of the defect. For example, an abnormal distribution (excessive high values) in the "Payment Amount" field may lead to an inflated sales forecast, while a semantic conflict in the "City-Province" field may cause logistics delivery errors. The report will indicate the impact level (serious / moderate / minor) and the recommended resolution priority (for example, semantic conflicts should be addressed first).

[0046] The final comprehensive quality assessment report is presented in a combination of text and graphics. For example, the report states: "The integrity defect map shows a 5% missing rate for the 'shipping address' field (medium-sized nodes) and two sets of duplicate records (connected by red dashed lines). The distribution anomaly heat map shows that the 'payment amount' range of 500-1000 yuan is dark red (serious anomaly). The semantic conflict link contains three pairs of conflicting fields, of which the 'city-province' conflict has a serious impact on logistics. Scores for each dimension: completeness 96.5, distribution 88, semantics 92, and a total score of 92.2. It is recommended to prioritize fixing semantic conflicts and high-value anomalies." S202: Based on the comprehensive quality assessment report, a policy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects. Specifically, the comprehensive quality assessment report can be parsed to extract the missing field coordinates and duplicate item identifiers in the integrity defect map, the high anomaly value intervals in the distribution anomaly heat map, and the conflicting field pairs in the semantic conflict link. These are then organized into a defect type-location-severity triple format to generate a defect-location-severity mapping table. A comprehensive quality assessment report is a comprehensive overview of data quality defects. The analysis process requires precise extraction of key defect information across all dimensions. Taking the comprehensive quality assessment report for e-commerce transaction data as an example, it includes a completeness defect map (e.g., missing "payment amount" fields, duplicate records 8 and 15), a heat map of numerical distribution anomalies (e.g., the KL divergence of "purchase quantity" in the range of 100-200 is 0.8, which is a high anomaly), semantic conflict links (e.g., the semantic similarity between "city = Guangzhou" and "province = Hunan" is 0.2 < 0.8), and scores for each dimension (85 for completeness, 78 for distribution rationality, and 80 for semantic consistency).

[0047] When extracting missing field coordinates, it's necessary to combine the visualization information in the completeness defect map to determine the missing field name and corresponding record index. For example, if the missing value in the "Shipping Address" field in the map is marked as a red node, and its associated record indexes are records 5, 12, and 20, then the missing field coordinate record is ("Shipping Address", [5, 12, 20]). Duplicate item identification requires specifying the index pair of the duplicate record. For example, if records 8 and 15, and 30 and 45 in the map are marked as duplicate links, then the duplicate item identifier is ([8, 15], [30, 45]).

[0048] Extracting high-outlier value intervals from a distribution anomaly heatmap requires considering the heatmap's color depth and the KL divergence threshold. Typically, intervals with a KL divergence ≥ 0.6 are defined as high-outliers. For example, in the heatmap for the "Purchase Quantity" field, the interval 100-200 appears dark red, corresponding to a KL divergence of 0.8 ≥ 0.6. Therefore, this interval is extracted as a high-outlier value interval ("Purchase Quantity," [100, 200]). In the "Product Unit Price" field, the interval 5000-10000 has a KL divergence of 0.5 < 0.6 and is therefore not considered a high-outlier.

[0049] To extract conflicting field pairs from semantically conflicting links, we need to focus on field combinations within the link whose similarity falls below a threshold. For example, if the similarity between "City = Shenzhen" and "Province = Guangxi Zhuang Autonomous Region" is 0.1 < 0.8, and the similarity between "Order Status = Paid" and "Payment Time = Null" is 0.3 < 0.7 (threshold 0.7), then the conflicting field pair record will be ("City - Province", "Order Status - Payment Time"), and the corresponding record indexes (items 25 and 33) will be associated.

[0050] The defect type-location-severity triples must be organized in a unified format. Defect types are categorized as "missing," "duplicated," "distribution anomaly," and "semantic conflict." Locations must be accurate to the field and record index, such as ("payment amount," [5,10]). Severity is scored on a 1-5 scale (5 being the most severe), based on the defect's impact on the business: missing core fields (such as "order ID") receive 5 points, while missing non-core fields (such as "remarks") receive 2 points. Duplicate records affecting statistical accuracy receive 3 points. Distribution anomalies that affect trend analysis (such as an abnormally high value for "payment amount") receive 4 points. Semantic conflicts leading to logical errors (such as a mismatch between "city" and "province") receive 5 points.

[0051] For example, a set of triples might be: ("missing", ("payment amount", [5,10]), 4 points), ("duplicate", ([8,15]), 3 points), ("abnormal distribution", ("purchase quantity", [100,200]), 4 points), and ("semantic conflict", ("city-province", [25,33]), 5 points). These triples are sorted by field to generate a defect-location-severity mapping table. Each record in the table corresponds to a triple, clearly displaying the specific information of each defect and providing precise targeting for subsequent remediation.

[0052] Based on the defect-location-severity mapping table and the business scenarios of the target data assets, we extract the repair constraints, including the value range of numeric fields, the format specifications of character fields, and cross-field business logic dependencies, to generate a repair constraint set. The defect-location-severity mapping table identifies the defects that need to be fixed, while the repair constraints define the boundaries of the repair operation, ensuring that the repaired data conforms to the logical rules of the business scenario. The business scenario determines the specific content of the constraints. For example, the constraints in e-commerce transactions and financial credit scenarios differ significantly: e-commerce scenarios focus more on the timing of transaction processes and the rationality of the amount, while financial scenarios emphasize the compliance of user information and the correlation with risk indicators.

[0053] Range constraints on numeric fields are the most basic type of constraint, used to limit the legal range of values ​​and prevent repair operations from introducing values ​​that violate business logic. In e-commerce scenarios, "Purchase Quantity" must be a positive integer (range ≥ 1), as it's impossible to purchase 0 or negative items. "Payment Amount" must be ≥ 0 and ≤ 100,000 RMB (assuming the platform's single order limit is 100,000 RMB). Negative values ​​or values ​​exceeding the upper limit are considered abnormal. "Discount Rate" must be between 0 and 1 (0 for no discount, 1 for the original price). Values ​​outside this range do not conform to the discount logic. For example, if a repair operation interpolates a missing value for "Purchase Quantity" to -2, this violates the range constraint and should be prohibited.

[0054] Formatting constraints for character fields ensure uniform formatting for text fields, facilitating data storage and retrieval. In e-commerce scenarios, the format for an "Order ID" is "ORD+8 digits" (e.g., "ORD20250725"). This format must include the prefix "ORD" followed by 8 digits, with all letters capitalized. A "Mobile Number" must be 11 digits (e.g., "13800138000") and must not contain any letters or symbols. A "Shipping Address" must include the province, city, district, and address (e.g., "No. 1, Kejiyuan Road, Nanshan District, Shenzhen, Guangdong Province"). Missing hierarchical information (e.g., "No. 1, Kejiyuan Road" alone) is not compliant. These specifications are often based on industry standards or internal enterprise data governance rules, and character values ​​generated by repair operations must strictly match this format.

[0055] Cross-field business logic dependency constraints are a more complex type of constraint, involving relationships between multiple fields to ensure logical consistency after repair. Typical logical dependencies in e-commerce scenarios include: "Payment Time" must be later than "Order Time" (a temporal dependency), as payment cannot occur before the order is placed; "Payment Amount" = "Unit Price" × "Purchase Quantity" × (1 - "Discount Rate") (a computational dependency), all three of which must satisfy this formula; and "Payment Time" must be present when the "Order Status" is "Paid" (a state dependency), otherwise the state and time will be inconsistent. For example, if a missing value for "Payment Time" is set to earlier than "Order Time" when repairing it, this violates the temporal dependency and must be corrected to a reasonable value later than the order time.

[0056] Extracting repair constraints requires integrating expert knowledge of the business scenario. By analyzing key nodes and rule documentation within the business process, implicit rules can be transformed into explicit constraints. For example, rules such as "Order ID Format" and "Payment Amount Limit" can be extracted from the e-commerce platform's "Order Data Specification Manual," and dependencies such as "Payment Time Later Than Order Time" can be derived from the transaction process. During the extraction process, it's important to prioritize constraints: core constraints (such as "Payment Amount ≥ 0") must not be violated, while secondary constraints (such as "Delivery Address Includes Street") can be flexible but must be met whenever possible.

[0057] The generated repair constraint set is a collection of these constraints, categorized by field type and dependency. For example, the constraint set for e-commerce scenarios includes: numerical constraints ("Purchase quantity ≥ 1," "Payment amount ≥ 0 and ≤ 100,000," "Discount rate ∈ [0, 1]"), character constraints ("Order ID format is 'ORD + 8 digits'," "Phone number is 11 digits"), and cross-field constraints ("Payment time > Order time," "Payment amount = Product price × Purchase quantity × (1 - Discount rate)," "Paid status → Payment time is not null"). Each constraint must clearly define the applicable fields, constraint content, and the handling method for violations (such as rejecting the repair or issuing a warning), providing rigid rules for subsequent repair instruction generation.

[0058] The defect-location-severity mapping table and the repair constraint set are input into the rule and learning fusion strategy generator, where the rule module generates basic repair instructions, and the learning module generates optimized repair instructions. The basic repair instructions and the optimized repair instructions are combined to obtain the initial repair operation instruction set; The policy generator, which integrates rules and learning, is the core module for generating remediation instructions. Its rule module ensures the fundamentals and compliance of remediation, while the learning module optimizes remediation results based on historical data. The integration of these two modules achieves the remediation goal of "minimum guarantee + improvement." The generator's input is a defect-location-severity mapping table (remediation target) and a set of remediation constraints (remediation boundaries). Its output is initial remediation instructions that cover all defect types.

[0059] The rule module generates basic remediation instructions based on a pre-set remediation rule library. Rules in the rule library correspond specifically to defect types, ensuring a clear solution for each defect. Rules are formulated based on the principle of "simple and effective," prioritizing industry-standard remediation methods. For example, for "missing value" defects, the rule library stipulates that core numeric fields (such as "payment amount") are interpolated using the median of all non-missing values ​​in the same field (because the median is robust to extreme values); non-core numeric fields (such as "discount rate") are interpolated using the mean; and character fields (such as "delivery address") are interpolated using the pattern (taking the most frequently occurring value). For "duplicate item" defects, the rule is "keep the first record and delete all other duplicates," as the first record is typically the original entry and is more likely to be accurate.

[0060] For "distribution anomaly" defects, the rule module develops strategies based on the anomaly type: For extremely high values ​​(e.g., "Payment Amount" = 1 million yuan, far exceeding the upper limit of 100,000 yuan), the value is truncated to the maximum value (100,000 yuan); for outlier low values ​​(e.g., "Purchase Quantity" = 0.5, which should be an integer), the value is rounded up to 1; for interval anomalies (e.g., "Purchase Quantity" is concentrated in the range of 100-200), the value is redistributed according to normal distribution probability (e.g., adjusting 150 to 50 for a smoother distribution). For "semantic conflict" defects, the rule prioritizes the "primary field modifies the secondary field" strategy. For example, in the case of a "city-province" conflict, the "city" is used to modify the "province" (because the city is more specific, for example, if "Guangzhou City" corresponds to "Guangdong Province," "Hunan Province" will be modified to "Guangdong Province").

[0061] For example, the basic instruction generated by the rule module for a missing value triple ("missing", ("payment amount", [5,10]), 4 points) is: "For the 'payment amount' field of the 5th and 10th records, use the median of the non-missing values ​​of this field (assuming the median is 299 yuan) to interpolate, and the interpolated value must be ≥0 and ≤100,000 yuan"; the instruction for the duplicate triple ("duplicate", ([8,15]), 3 points) is: "Retain the 8th record and delete the 15th record"; the instruction for the semantic conflict triple ("semantic conflict", ("city-province", [25,33]), 5 points) is: "Based on the value of the 'city' field, correct the 'province' of the 25th and 33rd records to the corresponding province (for example, 'Guangzhou City' → 'Guangdong Province')".

[0062] The learning module uses a machine learning model to learn optimal repair strategies from historical repair data and generate optimized repair instructions. The core of the learning module is a trained repair effect prediction model. This model uses "defect characteristics - repair method - repair effect" as training data (for example, historically, KNN interpolation of missing values ​​in the "payment amount" field has been more effective than median interpolation). It inputs the characteristics of the current defect (such as missing rate, field type, and distribution characteristics) and outputs the repair method with the best predicted effect. For example, when the missing rate of the "payment amount" field is less than 10%, the model predicts that KNN interpolation (based on the amounts of similar orders) will have a lower error (MAE = 5 yuan) than median interpolation (MAE = 15 yuan), and then generates optimized KNN interpolation instructions.

[0063] The training data for the learning module comes from the company's data repair logs, containing 10,000 repair records from the past six months. Each record includes the defect type, field name, missingness rate, historical repair method, and post-repair accuracy metrics (such as the error between the interpolated value and the actual value). The model uses the random forest algorithm (because it can handle mixed-type features and resists overfitting). Features include defect severity, field importance, and data distribution characteristics. The target variable is repair accuracy (such as the inverse of the error). After training, the model must achieve an accuracy of ≥85% on the test set to ensure the effectiveness of the predicted repair method.

[0064] For example, the learning module predicts that KNN interpolation (taking the average amount of the five most similar orders) is more effective for missing values ​​in the "payment amount" field (missing rate 8%, field importance score 4) through the model, and generates an optimization instruction: "For the 'payment amount' field of the 5th and 10th records, use KNN interpolation (k=5), referring to the order amounts of similar 'unit price' and 'purchase quantity'. The interpolated value must satisfy the following formula: payment amount = unit price × purchase quantity × (1 - discount rate)". For the abnormal distribution of "purchase quantity" (100-200 range), the model predicts that cluster correction (assigning abnormal interval values ​​to the adjacent normal cluster center) is more effective than simple truncation, and generates an instruction: "For records with 'purchase quantity' in the range of 100-200, according to the K-means clustering (k=3) results, correct the value to the corresponding cluster center (for example, 150→50)."

[0065] When merging basic and optimized repair instructions, possible conflicts must be resolved. If the optimized and basic repair instructions address the same defect and do not conflict (e.g., both use different interpolation methods), the optimized instruction is retained (due to its superior performance). If the optimized instruction violates the repair constraints (e.g., the interpolated value is outside the range), the optimized instruction is discarded and the basic instruction is retained. If the two instructions address different defects, both are retained. For example, if both the basic instruction (median interpolation) and the optimized instruction (KNN interpolation) for a missing "Payment Amount" field meet the constraints, the optimized instruction is retained. If the interpolated value of the optimized instruction is -50 yuan (violating the ≥0 constraint), the median interpolation of the basic instruction (299 yuan) is retained.

[0066] The merged initial repair operation instruction set includes all valid instructions. Each instruction must clearly specify the operation object (field, record index), operation type (interpolation, deletion, correction, etc.), operation parameters (such as the k value of KNN = 5), and constraint checking requirements. For example, the initial instruction set may include: "KNN interpolation 'payment amount' [5,10], k = 5, constraint ≥ 0", "Delete duplicate records

[15] ", "Correct 'province' [25,33] to 'Guangdong Province'", "Cluster correction 'purchase quantity' [100-200], k = 3", covering all extracted defects and laying the foundation for subsequent conflict detection and screening.

[0067] Conflict detection is performed on the initial repair operation instruction set. Instructions with a priority higher than a preset level are retained using a greedy algorithm. The repair cost of each instruction is calculated, and invalid instructions with a cost exceeding a cost threshold are filtered out to generate a candidate repair operation instruction set. The initial repair instruction set may contain conflicts and invalid instructions. Conflict detection and cost filtering are key steps to ensure the feasibility of the repair. A conflict occurs when two or more instructions operate on the same record in the same field, resulting in incompatible results (e.g., one instruction interpolates the "payment amount" to 299 yuan, while another instruction corrects it to 399 yuan). An invalid instruction is one where the repair cost is too high (e.g., the repair benefit is lower than the cost) or where it cannot be executed (e.g., the interpolated value consistently violates a constraint).

[0068] Conflict detection is performed by comparing the instruction's operands (field + record index). If two instructions have identical operands, a conflict is detected. For example, instruction A: "Correct article 25 of 'Province' to 'Guangdong Province'" and instruction B: "Correct article 25 of 'Province' to 'Guangxi Zhuang Autonomous Region'" both operate on ("Province", 25), but the results differ, resulting in a conflict. Conflict detection covers the entire initial instruction set, using a double loop to iterate over all instruction pairs to ensure no omissions. The detection frequency matches the instruction set generation frequency (e.g., once every 100 records).

[0069] A greedy algorithm is used to resolve conflicts. Its core principle is to prioritize instructions with higher priority. Priority is determined by the severity of the defect (higher severity scores indicate higher priority). The default level is typically set at 3 (on a 1-5 scale). Instructions with a score above this level (4-5) are considered high priority and must be retained; instructions with a score of 3 or less are considered low priority and can be replaced. For example, if there are conflicting instructions A (severity 5, semantic conflict) and B (severity 3, distribution anomaly), A will be retained because it has a higher priority, while B will be discarded.

[0070] If conflicting instructions have the same priority (e.g., a score of 5), the "operation relevance" check is used to determine: if the instructions target the upstream and downstream logic of the same field (e.g., "payment amount" and "unit price"), the instruction for the core field (e.g., "payment amount") is retained; if they are unrelated, one is randomly selected and retained (this decision will require expert judgment in practice). For example, if two instructions with a score of 5 each modify the same record for "payment amount" and "order ID," the instruction for "payment amount" is retained (as it has a greater impact on transaction validity).

[0071] The repair cost calculation comprehensively considers time cost, precision loss, and business impact. The higher the cost, the less worthwhile the instruction is to execute. Time cost refers to the computing resources required to execute the instruction (for example, KNN interpolation costs more than median interpolation due to the need to calculate similarity), and is expressed in milliseconds per record (for example, KNN interpolation = 10ms / record, median interpolation = 2ms / record). Precision loss refers to the expected error between the repaired value and the true value (for example, the precision loss of clustering correction is higher than that of truncation due to the potential deviation from the original data), and is expressed on a scale of 0-1 (0 = no loss, 1 = complete error). Business impact refers to the disruption of the repair operation to downstream businesses (for example, deleting duplicate records may affect statistical reports, with a business impact of 0.3).

[0072] The cost calculation formula is: Repair Cost = (Time Cost × 0.2) + (Precision Loss × 0.5) + (Business Impact × 0.3), where the weights reflect the importance of each factor (Precision Loss has the highest weight). For example, the time cost of KNN interpolation of "Payment Amount [5,10]" is 10ms / item × 2 items = 20ms (converted to 0.2), the precision loss is 0.1 (small error), and the business impact is 0.1 (small impact on downstream users). Therefore, the cost is 0.2 × 0.2 + 0.1 × 0.5 + 0.1 × 0.3 = 0.04 + 0.05 + 0.03 = 0.12.

[0073] The cost threshold is set based on the maximum acceptable cost for the business, typically 0.5 (in the 0-1 range). Instructions exceeding this threshold are considered invalid. For example, a command to correct an anomaly in the "purchase quantity" distribution requires invoking a complex deep learning model (time cost = 0.8), with a precision loss of 0.4 (possibly due to deviation from actual results after correction), and a business impact of 0.6 (affecting inventory statistics). The cost is 0.8 × 0.2 + 0.4 × 0.5 + 0.6 × 0.3 = 0.16 + 0.2 + 0.18 = 0.54. If this value exceeds 0.5, the command is considered invalid and filtered.

[0074] After conflict detection and cost filtering, a candidate repair operation instruction set is generated. The instructions in this set all meet the following requirements: no conflict, priority level (higher than a preset level), and cost ≤ threshold. For example, the candidate instruction set might retain: "KNN interpolation of 'payment amount' [5, 10]," "delete duplicate records

[15] ," and "correct 'province' [25, 33]" (all conflict-free, with costs of 0.12, 0.05, and 0.1, respectively, and priorities of 4-5). Conflicting instruction B and high-cost distribution anomaly repair instructions are filtered out to ensure that subsequent instructions are feasible and efficient.

[0075] According to the defect severity, repair urgency and operation relevance corresponding to each instruction in the candidate repair operation instruction set, the hierarchical analysis method is used to calculate the priority weight, and the priority sequence is generated by sorting the weights. The sequence is bound to the candidate instruction set to output the repair operation instruction set and the priority sequence.

[0076] The candidate remediation action set identifies the remediation actions to be performed, but the order in which they are performed affects remediation efficiency and effectiveness. Prioritizing remediation of high-severity, high-urgency defects can reduce business losses. Prioritizing closely related actions (for example, remediating the "payment amount" before remediating the dependent "discount rate") can avoid duplication of actions. The Analytic Hierarchy Process (AHP) quantifies the impact of each factor and scientifically calculates priority weights to ensure a reasonable ranking.

[0077] The first step in the AHP process is to construct a hierarchical structure. The objective layer is to prioritize repairs. The criterion layer includes three factors: defect severity (impact on data quality), repair urgency (loss of delayed repair), and operation relevance (dependencies with other instructions). The solution layer includes each instruction in the set of candidate repair operation instructions. For example, the weights of the three factors in the criterion layer are determined through pairwise comparison to reflect their importance in the prioritization process.

[0078] Criteria-level weights are determined on a 1-9 scale: 1 indicates both factors are equally important, 3 indicates slightly more important than the former, 5 indicates significantly more important, 7 indicates strongly more important, and 9 indicates extremely important. The reciprocal scale is used for any inverse of this scale. In e-commerce scenarios, "defect severity" (on a 5-point scale) is slightly more important than "fix urgency" (due to the more far-reaching impact of severe defects) and is assigned a scale of 3. "Fix urgency" is significantly more important than "operational relevance" (due to delayed fixes potentially causing order delays) and is assigned a scale of 5. The combined scale of "severity" and "relevance" is 3 × 5 = 15 (9 is chosen as the highest possible value). By constructing a judgment matrix and calculating the eigenvectors, the criterion-level weights are: severity 0.5, urgency 0.3, and relevance 0.2 (the sum of the weights is 1).

[0079] At the solution level, each instruction must be scored under each criterion. The defect severity score is directly based on the score in the triplet (1-5). For example, instruction A (semantic conflict) receives 5 points, and instruction B (duplicate) receives 3 points. The urgency of remediation is determined by the defect type: semantic conflicts (causing logical errors) and missing core fields (such as "payment amount") receive the highest urgency (5 points); distribution anomalies (affecting trends but not critical) receive the next highest urgency (3 points); and duplications (merely redundant) receive the lowest urgency (2 points). The operational dependency score measures the number of dependencies an instruction has on other instructions: if instruction C must be executed after instruction A (e.g., "discount rate" depends on "payment amount"), A's dependency score is increased by 1, and C's dependency score remains unchanged. Instructions with no dependencies receive a score of 0, and the more dependencies, the higher the score (maximum 5 points).

[0080] For example, there are 4 instructions in the candidate instruction set: Directive 1: Modify the "City-Province" conflict (Severity 5, Urgency 5, Linked 1 → Score 1); Instruction 2: KNN interpolation of “payment amount” (severity 4 points, urgency 5 points, 2 associations → score 2); Instruction 3: Delete duplicate records (severity 3 points, urgency 2 points, relevance 0 → score 0); Instruction 4: Cluster correction of “Purchase Quantity” (Severity 4 points, Urgency 3 points, 1 association → Score 1).

[0081] For each instruction, calculate the weighted sum of its score under each criterion and the criterion weight, that is, priority weight = (severity score / 5) × 0.5 + (urgency score / 5) × 0.3 + (relevance score / 5) × 0.2 (all scores are normalized to 0-1).

[0082] Instruction 1: (5 / 5) × 0.5 + (5 / 5) × 0.3 + (1 / 5) × 0.2 = 0.5 + 0.3 + 0.04 = 0.84; Instruction 2: (4 / 5) × 0.5 + (5 / 5) × 0.3 + (2 / 5) × 0.2 = 0.4 + 0.3 + 0.08 = 0.78; Instruction 3: (3 / 5) × 0.5 + (2 / 5) × 0.3 + 0 × 0.2 = 0.3 + 0.12 + 0 = 0.42; Instruction 4: (4 / 5) × 0.5 + (3 / 5) × 0.3 + (1 / 5) × 0.2 = 0.4 + 0.18 + 0.04 = 0.62.

[0083] Sorting by priority weight from high to low results in the following priority sequence: Instruction 1 (0.84) → Instruction 2 (0.78) → Instruction 4 (0.62) → Instruction 3 (0.42). This sequence ensures that the most serious and urgent semantic conflicts are fixed first, followed by addressing missing core fields, and finally addressing less impactful duplications and distribution anomalies.

[0084] Bind the priority sequence to the candidate instruction set and output the repair operation instruction set and priority sequence. In the binding result, each instruction has a corresponding priority ranking, for example: 1. Correct "city-province" [25,33]; 2. KNN interpolation "payment amount" [5,10]; 3. Cluster correction "purchase quantity" [100-200]; 4. Delete duplicate records

[15] . This result clarifies the order of repair and provides a clear operational guide for the multi-expert collaborative repair execution module, ensuring an efficient and orderly repair process.

[0085] Furthermore, the priority sequence must be verified for plausibility: checking for obvious logical inversions (e.g., repairing dependencies before dependents). If so, manual adjustments are required (e.g., if instruction C depends on instruction A, and A comes after C, the order must be swapped). Once verified, the final set of repair instructions and the priority sequence serve as input to the multi-expert collaborative repair execution module, initiating the actual repair process.

[0086] S203, input the target data asset, the repair operation instruction set and the priority sequence into a multi-expert collaborative repair execution module, which calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

[0087] Specifically, the repair operation instruction set and priority sequence can be parsed, and tasks can be assigned to corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type. The index and priority ranking of the pending tasks of each model can be recorded to generate a model-task allocation table; The repair operation instruction set and priority sequence serve as a guide for collaborative repair by multiple experts. The parsing process transforms abstract instructions into specific, executable tasks. The repair operation instruction set includes the instruction type (e.g., "numerical interpolation," "semantic reconstruction," "rule correction"), the operation object (field name, record index), and the operation parameters (e.g., the k value for the interpolation algorithm, the threshold for rule correction). The priority sequence is sorted by weight (e.g., [semantic reconstruction instruction 1, numerical interpolation instruction 2, rule correction instruction 3]), clarifying the execution order.

[0088] During parsing, each instruction's type label is first identified: instructions containing the keywords "missing value" and "interpolation" are classified as "numeric interpolation"; instructions containing the keywords "semantic conflict" and "field pair" are classified as "semantic reconstruction"; and instructions containing the keywords "distribution anomaly" and "interval adjustment" are classified as "rule modification." For example, the instruction "Perform KNN interpolation (k=5) on the 'payment amount' field in records 5 and 10" is labeled "numeric interpolation"; the instruction "Correct the semantic conflict in the 'city-province' field in records 25 and 33" is labeled "semantic reconstruction."

[0089] The multi-expert collaborative repair execution module includes three expert models, each with clear functional boundaries: the numerical interpolation expert model specializes in missing value repair and supports algorithms such as KNN and spatiotemporal attention interpolation; the semantic reconstruction expert model focuses on semantic conflict correction, relying on knowledge graphs and pre-trained language models; and the rule correction expert model is responsible for distribution anomaly adjustment, based on a business rule library and optimization algorithms. Task assignment must strictly match instruction type and model functionality to avoid cross-model assignment (for example, semantic conflict instructions cannot be assigned to the numerical interpolation model).

[0090] When recording pending task indices, it's important to clearly identify the record location for each instruction. For example, a numerical interpolation instruction involves records 5 and 10 in the "Payment Amount" field, and the task index is recorded as ("Payment Amount," [5, 10]); a semantic reconstruction instruction involves records 25 and 33 in the "City-Province" field pair, and the task index is recorded as ("City-Province," [25, 33]). Priority sorting directly follows the weight order of the priority sequence. For example, if semantic reconstruction instruction 1 has a weight of 0.84 and numerical interpolation instruction 2 has a weight of 0.78 in the priority sequence, the semantic reconstruction task will have a higher priority than the numerical interpolation task.

[0091] The model-task assignment table is a structured result of integrating the above information. It contains five columns: model name, task type, task index, priority weight, and operation parameters. Taking e-commerce data repair as an example, the assignment table might be: the numerical interpolation expert model corresponds to the "numerical interpolation" task, task index ("payment amount", [5, 10]), priority 0.78, parameters "k = 5, spatiotemporal attention window = 3"; the semantic reconstruction expert model corresponds to the "semantic reconstruction" task, task index ("city-province", [25, 33]), priority 0.84, parameters "knowledge graph = e-commerce geographic graph, pre-trained model = BERT-base"; the rule correction expert model corresponds to the "rule correction" task, task index ("purchase quantity", [100-200]), priority 0.62, parameters "threshold range = 1-50, gradient learning rate = 0.01".

[0092] After the allocation table is generated, it must be validated: the task index is checked to ensure it is within the valid record range (e.g., the record index cannot exceed 1000), the parameters meet the model requirements (e.g., the k value must be a positive integer), and the priority is consistent with the sequence. If a task index is incorrect (e.g., the 1001st record for "Payment Amount"), it is marked as invalid and returned to the strategy generator to regenerate instructions to ensure that all tasks in the allocation table are executable.

[0093] Using the numerical interpolation expert model, the missing value locations to be processed are obtained according to the model-task allocation table. The interpolation values ​​are calculated using the spatiotemporal attention interpolation algorithm. Cross-validation is performed to ensure that the interpolation values ​​conform to the distribution law. The interpolated data value fragments are output. The core of the numerical interpolation expert model is to accurately fill missing values ​​and ensure that the imputed values ​​conform to the inherent distribution of the data. The model first extracts the locations of the missing values ​​to be processed from the model-task assignment table. For example, ("Payment Amount", [5,10]) indicates that the "Payment Amount" field in records 5 and 10 is missing and requires interpolation. This location information must be aligned with the index of the original data asset to avoid positioning errors (such as confusing row and column indices).

[0094] The spatiotemporal attention interpolation algorithm is an advanced method for handling missing time series data. It simultaneously considers "spatial similarity" (correlations between other records with the same field) and "temporal correlation" (trends at adjacent time points), dynamically weighting each through an attention mechanism. Taking the case of missing "payment amount" in an e-commerce application as an example, spatial similarity refers to records with similarities to the missing record in fields such as "unit price," "purchase quantity," and "user level" (e.g., records with a user level of VIP and a purchase quantity of 2). Temporal correlation refers to the trend of "payment amount" in the three records before and after the missing record (e.g., record 3 shows an upward trend of 199 yuan, record 4 shows 299 yuan, and record 6 shows 399 yuan).

[0095] The attention mechanism assigns weights by calculating similarity scores: For the spatial dimension, cosine similarity is used to calculate the distance between the feature vectors of the record to be interpolated and the other records (e.g., the similarity between item 5 and item 7 is 0.9, with a weight of 0.6). For the temporal dimension, an exponential decay function is used to calculate the weight of the time difference (e.g., if the time difference between item 5 and item 4 is 1 minute, the weight is 0.3; if the time difference between item 5 and item 6 is 1 minute, the weight is 0.1). The interpolated value is the weighted sum of these values. For example, the interpolated value of "Payment Amount" for item 5 is (399 yuan for item 7 × 0.6) + (299 yuan for item 4 × 0.3) + (399 yuan for item 6 × 0.1) = 399 × 0.6 + 299 × 0.3 + 399 × 0.1 = 239.4 + 89.7 + 39.9 = 369 yuan.

[0096] Cross-validation is used to verify the rationality of the imputed values. A 5-fold cross-validation approach is used: the non-missing "payment amount" data is randomly divided into five groups, with each group serving as the validation set in turn (assuming the third group contains 100 records). Missing values ​​are artificially introduced for each record in the validation set, and the interpolation algorithm is used to impute the missing values. The mean absolute error (MAE) between the imputed value and the true value is calculated. If the MAE is less than or equal to a preset threshold (e.g., 20 yuan), the algorithm is stable; if the MAE is 15 yuan and less than 20 yuan, the imputed value is considered valid. For example, for the imputed value of 369 yuan for record 5, cross-validation shows that the MAE for the same type of interpolation is 12 yuan, which meets the requirements.

[0097] In addition, it is necessary to check whether the interpolated values ​​conform to the distribution law: calculate the difference in mean and standard deviation between the interpolated values ​​and the original non-missing values ​​(for example, the original mean is 300 yuan, the standard deviation is 100 yuan; the interpolated value is 369 yuan, which is within the range of mean ±1 standard deviation (200-400 yuan)), and does not violate the repair constraints (such as "payment amount ≥ 0", "payment amount = unit price of goods × purchase quantity × (1 - discount rate)", 369 yuan = 199 yuan × 2 × (1-0.1) = 358.2 yuan, the error is within 5%, which is considered reasonable).

[0098] The interpolated data value segments are partial datasets containing the interpolation results. For example, the "Payment Amount" field in records 5 and 10 changes from missing to 369 yuan and 299 yuan, respectively, while all other fields remain unchanged. Each segment must be marked with an interpolation identifier (e.g., "Interpolation_Spatiotemporal Attention") for easy traceability. The output segments are then integrated with other repaired segments to form the complete repaired data.

[0099] Using the semantic reconstruction expert model, we receive the field data corresponding to the semantic conflict link, call the pre-trained language model enhanced by the knowledge graph to generate candidate field values ​​that conform to the semantic logic, filter the candidate values ​​that do not conform to the business terms through the domain dictionary, select the one with the highest confidence as the reconstruction result, and output the semantically corrected data fragment; The Semantic Reconstruction Expert Model focuses on resolving cross-field semantic conflicts. Its input is the field data corresponding to the semantic conflict link. For example, for conflicting records with "City = Guangzhou" and "Province = Hunan" (items 25 and 33), the "Province" field value must be reconstructed to match the "City" field value. The core of the model is to combine the knowledge graph with a pre-trained language model to generate candidate values ​​that conform to semantic logic.

[0100] The knowledge graph-enhanced pre-trained language model is based on BERT and integrates domain knowledge graphs (e.g., the e-commerce geographic knowledge graph, which includes "city-province" mappings: Guangzhou → Guangdong Province, Changsha → Hunan Province, etc.). The model input is a text sequence with conflicting field pairs (e.g., "City: Guangzhou, Province:?"). The knowledge graph entity linking layer maps "Guangzhou" to the "Guangzhou" entity in the graph, activating its associated "Province = Guangdong Province" attribute. The BERT model then generates candidate province values.

[0101] When generating candidate field values, the model uses a masked language model (MLM) mechanism to predict the missing correct value. For example, given the input text "City is Guangzhou, Province is [MASK]," the model outputs candidate values ​​and confidence scores: "Guangdong Province" (confidence 0.95), "Guangxi Zhuang Autonomous Region" (0.03), and "Hunan Province" (0.02). Confidence is calculated by the model's softmax layer and reflects the reliability of the prediction; higher values ​​indicate greater semantic consistency.

[0102] The domain dictionary is used to filter candidate values ​​that do not conform to business terms. The e-commerce geography domain dictionary contains the standard names of all provinces and cities nationwide (for example, "Guangdong Province" and "Guangzhou City" are legal terms, while "Guang Province" and "Guang City" are illegal terms). For example, if the model generates the candidate value "Guang Province," the domain dictionary detects that the term does not exist and directly filters it out; legal terms such as "Guangdong Province" and "Guangxi Zhuang Autonomous Region" are retained.

[0103] The candidate value with the highest confidence is selected as the reconstruction result. For example, in the 25th record, "Guangdong Province" has the highest confidence of 0.95, so the "Province" field is reconstructed from "Hunan Province" to "Guangdong Province"; in the 33rd record, "City = Shenzhen City" conflicts with "Province = Guangxi Zhuang Autonomous Region", so the model generates the candidate value "Guangdong Province" (confidence 0.98), and the "Province" is reconstructed to "Guangdong Province".

[0104] After reconstruction, semantic similarity must be verified: Using the knowledge graph-enhanced BERT model, the semantic similarity of "city-province" is recalculated. The original conflicting pair had a similarity of 0.2. After reconstruction, the similarity of "Guangzhou City - Guangdong Province" is 0.95, which is greater than the threshold of 0.8, resolving the conflict. Furthermore, the reconstructed value is checked for compliance with business rules (for example, "province" must be the full name of a provincial-level administrative region, and "Guangdong Province" complies with the standard).

[0105] A semantically corrected data segment is a local dataset containing the reconstruction results. For example, the "Province" field in records 25 and 33 is changed from "Hunan Province" and "Guangxi Zhuang Autonomous Region" to "Guangdong Province." The segment is marked with a reconstruction identifier (e.g., "Reconstructed_Knowledge Graph + BERT"). The segment must preserve the correspondence between the original and reconstructed values ​​(e.g., "Original: Hunan Province → Reconstructed: Guangdong Province") to facilitate auditing and backtracking. The output segment will be used in subsequent integration steps to ensure that semantic conflicts are fully resolved.

[0106] Using the rule-corrected expert model, we adjust the abnormal values ​​based on the threshold range in the business rule library for the numerical interval marked by the distribution anomaly heat map, use the gradient descent method to fine-tune the values ​​to minimize the distribution deviation, and output the rule-corrected data fragments. The rule correction expert model is used to handle distribution anomalies in numeric fields. Its input is the high anomaly interval marked by the distribution anomaly heat map. For example, the KL divergence of "purchase quantity" in the range of 100-200 is 0.8 (high anomaly). The value in this range needs to be adjusted to conform to the normal distribution (for example, the range of 1-50 is the main distribution, with a KL divergence of 0.1).

[0107] The business rule library serves as the foundation for rule modification. It contains the legal threshold ranges for numeric fields, distribution characteristic parameters (such as mean and standard deviation), and business logic constraints. For "Purchase Quantity," the rule library defines: legal thresholds range from 1 to 100 (a single order typically purchases no more than 100 items); a normal distribution with a mean of 5 and a standard deviation of 3 (most orders purchase 1-10 items); and business logic constraints such as "Purchase Quantity = Integer" and "Purchase Quantity × Unit Price ≤ Payment Amount."

[0108] First, we preliminarily adjusted the outlier values ​​based on the threshold range: truncated the values ​​in the 100-200 range to the upper threshold limit of 100. For example, the 45th record "Purchase Quantity = 150" was adjusted to 100, and the 60th record "Purchase Quantity = 200" was adjusted to 100. However, simple truncation may lead to a new concentration of the distribution (too many values ​​near 100), so further fine-tuning is required.

[0109] Gradient descent is used to minimize distribution deviation. The objective function is the KL divergence between the actual distribution and the theoretical normal distribution, i.e., min KL (P||Q), where P is the adjusted actual distribution and Q is the theoretical normal distribution (mean 5, standard deviation 3). Gradient descent parameters include the learning rate (0.01, to control the adjustment amplitude) and the number of iterations (100 to ensure convergence).

[0110] For example, for a value initially adjusted to 100, its contribution to the KL divergence is calculated (100 deviates significantly from the mean of 5, contributing 0.3). The gradient is positive (the value needs to be reduced to reduce the deviation). The adjustment amount for each iteration = learning rate × gradient = 0.01 × 0.3 = 0.003, gradually moving 100 closer to 5. However, based on business rules, the adjusted value must be an integer and ≥ 1. After 100 iterations, 100 may be adjusted to 50 (still within the legal range and closer to a normal distribution).

[0111] For record 45, "Purchase quantity = 150 → Initial adjustment to 100 → Gradient descent fine-tuning to 50," 50 is now an integer between 1 and 100, and its deviation from the mean of 5 is less than 100. The KL divergence decreases from 0.8 to 0.2 (≤ the acceptable threshold of 0.3). The business logic is also checked: 50 × unit price (199 yuan) = 9,950 yuan ≤ payment amount (10,000 yuan), which meets the constraints.

[0112] A rule correction data segment is a local dataset containing the adjustment results. For example, the "Purchase Quantity" in records 45 and 60 changes from 150 and 200 to 50 and 60, respectively, while other fields remain unchanged. The segment must be labeled with the adjustment method (e.g., "rule truncation + gradient descent") and the distribution parameters before and after the adjustment (e.g., KL divergence 0.8 before adjustment, 0.2 after adjustment) to demonstrate that the distribution anomaly has been improved. The output segment is integrated with other repaired segments to complete the quality restoration of the distribution dimension.

[0113] The interpolated data numerical fragments, semantically corrected data fragments and rule-corrected data fragments are integrated in priority sequence to obtain a complete data set as the repaired data asset.

[0114] The interpolated data numerical fragments, semantically corrected data fragments and rule-corrected data fragments are local repair results. The integration process needs to follow the priority sequence to ensure the timing rationality and data consistency of the repair, and finally generate a complete repaired data asset.

[0115] The priority sequence determines the order of integration: high-priority segments must be overwritten with the original data first to prevent low-priority repairs from overwriting high-priority results. For example, if the priority sequence is [semantic correction segment (weight 0.84), numerical interpolation segment (0.78), rule correction segment (0.62)], the semantic correction segment will be integrated first, followed by the numerical interpolation segment, and finally the rule correction segment.

[0116] During integration, it is necessary to process overlapping records between fragments: if different fragments involve different fields of the same record, they are directly merged (for example, in the 25th record, the semantic correction fragment adjusts the "province", and the numerical interpolation fragment does not involve this record, so the results of both are directly retained); if the same field is involved, only the results of the high-priority fragment are retained (for example, assuming that the 5th record is involved in both numerical interpolation and rule correction, the numerical interpolation has a higher priority, so the interpolation result is retained).

[0117] Data consistency verification is key to integration, ensuring cross-field logic consistency: Check whether "Payment Amount = Product Unit Price × Purchase Quantity × (1 - Discount Rate)" is true. For example, after interpolation of the fifth record, "Payment Amount = 369 yuan", "Product Unit Price = 199 yuan", "Purchase Quantity = 2", and "Discount Rate = 0.1", 199 × 2 × 0.9 = 358.2 yuan. The error between 369 yuan and 358.2 yuan is ≤5% (within the business tolerance range) and is considered consistent; check "Payment Time > Order Time". All records after semantic correction meet this condition; check that "Purchase Quantity" is an integer in the range of 1-100. All values ​​after rule correction meet this requirement.

[0118] A completeness check ensures that all flagged defects have been fixed: Comparing the defect-location-severity mapping table, missing values ​​at positions [5, 10] have been interpolated, duplicates [8, 15] have been deleted, the distribution anomaly range [100-200] has been adjusted, and the semantically conflicting field pair [25, 33] has been reconstructed. There are no unfixed defects. Furthermore, the statistical repair coverage (number of repaired defects / total number of defects) must be ≥ 95%. This repair coverage is 100%, meeting the requirement.

[0119] A repaired data asset is a complete dataset containing all repair results. For example, out of 1,000 records, 50 missing values ​​are interpolated, 20 duplicates are removed, 30 distribution anomalies are corrected, and 40 semantic conflicts are reconstructed, ensuring that all fields conform to business rules and distribution patterns. The dataset must include repair metadata: the repair time (e.g., "2025-07-25 14:30:00"), the repair method (e.g., "spatiotemporal attention interpolation," "knowledge graph + BERT reconstruction"), and the quality scores before and after the repair (e.g., completeness score increases from 85 to 98, semantic consistency score increases from 80 to 96).

[0120] The final output of repaired data assets must be in a standardized format (e.g., CSV format, with commas as field delimiters and strings enclosed in double quotes) to facilitate direct use by downstream systems (e.g., data analysis platforms and business systems). A repair report should also be generated, documenting a comparison of key metrics before and after the repair, providing a basis for continuous improvement of data quality.

[0121] It can be seen that the target data assets to be processed are received and input into the multi-dimensional joint quality scoring engine to generate a comprehensive quality assessment report including an integrity defect map, a distribution anomaly heat map and a semantic conflict link; based on the comprehensive quality assessment report, a strategy generator that integrates rules and learning is used to parse the repair requirements and constraints contained in the defect map, anomaly heat map and conflict link, and dynamically generate repair operation instruction sets and priority sequences for different types of defects; the target data assets, repair operation instruction sets and priority sequences are input into the multi-expert collaborative repair execution module, the defects in the target data assets are repaired according to the priority sequence, and the repaired data assets are output, thereby achieving comprehensive and accurate data quality assessment and improving the automation level and accuracy of data repair.

[0122] Another embodiment of the present invention provides an AI-based data asset quality assessment and automatic repair system, see Figure 3 , the system may include: Evaluation module 301 is used to receive the target data assets to be processed and input them into the multi-dimensional joint quality scoring engine. The engine performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report including integrity defect maps, distribution anomaly heat maps, and semantic conflict links. The parsing module 302 is configured to parse the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links based on the comprehensive quality assessment report and utilize a policy generator that integrates rules and learning to dynamically generate repair operation instruction sets and priority sequences for different types of defects. The repair module 303 is used to input the target data assets, the repair operation instruction set and the priority sequence into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data assets according to the priority sequence, and outputs the repaired data assets.

[0123] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0124] Specifically, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for performing the following steps: S201, converting an input large data set into a quantized data set at a communication transmitting end; S202, extracting chaotic features from the quantized data set to generate an initial symmetric key; S203, performing deep chaotic transformation enhancement on the initial symmetric key to obtain an enhanced symmetric key; S204, performing an encryption operation on the quantized data set using the enhanced symmetric key to obtain an encrypted data set; S205, encoding the encrypted data set into a quantum bit carrier, and transmitting it to a communication receiving end through quantum entanglement, so that the communication receiving end can decrypt and calculate a data integrity index to verify the integrity and consistency of the data.

[0125] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0126] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0127] Specifically, in this embodiment, the processor may be configured to execute the following steps through a computer program: S201, converting an input large data set into a quantized data set at a communication transmitting end; S202, extracting chaotic features from the quantized data set to generate an initial symmetric key; S203, performing deep chaotic transformation enhancement on the initial symmetric key to obtain an enhanced symmetric key; S204, performing an encryption operation on the quantized data set using the enhanced symmetric key to obtain an encrypted data set; S205, encoding the encrypted data set into a quantum bit carrier, and transmitting it to a communication receiving end through quantum entanglement, so that the communication receiving end can decrypt and calculate a data integrity index to verify the integrity and consistency of the data.

[0128] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the scope of implementation of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which do not exceed the spirit covered by the description and drawings, should be within the scope of protection of the present invention.

Claims

1. An AI-based data asset quality assessment and automatic repair method, characterized in that: The method comprises: Receive the target data assets to be processed and input them into the multi-dimensional joint quality scoring engine, which performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes integrity defect maps, distribution anomaly heat maps, and semantic conflict links; Based on the comprehensive quality assessment report, a policy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects. The target data asset, the repair operation instruction set and the priority sequence are input into a multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

2. The method according to claim 1, characterized in that The target data assets to be processed are received and input into a multi-dimensional joint quality scoring engine, which performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report containing an integrity defect map, a distribution anomaly heat map, and semantic conflict links, including: Receive the target data assets to be processed, analyze their data structure, use an adaptive sliding window filtering algorithm to remove high-frequency noise, extract the data generation time, source identifier, and field dependencies to build a metadata index table, and output the preprocessed data assets and metadata index table; Based on the metadata index table, the integrity check is performed in parallel through multiple threads. A binary matrix notation method is used to record the missing value location of each field. The integrity score is calculated based on the field importance weight. A hash check is used to detect duplicate records and mark redundant locations. This generates an integrity defect matrix containing the missing value distribution, duplicate item coordinates, and integrity scores. Using the integrity defect matrix as the filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heat map to generate a heat map of numerical distribution anomalies. A cross-field semantic network is constructed using pre-processed data assets. The semantic similarity between fields is calculated using the knowledge graph-enhanced BERT model. When the similarity falls below the similarity threshold, it is marked as a semantic conflict. A time-series backtracking method is used to trace the propagation path of the conflict, generating a semantic conflict link that includes the conflict node, association strength, and propagation path. The integrity defect matrix is ​​converted into a visual defect map, combined with the numerical distribution anomaly heat map and semantic conflict links, and the quality scores of each dimension and the defect impact range assessment are added to generate a comprehensive quality assessment report.

3. The method according to claim 2, characterized in that Based on the comprehensive quality assessment report, the strategy generator that integrates rules and learning is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links, and dynamically generate repair operation instruction sets and priority sequences for different types of defects, including: Parse the comprehensive quality assessment report, extract missing field coordinates and duplicate identifiers from the integrity defect map, high anomaly value intervals from the distribution anomaly heat map, and conflicting field pairs from the semantic conflict link, organize them into a defect type-location-severity triple format, and generate a defect-location-severity mapping table. Based on the defect-location-severity mapping table and the business scenarios of the target data assets, we extract the repair constraints, including the value range of numeric fields, the format specifications of character fields, and cross-field business logic dependencies, to generate a repair constraint set. The defect-location-severity mapping table and the repair constraint set are input into the rule and learning fusion strategy generator, where the rule module generates basic repair instructions, and the learning module generates optimized repair instructions. The basic repair instructions and the optimized repair instructions are combined to obtain the initial repair operation instruction set; Conflict detection is performed on the initial repair operation instruction set. Instructions with a priority higher than a preset level are retained using a greedy algorithm. The repair cost of each instruction is calculated, and invalid instructions with a cost exceeding a cost threshold are filtered out to generate a candidate repair operation instruction set. According to the defect severity, repair urgency and operation relevance corresponding to each instruction in the candidate repair operation instruction set, the hierarchical analysis method is used to calculate the priority weight, and the priority sequence is generated by sorting the weights. The sequence is bound to the candidate instruction set to output the repair operation instruction set and the priority sequence.

4. The method according to claim 3, characterized in that The target data asset, the repair operation instruction set, and the priority sequence are input into a multi-expert collaborative repair execution module. The module calls a corresponding numerical interpolation expert model, a semantic reconstruction expert model, or a rule correction expert model according to the instruction set, repairs defects in the target data asset according to the priority sequence, and outputs a repaired data asset, including: Parse the repair operation instruction set and priority sequence, assign tasks to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type, record the pending task index and priority ranking of each model, and generate a model-task allocation table; Using the numerical interpolation expert model, the missing value locations to be processed are obtained according to the model-task allocation table. The interpolation values ​​are calculated using the spatiotemporal attention interpolation algorithm. Cross-validation is performed to ensure that the interpolation values ​​conform to the distribution law. The interpolated data value fragments are output. Using the semantic reconstruction expert model, we receive the field data corresponding to the semantic conflict link, call the pre-trained language model enhanced by the knowledge graph to generate candidate field values ​​that conform to the semantic logic, filter the candidate values ​​that do not conform to the business terms through the domain dictionary, select the one with the highest confidence as the reconstruction result, and output the semantically corrected data fragment; Using the rule-corrected expert model, we adjust the abnormal values ​​based on the threshold range in the business rule library for the numerical interval marked by the distribution anomaly heat map, use the gradient descent method to fine-tune the values ​​to minimize the distribution deviation, and output the rule-corrected data fragments. The interpolated data numerical fragments, semantically corrected data fragments and rule-corrected data fragments are integrated in priority sequence to obtain a complete data set as the repaired data asset.

5. An AI-based data asset quality assessment and automatic repair system, characterized by: The system comprises: The evaluation module receives the target data assets to be processed and feeds them into a multi-dimensional joint quality scoring engine. The engine performs data integrity verification, value distribution rationality analysis, and cross-field semantic dependency verification in parallel, generating a comprehensive quality assessment report that includes an integrity defect map, distribution anomaly heat map, and semantic conflict links. An analysis module is used to analyze the repair requirements and constraints contained in the defect map, anomaly heat map, and conflict links based on the comprehensive quality assessment report and utilize a policy generator that integrates rules and learning to dynamically generate repair operation instruction sets and priority sequences for different types of defects; The repair module is used to input the target data asset, the repair operation instruction set and the priority sequence into the multi-expert collaborative repair execution module. The module calls the corresponding numerical interpolation expert model, semantic reconstruction expert model or rule correction expert model according to the instruction set, repairs the defects in the target data asset according to the priority sequence, and outputs the repaired data asset.

6. The system according to claim 5, characterized in that The evaluation module is specifically used to: Receive the target data assets to be processed, analyze their data structure, use an adaptive sliding window filtering algorithm to remove high-frequency noise, extract the data generation time, source identifier, and field dependencies to build a metadata index table, and output the preprocessed data assets and metadata index table; Based on the metadata index table, the integrity check is performed in parallel through multiple threads. A binary matrix notation method is used to record the missing value location of each field. The integrity score is calculated based on the field importance weight. A hash check is used to detect duplicate records and mark redundant locations. This generates an integrity defect matrix containing the missing value distribution, duplicate item coordinates, and integrity scores. Using the integrity defect matrix as the filtering condition, kernel density estimation and Gaussian mixture model fitting are performed on the numerical fields of valid samples. The KL divergence between the actual distribution and the theoretical distribution is calculated, and the divergence value is mapped to the color depth of the heat map to generate a heat map of numerical distribution anomalies. A cross-field semantic network is constructed using pre-processed data assets. The semantic similarity between fields is calculated using the knowledge graph-enhanced BERT model. When the similarity falls below the similarity threshold, it is marked as a semantic conflict. A time-series backtracking method is used to trace the propagation path of the conflict, generating a semantic conflict link that includes the conflict node, association strength, and propagation path. The integrity defect matrix is ​​converted into a visual defect map, combined with the numerical distribution anomaly heat map and semantic conflict links, and the quality scores of each dimension and the defect impact range assessment are added to generate a comprehensive quality assessment report.

7. The system according to claim 6, characterized in that The parsing module is specifically used to: Parse the comprehensive quality assessment report, extract missing field coordinates and duplicate identifiers from the integrity defect map, high anomaly value intervals from the distribution anomaly heat map, and conflicting field pairs from the semantic conflict link, organize them into a defect type-location-severity triple format, and generate a defect-location-severity mapping table. Based on the defect-location-severity mapping table and the business scenarios of the target data assets, we extract the repair constraints, including the value range of numeric fields, the format specifications of character fields, and cross-field business logic dependencies, to generate a repair constraint set. The defect-location-severity mapping table and the repair constraint set are input into the rule and learning fusion strategy generator, where the rule module generates basic repair instructions, and the learning module generates optimized repair instructions. The basic repair instructions and the optimized repair instructions are combined to obtain the initial repair operation instruction set; Conflict detection is performed on the initial repair operation instruction set. Instructions with a priority higher than a preset level are retained using a greedy algorithm. The repair cost of each instruction is calculated, and invalid instructions with a cost exceeding a cost threshold are filtered out to generate a candidate repair operation instruction set. According to the defect severity, repair urgency and operation relevance corresponding to each instruction in the candidate repair operation instruction set, the hierarchical analysis method is used to calculate the priority weight, and the priority sequence is generated by sorting the weights. The sequence is bound to the candidate instruction set to output the repair operation instruction set and the priority sequence.

8. The system according to claim 7, characterized in that The repair module is specifically used to: Parse the repair operation instruction set and priority sequence, assign tasks to the corresponding expert models in the multi-expert collaborative repair execution module according to the instruction type, record the pending task index and priority ranking of each model, and generate a model-task allocation table; Using the numerical interpolation expert model, the missing value locations to be processed are obtained according to the model-task allocation table. The interpolation values ​​are calculated using the spatiotemporal attention interpolation algorithm. Cross-validation is performed to ensure that the interpolation values ​​conform to the distribution law. The interpolated data value fragments are output. Using the semantic reconstruction expert model, we receive the field data corresponding to the semantic conflict link, call the pre-trained language model enhanced by the knowledge graph to generate candidate field values ​​that conform to the semantic logic, filter the candidate values ​​that do not conform to the business terms through the domain dictionary, select the one with the highest confidence as the reconstruction result, and output the semantically corrected data fragment; Using the rule-corrected expert model, we adjust the abnormal values ​​based on the threshold range in the business rule library for the numerical interval marked by the distribution anomaly heat map, use the gradient descent method to fine-tune the values ​​to minimize the distribution deviation, and output the rule-corrected data fragments. The interpolated data numerical fragments, semantically corrected data fragments and rule-corrected data fragments are integrated in priority sequence to obtain a complete data set as the repaired data asset.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 4 when run.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

Images