A big data quality evaluation and repair method based on deep learning

Abstract

The application belongs to the technical field of computers, and particularly relates to a big data quality evaluation and repair method based on deep learning, which comprises the following steps: collecting multi-source heterogeneous data and performing semantic alignment integration; extracting features by using a deep network containing a self-attention mechanism, and identifying abnormalities based on reconstruction error; constructing a multi-dimensional index system to quantitatively evaluate quality; performing intelligent repair and semantic verification on abnormal data based on a generative adversarial network and a bidirectional long short-term memory network; and verifying repair effect and updating governance strategy. Through the above scheme, the application realizes an automatic closed loop from abnormality discovery to repair, significantly improves the detection rate of hidden errors and repair consistency, and guarantees processing efficiency while taking into account data privacy and security.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a method for big data quality assessment and repair based on deep learning. Background Technology

[0002] With the widespread application of big data technology, data has become a core element driving intelligent transformation and scientific decision-making in industries. During the collection, storage, and transmission of massive amounts of data, data quality directly impacts the accuracy and reliability of subsequent analysis results. High-quality data is not only the foundation for building accurate predictive models but also a prerequisite for ensuring the stable operation of business systems. Therefore, establishing an efficient data governance system to ensure the authenticity and usability of data has become a key focus of current data science research.

[0003] Among these technologies, big data quality assessment and repair, with its powerful feature representation capabilities, is gradually becoming a key means of solving complex data defect problems. This technology utilizes neural network models to automatically learn the deep distribution patterns of massive amounts of data, aiming to intelligently identify and accurately repair missing values, outliers, and logical conflicts in the dataset. By constructing a high-dimensional feature mapping space, this technology can achieve automated quality monitoring and data correction in large-scale dynamic data streams, thereby enhancing the overall value of data assets.

[0004] However, traditional quality assessment models often face the problem of insufficient feature extraction when dealing with high-dimensional sparse data, resulting in a low detection rate of hidden data defects. Meanwhile, the multi-source heterogeneous nature of big data scenarios makes the relationships between different dimensions extremely complex, and existing methods lack the ability to semantically align cross-domain features, making it difficult to achieve deep integration of assessment criteria and repair logic. Furthermore, existing repair mechanisms mostly rely on static mapping strategies, which cannot effectively capture the temporal dependencies and nonlinear fluctuations in the data evolution process. This can lead to secondary biases or semantic distortions in the repaired data, severely limiting the effectiveness of data application in complex business scenarios.

[0005] Therefore, a method for big data quality assessment and repair based on deep learning is desired. Summary of the Invention

[0006] The purpose of this invention is to provide a method for big data quality assessment and repair based on deep learning, which can effectively solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A deep learning-based method for big data quality assessment and remediation, the method comprising the following steps: S1. Collect and integrate multi-source heterogeneous big data to obtain a standardized initial dataset; S2. Extract high-dimensional features from the initial dataset and identify abnormal data based on reconstruction errors; S3. Construct a multidimensional evaluation model to quantitatively evaluate the quality of the dataset and generate a quality evaluation report; S4. An intelligent repair model constructed using generative adversarial networks and bidirectional long short-term memory networks is used to repair and correct the identified abnormal data; S5. Verify the repair effect, and encapsulate the effective repair logic and parameters into a dynamic governance strategy, store it in a knowledge base built on a graph database, and realize the closed-loop update of the governance strategy.

[0008] Preferably, in step S2, the identification of abnormal data based on reconstruction error specifically includes: Construct an autoencoder structure that includes an encoder and a decoder, wherein the encoder shares parameters with the deep feature extraction network; The data items are input into the autoencoder, and its reconstruction error is calculated. The calculation formula is: ; in, Represents the original input vector. This represents the output vector reconstructed by the decoder. Let be the dimension of the vector; The reconstruction error of the data item to be tested is compared with the baseline distribution obtained by training on normal samples and the preset outlier threshold. If it exceeds the threshold, the data item is determined to be abnormal data.

[0009] Preferably, in step S3, the multidimensional evaluation model includes evaluation indicators in five dimensions: completeness, accuracy, consistency, timeliness, and uniqueness. Among them, the completeness index The calculation formula is:

[0010] in, This indicates the total number of records in the dataset. This indicates the total number of fields contained in the record. Indicates the first Each field has a preset importance weight. This is an indicator function.

[0011] Preferably, in step S4, the loss function of the intelligent repair model Defined as mean squared error loss and adversarial losses Weighted sum: ; in, and These are the preset weighting coefficients. This is used to constrain the repair value to be numerically close to the true value. This is used to constrain the repair data to maintain an overall distribution consistent with the original normal data.

[0012] Preferably, in step S5, verifying the repair effect includes calculating the information entropy of the repaired dataset and comparing it with the information entropy before repair. If the repair causes an abnormal increase in information entropy, a manual review mechanism is triggered.

[0013] Preferably, in step S2, the deep feature extraction network is a deep neural network containing a self-attention mechanism layer; and / or, in step S4, the bidirectional long short-term memory network consists of a forward LSTM layer and a backward LSTM layer, used to generate an enhanced feature representation containing past and future contextual information of data items.

[0014] Preferably, when the multi-source heterogeneous big data includes images or video streams, in step S2, a hybrid architecture of convolutional neural network and visual Transformer is used for feature extraction, and peak signal-to-noise ratio and structural similarity index are calculated as measures of reconstruction error; in step S4, a super-resolution reconstruction or image inpainting algorithm based on generative adversarial network is used for data inpainting.

[0015] Preferably, when the multi-source heterogeneous big data is spatiotemporal trajectory data, a spatiotemporal graph convolutional network is used for feature extraction in step S2; in step S4, when the generator of the intelligent repair model generates the repair trajectory, it needs to ensure that the trajectory conforms to the constraints defined based on the urban road network topology and vehicle dynamics model.

[0016] Preferably, after performing intelligent repair in step S4 and before outputting the data, the method further includes a step of desensitizing sensitive information in the repaired data. The desensitization uses an algorithm that meets differential privacy requirements to add controlled Laplace noise to the data.

[0017] Preferably, the method employs hardware co-optimization at runtime, including: decomposing model operators using computation graph optimization techniques for parallel execution on tensor processing units; reading data from a distributed message queue using remote direct memory access techniques; and accelerating the matching and retrieval process of policies in the knowledge base using FPGA hardware accelerators.

[0018] Compared with the prior art, the present invention has the following beneficial effects: By introducing deep residual networks and self-attention mechanisms, this invention can extract high-dimensional nonlinear features from massive heterogeneous data that are difficult to detect using traditional statistical methods. Compared with traditional detection methods based on heuristic rules, this invention significantly improves the detection rate of hidden logical errors and minor noise interference, and the accuracy of anomaly identification reaches a preset high precision level.

[0019] The established multidimensional quality indicator system covers the core dimensions of the data lifecycle. Weights are dynamically determined using the analytic hierarchy process (AHP), addressing the issues of single evaluation dimensions and strong subjectivity in existing technologies. The evaluation results are accurate down to individual data items, providing a reliable basis for subsequent precise repair decisions.

[0020] By leveraging the collaborative work of generative adversarial networks and bidirectional long short-term memory networks, this invention achieves a fully automated closed loop from anomaly detection to automatic repair. The generative model can automatically complete missing information based on contextual semantics, avoiding the semantic distortion problems caused by traditional static mapping methods. The semantic consistency of the repaired data is significantly improved compared to existing technologies.

[0021] By establishing a remediation effectiveness verification process and a dynamic governance knowledge base, this invention can continuously optimize governance strategies based on remediation feedback. This self-learning mechanism enables the system to cope with constantly changing data distributions and new anomaly patterns in a big data environment, ensuring the long-term effectiveness of the governance logic and reducing manual maintenance costs.

[0022] By deeply integrating distributed computing nodes and graphics processing unit (GPU) acceleration technologies, this invention possesses the capability to process massive amounts of data. With a preset throughput and inference latency within a predetermined range, it meets the stringent requirements of real-time big data stream processing, significantly improving execution efficiency compared to traditional solutions.

[0023] By employing a distributed multi-replica mechanism and differential privacy algorithms, this invention ensures efficient data governance while simultaneously maintaining system stability and data privacy security. The repaired data effectively prevents the leakage of sensitive information while maintaining extremely high availability, thus complying with data security compliance requirements. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall technical solution architecture of the big data quality assessment and repair method based on deep learning proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework for high-dimensional feature extraction based on self-attention mechanism and residual network in this invention; Figure 3 This is a flowchart illustrating the logical process of intelligent data repair based on generative adversarial networks and bidirectional long short-term memory networks in this invention. Detailed Implementation

[0025] Example 1 To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0026] In the deep learning-based big data quality assessment and repair method, step S1 involves collecting and integrating multi-source heterogeneous big data. The specific execution logic of this step is as follows: First, a distributed data acquisition architecture is constructed, deployed in a cluster environment composed of multiple computing nodes interconnected via a 10 Gigabit high-speed fiber optic network. At the physical layer, each computing node is equipped with multiple high-performance processors and a large-capacity synchronous dynamic random access memory, ensuring sufficient computing and caching resources when processing massive concurrent data streams. At the software logic layer, the distributed data acquisition architecture adopts a Master-Worker control mode, where the Master node is responsible for global task scheduling and load balancing, dynamically allocating acquisition tasks based on the real-time CPU utilization, remaining memory, and network bandwidth throughput of each Worker node. Worker nodes execute specific acquisition instructions, establishing persistent connections with external data sources through a pre-defined asynchronous non-blocking input / output mechanism.

[0027] For structured databases, such as relational database management systems, worker nodes call specific database driver interfaces, utilize multi-threaded parallel scanning technology to acquire incremental data, and encapsulate the read data stream into standardized data packets according to a preset concurrency scale. For semi-structured log files, such as web server logs or system operation logs, a regular expression-based streaming parsing engine is used to capture appended content at the end of the file in real time and convert it into a structured key-value pair format. For unstructured real-time streaming data sources, such as binary streams generated by sensors or video surveillance metadata, a dedicated stream processing adapter is used to truncate and encapsulate the data according to a predetermined sampling frequency.

[0028] During data integration, semantic alignment techniques based on ontology mapping are used to unify data patterns. Specifically, the system predefines a globally shared conceptual model, which includes all core entities, attributes, and logical relationships within the business domain. When data from different sources enters the integration engine, the system first extracts their metadata tags and uses natural language processing to calculate the semantic similarity between tags. By constructing an ontology mapping matrix, fields from different business systems with inconsistent naming conventions are mapped to standard fields in the globally shared conceptual model. For example, fields from different business systems with inconsistent naming conventions are mapped to standard fields in the globally shared conceptual model. " "and "User ID" are uniformly mapped to " ".

[0029] The integrated data stream is cached and transmitted via a distributed message queue. This message queue employs a multi-replica fault-tolerant mechanism, dividing the data into multiple partitions and storing them on different physical disks. Each partition is configured with a predetermined number of follower replicas. When the primary replica experiences a hardware failure, the system automatically switches to a follower replica, ensuring that the reliability of the data acquisition process meets predetermined standards. Finally, after cleaning, transformation, and alignment, a standardized and integrated initial dataset is obtained.

[0030] In the above method, step S2 extracts high-dimensional features and identifies abnormal data.

[0031] The initial dataset is input into a pre-constructed deep feature extraction network. This network consists of an input layer, a predetermined number of residual blocks, a self-attention mechanism layer, a global average pooling layer, and an output layer connected sequentially. The input layer is responsible for receiving the standardized raw data vector and mapping it to a high-dimensional continuous vector space.

[0032] Each residual block contains two convolutional layers and one skip connection branch. The convolutional layers use 3x3 or 1x1 kernels to capture correlations in the data in local space or feature dimensions. Following the convolutional operation are batch normalization layers and linear rectified function activation layers to accelerate network convergence and enhance non-linear expressive power. The skip connection branch directly adds the input of the residual block to the output, effectively mitigating the vanishing gradient problem during deep neural network training.

[0033] The self-attention mechanism layer models long-range dependencies in a data sequence by calculating the inner product of the query vector, key vector, and value vector. Specifically, the system multiplies the feature matrix output from the residual block with three learnable weight matrices to obtain the Query, Key, and Value matrices. By calculating the product of the transpose of the Query and Key, and then normalizing it using Softmax, an attention weight allocation map is obtained. This weight map reflects the strength of the correlation between different dimensions of features in the data item, enabling the model to automatically focus on the feature dimensions that contribute more to the quality assessment.

[0034] Deep feature extraction networks generate high-dimensional feature vectors that represent the essential attributes of the original data through multiple layers of abstraction. Based on the reconstruction error distribution, missing values, noise points, and logical inconsistencies in the dataset are identified. Specifically, a reconstruction network based on an autoencoder structure is constructed, where the encoding part shares parameters with the aforementioned feature extraction network, and the decoding part attempts to restore the high-dimensional feature vectors to the original input. The reconstruction error is calculated as follows: Let the original input vector be The restored vector is Reconstruction error Defined as: ; The system trains on a large number of normal samples to obtain a baseline distribution of reconstruction errors. During the actual detection phase, the reconstruction error of each data item is calculated and compared to a preset outlier threshold. If the reconstruction error exceeds this threshold, the data item is considered anomalous. For example, missing values ​​can cause significant deviations in the corresponding dimensions of the reconstructed vector; noise points can disrupt the manifold structure of the feature space, leading to a decrease in overall reconstruction accuracy. Logically contradictory items, such as negative ages or employment dates earlier than birth dates, will also produce significant reconstruction errors because they do not conform to the statistical patterns learned by the model. Finally, the system outputs a list of anomalous data to be repaired, recording the physical location, anomaly type, and preliminary severity of the anomalous data.

[0035] In the above method, step S3 involves constructing a multidimensional evaluation model for quality quantification.

[0036] A multi-dimensional quality assessment index system is established, encompassing completeness, accuracy, consistency, timeliness, and uniqueness. The completeness index is calculated by the ratio of non-null records to the total number of records in the dataset, reflecting the comprehensiveness of information collection. The accuracy index is derived by comparing data items with authoritative reference libraries or prior rules, such as verifying whether an ID number conforms to the national standard checksum logic. The consistency index is derived through cross-table and cross-system logical rule verification, ensuring that the attribute descriptions of the same entity maintain a logical closed loop across different data sources. The timeliness index is calculated by the difference between the timestamp of data generation and the timestamp of data entry into the governance system; the smaller the difference, the more real-time valuable the data. The uniqueness index is derived by hashing key fields and performing conflict detection, used to identify duplicate records.

[0037] A weighted fusion algorithm is used to discriminate the extracted high-dimensional features. First, the weight coefficients of each dimension are determined using the analytic hierarchy process (AHP). Domain experts are invited to conduct pairwise comparisons of the importance of the five indicators based on business requirements, constructing a judgment matrix. The largest eigenvalue and its corresponding eigenvector of this matrix are calculated, and after normalization, the weight vector is obtained. In practical applications, accuracy and consistency are given higher weights, and their sum is set to a predetermined weight ratio of 0.6.

[0038] Calculate the quality score for each data item. Combine the high-dimensional feature vector extracted by S2 with the judgment logic of each quality indicator, and output the raw scores for each dimension through a multilayer perceptron network.

[0039] For each data item, extract its corresponding high-dimensional features from step S2, input them into the trained multilayer perceptron, and output its original score vector in each dimension: ; in, Indicates the first The score vector for each data item; Then, the original scores are multiplied by the corresponding weighting coefficients and summed to obtain the overall quality score of the data item.

[0040] Weighted composite quality score function: .

[0041] Furthermore, for all data items Kernel density estimation is used to fit the probability density curve of the quality scores. The overall health of the dataset is assessed by analyzing its peak (average quality level), skewness (whether the quality distribution is symmetrical), and variance (quality volatility).

[0042] Generate a quality heat map, mapping the quality score of data items to multiple dimensions such as business line, data production time, and data source system for cross-display, intuitively revealing "when, where, and what business" the data quality is worst.

[0043] In the above method, step S4 performs intelligent repair and data correction.

[0044] A data repair model is constructed based on generative adversarial networks (GANs) and bidirectional long short-term memory (LSTM) networks. The GAN consists of two core components: a generator and a discriminator.

[0045] The generator employs a deep transposed convolutional structure, taking as input a contextual feature vector of the outlier data and a noise vector conforming to a specific distribution. The contextual feature vector contains information about the normal data surrounding the outlier, providing semantic constraints for the repair process. The generator's task is to learn the latent distribution of the real data and generate candidate repair values ​​capable of filling in missing information or correcting errors.

[0046] The discriminator employs a deep classification network containing multiple convolutional layers, pooling layers, and fully connected layers. Its task is to receive a data sample and output the probability that the sample belongs to the real data. During training, the generator attempts to deceive the discriminator, making it unable to distinguish between the repaired data and the real, normal data; while the discriminator continuously improves its discriminative ability.

[0047] To handle large datasets with time-series characteristics, a bidirectional long short-term memory (LSTM) network is integrated into the generative network. The bidirectional LSTM network consists of a forward LSTM and a backward LSTM, capable of simultaneously capturing both past and future information of data items along the timeline. For each data point at each time step, the forward layer calculates the hidden state from the beginning to the present, and the backward layer calculates the hidden state from the end to the present; the two are concatenated as the enhanced feature representation for that point. This structure is particularly effective for repairing sensor data or transaction logs with continuous characteristics.

[0048] Loss function during the repair process Combined with mean squared error loss and adversarial losses The definition is as follows: ; in, and These are the preset weighting coefficients. The mean squared error loss ensures that the repaired values ​​are numerically close to the expected values ​​of the true distribution, while the adversarial loss ensures that the repaired data maintains a high degree of consistency with the original dataset in terms of overall distribution and semantic logic.

[0049] The system employs backpropagation and the Adam optimizer to iteratively update model parameters. In each training round, the generator is first fixed, and the discriminator's weights are updated to enhance its classification accuracy. Subsequently, the discriminator is fixed, and the generator's weights are updated to reduce its adversarial loss. The model is considered optimal when the probability that the discriminator cannot distinguish between the repaired data and the real data stabilizes at a preset equilibrium state of around 0.5. At this point, the abnormal data to be repaired and its context are input into the model, and the generator outputs the optimal repair value, achieving intelligent imputation of missing data and automatic correction of erroneous data.

[0050] In the above method, step S5 verifies the repair effect and updates the governance strategy.

[0051] The repaired dataset is re-input into the quality assessment model described in S3. The system recalculates various quality indicators such as completeness, accuracy, and consistency, and compares them with the initial indicators before the repair. The improvement rate is calculated, and if the overall improvement rate of all indicators exceeds the preset accuracy threshold, the repair is deemed effective.

[0052] The verification process also includes calculating the data entropy value. By analyzing the changes in information entropy of the dataset before and after the repair, it is ensured that the repair operation eliminates anomalies without introducing additional uncertainty or false information. If the data entropy increases abnormally after repair, a manual review mechanism is triggered.

[0053] If the repair results meet the preset requirements, the logic, model parameters, and feature mapping relationships used in this repair will be encapsulated into a dynamic governance strategy. This strategy includes cleaning rules for specific data sources, repair weights for specific anomaly types, and evaluation criteria for specific business scenarios.

[0054] The encapsulated dynamic governance strategies are stored in a knowledge base built on a graph database. The knowledge base uses governance strategies as nodes and data feature similarity and business relevance as edges, constructing a complex network of strategy relationships. When new raw data enters the system, the system first extracts the feature fingerprints of the new data and performs a similarity search in the knowledge base. If a historical governance record with a high degree of matching is found, the relevant repair logic and parameter configurations are directly reused, achieving closed-loop evolution and dynamic updates of the governance strategies. This knowledge base-based feedback mechanism enables the system to continuously absorb successful governance experiences and improve its ability to respond to new data anomalies.

[0055] Example 2 Based on Example 1, this example details the specific application of this method in the big data platform of the smart city operation center.

[0056] The smart city platform receives heterogeneous data in real time from multiple fields such as traffic monitoring, environmental monitoring, public safety, and municipal management. It can process over 100 terabytes of data per day, according to a pre-set processing capacity.

[0057] When executing S1, the distributed data acquisition architecture collects monitoring data from over 1 million sensors in real time through edge computing nodes deployed throughout the city. Semantic alignment technology unifies the proprietary protocol data output by traffic cameras from different manufacturers into a standard urban geographic information vector format. The distributed message queue uses a Kafka cluster configured with 3 replicas to ensure data reliability under extreme network fluctuations.

[0058] During S2 execution, the deep feature extraction network uses a self-attention mechanism to identify the spatial correlation between traffic flows at different intersections based on the traffic flow data. Reconstruction error distribution detection revealed an abnormal surge in traffic flow data for a certain road segment during the early morning hours, with the reconstruction error significantly exceeding the preset outlier threshold. The system determined that this anomaly was caused by transient noise generated by strong electromagnetic interference affecting the sensors.

[0059] During S3, the constructed evaluation model quantified the data for that road segment. Due to noise interference, the accuracy score for this period was only 0.2, causing the overall quality score to drop below the preset passing threshold. The evaluation report automatically triggered the S4 repair process.

[0060] During S4 execution, the repair model invokes a pre-trained generative adversarial network. The generator combines historical traffic flow distribution patterns for the same period on that road segment with real-time traffic flow trends from surrounding related road segments to generate smoothed repair values. A bidirectional long short-term memory network ensures that the repaired traffic flow curve maintains the physical continuity of vehicle movement over time. The probability of the repaired data matching the actual traffic flow distribution reaches 0.49, meeting the repair requirements.

[0061] When S5 was executed, the repaired data was re-evaluated, and the accuracy index improved to over 0.985, exceeding the preset accuracy threshold. The system encapsulates the repair parameters for this type of "electromagnetic interference-type flow noise" into a new strategy and stores it in the Neo4j graph database knowledge base. If similar anomalies occur in other areas subsequently, the system will automatically invoke this strategy for repair within milliseconds.

[0062] In this embodiment, multiple high-performance graphics processing units (GPUs) are used to perform parallel acceleration computation on the deep learning model. The data volume for processing in a single batch is set to 2048 records, and the model inference latency is controlled within a preset time range of less than 50 milliseconds. When a large-scale event is held in the city, causing a surge in the amount of data to be processed, exceeding 80% of the current cluster's capacity, the system automatically triggers the online addition of new computing nodes through the container cloud platform, achieving horizontal elastic scaling of computing power.

[0063] Furthermore, this embodiment performs automatic desensitization of sensitive information before outputting the repaired data. A differential privacy algorithm is employed to add controlled Laplace noise to the repaired statistical data according to a preset privacy budget. For example, when outputting repaired pedestrian density data for a certain area, noise is added to ensure that the precise location of a specific individual cannot be deduced in reverse, thereby improving data quality, ensuring the accuracy of urban operational decisions, and strictly protecting citizens' privacy. The desensitized data meets the preset k-anonymity requirements, and the deviation of key statistical characteristics is controlled within a preset allowable error range of less than 2%.

[0064] Example 3 Based on the above embodiments, this embodiment demonstrates the specific implementation details of this method in the scenario of predictive maintenance of equipment on an industrial internet platform.

[0065] Vibration sensors, temperature sensors, and pressure sensors in industrial environments generate high-frequency sampling data. In S1, the system acquires the raw sampling sequence through an industrial protocol gateway (such as Modbus-TCP or OPC-UA). Due to the complexity of the industrial environment, data often becomes intermittent and missing due to signal attenuation.

[0066] In S2, the deep feature extraction network is specifically optimized for one-dimensional time series. Dilated convolution is used in the residual block to expand the receptive field, thereby capturing long-range features within the equipment's operating cycle. Reconstruction error calculation revealed that the vibration signal of a certain bearing was missing in a specific frequency band, resulting in a significant increase in the Euclidean distance between the reconstructed vector and the original signal.

[0067] In S3, the multidimensional evaluation model focuses on consistency metrics. The system verifies the physical constraints between vibration frequency and motor speed, discovering that missing vibration data leads to logical inconsistencies. The quality score indicates that this batch of data is in an "unusable" state.

[0068] In S4, a generative adversarial network is used for signal completion. The generator predicts the vibration amplitude of the missing segment based on the phase information extracted by Bi-LSTM. The discriminator verifies the authenticity of the repaired signal from the perspective of spectral distribution, ensuring that the repaired signal does not produce distortion in the frequency domain characteristics.

[0069] In S5, the experience database plays a crucial role. This database has stored over 1 million historical equipment failure and maintenance records. The system utilizes a reinforcement learning algorithm, using the quality of the repaired signal as a reward signal to continuously optimize the hyperparameters of the repair model. Through a large-scale search of the experience database, the system discovered that for this type of bearing vibration deficiency, employing a specific learning rate decay strategy can shorten the model evolution cycle to less than 4 hours.

[0070] The restored high-value industrial big data was used in subsequent fault prediction models, significantly improving the accuracy of equipment downtime prediction from 0.75 in the original data to 0.96. This demonstrates that the proposed method has extremely high engineering application value and robustness in dealing with data quality issues in complex industrial environments.

[0071] Example 4 Based on the above embodiments, this embodiment describes the application of this method in the compliance review of cross-border financial transactions.

[0072] Financial transaction data has extremely high consistency requirements. In S1, the system integrates transaction records from different clearing organizations and different monetary systems around the world. Due to differences in interbank communication protocols among countries, transaction messages often exhibit field misalignment or semantic ambiguity.

[0073] When executing S2, the deep feature extraction network uses the encoder part of the Transformer architecture, employing a multi-head attention mechanism to capture the complex relationships between the transferor, intermediary bank, and recipient in the transaction chain. Reconstruction error analysis identifies a logical contradiction between the currency information and exchange rate field in a large cross-border transfer, triggering an alert due to the reconstruction error.

[0074] In S3, the evaluation model sets the consistency metric for this transaction to an extremely low value. The weighted fusion algorithm, based on financial regulatory requirements, increases the consistency weight to 0.8. The overall score indicates that this data item carries a risk of money laundering or is suspected of being entered incorrectly.

[0075] In S4, the intelligent repair model combines the historical message structure of the SWIFT standard to perform logical shifting and correction on misaligned fields, and corrects erroneous local currency amounts based on the real-time exchange rate interface at the time of the transaction. The discriminator ensures that the repaired message complies with the compliance verification logic of the International Financial Telecommunications Association (IFTA).

[0076] In S5, the verification process incorporates knowledge graph-based association verification. The system retrieves the historical behavioral patterns of the transaction entity from the knowledge base, confirming that the corrected logic aligns with the entity's consistent transaction characteristics. The dynamic governance strategy then updates the exchange rate conversion logic for specific niche currencies and synchronizes it to governance nodes across global branches.

[0077] By implementing this embodiment, financial institutions can significantly improve the accuracy of anomaly detection when processing large-scale cross-border transaction data, and significantly reduce compliance risks and manual auditing costs.

[0078] Example 5 This embodiment further refines the hardware co-optimization scheme for the deep feature extraction network and the data repair model.

[0079] In a large-scale deployment environment, to support the real-time processing requirement of more than 2 million records per second, the system introduces a heterogeneous computing resource pool at the hardware level. During the deep learning inference stages of S2 and S4, the system uses computation graph optimization technology to decompose model operators into micro-operations that can be executed in parallel on tensor processing units (TPUs).

[0080] To address the large-scale matrix multiplication in the self-attention mechanism, a block-based computation strategy is adopted, dividing the Query, Key, and Value matrices into sub-blocks adapted to the cache size, significantly reducing the frequency of data exchange between main memory and the processor. Between the distributed message queue and the compute nodes, Remote Direct Memory Access (RDMA) technology is used, allowing Worker nodes to directly read data packets from the queue server's memory, bypassing complex operating system protocol stack processing and reducing data access latency by more than 60%.

[0081] In the knowledge base retrieval phase of S5, to accelerate policy matching, the system utilizes an FPGA hardware accelerator to implement hash indexing and vector similarity calculation. When the feature vector of the data to be processed enters the retrieval phase, the FPGA completes the matching with millions of policies in the knowledge base within nanoseconds, ensuring end-to-end real-time performance of the entire quality governance process.

[0082] Furthermore, the system is equipped with robust exception handling and fault tolerance mechanisms. If a deep learning model experiences memory overflow or computation timeout during inference, the system automatically redirects the batch of data to a backup lightweight rule engine for preliminary processing, while simultaneously recording error logs, triggering automatic model restart and state recovery. This multi-level degradation governance scheme ensures that even with a surge in big data traffic or partial hardware failure, the system can still maintain basic data quality governance capabilities and maintain uninterrupted business operations.

[0083] Example 6 This embodiment details the mathematical calculation process and engineering implementation details of each indicator in the multidimensional evaluation model.

[0084] In S3, the completeness index The calculation formula is as follows:

[0085] in, To record the total, Total number of fields For fields Importance weights This is an indicator function. In engineering implementation, this computational task is assigned to an aggregation operator in a distributed computing framework, which uses MapReduce to count non-empty items in each partition in parallel.

[0086] Accuracy indicators A probabilistic graphical model-based decision-making method is adopted. For numerical fields, the system uses features extracted by S2 and combines them with the mean and variance of historical distributions to calculate the probability that the current value falls into the high-confidence interval. For enumerated fields, a hash forest structure is used to quickly compare the set of valid values.

[0087] Consistency Indicators The validation is performed through a pre-defined logical rule engine. For example, in e-commerce data governance, the rule engine validates the equation "Total order amount = Unit price of goods × Quantity + Shipping fee - Discount amount" in real time. If the equation is not true, the consistency score is deducted according to the deviation ratio.

[0088] Timeliness indicators The system considers transmission latency, processing latency, and storage latency throughout the data lifecycle. High-precision timestamps are embedded at the data source, and arrival and departure times are recorded at each processing node. The system evaluates its ability to support real-time decision-making by calculating a weighted average latency.

[0089] Uniqueness Indicator A Bloom filter is used for initial screening. For records suspected of being duplicates, a deeper comparison is performed using a similarity algorithm based on edit distance or Jaro-Winkler distance. This two-stage filtering mechanism ensures high accuracy in duplicate detection while significantly reducing memory consumption.

[0090] The final overall score is calculated using a non-linear mapping function, mapping each dimension indicator to a standard score range of 0 to 100. The system also introduces a dynamic penalty factor: if any core dimension (such as accuracy) falls below a preset minimum threshold, the overall score is forcibly reduced to the unqualified range, regardless of how high the scores of other dimensions are, thereby ensuring the seriousness of the quality assessment results and business security.

[0091] Example 7 This embodiment describes a specific implementation of the present invention in processing the quality of unstructured multimedia big data.

[0092] In smart city or internet content regulation scenarios, large amounts of raw data manifest as image, video, or audio streams. In S1, the distributed acquisition architecture integrates a high-performance decoding plugin to segment the video stream into keyframe image groups.

[0093] In S2, the deep feature extraction network employs a hybrid architecture of Convolutional Neural Network (CNN) and Visual Transformer (ViT). CNN is responsible for extracting low-level features such as texture and color from the image, while ViT utilizes a global self-attention mechanism to identify semantic subjects and their interactions within the image. Reconstruction error analysis identifies quality anomalies such as image blurring, screen tearing, or missing content by calculating the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) between the original frame and the reconstructed frame.

[0094] In S3, the evaluation metrics system adds the dimensions of "content compliance" and "visual clarity". Using a pre-trained large-scale visual language model, semantic annotation of image content is performed to verify whether it conforms to the preset business scenario logic.

[0095] In S4, the data restoration model employs super-resolution reconstruction technology and image inpainting algorithms. For blurry surveillance footage, the generative adversarial network learns the distribution of high-frequency details to upscale low-resolution images to high-definition levels. For occlusions or dead pixels in the image, the generator intelligently fills in the gaps based on the texture features of surrounding pixels, while the discriminator ensures that the restored image has no artificial traces in terms of visual coherence.

[0096] In S5, the restoration effect is verified by comparing the visual quality indicators and semantic recognition accuracy before and after restoration. If the restored image can improve the accuracy of subsequent face recognition or license plate recognition models by more than 20%, the restoration is considered successful. The relevant enhancement operators and model weights are encapsulated as dynamic governance strategies for "low-light environments" or "severe weather" and stored in a knowledge base for real-time retrieval.

[0097] Example 8 This embodiment details the integration and operation mode of the present invention in the construction of a large-scale enterprise-level data platform.

[0098] The enterprise data platform aggregates hundreds of millions of data points from ERP, CRM, HR, and manufacturing execution systems (MES). This methodology, as the core governance component of the data platform, is deployed using a microservices architecture.

[0099] In S1, the ChangeDataCapture (CDC) technology is used to capture database changes of various business systems in real time through the data integration bus, ensuring that the data acquisition latency is less than a second.

[0100] In S2, the system configures an independent feature extraction model for each business segment. For example, the HR system's model focuses on the logical relationships between personnel attributes, while the ERP system focuses on the balance verification of financial indicators. The refactored error detection engine monitors each data pipeline in real time, and immediately notifies the relevant business leaders via message push mechanism once an anomaly is detected.

[0101] In S3, quality assessment reports are displayed in real-time as dashboards. Enterprise management can intuitively view data quality rankings and trend analyses for each department and region. The system automatically identifies weak points in quality and provides improvement suggestions.

[0102] In S4, the intelligent repair model works in conjunction with the business process engine. For errors that can be automatically corrected (such as non-standard formatting or simple logical omissions), the system directly performs corrective actions; for anomalies involving core business logic, the system generates repair suggestions and pushes them to the business terminal, where they are executed with a single click after manual confirmation.

[0103] In S5, the governance strategy knowledge base enables cross-departmental experience sharing. For example, the supplier name standardization strategy summarized by the finance department can be directly reused by the procurement department, thereby eliminating information silos within the enterprise and achieving a collaborative improvement in overall data quality.

[0104] In this embodiment, the system supports multi-tenant isolation, ensuring that data from different business segments do not interfere with each other during the governance process, while sharing underlying computing resources and governance algorithm libraries. This cloud-native architecture enables this method to easily support the long-term governance and value mining of trillions of data assets.

[0105] Example 9 This embodiment describes the special optimization logic of this method when processing large-scale spatiotemporal trajectory big data.

[0106] In logistics delivery or shared mobility scenarios, spatiotemporal trajectory data has extremely high sampling frequency and complex spatial constraints. In S1, the system uses GeoHash encoding technology to spatially index trajectory points, enabling rapid integration and aggregation of massive coordinate data.

[0107] In S2, a spatiotemporal convolutional network (ST-GCN) is introduced into the deep feature extraction network. This network models the urban road network as a graph structure, using trajectory points as dynamic node features on the graph. A self-attention mechanism is used to identify the traffic flow propagation effects between different road segments. Reconstruction error analysis can accurately identify "flying point" anomalies caused by GPS signal drift and trajectory interruptions caused by device power failure.

[0108] In S3, the evaluation index system includes "path rationality" and "temporal continuity". The system uses a Hidden Markov Model (HMM) for map matching and calculates the deviation between trajectory points and the actual road centerline.

[0109] In S4, the repair model utilizes the sequence modeling capabilities of Bi-LSTM, combined with the topological constraints of the urban road network, to extrapolate paths for missing trajectory segments. The generator not only generates coordinate values ​​but also ensures that the generated trajectory conforms to vehicle dynamics constraints in terms of speed, acceleration, and steering angle. The discriminator then compares the trajectory with a database of real driving trajectories, eliminating repair schemes that do not comply with traffic rules.

[0110] In S5, the effectiveness of the governance is verified by calculating the matching degree between the repaired trajectory and the real road network. The dynamic governance strategy library records trajectory correction parameters for different weather conditions and congestion levels, enabling the system to maintain extremely high trajectory restoration accuracy even in complex urban environments.

[0111] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for big data quality assessment and repair based on deep learning, characterized in that, The method includes the following steps: S1. Collect and integrate multi-source heterogeneous big data to obtain a standardized initial dataset; S2. Extract high-dimensional features from the initial dataset and identify abnormal data based on reconstruction errors; S3. Construct a multidimensional evaluation model to quantitatively evaluate the quality of the dataset and generate a quality evaluation report; The multidimensional evaluation model includes evaluation indicators in five dimensions: completeness, accuracy, consistency, timeliness, and uniqueness. Among them, the completeness index The calculation formula is: ； in, This indicates the total number of records in the dataset. This indicates the total number of fields contained in the record. Indicates the first Each field has a preset importance weight. For indicator functions; S4. An intelligent repair model constructed using generative adversarial networks and bidirectional long short-term memory networks is used to repair and correct the identified abnormal data; S5. Verify the repair effect, and encapsulate the effective repair logic and parameters into a dynamic governance strategy, store it in a knowledge base built on a graph database, and realize the closed-loop update of the governance strategy.

2. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, In step S2, the identification of abnormal data based on reconstruction error specifically includes: Construct an autoencoder structure that includes an encoder and a decoder, wherein the encoder shares parameters with the deep feature extraction network; The data items are input into the autoencoder, and its reconstruction error is calculated. The calculation formula is: ； in, Represents the original input vector. This represents the output vector reconstructed by the decoder. Let be the dimension of the vector; The reconstruction error of the data item to be tested is compared with the baseline distribution obtained by training on normal samples and the preset outlier threshold. If it exceeds the threshold, the data item is determined to be abnormal data.

3. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, In step S4, the loss function of the intelligent repair model Defined as mean squared error loss and adversarial losses Weighted sum: ； in, and These are the preset weighting coefficients. This is used to constrain the repair value to be numerically close to the true value. This is used to constrain the repair data to maintain an overall distribution consistent with the original normal data.

4. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, In step S5, verifying the repair effect includes calculating the information entropy of the repaired dataset and comparing it with the information entropy before repair. If the repair causes an abnormal increase in information entropy, a manual review mechanism is triggered.

5. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, In step S2, the deep feature extraction network is a deep neural network containing a self-attention mechanism layer; And / or, in step S4, the bidirectional long short-term memory network consists of a forward LSTM layer and a backward LSTM layer, used to generate an enhanced feature representation containing past and future contextual information of data items.

6. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, When the multi-source heterogeneous big data includes images or video streams, in step S2, a hybrid architecture of convolutional neural network and visual Transformer is used for feature extraction, and peak signal-to-noise ratio and structural similarity index are calculated as measures of reconstruction error; in step S4, a super-resolution reconstruction or image inpainting algorithm based on generative adversarial network is used for data inpainting.

7. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, When the multi-source heterogeneous big data is spatiotemporal trajectory data, in step S2, a spatiotemporal graph convolutional network is used for feature extraction; in step S4, when the generator of the intelligent repair model generates the repair trajectory, it needs to ensure that the trajectory conforms to the constraints defined based on the urban road network topology and vehicle dynamics model.

8. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, After performing intelligent repair in step S4 and before outputting the data, the process also includes a step of desensitizing sensitive information in the repaired data. The desensitization uses an algorithm that meets differential privacy requirements to add controlled Laplace noise to the data.

9. The method for big data quality assessment and repair based on deep learning according to claim 1, characterized in that, The method employs hardware co-optimization at runtime, including: decomposing model operators using computation graph optimization techniques for parallel execution on tensor processing units; reading data from a distributed message queue using remote direct memory access techniques; and accelerating the matching and retrieval process of policies in the knowledge base using FPGA hardware accelerators.

Images