Large model-based treatment method for improving data quality

Abstract

The invention discloses a treatment method for improving data quality based on a large model, and the method comprises the following steps: a, receiving a multi-source heterogeneous input data stream, analyzing modal features of a text, an image and a time sequence signal, and generating a mixed feature vector through a pre-trained multi-modal encoder; b, generating a data cleaning rule based on a large language model, and dynamically executing missing value interpolation and abnormal value detection; c, applying a differential privacy policy to the sensitive field, and selecting k-anonymous or homomorphic encryption according to a data grading label; and d, executing preprocessing quality verification, including calculating an integrity score # imgabs0 # and a consistency score # imgabs1 #, and updating a cleaning rule when a comprehensive quality index # imgabs2 # is lower than a dynamic threshold value # imgabs3 #. Through the preprocessing quality verification and incremental learning mechanism, the model response speed is increased by three times, and the result is delayed; the misjudgment rate is reduced by 42% within one minute; and the cross-modal contrast learning enables the multi-modal task F1-score to be improved by 17%-23%.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and specifically to a governance method for improving data quality based on a large model. Background Art

[0002] With the exponential growth of multimodal data, traditional data governance methods face the following problems: Limitations of static rules: Existing technologies rely on manually predefined cleaning rules, which make it difficult to dynamically adapt to data distribution drift (such as sudden anomalies in sensor data), resulting in delayed governance.

[0003] Multimodal fragmentation: Heterogeneous data such as text and images are processed independently, lacking feature alignment and joint optimization, affecting the accuracy of downstream model reasoning (the average error increases by 15%-30%).

[0004] One-way quality assessment: Relying on manual sampling and static indicators, it is impossible to reversely optimize data governance strategies through model feedback.

[0005] Conflict between privacy and efficiency: Traditional encryption technologies (such as AES) lead to reduced data availability (model performance loss ≥ 25%), and solutions such as federated learning have excessive communication overhead (delay > 5 minutes).

[0006] To this end, we propose a governance method based on large models to improve data quality. Summary of the Invention

[0007] The purpose of the present invention is to provide a governance method for improving data quality based on a large model to solve the problems raised in the above background technology.

[0008] To achieve the above objectives, the present invention provides the following technical solutions: a governance method for improving data quality based on a large model, comprising the following steps: a. receiving multi-source heterogeneous input data streams, parsing text, image, and time series signal modal features, and generating mixed feature vectors through a pre-trained multimodal encoder; b. generating data cleaning rules based on a large language model, dynamically performing missing value interpolation and outlier detection; c. applying differential privacy strategies to sensitive fields, selecting k-anonymity or homomorphic encryption based on data classification labels; d. performing preprocessing quality verification, including calculating integrity scores and consistency scores , when the comprehensive quality index Below dynamic threshold Update the cleaning rules; e. Build a cross-modal contrastive learning network to align text-image features; f. Deploy the federated learning framework, encrypt and transmit the gradient to the core node after local training on the edge node; g. Perform model performance verification, when the performance indicator P is lower than the dynamic threshold Adjust the contrast loss weight and resample the data; h. Monitor data distribution drift and trigger incremental learning; i. Generate governance results including data lineage graph, natural language causal chain and visualization report.

[0009] In the step a: The multimodal encoder is the ViT-BERT model based on the Transformer architecture, and the pre-training dataset is Wiki-ImagePairs; During fine-tuning, the visual encoding layer is frozen and the learning rate of the text encoding layer is set to 0.0001.

[0010] In the step b: Missing value interpolation was performed using the MICE algorithm with 5 iterations; The isolation forest algorithm is used for outlier detection, and the anomaly threshold α=Q3+1.5IQR.

[0011] In the step c: k-anonymity k value is an integer from 3 to 7; Homomorphic encryption uses the Paillier algorithm with a key length of 2048 bits.

[0012] In the step d: = the number of non-empty fields divided by the total number of fields multiplied by the time decay factor γ, where γ is equal to the negative λ of the natural exponential function exp multiplied by the time interval t, that is, γ=exp(-λt), and the value range of λ is 0.05 to 0.15; =1 minus the proportion of conflicting entity pairs to the total entity pairs. The conflicting entity pairs are calculated using the BERT model to measure semantic similarity. When the similarity is less than or equal to 0.85, it is considered a conflict. =0.6 +0.4 , =μ−3σ.

[0013] In the step e: Contrastive learning uses the InfoNCE loss function with a temperature coefficient of τ = 0.07; When aligning text-image features, the visual feature weight is 0.6 and the text weight is 0.4.

[0014] In the step f: The learning rate η of the local training of edge nodes is 0.01, and the momentum coefficient β is 0.9; Gradient transmission uses BFV homomorphic encryption, and the key is rotated every hour.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Dynamic Adaptability: Through pre-processing quality verification and incremental learning mechanisms, the model response speed is increased by 3 times, the delay is less than 1 minute, and the false positive rate is reduced by 42%; 2. Multimodal collaboration: Cross-modal contrastive learning improves the F1-score of multimodal tasks by 17%-23%; 3. Privacy-Utility Balance: BFV encryption combined with gradient masking (top-10% sparsification) improves privacy protection by 4 times with less than 8% performance loss. 4. Enhanced explainability: The time required to locate defects in data lineage maps is reduced by 90%, and the accuracy of causal chain generation is ≥92%. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] See also Figure 1 The present invention provides a technical solution: a governance method for improving data quality based on a large model, comprising the following steps: a. receiving multi-source heterogeneous input data streams, parsing text, image, and time series signal modal features, and generating mixed feature vectors through a pre-trained multimodal encoder; b. generating data cleaning rules based on a large language model, dynamically performing missing value interpolation and outlier detection; c. applying differential privacy strategies to sensitive fields, selecting k-anonymity or homomorphic encryption based on data classification labels; d. performing preprocessing quality verification, including calculating integrity scores and consistency scores , when the comprehensive quality index Below dynamic threshold Update the cleaning rules; e. Build a cross-modal contrastive learning network to align text-image features; f. Deploy the federated learning framework, encrypt and transmit the gradient to the core node after local training on the edge node; g. Perform model performance verification, when the performance indicator P is lower than the dynamic threshold Adjust the contrast loss weight and resample the data; h. Monitor data distribution drift and trigger incremental learning; i. Generate governance results including data lineage graph, natural language causal chain and visualization report.

[0019] In the step a: The multimodal encoder is the ViT-BERT model based on the Transformer architecture, and the pre-training dataset is Wiki-ImagePairs; During fine-tuning, the visual encoding layer is frozen and the learning rate of the text encoding layer is set to 0.0001.

[0020] In the step b: Missing value interpolation was performed using the MICE algorithm with 5 iterations; The isolation forest algorithm is used for outlier detection, and the anomaly threshold α=Q3+1.5IQR.

[0021] In the step c: k-anonymity k value is an integer from 3 to 7; Homomorphic encryption uses the Paillier algorithm with a key length of 2048 bits.

[0022] In the step d: = The number of non-empty fields divided by the total number of fields multiplied by the time decay factor γ, where γ is equal to the negative λ of the natural exponential function exp multiplied by the time interval t, where t is the data collection time interval (hours), that is, γ=exp(-λt). It is set according to the data timeliness requirements. The value range of λ is 0.05 to 0.15; =1 minus the proportion of conflicting entity pairs to the total entity pairs. The conflicting entity pairs are calculated using the BERT model to measure semantic similarity. When the similarity is less than or equal to 0.85, it is considered a conflict. =0.6 +0.4 , =μ−3σ; μ is the mean of the quality indicators of the most recent N data, N = 1000 ± 200.

[0023] The specific methods of triggering the rule generator to update the cleaning rules in step d include: Rule generation based on reinforcement learning: The current data quality index is used as the state space, the rules are added, deleted and modified into the action space, and the reward function R= Improvement − Computational overhead; Iterative optimization of large language models: Low-quality data samples are input into the LLM to generate improved cleaning rule prompts, such as "adjust the missing value interpolation strategy to linear interpolation of adjacent time windows."

[0024] In the step e: Contrastive learning uses the InfoNCE loss function with a temperature coefficient of τ = 0.07. τ is used to adjust the feature similarity distribution, with a value range of 0.05 to 0.09; When aligning text and image features, the weights are dynamically adjusted according to the information entropy of the modal features, with the visual feature weight being 0.6 and the text weight being 0.4.

[0025] In the step f: The learning rate η of the local training of edge nodes is 0.01, and the momentum coefficient β is 0.9; Gradient transmission uses BFV homomorphic encryption, and the key is rotated every hour.

[0026] The method for monitoring data distribution drift in step h includes: Calculate the KL divergence of the data in the window, with a window size of 1000 and a sliding step size of 100. The window size and step size are dynamically adjusted based on the data throughput, with a baseline value of 20% of the throughput. When the KL divergence is > 0.15 for three consecutive windows, incremental learning is triggered.

[0027] Example 1: Training of the ViT-BERT Multimodal Encoder Application scenario: Multimodal data management of products on e-commerce platforms (text description + product images). Implementation steps: Pre-training phase: Dataset: The Wiki-ImagePairs dataset (1 million image-text pairs) is used. The image resolution is adjusted to 224×224, and the text length is truncated to 256 words. Model architecture: ViT-BERT consists of VisionTransformer (ViT-B / 16) and BERT-base, with image patch size of 16×16 and text encoding dimension of 768; Training parameters: Adam optimizer (lr=3e-5), batch size 64, training for 50 epochs, joint training of MLM (masked language modeling) and ITM (image-text matching) tasks.

[0028] Fine-tuning phase: Data preparation: e-commerce product data (500,000 items), including product titles (text) and main images (images); Freezing strategy: Freeze the visual encoding layer of ViT and only fine-tune the BERT text encoding layer with a learning rate of 0.0001; Feature fusion: After the text features and image features are residually connected, a 1024-dimensional mixed vector is generated through a fully connected layer.

[0029] Experimental results: The top-5 accuracy of cross-modal retrieval increased to 89.3% (the baseline model CLIP was 82.1%). Fine-tuning time is reduced by 37% (only 12 hours, compared to 19 hours for full parameter fine-tuning).

[0030] Example 2: Missing Value Interpolation and Anomaly Detection Application scenario: Industrial sensor time series data management. Implementation steps: Missing value handling: Algorithm selection: MICE algorithm (Multiple Interpolation), 5 iterations, convergence check after each round of interpolation (relative error <1%); Field matching: When interpolating temperature and pressure fields, the mean of the three adjacent sensors is used as a priori.

[0031] Anomaly Detection: Algorithm parameters: number of isolation forest trees 100, number of subsampling 256, anomaly threshold α = Q3 + 1.5IQR; Dynamic adjustment: When the data distribution shifts (KL divergence > 0.1), α is adaptively adjusted to Q3 + 2.0 IQR.

[0032] Experimental results: Data completeness increased from 78% to 99.2% after missing value interpolation; The anomaly detection false alarm rate is reduced to 3.7% (compared to 12.5% for the traditional Z-score method).

[0033] Example 3: Quality Index Formula Verification Application scenario: Financial transaction data consistency management. Implementation steps: Completeness Score calculate: Data sample: 1 million transaction records, with fields including user ID, amount, and time; Calculation logic: = (number of non-empty fields / total number of fields) × exp(-0.1t), where t is the data collection time interval (hours); Test conditions: t=24 hours, non-empty fields account for 95%, but =0.95×exp(-2.4)=0.95×0.0907≈0.086.

[0034] Calculation of consistency score U1: Conflict detection: Transaction address conflicts for the same user ID (BERT semantic similarity < 0.85 is considered a conflict); Statistical results: There are 100,000 total entity pairs and 1,500 conflicting entity pairs. =1−1500 / 100000=0.985.

[0035] Comprehensive quality indicators Evaluate: =0.6×0.086+0.4×0.985=0.0516+0.394=0.4456; Threshold =μ−3σ=0.52−3×0.08=0.28, because =0.4456> , no rule update is triggered.

[0036] Experimental results: Formula sensitivity verification: When the number of conflicting entity pairs increases to 5000, =0.95, =0.6×0.086+0.4×0.95=0.476, still higher than =0.28, proving the rationality of the threshold setting.

[0037] Example 4: Contrastive Learning Parameter Optimization (Corresponding to Claim 6) Application scenario: Multimodal alignment of medical imaging reports (CT images + diagnostic text). Implementation steps: Loss function configuration: Loss type: InfoNCE loss, temperature coefficient τ=0.07; Weight distribution: visual feature weight 0.6, text weight 0.4.

[0038] Training process: Dataset: MIMIC-CXR dataset (chest X-ray + radiology report); Optimizer: AdamW (lr=5e-5), batch size 32; Evaluation metric: cross-modal retrieval mean average precision (mAP).

[0039] Experimental results: When τ=0.07, mAP=76.3%, which is better than τ=0.05 (mAP=72.1%) and τ=0.09 (mAP=74.8%); When the visual weight is 0.6, the image-to-text retrieval accuracy is improved by 9%.

[0040] Example 5: Federated Learning Communication Efficiency Test Application scenario: cross-bank user credit assessment model training. Implementation steps: Parameter configuration: Local training: SGD optimizer, η=0.01, momentum β=0.9; Gradient processing: Top-10% sparse masking, BFV encryption (key length 2048 bits); Aggregation strategy: Momentum FedAvg, key rotation every 60 minutes.

[0041] Comparative experiment: Experimental group: gradient mask + BFV encryption of the present invention; Control group: traditional FedAvg+AES encryption; Test indicators: single-round communication time, model AUC-ROC.

[0042] Experimental results: Communication time: 2.3 minutes in the experimental group vs. 5.1 minutes in the control group (55% reduction in time); Model performance: AUC = 0.891 for the experimental group vs AUC = 0.843 for the control group (accuracy improved by 5.7%). Security: The cost of cracking BFV encryption is 7 times higher than that of AES.

[0043] Example 6: Full-process application scenario of real-time data governance on e-commerce platforms: multimodal governance of e-commerce user behavior logs (clickstream + purchase records) and product images.

[0044] Implementation steps: Step a (data reception and feature extraction): Connect to Kafka real-time stream (TPS = 5000) to parse user ID, click time, and product image URL; ViT-BERT extracts product image features (ViT-B / 16) and text description features (BERT) to generate a 1024-dimensional mixed vector.

[0045] Step b (rule generation and cleaning): GPT-4 generation rule: "Missing user geographic location fields are completed by reverse parsing the IP address"; Isolation forest detects abnormal click times (α = Q3 + 1.5IQR = 120s + 1.5 × 60s = 210s), filtering out clicks greater than 10 per user within 1 second.

[0046] Step c (Privacy protection): User IDs are k-anonymized (k=5) and IP addresses are Paillier encrypted (key 2048 bits).

[0047] Step d: calculate = (Number of non-empty fields / 5)×exp(-0.1×1)=0.98×0.904=0.886; Conflict detection: Purchase address conflict rate for the same user =1−50 / 100000=0.9995; =0.6×0.886+0.4×0.9995=0.5316+0.3998=0.9314> =0.85, no update is triggered.

[0048] Step e (multimodal alignment): InfoNCE loss aligns user click sequences with product image features, τ=0.07; The visual weight is 0.6, the text weight is 0.4, and the cross-modal retrieval mAP after training is 82.1%.

[0049] Step f (federated learning training): 10 edge nodes locally train the click prediction model (SGD, η = 0.01), with gradient top-10% sparsification and BFV encryption; Core node aggregation (momentum β=0.9) improves the AUC to 0.89.

[0050] Step g: Validation set P = 0.7 × 0.87 + 0.3 × 0.89 = 0.876> =0.82, no parameter adjustment.

[0051] Step h (incremental learning): When monitoring KL divergence = 0.12 < 0.15, no update is triggered. When simulating burst traffic (KL = 0.21), EWC updates the model and the buffer recall rate reaches 96%.

[0052] Step i (result generation): GraphSAGE builds a user-product-click time relationship graph to locate the source of abnormal clicks; T5 generates a report: "User #12345's abnormal clicks are concentrated on product #678. It is recommended to investigate crawler behavior." D3.js visualization heat map marks high-risk commodities.

[0053] Experimental results: Time consumption for the entire process: from data access to report generation <3 minutes; Governance effect: The abnormal click recognition rate is 98.7%, and the risk of privacy leakage is reduced by 91%.

[0054] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

Claims

1. A data quality improvement governance method based on a large model, characterized in that: The following steps are involved: a. Receive multi-source heterogeneous input data streams, parse text, image, and time series signal modal features, and generate hybrid feature vectors using a pre-trained multimodal encoder; b. Generate data cleaning rules based on a large language model, dynamically perform missing value interpolation and outlier detection; c. Apply differential privacy strategies to sensitive fields, choosing between k-anonymity or homomorphic encryption based on data classification labels; d. Perform pre-processing quality verification, including calculation of integrity scores and consistency scores , when the comprehensive quality index Below dynamic threshold Update the cleaning rules; e. Build a cross-modal contrastive learning network to align text-image features; f. Deploy the federated learning framework, encrypt and transmit the gradients after local training on the edge nodes to the core nodes for aggregation; g. Perform model performance verification, when the performance index P is lower than the dynamic threshold Adjust the contrast loss weight and resample the data; h. Monitor data distribution drift and trigger incremental learning; i. Generate governance results including data lineage graph, natural language causal chain and visualization report.

2. The method for improving data quality based on a large model according to claim 1, characterized in that: In the step a: The multimodal encoder is the ViT-BERT model based on the Transformer architecture, and the pre-training dataset is Wiki-ImagePairs; During fine-tuning, the visual encoding layer is frozen and the learning rate of the text encoding layer is set to 0.0001.

3. The method for improving data quality based on a large model according to claim 1, characterized in that: In the step b: Missing value interpolation was performed using the MICE algorithm with 5 iterations; The isolation forest algorithm is used for outlier detection, and the anomaly threshold α=Q3+1.5IQR.

4. The method for improving data quality based on a large model according to claim 1 is characterized by: In the step c: k-anonymity k value is an integer from 3 to 7; Homomorphic encryption uses the Paillier algorithm with a key length of 2048 bits.

5. The method for improving data quality based on a large model according to claim 1 is characterized by: In the step d: = the number of non-empty fields divided by the total number of fields multiplied by the time decay factor γ, where γ is equal to the negative λ of the natural exponential function exp multiplied by the time interval t, that is, γ=exp(-λt), and the value range of λ is 0.05 to 0.15; =1 minus the proportion of conflicting entity pairs to the total entity pairs. The conflicting entity pairs are calculated using the BERT model to measure semantic similarity. When the similarity is less than or equal to 0.85, it is considered a conflict. =0.6 +0.4 , =μ−3σ, where μ is the value of the most recent 1000±200 data is the mean, and σ is the standard deviation.

6. The method for improving data quality based on a large model according to claim 1 is characterized by: In the step e: Contrastive learning uses the InfoNCE loss function with a temperature coefficient of τ = 0.07; When aligning text-image features, the visual feature weight is 0.6 and the text weight is 0.

4.

7. The method for improving data quality based on a large model according to claim 1, characterized in that: In the step f: The learning rate η of the local training of edge nodes is 0.01, and the momentum coefficient β is 0.9; Gradient transmission uses BFV homomorphic encryption, and the key is rotated every hour.

Images