Automatic cleaning and labeling method for data assets

Abstract

The invention discloses an automatic cleaning and labeling method for data assets, relates to the technical field of data processing, and solves the problems of high manual dependency degree, flow splitting, insufficient dynamic adaptability and poor traceability in existing data cleaning and labeling. According to the automatic cleaning and labeling method for the data assets, a topology analysis module, a containerization cleaning engine, a multi-modal labeling center, a feedback console and a block chain evidence storage module are included, and the topology analysis module comprises a matrix generation algorithm and a function dependency detector; the topology analysis module is used for outputting a dynamic topology map and positioning an abnormal propagation path. According to the method, high-precision automatic processing of multi-modal data is realized on the basis of an abnormal positioning and containerization cleaning engine of a data topological graph, and the preprocessing period is greatly shortened; the confidence arbitration mechanism of the intelligent labeling center significantly reduces invalid manual auditing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data asset automatic cleaning and labeling method. BACKGROUND

[0002] With the development of big data and artificial intelligence technology, high-quality data assets have become the core resources for model training and business decision-making. However, traditional data cleaning and labeling has the following defects:

[0003] High dependence on manual work: cleaning rules, abnormal value identification, and data labeling require a lot of manual intervention, which is low in efficiency and poor in consistency;

[0004] Process fragmentation: cleaning and labeling belong to independent links, lacking a collaborative mechanism, resulting in repeated operations and resource waste;

[0005] Insufficient dynamic adaptability: unable to automatically optimize cleaning strategies and labeling rules according to changes in data distribution;

[0006] Poor traceability: the data processing process lacks version management and audit capabilities, affecting the credibility of data assets.

[0007] Therefore, the existing needs are not met, and for this we propose a data asset automatic cleaning and labeling method. SUMMARY

[0008] The purpose of the present application is to provide a data asset automatic cleaning and labeling method to solve the problems of high dependence on manual work, process fragmentation, insufficient dynamic adaptability, and poor traceability in the existing data cleaning and labeling.

[0009] To achieve the above purpose, the present application provides the following technical scheme: a data asset automatic cleaning and labeling system, comprising a topology analysis module, a containerized cleaning engine, a multi-modal labeling center, a feedback console, and a blockchain storage module;

[0010] The topology analysis module includes a matrix generation algorithm and a function dependency detector, and is used to output a dynamic topology map and locate an abnormal propagation path;

[0011] The containerized cleaning engine includes a rule cleaning container, an AI cleaning container, and a cleaning scheduler;

[0012] The multi-modal labeling center includes a structured labeling unit, an unstructured labeling unit, and a confidence arbiter;

[0013] The feedback console includes a cleaning feedback channel and a labeling feedback channel, and is used to display cleaning coverage and labeling accuracy indicators on a visual panel;

[0014] The blockchain storage module comprises a data fingerprint generator, a distributed ledger unit and a version snapshot manager.

[0015] Preferably, the matrix generation algorithm is used to calculate the inter-field statistical dependence, and the function dependence detector is used to identify the business rule constraints.

[0016] Preferably, the rule cleaning container is used to execute a predefined consistency checking rule, the AI cleaning container is used to integrate a pre-trained anomaly detection model and support dynamic loading of a custom model, and the cleaning scheduler selects a cleaning container based on the dynamic topology map and records an operation flow log.

[0017] Preferably, the structured labeling unit generates labels based on decision tree rules and automatic encoders, the unstructured labeling unit is used to deploy an AI labeling model adapted to modalities, and the confidence arbiter assigns tasks according to a three-level threshold, specifically, when the confidence is > 90%, an automatic warehousing instruction is executed; when 60% < confidence ≤ 90%, cross-validation is started; and when the confidence is ≤ 60%, manual review is transferred.

[0018] Preferably, the cleaning feedback channel is used to inject residual anomalies found in the labeling link into the AI cleaning container for retraining, and the labeling feedback channel is used to monitor data distribution deviation and dynamically adjust the labeling model threshold.

[0019] Preferably, the data fingerprint generator is used to calculate SHA-256 hash on operation logs and construct a Merkle tree, the distributed ledger unit is used to store hash values on a consortium chain and support zero-knowledge proof verification, and the version snapshot manager generates Delta-encoded asset version snapshots in batches.

[0020] A labeling method of a data asset automatic cleaning and labeling system, comprising the following steps:

[0021] Step S1: Accessing multi-source heterogeneous data through a unified data interface, and automatically constructing a data topology map comprising field association and statistical dependence;

[0022] Step S2: Locating an abnormal propagation path based on the topology map, calling a multi-modal quality detection engine to perform abnormal cleaning, and generating a quality repair report;

[0023] Step S3: Activating intelligent labeling tasks according to data characteristics after cleaning:

[0024] First, generating initial labels through a preset rule template, second, screening low-confidence samples for manual review using a labeling confidence model, and third, updating labeling model parameters by fusing manual feedback;

[0025] Step S4: Establishing a double-channel feedback mechanism:

[0026] Cleaning feedback channel: update cleaning rule base according to hidden abnormalities found in the labeling link;

[0027] Labeling feedback channel: adjust labeling model threshold based on changes in data distribution after cleaning;

[0028] Step S5: store the whole process operation in the blockchain, and generate a versioned data asset snapshot.

[0029] Preferably, the multi-modal quality detection engine in step S2 includes:

[0030] Structured data detection module: verify integrity, consistency, and uniqueness constraints based on field topology relationships;

[0031] Unstructured data detection module: identify image blur, text repetition, and audio noise defects through CV / NLP models.

[0032] Preferably, the intelligent labeling task activation conditions in step S3 are as follows: the newly added data volume after cleaning exceeds the threshold, the data distribution offset is greater than the preset standard deviation, and the artificial forced trigger instruction.

[0033] Compared with the prior art, the beneficial effects of the present application are:

[0034] 1. The present application is based on the abnormal positioning and containerized cleaning engine of data topology graph, which realizes high-precision automatic processing of multi-modal data, greatly shortens the preprocessing period, and the confidence arbitration mechanism of the intelligent labeling center significantly reduces invalid manual review.

[0035] 2. The present application connects the cleaning and labeling links through a double-channel feedback mechanism, uses hidden abnormalities found by labeling to optimize the cleaning model in reverse, and dynamically calibrates the labeling strategy according to the data distribution changes to continuously improve the credibility of data assets;

[0036] 3. The present application ensures that the whole process operation is traceable through the blockchain storage module, meets the audit requirements of high compliance scenarios, and supports data asset state rollback through versioned snapshot management, effectively reducing the business risk caused by misoperation;

[0037] 4. The present application supports hot plugging of cleaning operators and labeling models through containerized architecture, which can quickly adapt to the differentiated needs of financial, medical, and Internet of Things fields. BRIEF DESCRIPTION OF DRAWINGS

[0038] Fig. 1 The system framework diagram of the labeling system of the present application;

[0039] Fig. 2 The system flowchart of the labeling method of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0041] Please refer to Figs. 1-2 An embodiment provided by the present application is a data asset automatic cleaning and labeling system, comprising a topology analysis module, a containerized cleaning engine, a multi-modal labeling center, a feedback console and a blockchain storage module.

[0042] The topology analysis module comprises a matrix generation algorithm and a function dependency detector, and is used to output a dynamic topology graph and locate an abnormal propagation path.

[0043] The containerized cleaning engine comprises a rule cleaning container, an AI cleaning container and a cleaning scheduler.

[0044] The multi-modal labeling center comprises a structured labeling unit, an unstructured labeling unit and a confidence arbiter.

[0045] The feedback console comprises a cleaning feedback channel and a labeling feedback channel, and is used to display cleaning coverage and labeling accuracy indicators on a visual panel.

[0046] The blockchain storage module comprises a data fingerprint generator, a distributed ledger unit and a version snapshot manager.

[0047] The matrix generation algorithm is used to calculate the statistical dependency between fields, and the function dependency detector is used to identify business rule constraints.

[0048] The rule cleaning container is used to execute predefined consistency check rules, the AI cleaning container is used to integrate pre-trained anomaly detection models and support dynamic loading of custom models, and the cleaning scheduler selects cleaning containers based on a dynamic topology graph and records operation flow logs.

[0049] The structured labeling unit generates labels based on decision tree rules and autoencoders, the unstructured labeling unit deploys AI labeling models adapted to modalities, and the confidence arbiter assigns tasks according to three threshold values, specifically, when the confidence is > 90%, an automatic warehousing instruction is executed; when 60% < confidence ≤ 90%, cross-validation is started; and when the confidence ≤ 60%, manual review is transferred.

[0050] The cleaning feedback channel is used to inject residual abnormalities found in the labeling process into the AI cleaning container for retraining, and the labeling feedback channel is used to monitor data distribution shifts and dynamically adjust labeling model thresholds.

[0051] The data fingerprint generator is configured to calculate SHA-256 hash and build a Merkle tree for the operation log, the distributed ledger unit is configured to store the hash value on the consortium chain and support zero-knowledge proof verification, and the version snapshot manager is configured to generate a Delta encoded asset version snapshot in batches.

[0052] A labeling method of a data asset automatic cleaning and labeling system, comprising the following steps:

[0053] Step S1: Access multi-source heterogeneous data through a unified data interface, and automatically build a data topology graph containing field correlation and statistical dependency;

[0054] Step S2: Locate the abnormal propagation path based on the topology graph, call the multi-modal quality detection engine to perform abnormal cleaning, and generate a quality repair report;

[0055] Step S3: According to the data characteristics after cleaning, activate the intelligent labeling task:

[0056] First, generate initial labeling through a preset rule template, second, filter low-confidence samples for manual review using a labeling confidence model, and third, update labeling model parameters by fusing manual feedback;

[0057] Step S4: Establish a double-channel feedback mechanism:

[0058] Cleaning feedback channel: update the cleaning rule library according to hidden abnormalities found in the labeling link;

[0059] Labeling feedback channel: adjust the labeling model threshold based on the distribution changes of the cleaned data;

[0060] Step S5: Store the whole process operation in the blockchain, and generate a versioned data asset snapshot.

[0061] The multi-modal quality detection engine in step S2 includes:

[0062] Structured data detection module: verify integrity, consistency, and uniqueness constraints based on field topology relationships;

[0063] Unstructured data detection module: identify image blur, text repetition, and audio noise defects through CV / NLP models.

[0064] The intelligent labeling task activation conditions in step S3 are as follows: the amount of newly added data after cleaning exceeds the threshold, the data distribution deviation is greater than the preset standard deviation, and the manual forced trigger instruction.

[0065] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. An automated data asset cleaning and labeling system, characterized in that: It includes a topology analysis module, a containerized cleaning engine, a multimodal annotation center, a feedback console, and a blockchain evidence storage module; The topology analysis module includes a matrix generation algorithm and a functional dependency detector. The topology analysis module is used to output a dynamic topology map and locate abnormal propagation paths. The containerized cleaning engine includes rule-based cleaning containers, AI-based cleaning containers, and a cleaning scheduler. The multimodal annotation center includes structured annotation units, unstructured annotation units, and a confidence arbitrator; The feedback console includes a cleaning feedback channel and a labeling feedback channel. The feedback console is used to display cleaning coverage and labeling accuracy indicators on a visualization panel. The blockchain evidence storage module includes a data fingerprint generator, a distributed ledger unit, and a version snapshot manager.

2. The automated data asset cleaning and labeling system according to claim 1, characterized in that: The matrix generation algorithm is used to calculate the statistical dependency between fields, and the functional dependency detector is used to identify business rule constraints.

3. The automated data asset cleaning and labeling system according to claim 1, characterized in that: The rule-based cleaning container is used to execute predefined consistency verification rules, the AI ​​cleaning container is used to integrate pre-trained anomaly detection models and support dynamic loading of custom models, and the cleaning scheduler selects cleaning containers based on the dynamic topology graph and records operation logs.

4. The automated data asset cleaning and labeling system according to claim 1, characterized in that: The structured annotation unit generates labels based on decision tree rules and an autoencoder. The unstructured annotation unit is used to deploy a modality-adaptive AI annotation model. The confidence arbitrator allocates tasks according to a triple threshold: when the confidence level is >90%, an automatic data entry instruction is executed; when the confidence level is 60% < confidence level ≤ 90%, cross-validation is initiated; and when the confidence level is ≤ 60%, the task is transferred to manual review.

5. The automated data asset cleaning and labeling system according to claim 1, characterized in that: The cleaning feedback channel is used to inject residual anomalies found in the annotation process into the AI ​​cleaning container for retraining, and the annotation feedback channel is used to monitor data distribution shifts and dynamically adjust the annotation model threshold.

6. The automated data asset cleaning and labeling system according to claim 1, characterized in that: The data fingerprint generator is used to calculate SHA-256 hashes on operation logs and construct Merkle trees. The distributed ledger unit is used to store hash values ​​on the consortium blockchain and support zero-knowledge proof verification. The version snapshot manager generates Delta-encoded asset version snapshots in batches.

7. A labeling method for an automated data asset cleaning and labeling system according to any one of claims 1-6, characterized in that: Includes the following steps: Step S1: Access multi-source heterogeneous data through a unified data interface and automatically construct a data topology map that includes field relationships and statistical dependencies; Step S2: Based on the topology map, locate the anomaly propagation path, call the multimodal quality detection engine to perform anomaly cleaning, and generate a quality repair report; Step S3: Activate the intelligent annotation task based on the characteristics of the cleaned data: First, initial labels are generated using a preset rule template. Second, low-confidence samples are selected using a label confidence model and submitted for manual review. Third, manual feedback is incorporated to update the label model parameters. Step S4: Establish a dual-channel feedback mechanism: Cleaning feedback channel: Update the cleaning rule base based on hidden anomalies discovered during the annotation process; Annotation feedback channel: Adjust the annotation model threshold based on changes in data distribution after cleaning; Step S5: Perform blockchain-based notarization of the entire process and generate a versioned data asset snapshot.

8. The annotation method of the automated data asset cleaning and annotation system according to claim 1, characterized in that: The multimodal quality detection engine in step S2 includes: Structured data inspection module: Verifies integrity, consistency, and uniqueness constraints based on field topological relationships; Unstructured data detection module: Identifies image blurring, text repetition, and audio noise defects using CV / NLP models.

9. The annotation method of the automated data asset cleaning and annotation system according to claim 1, characterized in that: The activation conditions for the intelligent annotation task in step S3 are as follows: the amount of newly added data after cleaning exceeds the threshold, the data distribution offset is greater than the preset standard deviation, and a manual forced trigger command is issued.

Images