A method for data cleaning decision and automatic tuning using AI workflow

Abstract

The application discloses a method for data cleaning decision and automatic optimization using AI workflow, relates to the technical field of data processing, and obtains original data from a plurality of heterogeneous data sources, generates relevant tables and results through field extraction, structure analysis and language attribute recognition; based on this, storage and task generation are carried out, standardized data and task queue information are output; quality problems are identified through feature analysis, targets are defined, and a cleaning task graph is constructed, then cleaning metadata is generated through preliminary screening marking; an AI cleaning workflow containing a plurality of strategy paths is constructed and optimized, and after execution, the effect is evaluated in combination with downstream feedback and adjusted, and finally, a plurality of versions of cleaning results are integrated and output. The method effectively improves the adaptation capability to different types of data sources, reduces the dependence on static preset strategies, reduces the demand for manual intervention, improves the optimization efficiency, and enhances the systematicness and standardization of data cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method for data cleaning decision and automatic optimization using AI workflow. BACKGROUND

[0002] Under the background of rapid development of big data, artificial intelligence and other technologies, intelligent knowledge retrieval systems are widely used in search engines, enterprise knowledge management and other fields, and their performance is highly dependent on data quality and structured degree. Before data enters the retrieval link, it needs to go through cleaning processes such as missing value completion, format unification, redundancy removal and text processing.

[0003] Current data cleaning relies on artificial rules, ETL tools and preset processing libraries, supplemented by machine learning local processing and AI platform automatic arrangement. However, in the face of structured and unstructured mixed multi-source heterogeneous data, the existing static strategy is difficult to adapt to changes in data structure, type and quality due to the lack of downstream task feedback optimization mechanism, and frequent manual intervention is required, resulting in unstable cleaning effect and low optimization efficiency, which cannot meet the dynamic needs of high semantic scenarios. SUMMARY

[0004] To solve the technical problems existing in the prior art, the present application provides a method for data cleaning decision and automatic optimization using AI workflow.

[0005] The technical solution adopted by the present application is: a method for data cleaning decision and automatic optimization using AI workflow, comprising the following steps:

[0006] Step 1: obtaining original data from multiple heterogeneous data sources, performing field extraction, structure analysis and language attribute recognition on the original data, and generating field definition table, structured metadata and quality preliminary evaluation results;

[0007] Step 2: based on the field definition table, structured metadata and quality preliminary evaluation results generated in step 1, storing the structure recognition results and generating tasks, outputting standardized storage data and task queue information;

[0008] Step 3: based on the structured metadata and quality preliminary evaluation results of step 1 and the standardized storage data of step 2, performing content feature analysis on the fields in the field definition table, identifying the quality problems of the fields and defining the cleaning target, and constructing a cleaning task graph containing field information, problem type, strategy candidate and dependency relationship;

[0009] Step 4: based on the cleaning task graph generated in step 3 and the task queue information of step 2, performing data quality preliminary screening and marking on the fields, and outputting cleaning metadata;

[0010] Step 5: Based on the cleaning task graph of step 3, the cleaning metadata of step 4, and the standardized storage data of step 2, call the adaptive components in the AI cleaning component library to automatically build a first AI data cleaning workflow containing multiple strategy candidate paths, and the first AI data cleaning workflow is embedded with a feedback collection mechanism to record execution process data and result indicators;

[0011] Step 6: Based on the first AI data cleaning workflow built in step 5, model the multiple strategy candidate paths and make optimization decisions, filter the optimal strategy path and update it to the first AI data cleaning workflow, and output the optimized second AI data cleaning workflow;

[0012] Step 7: Execute the second AI data cleaning workflow optimized in step 6 to output the cleaning result, collect the effect feedback data of the cleaning result from the downstream tasks, evaluate the cleaning effect based on the process data and result indicators recorded in step 5, and generate the evaluation result; based on the evaluation result, adjust the strategy path, replace the component, or reconfigure the cleaning task graph, and generate the evaluation adjustment information;

[0013] Step 8: Based on the cleaning result and evaluation adjustment information output in step 7, integrate and output multiple versions of the cleaning result.

[0014] Preferably, the step 1 includes the following sub-steps:

[0015] Step 1.1: Configure the access parameters of the heterogeneous data source;

[0016] Step 1.2: Based on the access parameters, perform data grabbing to obtain the original data; wherein the original data includes structured data, semi-structured data, and unstructured data;

[0017] Step 1.3: Extract field information for the structured data, semi-structured data, and unstructured data, respectively; generate a field definition table;

[0018] Step 1.4: Filter the text type fields in the field definition table; use a multi-language model to identify the language type and distribution of the text type fields, extract the semantic features of the text type fields, and form a field semantic metadata table;

[0019] Step 1.5: Based on the field definition table and the semantic metadata table, calculate the missing rate, duplication rate, format consistency, and noise probability of the fields; generate a quality score and mark the priority cleaning fields;

[0020] Step 1.6: Integrate the field definition table, semantic metadata table, and quality pre-judgment result, and output the structured metadata and quality preliminary evaluation result.

[0021] Preferably, the step 2 includes the following sub-steps:

[0022] Step 2.1: converting the field definition table, structured metadata and quality preliminary evaluation results generated in step 1 into standardized storage data in a unified format, and establishing an association index of fields and metadata through the binding of field identification and metadata identification, and outputting the standardized storage data and the association index;

[0023] Step 2.2: based on the standardized storage data and the association index, the hierarchical relationship between fields and the metadata association are combed through the association index to generate a structure identification report containing field statistical profiles, structure relationships and quality problem summaries;

[0024] Step 2.3: based on the structure identification report, the quality preliminary evaluation results, the standardized storage data and the association index, a mapping relationship between fields and cleaning tasks is established, a preset strategy template is bound, and fields requiring multi-strategy processing are marked;

[0025] Step 2.4: based on the mapping relationship established in step 2.3, a cleaning task queue is registered, the task identification, associated fields and processing priority are recorded, and task queue information is generated.

[0026] Preferably, the step 3 includes the following sub-steps:

[0027] Step 3.1: based on the structured metadata, quality preliminary evaluation results of step 1 and the standardized storage data of step 2, the content feature analysis of value domain distribution, frequency of occurrence and format characteristics is performed on each field in the field definition table combined with the association index of step 2; and a feature analysis result is generated;

[0028] Step 3.2: combined with the feature analysis result and the quality problem summary in the structure identification report of step 2, the quality problems of the fields are identified; the quality problems include missing, redundancy, format error and semantic conflict;

[0029] Step 3.3: defining cleaning targets according to business requirements and data application scenarios; the cleaning targets include improving field integrity, ensuring format uniformity and eliminating semantic conflicts;

[0030] Step 3.4: based on the field information, quality problem type, preset strategy library and cleaning target defined in step 3.3, a preliminary cleaning strategy candidate is matched;

[0031] Step 3.5: combined with the field relationship combed by the association index of step 2, the association rules and dependent logic between fields are analyzed, and a cleaning task graph containing task nodes, problem type labels, strategy candidates and dependent edges is constructed.

[0032] Preferably, the step 4 includes the following sub-steps:

[0033] Step 4.1: Based on the cleaning task graph generated in step 3 and the task queue information in step 2, combined with the cleaning target, determine the focus of the evaluation indicators, calculate the missing proportion, abnormal value proportion and cross-field conflict times of each field, and form the field evaluation indicators;

[0034] Step 4.2: According to the field evaluation indicators, preset rules and cleaning target, identify high-risk fields and generate abnormal labels;

[0035] Step 4.3: Combined with the dependency relationship in the cleaning task graph, the association index in step 2 and the cleaning target, perform multi-field joint verification to identify logical contradictions and duplicate records, and generate cross-field risk markers;

[0036] Step 4.4: Based on the abnormal label, cross-field risk marker, priority of task queue and cleaning target, construct a field cleaning priority list and output processing suggestions;

[0037] Step 4.5: Integrate field evaluation indicators, abnormal labels, cross-field risk markers and field cleaning priority list to form cleaning metadata and output.

[0038] Preferably, the step 5 includes the following sub-steps:

[0039] Step 5.1: Based on the strategy candidate in the cleaning task graph in step 3, the cleaning metadata in step 4 and the standardized storage data in step 2, call the appropriate processing components from the AI cleaning component library; The processing components include missing value processing components, format conversion components and duplicate data processing components;

[0040] Step 5.2: Generate multiple strategy candidate paths for each field cleaning task; Each path contains component call sequence and parameter configuration scheme;

[0041] Step 5.3: According to the dependency relationship of the cleaning task graph, bind the strategy candidate path with the processing component, and construct the first AI data cleaning workflow containing multiple strategy candidate paths;

[0042] Step 5.4: Embed a feedback collection mechanism after each component node in the first AI data cleaning workflow, configure the execution process data and result indicators that need to be recorded; Execution process data includes processing time, modification record; Result indicators include cleaning completion rate;

[0043] Step 5.5: Output the first AI data cleaning workflow containing multiple strategy candidate paths and feedback collection mechanism.

[0044] Preferably, the step 6 includes the following sub-steps:

[0045] Step 6.1: Based on the first AI data cleaning workflow constructed in step 5, a directed graph is used to structurally model multi-strategy candidate paths, define the execution order of sub-tasks and pre-requisite dependencies;

[0046] Step 6.2: Retrieve rule-based, model-based and hybrid strategies from the strategy library and supplement to the set of strategy candidate paths;

[0047] Step 6.3: Based on the adaptation degree of strategies and fields, estimated processing efficiency and historical execution effect, the candidate paths are comprehensively evaluated to select the optimal strategy path;

[0048] Step 6.4: Update the optimal strategy path to the first AI data cleaning workflow to form the optimized second AI data cleaning workflow.

[0049] Preferably, the step 7 includes the following sub-steps:

[0050] Step 7.1: Execute the second AI data cleaning workflow optimized in step 6 to output the cleaning result;

[0051] Step 7.2: Collect the effect feedback data of the cleaning result from the downstream tasks; the effect feedback data includes data availability and business rule compliance;

[0052] Step 7.3: Based on the execution process data recorded in step 5, the result indicators and the effect feedback data in step 7.2, the cleaning effect is comprehensively evaluated to generate an evaluation result;

[0053] Step 7.4: Based on the evaluation result, adjust the parameter configuration of the strategy path, replace the components that do not achieve the expected effect, or reconfigure the dependency relationship of the cleaning task graph to generate evaluation adjustment information.

[0054] Preferably, the step 8 includes the following sub-steps:

[0055] Step 8.1: Based on the cleaning result and evaluation adjustment information output in step 7, mark the cleaning state of each field and inject the version backtracking identifier to generate a preliminary cleaning data set; the cleaning state includes cleaned and pending manual review.

[0056] Step 8.2: Based on the preliminary cleaning data set, generate a unique version identifier for this cleaning, associate the version identifier with the original data version, the second AI data cleaning workflow version and the cleaning result sample to form a versioned cleaning data set and store it in the version management library.

[0057] Step 8.3: Based on the versioned cleaning data set, convert the output format according to the requirements of the downstream system, perform desensitization processing on sensitive fields to generate a cleaning data set adapted to the downstream system.

[0058] Step 8.4: Based on the cleaning data set adapted to the downstream system, generate a comparison analysis of data before and after cleaning, abnormal field processing statistics and strategy execution effect description, form a cleaning effect analysis report for business audit.

[0059] Step 8.5: Based on the cleaning effect analysis report, the field structure after cleaning, the optimal strategy path and the evaluation result, feedback to the metadata center and update the data asset information.

[0060] The beneficial effects of the present application are at least one of the following: capable of adapting to the processing needs of multiple heterogeneous data sources, providing a structured basis for subsequent cleaning process through field extraction, structure analysis and language attribute recognition of original data, which helps to improve the adaptability of multi-source heterogeneous data processing;

[0061] By constructing an AI data cleaning workflow containing multiple strategy candidate paths, and combining the feedback collection mechanism to model and optimize the strategy path, the dependence on static preset strategies is reduced, and the flexibility of cleaning strategies is improved to a certain extent;

[0062] Introducing cleaning effect evaluation and dynamic adjustment mechanism, optimizing strategy path, replacing components or reconstructing task graph based on downstream task feedback and execution process data, which helps to reduce the need for manual intervention and improve the self-adaptive ability of cleaning process;

[0063] Through the cooperation of cleaning task graph construction, quality preliminary screening mark and result integration output, the systematicness and standardization of data cleaning can be improved to a certain extent, which helps to provide more stable data quality support for high semantic requirement scenarios such as knowledge retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The method flowchart of the present embodiment is shown. DETAILED DESCRIPTION

[0065] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0066] Current data cleaning relies on artificial rules, ETL tools and preset processing library, supplemented by machine learning local processing and AI platform automatic arrangement. However, in the face of structured and unstructured mixed multi-source heterogeneous data, the existing static strategy is difficult to adapt to the changes of data structure, type and quality due to the lack of downstream task feedback optimization mechanism, and frequent manual intervention is required, resulting in unstable cleaning effect and low optimization efficiency, which cannot meet the dynamic needs of high semantic scenarios.

[0067] In order to solve the above problems, the present embodiment provides a method for data cleaning decision and automatic optimization using AI workflow, comprising the following steps:

[0068] Step 1: Obtain raw data from multiple heterogeneous data sources, perform field extraction, structure parsing and language attribute recognition on the raw data, and generate a field definition table, structured metadata and quality preliminary evaluation results.

[0069] In one possible implementation, the step 1 includes the following sub-steps:

[0070] Step 1.1: Configure access parameters of the heterogeneous data sources.

[0071] For example, the system provides a configurable data source access module, and a user defines access parameters through an interface or an API, including data source types, connection methods, pulling strategies and security authentication information. For non-static content, the system sets a polling frequency or a listening event trigger to achieve dynamic data acquisition.

[0072] Step 1.2: Perform data grabbing based on the access parameters to obtain raw data; wherein the raw data includes structured data, semi-structured data and unstructured data.

[0073] For example, according to the configured access parameters, structured table data is grabbed from a relational database, semi-structured document data is grabbed from a NoSQL database, structured file data such as CSV and Excel and unstructured file data such as TXT and PDF are grabbed from a file system, and semi-structured data in JSON and XML formats is grabbed from a RESTAPI interface.

[0074] It should be noted that preliminary standardization processing is performed synchronously during data grabbing: character sets are uniformly converted to UTF-8; special characters such as tab characters and carriage return characters are uniformly converted to standard spaces, and non-printing characters are automatically filtered; blank values, NULL and illegal encodings are uniformly marked as standard missing symbols. For compressed or encapsulated formats, the system automatically decompresses and recursively extracts internal content.

[0075] It should be noted that for large files exceeding 1 GB, a block grabbing mechanism is adopted, and multi-thread concurrent processing is used to avoid memory overflow, block data is automatically merged into a complete data set, and multiple types of raw data sets are formed.

[0076] Step 1.3: Extract field information for the structured data, semi-structured data and unstructured data, respectively; and generate a field definition table.

[0077] Exemplarily, different types of data are adopted with different structure identification strategies to generate the field definition table: structured data directly extracts field name, field type and field sample value and records to the field definition table; semi-structured data extracts nested fields through hierarchical traversal, establishes "path + key name" as field identification and records to the field definition table; unstructured data uses rule templates and NLP models for potential field extraction, such as using syntax analysis to identify time, place, number, noun phrases, extract title + text, question + answer pairs and other common structures, and output the results to the preliminary field definition table, which contains field path, original name, field type preliminary judgment, sample value and other information.

[0078] Step 1.4: Screening text type fields in the field definition table; using multilingual models to identify the language type and distribution of the text type fields, extracting semantic features of the text type fields to form a field semantic metadata table.

[0079] Exemplarily, the system screens out text type fields from the field definition table, automatically identifies the language type of these text field contents, and supports multilingual models; the judgment result supports "field level" language unification and "sample level" mixed language recognition; if the field is mixed language, record the dominant language probability and language distribution; for text type fields, extract semantic features such as average length, syntactic complexity, named entity density, keyword frequency distribution, and whether to contain HTML, Markdown and other mark structure information, which together with the field structure form the "field semantic metadata table".

[0080] Step 1.5: Based on the field definition table and the semantic metadata table, calculate the missing rate, duplication rate, format consistency and noise probability of the field; generate quality score and mark priority cleaning field.

[0081] Exemplarily, the system calculates the following quality indicators according to the field samples in the field definition table and the semantic metadata table: missing rate, duplication rate, format consistency, noise probability. Each field calculates a preliminary "quality score" Q (0-1), which is used for subsequent process scheduling and sorting. For fields with a score below a certain threshold, the system marks them as "fields that need priority cleaning".

[0082] Step 1.6: Integrate the field definition table, semantic metadata table and quality preliminary evaluation results, output structured metadata and quality preliminary evaluation results.

[0083] Exemplarily, the system integrates the field structure information in the field definition table, the language identification information in the semantic metadata table, and the quality score, and outputs the contents in a structured manner as a metadata configuration item to form structured metadata; meanwhile, the system aggregates quality assessment related information as a quality preliminary evaluation result. The system also generates a "structure identification log" to record data sources and connection information, key decisions in field parsing process, abnormal fields and structure conflicts, and other alarm information, and the log is used for process tracing and debugging assistance.

[0084] Step 2: Based on the field definition table, the structured metadata, and the quality preliminary evaluation result generated in step 1, the system performs structure identification result storage and task generation, and outputs standardized storage data and task queue information.

[0085] In a possible implementation, the step 2 includes the following sub-steps:

[0086] Step 2.1: The system converts the field definition table, the structured metadata, and the quality preliminary evaluation result generated in step 1 into standardized storage data in a unified format, and establishes an association index of the fields and the metadata through the binding of the field identifiers and the metadata identifiers, and finally outputs the standardized storage data and the association index.

[0087] Exemplarily, the system uniformly converts the field definition table, the structured metadata, and the quality preliminary evaluation result generated in step 1 into a structured description format that is readable and controllable, to form the standardized storage data. The main standardization processing includes field description standardization, additional quality attribute integration, and standard format structure packaging. Meanwhile, the system establishes an association index of the fields and the metadata through the binding of the field identifiers and the metadata identifiers, and finally outputs the standardized storage data and the association index.

[0088] Step 2.2: Based on the standardized storage data and the association index, the system combs the hierarchical relationship between the fields and the metadata association through the association index, and generates a structure identification report containing field statistical overview, structure relationship, and quality problem summary.

[0089] Exemplarily, based on the standardized storage data and the association index, the system automatically generates a structure identification visual report by combing the hierarchical relationship between the fields and the metadata association through the association index. The report content includes field statistical overview, data quality scoring chart, and field processing suggestion, to form a structure identification report containing field statistical overview, structure relationship, and quality problem summary, which can be generated in HTML, PDF, Markdown, or other formats.

[0090] Step 2.3: Based on the structure identification report, the quality preliminary evaluation result, the standardized storage data, and the association index, the system establishes a mapping relationship between the fields and the cleaning tasks, binds a preset strategy template, and marks the fields that need to be processed by multiple strategies.

[0091] Exemplarily, the system establishes a mapping relationship between the fields and the cleaning tasks according to the structure identification report, the quality preliminary evaluation result, the standardized storage data and the association index, forms a field-task mapping relationship table, and clearly shows the cleaning operation list corresponding to each field, whether to enable the rule module or the model module and the like. Meanwhile, the system binds the existing cleaning strategy template in the system for the corresponding field, and generates a "to be configured" state task if there is no matching strategy. The field or structure with uncertainty is automatically marked as a field requiring multi-strategy processing, triggering the subsequent strategy competition and dynamic learning mechanism.

[0092] Step 2.4: Based on the mapping relationship established in step 2.3, register the cleaning task queue, record the task identification, associated field and processing priority, and generate the task queue information.

[0093] Exemplarily, the system registers the cleaning task queue to the task scheduler based on the mapping relationship between the fields and the cleaning tasks established in step 2.3, records the identification, associated field and processing priority of each task and the like. If there is an asynchronous model loading / rule generation requirement, an initialization request is initiated to the corresponding module; the cleaning metadata version and the process state are updated, and finally the task queue information is generated for the front end or monitoring system to visually track.

[0094] Step 3: Based on the structured metadata in step 1, the quality preliminary evaluation result and the standardized storage data in step 2, content feature analysis is performed on the fields in the field definition table, quality problems of the fields are identified and cleaning targets are defined, and a cleaning task graph containing field information, problem type, strategy candidate and dependency relationship is constructed.

[0095] In a possible implementation, the step 3 includes the following sub-steps:

[0096] Step 3.1: Based on the structured metadata in step 1, the quality preliminary evaluation result and the standardized storage data in step 2, and in combination with the association index in step 2, value domain distribution, frequency of occurrence and format feature content feature analysis is performed on each field in the field definition table; and a feature analysis result is generated.

[0097] Exemplarily, the system performs content feature analysis on each field in the field definition table based on the structured metadata in step 1, the quality preliminary evaluation result and the standardized storage data in step 2, and in combination with the association index in step 2, including distribution statistics and anomaly detection, and each field forms a feature analysis result containing a list of statistical attributes + anomaly labels.

[0098] Step 3.2: In combination with the feature analysis result and the quality problem summary in the structure identification report in step 2, the quality problems of the fields are identified; the quality problems include missing, redundancy, format error and semantic conflict.

[0099] Exemplarily, the system combines the feature analysis result and the quality problem summary in the structure recognition report of step 2, automatically identifies the quality problems of the field based on the field statistical result and the predefined quality rules, and the quality problems mainly include missing problems, redundancy problems, ambiguity problems, format problems and semantic problems, which are classified into standard quality problem types such as missing, redundancy, format error and semantic conflict.

[0100] Step 3.3: Defining cleaning targets according to business requirements and data application scenarios; the cleaning targets include improving field integrity, ensuring format uniformity and eliminating semantic conflict.

[0101] Exemplarily, the system defines the cleaning targets according to business requirements and data application scenarios, combines the field importance score and the problem severity, and the cleaning targets include improving field integrity, ensuring format uniformity and eliminating semantic conflict. Meanwhile, the priority score of the cleaning task is automatically generated, and a cleaning strategy label is marked for each field.

[0102] Step 3.4: Matching preliminary cleaning strategy candidates based on field information, quality problem types, preset strategy library and cleaning targets defined in step 3.3.

[0103] Exemplarily, the system analyzes the structural dependency relationship and semantic dependency relationship among the field information based on the field information, the identified quality problem types, the preset strategy library and the cleaning targets defined in step 3.3, constructs a dependency graph for guiding the execution order of the cleaning task, matches preliminary cleaning strategy candidates for each field, and marks the fields with dependency relationship with a "cleaning needs to be synchronized" mark.

[0104] Step 3.5: Combining the field relationship combed by the association index in step 2, analyzing the association rules and dependency logic among the fields, and constructing a cleaning task graph containing task nodes, problem type labels, strategy candidates and dependency edges.

[0105] Exemplarily, the system combines the field relationship combed by the association index in step 2, analyzes the association rules and dependency logic among the fields, constructs a cleaning task graph according to the field problem classification, priority score, strategy label and dependency relationship. In the graph, the node represents a field + target cleaning task combination, the edge represents the field dependency relationship or the processing order, each node contains the field name, the cleaning type, the recommended model and other attributes, forming a cleaning task graph containing task nodes, problem type labels, strategy candidates and dependency edges, and supporting the graph database persistence.

[0106] Step 4: Based on the cleaning task graph generated in step 3 and the task queue information in step 2, performing data quality preliminary screening and marking on the fields, and outputting cleaning metadata.

[0107] In one possible implementation, the step 4 includes the following substeps:

[0108] Step 4.1: Based on the cleaning task graph generated in step 3 and the task queue information in step 2, the system determines the focus of the evaluation indicators in combination with the cleaning target, calculates the missing proportion, abnormal value proportion and cross-field conflict times of each field, and forms field evaluation indicators.

[0109] For example, based on the cleaning task graph generated in step 3 and the task queue information in step 2, the system determines the focus of the evaluation indicators in combination with the cleaning target, performs statistical data quality evaluation at the field level for all identified fields, calculates indicators such as missing proportion, abnormal value proportion and cross-field conflict times, including missing value proportion statistics, repetition rate, uniqueness detection, etc., and forms field evaluation indicators.

[0110] Step 4.2: According to the field evaluation indicators, preset rules and cleaning target, identify high-risk fields and generate abnormal labels.

[0111] For example, based on built-in rules and threshold judgments, the system identifies high-risk fields with quality risks, including missing rate exceeding the set threshold and containing high-frequency illegal characters, according to the problem type, and generates corresponding abnormal labels for these fields.

[0112] Step 4.3: In combination with the dependency relationship in the cleaning task graph, the association index in step 2 and the cleaning target, perform multi-field joint verification, identify logical contradictions and duplicate records, and generate cross-field risk markers.

[0113] For example, in combination with the dependency relationship in the cleaning task graph, the association index in step 2 and the cleaning target, the system performs multi-field joint verification, including joint uniqueness detection, inter-field logical conflict identification and field dependency analysis, identifies logical contradictions and duplicate records, and generates cross-field risk markers for use in complex strategy generation or model intervention modules.

[0114] Step 4.4: Based on the abnormal labels, cross-field risk markers, task queue priorities and cleaning target, construct a field cleaning priority list and output processing suggestions.

[0115] For example, based on the abnormal labels, cross-field risk markers, task queue priorities and cleaning target, the system automatically constructs a field cleaning priority list in combination with the field quality evaluation results and label information, clearly identifies high-risk fields for priority processing and low-risk fields for deferred cleaning, and outputs corresponding processing suggestions as decision signals to the cleaning strategy path modeling module.

[0116] Step 4.5: Integrate field evaluation indicators, abnormal labels, cross-field risk markers and field cleaning priority lists to form cleaning metadata and output.

[0117] Exemplarily, the system integrates the field evaluation index, the exception label, the cross-field risk label, and the field cleaning priority list, structures the information into the metadata storage layer of the system to form cleaning metadata including the field name, type, each quality index value, system automatic labeling result, and the like, and outputs the cleaning metadata as the basic input for modeling the cleaning task.

[0118] Step 5: Based on the cleaning task graph in step 3, the cleaning metadata in step 4, and the standardized storage data in step 2, an adaptive component in the AI cleaning component library is called to automatically construct a first AI data cleaning workflow containing multiple strategy candidate paths, and the first AI data cleaning workflow is embedded with a feedback collection mechanism to record the execution process data and result index.

[0119] In a possible implementation, the step 5 includes the following sub-steps:

[0120] Step 5.1: Based on the strategy candidate in the cleaning task graph in step 3, the cleaning metadata in step 4, and the standardized storage data in step 2, an adaptive processing component is called from the AI cleaning component library; the processing component includes a missing value processing component, a format conversion component, and a duplicate data processing component.

[0121] Exemplarily, based on the strategy candidate in the cleaning task graph in step 3, the cleaning metadata in step 4, and the standardized storage data in step 2, an adaptive processing component is called from the AI cleaning component library according to the cleaning strategy label and field type of each task node, including a missing value processing component, a format conversion component, and a duplicate data processing component, and a language model is automatically selected according to the field language attribute.

[0122] Step 5.2: Multiple strategy candidate paths are generated for the cleaning task of each field; each path contains the component calling sequence and parameter configuration scheme.

[0123] Exemplarily, multiple strategy candidate paths are generated for the cleaning task of each field, and each path explicitly contains the component calling sequence and parameter configuration scheme. For example, the "missing completion" path is [rule judgment]→[model inference]→[confidence filtering], the "semantic correction" path is [regular preliminary screening]→[context-aware correction model]→[manual intervention candidate], and corresponding parameters are injected into each module in each path.

[0124] Step 5.3: According to the dependency relationship of the cleaning task graph, the strategy candidate path is bound with the processing component to construct a first AI data cleaning workflow containing multiple strategy candidate paths.

[0125] Exemplarily, the system binds the strategy candidate path generated in step 5.2 with the processing component called in step 5.1 according to the dependency relationship of the cleaning task graph, and constructs a first AI data cleaning workflow. The workflow adopts a directed acyclic graph structure, wherein a node represents a cleaning task, an edge represents a field dependency or a sequence relationship, synchronous / asynchronous tasks in the graph can be automatically and concurrently divided, and a unified normalization node is introduced as a strategy scheduling point for a node with an aggregation dependency, so as to form a first AI data cleaning workflow containing multiple strategy candidate paths.

[0126] Step 5.4: Embed a feedback collection mechanism after each component node of the first AI data cleaning workflow, and configure the execution process data and result indicators that need to be recorded. The execution process data includes processing time length and modification record. The result indicators include cleaning completion rate.

[0127] Exemplarily, the system embeds a feedback collection mechanism after each component node of the first AI data cleaning workflow, and configures the execution process data and result indicators that need to be recorded. At the same time, intelligent decision triggers are set for key steps in the workflow, the conditions include model confidence lower than threshold value→fallback rule module, and the condition branch node supports Boolean expression and strategy script language configuration, which improves flexibility.

[0128] Step 5.5: Output the first AI data cleaning workflow containing multiple strategy candidate paths and feedback collection mechanisms.

[0129] Exemplarily, the system outputs the first AI data cleaning workflow constructed, which contains multiple strategy candidate paths and feedback collection mechanisms. The feedback collection component embedded after each cleaning module node supports automatic recording of field quality indicators before / after cleaning, module execution success rate and other information, and all indicators are written into a log database. The generated first AI data cleaning workflow is registered as a traceable version, containing workflow version ID, task node graph and other information, and can be exported in standard formats such as YAML, JSON or BPMN for integration and calling.

[0130] Step 6: Based on the first AI data cleaning workflow constructed in step 5, model the multiple strategy candidate paths and make optimization decisions, filter the optimal strategy path and update it to the first AI data cleaning workflow, and output the optimized second AI data cleaning workflow.

[0131] In one possible implementation, the step 6 includes the following sub-steps:

[0132] Step 6.1: Based on the first AI data cleaning workflow constructed in step 5, structure the multiple strategy candidate paths by using a directed graph to define the execution order and pre-dependence of sub-tasks.

[0133] Exemplarily, the system constructs a first AI data cleaning workflow based on step 5, uses a directed graph to structurally model the multi-strategy candidate paths therein, clearly defines the cleaning sub-tasks required for each field, the sub-task execution order and dependency relationship, and the list of optional strategies for each sub-task, defines the execution order and pre-requisite dependencies of the sub-tasks, and the structural modeling can be in the form of a DAG or a hierarchical state machine.

[0134] Step 6.2: Retrieve rule-type, model-type and hybrid strategies from the strategy library and supplement to the strategy candidate path set.

[0135] Exemplarily, the system retrieves rule-type strategies, model-type strategies and hybrid strategies from the strategy library and supplements them to the strategy candidate path set of the first AI data cleaning workflow, forming a "multi-path cleaning scheme" alternative set, each strategy with meta-attribute descriptions such as applicable conditions, past effect scores, etc.

[0136] Step 6.3: Based on the degree of adaptation of strategies to fields, estimated processing efficiency and historical execution effect, comprehensively evaluate the candidate paths and select the optimal strategy path.

[0137] Exemplarily, the system comprehensively evaluates and scores the candidate paths in the first AI data cleaning workflow based on the degree of adaptation of strategies to fields, estimated processing efficiency and historical execution effect, etc. as shown in Table 1, and the scoring model can be driven by a lightweight machine learning model.

[0138] Table 1 is a dimension specification table for comprehensive evaluation and scoring of candidate paths

[0139] Score dimensions Explanation Fitting score The degree of matching between the current field structure + data sample and the historical performance of the strategy Performance prediction Execution time, resource consumption, model complexity, etc. Quality prediction Integrity, standardization, semantic accuracy prediction of the field after cleaning User preference guidance If there is a business user annotated preference, weighted reference in the strategy score Historical feedback weight The historical accuracy or problem rate of the strategy in similar tasks

[0140] Finally, the path with the highest score is selected as the optimal strategy path, and other candidate paths are retained for AB testing or subsequent optimization.

[0141] Step 6.4: Update the optimal strategy path to the first AI data cleaning workflow to form an optimized second AI data cleaning workflow.

[0142] Exemplarily, the system updates the optimal strategy path selected in step 6.3 to the first AI data cleaning workflow, replacing the original default path or adjusting the path priority, to form an optimized second AI data cleaning workflow. This workflow maintains the overall structure while improving the cleaning efficiency and reliability of the effect.

[0143] Step 7: execute the second AI data cleaning workflow optimized in step 6 to output the cleaning result, collect the effect feedback data of the downstream task on the cleaning result, combine the process data recorded in step 5 and the result index to evaluate the cleaning effect, and generate an evaluation result; based on the evaluation result, adjust the parameter configuration of the strategy path, replace the components that do not achieve the expected effect, or reconstruct the dependency relationship of the cleaning task graph, and generate evaluation adjustment information.

[0144] In one possible implementation, step 7 includes the following sub-steps:

[0145] Step 7.1: execute the second AI data cleaning workflow optimized in step 6 to output the cleaning result.

[0146] For example, the system executes the second AI data cleaning workflow optimized in step 6, automatically executes the cleaning task according to the path structure, and supports state monitoring, intermediate result caching, multi-path parallel cleaning, and exception branch jump mechanism at the sub-task level during the execution process. After the task execution is completed, the system marks the version number of the cleaning result and outputs it for subsequent evaluation modules to reference.

[0147] Step 7.2: collect the effect feedback data of the downstream task on the cleaning result; the effect feedback data includes data availability and business rule compliance.

[0148] For example, after the cleaning result is output, the system automatically collects the effect feedback data of the downstream task on the cleaning result, including the integrity, uniqueness, and distribution changes of the cleaned fields, the accuracy and recall rate compared with manual annotation or existing benchmarks, user manual revision records, whether the result is adopted or abandoned by the downstream system, etc. These data cover data availability and business rule compliance, and will be input into the model retraining mechanism or the strategy scoring system as reinforcement learning signals or training samples.

[0149] Step 7.3: combine the execution process data recorded in step 5, the result index, and the effect feedback data in step 7.2 to comprehensively evaluate the cleaning effect and generate an evaluation result.

[0150] For example, the system combines the execution process data recorded in step 5, the result index, and the effect feedback data in step 7.2 to comprehensively evaluate the cleaning effect from multiple dimensions such as cleaning quality, efficiency, and resource consumption, and generates an evaluation result containing various evaluation indexes and overall evaluation.

[0151] Step 7.4: based on the evaluation result, adjust the parameter configuration of the strategy path, replace the components that do not achieve the expected effect, or reconstruct the dependency relationship of the cleaning task graph, and generate evaluation adjustment information.

[0152] Exemplarily, the system optimizes and adjusts the second AI data cleaning workflow based on the evaluation result: if the current cleaning strategy effect continuously falls below the historical average, the last version strategy is automatically switched or restored; for the case where there are multiple candidate paths, the winner is selected for replacement through evaluation score; the cleaning task path is modeled as a Markov decision process, and the RL strategy is used for path update; the strategy with long-term poor performance is marked as "candidate elimination" and a new strategy version is introduced. These adjustments include parameter configuration adjustment of the strategy path, replacement of components with unexpected effect, or reconstruction of the cleaning task graph dependency, and finally generate evaluation adjustment information.

[0153] Step 8: Based on the cleaning result and evaluation adjustment information output in step 7, the cleaning result is integrated and multiple version outputs are generated.

[0154] In one possible implementation, the step 8 includes the following sub-steps:

[0155] Step 8.1: Based on the cleaning result and evaluation adjustment information output in step 7, mark the cleaning state of each field and inject version backtracking identifier to generate a preliminary cleaning dataset; the cleaning state includes cleaned and pending manual review.

[0156] Exemplarily, the system integrates and processes the cleaning results from different sub-modules based on the cleaning result and evaluation adjustment information output in step 7, and maps and binds the cleaning result with the original data by field. For the multiple candidate cleaning results generated under the "multi-path strategy", the optimal one is selected, the cleaning state of each field is marked, and the version backtracking identifier is injected, and finally a preliminary cleaning dataset containing complete field mapping relationship, state identifier and backtracking information is generated.

[0157] Step 8.2: Based on the preliminary cleaning dataset, generate a unique version identifier for this cleaning, associate the version identifier with the original data version, the second AI data cleaning workflow version and the cleaning result sample, form a versioned cleaning dataset and store it in the version management library.

[0158] Exemplarily, the system generates a unique version identifier based on the preliminary cleaning dataset, binds the identifier with the original data version, the second AI data cleaning workflow version and the cleaning result sample through the version association mechanism, forms a versioned cleaning dataset containing complete version pedigree, and stores it in the version management library through the distributed storage engine, supports incremental update and full backtracking.

[0159] Step 8.3: Based on the versioned cleaning dataset, convert the output format according to the downstream system requirements, perform desensitization processing on sensitive fields, and generate a cleaning dataset adapted to the downstream system.

[0160] Exemplarily, the system is based on the versioned cleaning data set, parses the interface protocol and data format requirements of the downstream system: converts into RDF triple format for the knowledge graph system, converts into JSONLines format for the search engine, and converts into Parquet columnar storage format for the BI tool; at the same time, desensitization is performed on sensitive fields such as "ID card number" and "bank card number", and through dynamic desensitization rules, the desensitization granularity of different downstream systems is adapted to the permission level, and finally the cleaning data set adapted to each downstream system is generated.

[0161] Step 8.4: Based on the cleaning data set adapted to the downstream system, generate a data comparison analysis before and after cleaning, an abnormal field processing statistics and a strategy execution effect description, form a cleaning effect analysis report for business audit.

[0162] Exemplarily, the system automatically generates a cleaning effect analysis report based on the cleaning data set adapted to the downstream system: the comparison part before and after cleaning shows the distribution change of numerical fields through the box plot, and shows the text correction accuracy through the confusion matrix; the abnormal processing statistics part summarizes the number of missing data filling, the number of format correction and the number of duplicate elimination of each field; the strategy effect description part compares the execution accuracy of different strategies through the ROC curve, and marks the parameter configuration and applicable scene of the optimal strategy. The report supports online preview and PDF export, which can be used by business personnel to audit the cleaning quality and strategy effectiveness.

[0163] Step 8.5: Based on the cleaning effect analysis report, the field structure after cleaning, the optimal strategy path and the evaluation result, feedback to the metadata center and update the data asset information.

[0164] Exemplarily, the system packs the key indicators in the cleaning effect analysis report, the field structure after cleaning, the optimal strategy path in the second AI data cleaning workflow and the evaluation result generated in step 7 and feeds back to the metadata center, updates the field quality rating, recommended cleaning strategy and version evolution record in the data asset directory, provides optimization basis for subsequent data access and cleaning tasks, and forms a full life cycle management closed loop of data assets.

[0165] In summary, through the multi-path strategy candidate and dynamic scoring mechanism, the application can automatically select the optimal cleaning scheme for different data fields and scenes, avoid the blind application of traditional single rule or model, significantly improve the accuracy and semantic rationality of the cleaning effect, and enhance the intelligent level of data cleaning.

[0166] The application constructs a closed loop mechanism from cleaning execution, effect monitoring, feedback collection to strategy path self-adjustment, supports automatic retraining of the model and dynamic optimization of the strategy based on the actual effect, solves the problem of frequent dependence on manual intervention and long tuning period in the prior art, greatly reduces the operation and maintenance cost, and improves the system adaptability and robustness.

[0167] The application supports version control and format diversification output of cleaning results, meets the access requirements of different downstream business systems and data platforms, facilitates historical version tracing and data quality comparison, enhances the standardization and transparency of data governance, and effectively improves the management efficiency and reuse value of data assets.

[0168] Through automatic identification and self-recovery scheduling functions, the application can timely discover potential abnormalities and bottlenecks in the cleaning process, automatically adjust the execution strategy or switch to a backup solution, ensure the continuity and stability of data cleaning work, reduce system failure rate, and improve overall production efficiency.

[0169] By using AI workflow automatic optimization, the dependence on professional data engineers and the frequency of manual intervention are greatly reduced, the cleaning cycle is shortened, and the labor cost is reduced. At the same time, by optimizing the model training and inference process, the utilization rate of computing resources is improved, the cleaning task is efficiently run, and the cost performance is significantly better than traditional manual rule cleaning or static model cleaning scheme.

[0170] The application supports structure recognition, quality evaluation and task splitting of multi-source data, can flexibly combine multiple cleaning modules and strategies, adapts to complex and variable data scenarios, has good expansibility and adaptability, and provides a solid foundation for subsequent function iteration and technology upgrade.

[0171] The above-described embodiments only express the specific implementation of the application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are within the scope of protection of the application.

Claims

1. A method for data cleaning decision-making and automated optimization using AI workflow, characterized in that, Includes the following steps: Step 1: Obtain raw data from multiple heterogeneous data sources, extract fields, parse the structure, and identify language attributes from the raw data to generate a field definition table, structured metadata, and preliminary quality assessment results; Step 2: Based on the field definition table, structured metadata and initial quality assessment results generated in Step 1, store the structure recognition results and generate tasks, and output standardized storage data and task queue information; Step 3: Based on the structured metadata and initial quality assessment results from Step 1 and the standardized stored data from Step 2, perform content feature analysis on the fields in the field definition table, identify the quality problems of the fields and define the cleaning targets, and construct a cleaning task map that includes field information, problem types, strategy candidates and dependencies. Step 4: Based on the cleaning task map generated in Step 3 and the task queue information in Step 2, perform initial data quality screening and labeling on the fields, and output cleaning metadata; Step 5: Based on the cleaning task map in Step 3, the cleaning metadata in Step 4, and the standardized storage data in Step 2, call the adaptable components in the AI ​​cleaning component library to automatically construct the first AI data cleaning workflow containing multiple strategy candidate paths. The first AI data cleaning workflow embeds a feedback collection mechanism to record execution process data and result indicators. Step 6: Based on the first AI data cleaning workflow built in Step 5, model and optimize the multi-strategy candidate paths, select the optimal strategy path and update it to the first AI data cleaning workflow, and output the optimized second AI data cleaning workflow. Step 7: Execute the second AI data cleaning workflow optimized in Step 6 to output cleaning results, collect feedback data on the effect of downstream tasks on the cleaning results, evaluate the cleaning effect by combining the process data and result indicators recorded in Step 5, and generate evaluation results; adjust the strategy path, replace components or reconstruct the cleaning task map based on the evaluation results, and generate evaluation adjustment information. Step 8: Based on the cleaning results and evaluation and adjustment information output in Step 7, integrate the cleaning results and output multiple versions.

2. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: Configure the access parameters for heterogeneous data sources; Step 1.2: Perform data capture based on the access parameters to obtain raw data; wherein, the raw data includes structured data, semi-structured data, and unstructured data; Step 1.3: Extract field information for the structured data, semi-structured data, and unstructured data respectively; generate a field definition table; Step 1.4: Filter the text fields in the field definition table; use a multilingual model to identify the language type and distribution of the text fields, extract the semantic features of the text fields, and form a field semantic metadata table; Step 1.5: Based on the field definition table and the semantic metadata table, calculate the field's missing rate, duplication rate, format consistency, and noise probability; generate a quality score and mark fields that should be cleaned first; Step 1.6: Integrate the field definition table, semantic metadata table, and quality prediction results, and output structured metadata and initial quality assessment results.

3. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 2, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Convert the field definition table, structured metadata, and initial quality assessment results generated in Step 1 into standardized storage data in a unified format, and establish an association index between fields and metadata by binding field identifiers and metadata identifiers, and output the standardized storage data and association index. Step 2.2: Based on the standardized stored data and associated indexes, the hierarchical relationships and metadata associations between fields are sorted out through the associated indexes, and a structure identification report containing field statistical overview, structural relationships and quality problem summaries is generated; Step 2.3: Based on the structure identification report, the initial quality assessment results, standardized storage data and associated indexes, establish a mapping relationship between fields and cleaning tasks, bind preset strategy templates, and mark fields that require multi-strategy processing; Step 2.4: Based on the mapping relationship established in Step 2.3, register the cleaning task queue, record the task identifier, associated fields and processing priority, and generate task queue information.

4. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Based on the structured metadata and initial quality assessment results from Step 1 and the standardized stored data from Step 2, and combined with the associated index from Step 2, perform content feature analysis on the value range distribution, frequency of occurrence, and format characteristics of each field in the field definition table; Generate feature analysis results; Step 3.2: Combining the feature analysis results and the quality problem summary in the structure identification report of Step 2, identify the quality problems of the fields; quality problems include missing, redundancy, format errors and semantic conflicts; Step 3.3: Define the cleaning objectives based on business needs and data application scenarios; the cleaning objectives include improving field integrity, ensuring format consistency, and eliminating semantic conflicts; Step 3.4: Based on field information, quality problem type, preset strategy library, and cleaning target defined in Step 3.3, match preliminary cleaning strategy candidates; Step 3.5: Based on the field relationships identified in Step 2, analyze the association rules and dependency logic between fields, and construct a cleaning task graph that includes task nodes, problem type labels, strategy candidates, and dependency edges.

5. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 4, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Based on the cleaning task map generated in Step 3 and the task queue information in Step 2, determine the focus of the evaluation indicators in combination with the cleaning objectives, calculate the missing ratio, outlier ratio and cross-field conflict number of each field, and form field evaluation indicators. Step 4.2: Based on the field evaluation indicators, preset rules, and cleaning objectives, identify high-risk fields and generate anomaly tags; Step 4.3: Combining the dependencies in the cleaning task graph, the association index in Step 2, and the cleaning target, perform multi-field joint verification to identify logical contradictions and duplicate records, and generate cross-field risk markers; Step 4.4: Based on anomaly tags, cross-field risk markers, task queue priorities, and cleaning targets, construct a field cleaning priority list and output processing suggestions; Step 4.5: Integrate field evaluation metrics, anomaly tags, cross-field risk markers, and field cleaning priority lists to form cleaning metadata and output it.

6. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 3, characterized in that, Step 5 includes the following sub-steps: Step 5.1: Based on the strategy candidates in the cleaning task graph in Step 3, the cleaning metadata in Step 4, and the standardized storage data in Step 2, call the appropriate processing components from the AI ​​cleaning component library; the processing components include missing value processing components, format conversion components, and duplicate data processing components; Step 5.2: Generate multiple policy candidate paths for each field's cleaning task; each path includes the component call order and parameter configuration scheme; Step 5.3: Based on the dependencies of the cleaning task graph, bind the policy candidate paths to the processing components to build the first AI data cleaning workflow containing multiple policy candidate paths; Step 5.4: Embed a feedback collection mechanism after each component node of the first AI data cleaning workflow, and configure the execution process data and result indicators to be recorded; the execution process data includes processing time and modification records; the result indicators include cleaning completion rate; Step 5.5: Output the first AI data cleaning workflow, which includes multiple strategy candidate paths and feedback collection mechanisms.

7. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 1, characterized in that, Step 6 includes the following sub-steps: Step 6.1: Based on the first AI data cleaning workflow built in Step 5, a directed graph is used to perform structured modeling of multi-strategy candidate paths, defining the execution order and prerequisite dependencies of subtasks; Step 6.2: Retrieve rule-based, model-based, and hybrid strategies from the strategy library and add them to the strategy candidate path set; Step 6.3: Based on the compatibility between the strategy and the fields, the estimated processing efficiency, and the historical execution results, comprehensively evaluate the candidate paths and select the optimal strategy path; Step 6.4: Update the optimal strategy path to the first AI data cleaning workflow to form the optimized second AI data cleaning workflow.

8. The method for data cleaning decision-making and automated optimization using AI workflow according to claim 1, characterized in that, Step 7 includes the following sub-steps: Step 7.1: Execute the second AI data cleaning workflow optimized in Step 6 and output the cleaning results; Step 7.2: Collect feedback data on the effectiveness of downstream tasks on the cleaning results; the feedback data includes data availability and compliance with business rules; Step 7.3: Combining the execution process data and result indicators recorded in Step 5 with the effect feedback data in Step 7.2, conduct a comprehensive evaluation of the cleaning effect and generate an evaluation result; Step 7.4: Based on the evaluation results, adjust the parameter configuration of the strategy path, replace components that do not achieve the expected results, or reconstruct the dependencies of the cleaning task graph to generate evaluation adjustment information.

9. A method for data cleaning decision-making and automated optimization using AI workflow according to claim 1, characterized in that, Step 8 includes the following sub-steps: Step 8.1: Based on the cleaning results and evaluation and adjustment information output in Step 7, mark the cleaning status of each field and inject a version backtracking identifier to generate a preliminary cleaned dataset; the cleaning status includes cleaned and awaiting manual review; Step 8.2: Based on the preliminary cleaned dataset, generate a unique version identifier for this cleaned dataset, associate this version identifier with the original data version, the second AI data cleaned workflow version, and the cleaned result sample to form a versioned cleaned dataset and store it in the version management library; Step 8.3: Based on the versioned clean dataset, convert the output format according to the requirements of the downstream system, perform desensitization processing on sensitive fields, and generate a clean dataset adapted to the downstream system; Step 8.4: Based on the cleaned dataset adapted to the downstream system, generate a comparative analysis of data before and after cleaning, statistics on handling abnormal fields, and an explanation of the strategy execution effect, forming a cleaning effect analysis report for business review. Step 8.5: Based on the cleaning effect analysis report, the cleaned field structure, the optimal strategy path and evaluation results, feed back to the metadata center to update the data asset information.

Images