Data intelligent cleaning methods, devices, computer equipment and storage media

Abstract

This invention discloses a data intelligent cleaning method, apparatus, computer equipment, and storage medium. The method includes: acquiring data to be cleaned; performing preliminary processing on the data to be cleaned to obtain preliminary processing results; performing rule encoding on the preliminary processing results to obtain encoded results; inputting the preliminary processing results and the encoded results into a dynamic GAN network model for cleaning to obtain cleaned results; wherein the dynamic GAN network model is trained using a rule-constrained generative adversarial architecture combined with real-time performance monitoring and adaptive adjustment mechanisms. The cleaning results are processed to obtain cleaned data; the cleaned data and a cleaning report are output, wherein the cleaning report includes the cleaning process. By implementing the method of this invention, high-precision cleaning of policy information, maintenance of semantic consistency, and low-latency processing can be achieved, significantly improving data quality and business response efficiency.

Description

Technical Field

[0001] This invention relates to data cleaning methods, and more specifically to intelligent data cleaning methods, apparatus, computer equipment, and storage media. Background Technology

[0002] In the field of intelligent data cleaning, current technologies primarily rely on traditional ETL (Extract, Transform, Load) methods based on hard-coded rules, KNN (K-Nearest Neighbors) and random forest anomaly detection algorithms, and generative adversarial networks. Traditional ETL techniques implement data processing logic by writing specific code. While this method can be customized for specific data sources and environments, it requires code rewriting and debugging when business needs or data structures change, a complex and time-consuming process. Furthermore, because traditional ETL systems typically employ timed scheduling mechanisms, their real-time data processing capabilities are limited, making it difficult to meet the demands of real-time data analysis.

[0003] For anomaly detection, KNN and Random Forest are two commonly used algorithms. KNN is suitable for small datasets and scenarios requiring fast response; however, it has high computational cost and its performance is greatly affected by parameter selection. In contrast, Random Forest is more suitable for handling complex datasets. Although it has high accuracy and scalability, it has a long training time, poor model interpretability, and is easily affected by noisy features, performing poorly with imbalanced data. These issues limit the widespread application and efficiency improvement of these two algorithms in intelligent data cleaning.

[0004] Finally, Generative Adversarial Networks (GANs), as a powerful data simulation tool, have shown great potential in learning real-world data distributions. However, GANs face challenges in practical applications such as pattern collapse, vanishing gradients, or exploding gradients, resulting in a lack of diversity and stability in the generated results. These technical bottlenecks make GANs inadequate when dealing with the real distribution of data, especially rare but crucial anomalous patterns.

[0005] Therefore, it is necessary to design a new method to achieve high-precision cleaning, semantic consistency maintenance, and low-latency processing of policy information, thereby significantly improving data quality and business response efficiency; in order to address the shortcomings of existing technologies in terms of flexibility, real-time performance, data diversity processing, model robustness, and generalization ability. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data intelligent cleaning method, apparatus, computer equipment and storage medium.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a data intelligent cleaning method, comprising:

[0008] Obtain the data to be cleaned;

[0009] The data to be cleaned is subjected to preliminary processing to obtain preliminary processing results;

[0010] The preliminary processing results are then encoded according to rules to obtain the encoded results;

[0011] The preliminary processing results and the encoding results are input into the dynamic GAN network model for cleaning to obtain the cleaning results; wherein, the dynamic GAN network model is trained by rule-constrained generative adversarial architecture, combined with real-time performance monitoring and adaptive adjustment mechanism.

[0012] The cleaning results are processed to obtain cleaned data;

[0013] Output the cleaned data and a cleaning report, wherein the cleaning report includes the cleaning process.

[0014] The further technical solution is as follows: the preliminary processing of the data to be cleaned to obtain preliminary processing results includes:

[0015] Intelligently selects reading strategies based on the size of the data to be cleaned and automatically adapts to the encoding format to parse the data;

[0016] The key metadata information of the data to be cleaned is automatically extracted and standardized and stored in the database;

[0017] The data to be cleaned is subjected to basic cleaning and format standardization to obtain preliminary processing results.

[0018] The further technical solution is as follows: The preliminary processing result is encoded according to rules to obtain the encoded result, including:

[0019] The field names of the preliminary processing results are converted to Unicode format to obtain the converted field names;

[0020] Create a mapping relationship between the transformed field names and the data of the preliminary processing result;

[0021] Business rules configured by registered users;

[0022] A rule matrix is ​​generated based on the business rules and the mapping relationship to obtain the encoding result.

[0023] Its further technical solution is as follows: the dynamic GAN network model includes a rule-aware generator, a dual discriminator, a dynamic regulator, and a rule feature encoder;

[0024] The rule-aware generator is based on a deep residual network structure, integrates a rule attention mechanism, receives noise input and rule features, and outputs data repair results that meet the requirements.

[0025] The dual discriminator includes a data authenticity judgment channel and a rule compliance verification channel, and provides a comprehensive evaluation by fusing weights;

[0026] The dynamic regulator monitors performance indicators in real time, automatically triggers structural adjustment strategies, and optimizes network parameters.

[0027] The rule feature encoder is used to convert various business rules into numerical feature representations, supporting range and classification type encoding conversions.

[0028] Its further technical solution is as follows: the training process of the dynamic GAN network model includes:

[0029] Obtain the preliminary processing results and corresponding encoding results from the historical data;

[0030] The preliminary historical processing results and corresponding encoding results are standardized, and the standardized data are fused into the same feature space through a fully connected layer. The feature weights are dynamically adjusted using an improved residual network architecture to obtain the generated data.

[0031] The generated data is processed using the Tanh activation function to ensure that the generated data falls within a reasonable range, thereby obtaining the generation result;

[0032] The total loss value is obtained by weighted summation of the difference between the generated results and the actual data distribution and the compliance with business rules. The Adam optimizer is used for backpropagation and gradient pruning is applied to ensure the stability of the training process. At the same time, multi-level features are extracted through convolutional layers, and batch normalization and LeakyReLU activation function are used to improve feature representation.

[0033] The encoded results are processed one by one, and the matching score between the generated data and the rules is calculated to ensure data compliance.

[0034] The fusion weights of the two independent channels, data authenticity judgment and rule compliance verification, are automatically adjusted during the training phase.

[0035] Real-time performance metrics are collected, and the optimal adjustment scheme is selected to maintain the model's best state, resulting in a dynamic GAN network model.

[0036] The further technical solution is as follows: processing the cleaning result to obtain cleaned data includes:

[0037] Verify the consistency of the cleaning results and the integrity of the data;

[0038] Select the output format of the cleaning results according to the requirements and perform flattening processing on the nested structure of the cleaning results to generate a type metadata description file;

[0039] The optimization results are obtained by memory compression and type optimization of the type metadata description file;

[0040] Analyze the optimization results and calculate the optimal sharding strategy;

[0041] Based on the optimal sharding strategy, a multi-threaded asynchronous writing mode is adopted, and RowGroup is applied to optimize the Parquet format of the cleaning results to obtain optimized data;

[0042] A checksum is generated for the optimized data and a manifest file is created. A final consistency check is then performed to obtain the cleaned data.

[0043] The further technical solution is as follows: processing the cleaning result to obtain cleaned data includes:

[0044] The cleaning results are then subjected to inverse transformation, decoding, and normalization to obtain the cleaned data.

[0045] The present invention also provides a data intelligent cleaning device, comprising:

[0046] The acquisition unit is used to acquire the data to be cleaned.

[0047] A preliminary processing unit is used to perform preliminary processing on the data to be cleaned in order to obtain preliminary processing results;

[0048] An encoding unit is used to perform rule encoding on the preliminary processing results to obtain an encoded result;

[0049] The cleaning unit is used to input the preliminary processing result and the encoding result into the dynamic GAN network model for cleaning to obtain the cleaning result; wherein, the dynamic GAN network model is trained by rule-constrained generative adversarial architecture, combined with real-time performance monitoring and adaptive adjustment mechanism.

[0050] The post-processing unit is used to process the cleaning results to obtain cleaned data;

[0051] The output unit is used to output the cleaned data and the cleaning report, wherein the cleaning report includes the cleaning process.

[0052] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0053] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0054] The advantages of this invention compared to existing technologies are as follows: This invention achieves fully automated processing from the acquisition of data to be cleaned to the final output of cleaned data and reports through the collaborative optimization of a dynamic rule engine and a GAN network. First, the data to be cleaned undergoes preliminary processing. Then, the processing results are converted into a machine-understandable form through rule encoding and input into a dynamic GAN network model that integrates real-time performance monitoring and adaptive adjustment mechanisms for deep cleaning. This process not only ensures high accuracy and semantic consistency in data cleaning but also enhances the flexibility and real-time response capability for processing different types of data through adaptive adjustment mechanisms. The final output data and cleaning report provide a complete audit trail from raw data to cleaning results, significantly improving data quality and business response efficiency, and effectively addressing the shortcomings of existing technologies in terms of flexibility, real-time performance, data diversity processing, model robustness, and generalization ability.

[0055] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram illustrating an application scenario of the intelligent data cleaning method provided in this embodiment of the invention;

[0058] Figure 2 A schematic flowchart of the intelligent data cleaning method provided in an embodiment of the present invention;

[0059] Figure 3 A schematic block diagram of the intelligent data cleaning device provided in an embodiment of the present invention;

[0060] Figure 4 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0063] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0065] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the intelligent data cleaning method provided in an embodiment of the present invention. Figure 2This is a schematic flowchart illustrating the intelligent data cleaning method provided in this embodiment of the invention. The method is applied to a server. The server interacts with the terminal, integrating a dynamic GAN network model and a series of preprocessing and post-processing steps to achieve high-precision cleaning, semantic consistency maintenance, and low-latency processing of policy information. First, it uses an intelligent reading strategy and standardized metadata extraction to initially process the raw data, and generates a rule matrix through rule encoding to guide the subsequent data cleaning process. Next, a dynamic GAN network model composed of components such as a rule-aware generator and a dual discriminator is used for data repair, ensuring that the generated data not only meets business rule requirements but also achieves a balance between authenticity and rule compliance. During training, model performance is optimized by fusing weight adjustment, gradient pruning techniques, and multi-level feature extraction to enhance its robustness and generalization ability. Finally, through steps such as consistency verification of the cleaning results, format optimization, memory compression, and the application of sharding strategies, data quality and output efficiency are further improved, thereby significantly improving business response speed and decision support capabilities. This series of operations effectively addresses the shortcomings of existing technologies in terms of flexibility, real-time performance, and data diversity processing, providing a more reliable data cleaning solution.

[0066] Figure 2 This is a schematic flowchart of the intelligent data cleaning method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S160.

[0067] S110. Obtain the data to be cleaned.

[0068] In this embodiment, during stage S110, the system's main task is to obtain the data that needs to be cleaned from the user or a third-party interface. This data comes from different formats and sources, such as Excel documents, CSV files, or other types of structured data files. This step involves not only data access but also basic data quality checks and preprocessing to ensure the efficiency and accuracy of subsequent processing.

[0069] S120. Perform preliminary processing on the data to be cleaned to obtain preliminary processing results.

[0070] In this embodiment, the preliminary processing result refers to a standardized dataset with initially improved quality obtained after intelligent selection of reading strategies, automatic adaptation to encoding formats for data parsing, standardization of key metadata information, and basic cleaning and format unification. In short, the preliminary processing result is a preliminarily cleaned dataset that ensures data consistency, accuracy, and reliability.

[0071] In one embodiment, step S120 described above may include steps S121 to S123.

[0072] S121. Intelligently select a reading strategy based on the size of the data to be cleaned and automatically adapt to the encoding format to parse the data.

[0073] In this embodiment, the system first assesses the size of the data file to be cleaned and selects the optimal reading strategy accordingly. For small files, the system may directly use memory mapping for loading, while for large files, a streaming block processing method is used to reduce memory usage.

[0074] During data reading, the system automatically identifies and adapts to the file's encoding format (such as UTF-8, GBK, etc.) to ensure correct parsing of mixed-encoding content. Furthermore, the system can also identify table structure features, such as merged cells or multiple headers, to accurately extract data information.

[0075] S122. Automatically extract and standardize the key metadata information of the data to be cleaned and store it in the database.

[0076] In this embodiment, the system automatically extracts key metadata information from the data to be cleaned, including basic attributes such as field names, data types, and value ranges. The statistical analysis module then calculates quality indicators such as the numerical distribution and the proportion of null values ​​for each field.

[0077] The extracted metadata information is standardized and then stored in a metadata database. This process not only ensures data consistency and accuracy but also provides a reference for subsequent data processing.

[0078] S123. Perform basic cleaning and format unification on the data to be cleaned to obtain preliminary processing results.

[0079] In this embodiment, the basic cleaning operation is as follows:

[0080] Unified character encoding: All text data is uniformly converted to UTF-8 format to ensure consistency.

[0081] Date and time standardization: unifying the expression of dates and times to facilitate subsequent processing and analysis.

[0082] Remove extra spaces: Remove extra whitespace characters at the beginning and end of fields to improve data cleanliness.

[0083] Special format conversion: Automatically recognizes and converts special formats such as thousands separators in numeric fields.

[0084] Retain a copy of the original data: To ensure the traceability of data modifications, the system retains a copy of the original data and logs all modification operations for future review or retrospection.

[0085] Preliminary Quality Check: At this stage, the system also performs multi-dimensional quality checks, including field integrity verification, value range compliance checks, and business rule pre-validation. These checks help quickly identify potential problems and provide guidance for the next step of data cleaning.

[0086] After the above three sub-steps, the system obtains a preliminary processing result, which includes data that has undergone basic cleaning and format standardization. This data is not only more standardized in form, but its quality and reliability are also initially guaranteed, laying a solid foundation for further data cleaning and in-depth processing. At the same time, the metadata and log information collected during this process also provide important support for subsequent steps.

[0087] In this embodiment, a modular design is employed to automatically identify and parse structured data files in various formats, including different versions of Excel documents and CSV files. During this process, the system simultaneously performs basic quality checks, such as field integrity verification, data type checks, and outlier identification. Standardization processing is also performed through a built-in preprocessing pipeline, including unifying character encoding, removing redundant spaces, and converting date formats, laying a solid foundation for subsequent deep cleaning. Furthermore, this layer is equipped with real-time monitoring capabilities, generating detailed data quality assessment reports that allow users to quickly understand the overall status of their input data.

[0088] Specifically, the input layer consists of four core modules: the file parsing engine module is responsible for handling the reading tasks of various data files, with the ability to automatically detect file formats and encoding types, and supports the chunked loading of large data files; the data verification system module ensures data quality, including a regular expression validator, an integrity checker, and a deep verification model based on a lightweight generative adversarial network, ensuring the reliability of input data from multiple dimensions; the preprocessing pipeline module focuses on the standardization transformation of data, including sub-modules such as null value handling and type conversion, and adopts a memory-optimized design to improve processing efficiency; the monitoring and reporting module records operation logs in real time, tracks abnormal data, and generates a visual quality score, providing users with a comprehensive view of the processing process.

[0089] In terms of implementation, firstly, during the file receiving and verification phase, the system receives uploaded files through a secure channel and automatically performs integrity checks, including file signature detection, virus scanning, and format recognition, ensuring the legality and security of the files. Next, during the data loading and parsing phase, the system intelligently selects the optimal reading strategy based on file size: small files are directly loaded via memory mapping, while large files are processed using streaming chunking, and mixed-encoding content and table structure features are automatically handled. The metadata extraction phase involves automatically extracting key metadata information, including field names and data types, and standardizing it before storing it in a metadata database for subsequent processing. In the subsequent preliminary cleaning and standardization phase, the system performs a series of basic cleaning operations, such as unifying character encoding to UTF-8 format, standardizing date and time expressions, and removing whitespace characters. The following quality assessment and report generation phase runs multi-dimensional checks and automatically generates a problem report, visually displaying the quality distribution of each field and the main problem types. The data buffering and transmission preparation phase stores the cleaned data in a distributed cache system, optimizing I / O performance while establishing data lineage tracking records to ensure data integrity and the traceability of the processing process. Finally, the exception handling and recovery mechanism monitors exceptions in real time, automatically retrying recoverable errors and triggering alarms to notify the administrator for serious errors, ensuring the stable operation of the system.

[0090] S130. The preliminary processing result is encoded according to rules to obtain the encoded result.

[0091] In this embodiment, the encoding result refers to the rule matrix generated based on the business rules configured by the registered user after converting the field names of the preliminary processing results to Unicode format, establishing a mapping relationship between them and the data, and then using these rules and mapping relationships to guide subsequent data processing. In short, the encoding result is a structured representation of the data after standardization and regularization according to business rules, used to ensure the consistency and accuracy of data processing.

[0092] In one embodiment, step S130 described above may include steps S131 to S134.

[0093] S131. Convert the field names of the preliminary processing results to Unicode format to obtain the converted field names.

[0094] In this embodiment, the converted field name refers to a consistent identifier used internally by the system for data processing, obtained by standardizing the original field name using Unicode format according to preset business rules and data mapping relationships. In short, it is the system's internal representation of the original field name after standardization, ensuring consistency and accuracy during data processing.

[0095] This step first requires performing a Unicode standard form KC conversion on all field names in the initial processing results. The purpose is to eliminate inconsistencies caused by differences in simplified / traditional Chinese characters, capitalization, or special characters. This not only ensures consistency in field names but also lays the foundation for subsequent data processing and rule application. By using the Unicode standard, it is ensured that data from different sources can be uniformly identified and processed, which is crucial for maintaining data integrity and accuracy.

[0096] S132. Create a mapping relationship between the transformed field names and the data of the preliminary processing result.

[0097] In this embodiment, after completing the Unicode standardization of field names, the next step is to establish a mapping relationship between these standardized field names and their corresponding data. This process helps to build a clear data structure framework, ensuring that each field name accurately points to its corresponding value or dataset. This mapping relationship is crucial for subsequent application logic processing, data analysis, and rule engine operations, as it provides a reliable data access path.

[0098] S133, Business rules configured by registered users.

[0099] In this embodiment, the system next needs to register user-defined business rules. These rules may include range rules (such as the legal value range of numeric fields), regular expression rules (for pattern matching of text fields), and other constraints specific to business requirements. Registering these rules involves parsing the user's rule definitions and converting them into an executable form within the system. Furthermore, appropriate exception handling strategies need to be set up to address potential rule conflicts or unexpected data discrepancies.

[0100] S134. Generate a rule matrix based on the business rules and the mapping relationship to obtain the encoding result.

[0101] Finally, having completed the above preparations, the system will generate a rule matrix based on the registered business rules and the previously established mapping relationship between field names and data. This rule matrix is ​​essentially a set of feature vectors that quantifies the business rules that each field must follow and serves as one of the input conditions for advanced data processing techniques such as GANs (Generative Adversarial Networks). In this way, not only can the data processing process be ensured to strictly adhere to preset business specifications, but intelligent methods can also be used to further improve data quality and processing efficiency.

[0102] In summary, step S130 effectively achieves rule encoding of the data by performing a series of carefully designed operations on the preliminary processing results—from Unicode standardization of field names to the establishment of mapping relationships, to the registration of business rules and the generation of the final rule matrix—providing a solid foundation for subsequent data cleaning, feature engineering, and even the entire data processing pipeline.

[0103] Specifically, the S130 data processing pipeline described above implements an end-to-end processing flow from raw data to cleaned results. It adopts a phased processing model, ensuring that data quality gradually improves during the process through multi-level data transformation and quality control nodes. The system supports fully automated processing of structured data, covering key cleaning operations such as anomaly detection, missing value repair, and format standardization, and strictly adheres to business rules. Furthermore, the pipeline design has flexible scalability, automatically adjusting processing strategies based on data characteristics to meet the data cleaning needs of different business scenarios.

[0104] The data processing pipeline consists of a hierarchical processing system comprised of six core modules:

[0105] Data preprocessing module: responsible for the initial standardization of raw data.

[0106] Rule Engine Module: Executes the parsing and application of business rules.

[0107] Exception retry mechanism: responsible for handling exceptions and retrying.

[0108] Intelligent cleaning module: The core cleaning component based on generative adversarial networks (GAN).

[0109] Quality verification module: Multi-dimensional result verification system.

[0110] Exception handling module: A hierarchical processing mechanism for problem data.

[0111] The system employs a dual-buffering mechanism to receive input data. The main buffer handles temporary storage of batch data, while the working buffer performs real-time processing. The preprocessing stage automatically identifies data types and extracts metadata, establishing field-level quality baselines. A smart block-splitting algorithm divides the large dataset into optimal processing units, with each data block accompanied by an integrity checksum to ensure data consistency in subsequent processing.

[0112] The system automatically analyzes data structure characteristics, including numerical distribution statistics, text feature encoding, and temporal pattern recognition. A quality analysis engine runs concurrently, detecting missing values, outliers, and business rule conflicts, generating structured quality reports. A dynamic feature selector automatically determines the optimal feature combination and processing strategy based on the analysis results.

[0113] The rule compiler converts business rules into executable instruction sets and establishes a rule dependency graph. The parallel rule executor dispatches processing tasks, supporting a three-tiered processing pipeline of conditional rules, computational rules, and validation rules. The rule conflict detection module monitors the execution process in real time and resolves rule conflicts through a voting mechanism, ensuring consistency in processing logic.

[0114] The generator network receives noisy input data, combines it with rule-constrained features, and outputs preliminary repair results. The discriminator network performs double verification, simultaneously evaluating data quality and rule compliance. The dynamic weight adjuster optimizes network parameters in real time based on verification feedback, achieving adaptive adjustment of the processing strategy. Multiple rounds of iterative cleaning ensure that data quality reaches a predetermined threshold.

[0115] The system includes field-level format validation, record-level business logic validation, and dataset-level distribution validation. A difference analysis engine compares data before and after cleaning, identifying and categorizing all changes. A feedback learning module collects validation results, automatically updating the rule base and model parameters to form a closed-loop optimization system.

[0116] The standardized output converter transforms the cleaning results into a specified format, supporting multi-volume storage and encrypted transmission. The metadata management system automatically records key metrics and operation logs throughout the processing. The data lineage tracer establishes a complete processing trajectory, supporting result traceability and impact analysis. The final output includes three parts: cleaned data, a quality report, and an operation audit.

[0117] A multi-level anomaly detection system monitors each processing stage in real time, including data anomalies, processing anomalies, and system anomalies. An intelligent routing controller automatically isolates problematic data and initiates retry, repair, or manual intervention processes based on the anomaly type. A checkpoint recovery mechanism periodically saves the processing status, supporting rapid recovery from the most recent stable state and ensuring the reliability of the processing.

[0118] Specifically, the rule engine adopts a hybrid rule processing mode, supporting the collaborative work of structured rules and machine learning models. It can simultaneously handle multi-dimensional business constraints such as field-level, record-level association rules, and dataset-level distribution rules. Through dynamic rule loading and real-time evaluation mechanisms, the system ensures that the data processing process strictly adheres to the latest business specifications and quality standards, while maintaining millisecond-level rule response speed. The rule engine deeply integrates business rule logic, incorporating three types of core business rules into the data processing flow through a dynamic rule injection mechanism: numerical range rules (such as the 0.2-1.0 range limit for the field "subscribed_contribution_ratio"), forced assignment rules (such as forcibly setting the field "register_province" to "D province"), and format validation rules (such as regular expression pattern matching).

[0119] The rules engine consists of the following three main modules:

[0120] Rule Registrant Module: Handles the registration of rule types such as range and regular expression, and manages mandatory rules such as force_value.

[0121] Rule executor module: responsible for operations such as numeric range pruning, forced value overriding, and regular expression validation.

[0122] The rule compiler module converts business rules into feature matrices, providing rule constraints for generative adversarial networks (GANs).

[0123] The rule registration function supports two core business rules: range rules (range type) and regular expression rules (regex type). Range rules define the legal value range of a field and automatically trigger out-of-bounds value correction; regular expression rules include a forced assignment function (force_value) and support pattern matching validation. For example:

[0124] The value of the field “subscribed_contribution_ratio” must be greater than or equal to 20%.

[0125] The value of the field “register_province” must be “D province”.

[0126] The value of the field “total_score” must be greater than or equal to 0.78.

[0127] The feature vector of the range rule (numerical type) is represented as: Vec = [min, max, (min + max) / 2, log(1 + max - min)]. For example:

[0128] The field “subscribed_contribution_ratio” is: [0.2, 1.0, 0.6, log(1+0.8)]≈[0.2, 1.0, 0.6, 0.693];

[0129] The field "total_score": [0.78, 1.0, 0.89, log(1+0.22)]≈[0.78, 1.0, 0.89, 0.211]

[0130] The feature vector of a mandatory rule (textual type) is represented as: Vec = [numerical percentage, character complexity, uniqueness, strictness];

[0131] Receive rule definitions in JSON format via the REST API. The parser extracts key elements, including rule names, applicable fields, constraints, execution actions, and exception handling strategies. Conduct pre-compilation checks on script-type rules to ensure syntax correctness.

[0132] Normalize field names using Unicode Normalization Form KC to eliminate differences in Chinese simplified and traditional characters, case, and special characters. Establish a field alias mapping table to support business term conversions such as "registered province - register_province".

[0133] The rule library adopts a hierarchical storage architecture. Hot rules are stored in the Redis cache (TTL 24 hours), and all rules are persisted to MongoDB. The storage structure includes rule metadata, AST syntax trees, and compiled bytecode.

[0134] Establish a triple-index system, including field-rule index (inverted), rule-field index (forward), and rule-rule dependency graph. Index updates trigger incremental construction to ensure millisecond-level effectiveness.

[0135] Select an encoding strategy based on the rule type. For range rules, use [min, max, mean, range]; for regular rules, use [pattern length, number of special characters, number of capture groups, strict level]. The feature dimension is uniformly a multiple of 4 for easy GAN processing. The feature matrix is represented as:

[0136] ;

[0137] Business rule execution supports three rule execution logics: Numeric field processing: Retain compliant original values (valid_mask), automatically correct out-of-bounds values (clip + rescale); Text field processing: Force value direct overwrite, inverse label encoding conversion; Mixed type support: Simultaneously process numeric / text fields while maintaining inter-field association constraints.

[0138] Numeric correction formula:

[0139] ;

[0140] An example of the correction is shown in Table 1.

[0141] Table 1. Example of Correction

[0142] Fields Original value Correction operation Correction results subscribed_contribution_ratio 0.15 clip(0.15, 0.2, 1.0) 0.2 total_score 1.05 clip(1.05, 0.78, 1.0) 1.0

[0143] Forced assignment formula: register_province = force_value = "Province D";

[0144] An example of the correction is shown in Table 2.

[0145] Table 2. Examples of Corrections

[0146] Before revision Revised [Province A, City B, Province C] ["D Province", "D Province", D Province]

[0147] Specifically, field types (numeric / text / date, etc.) are automatically identified through metadata, and a type inference algorithm is enabled for mixed-type fields. A field processing priority queue is built to ensure that critical fields are processed first.

[0148] First, check for NULL values, then perform range validation (including open and closed interval processing), and calculate the deviation degree for out-of-bounds values; first, standardize the encoding, then perform regular expression matching (supporting partial matching and fuzzy matching), and force the assigned field to directly overwrite the original value.

[0149] For multiple rules on the same field, a decision matrix is ​​used to integrate the results. Strict rules have veto power, while non-strict rules accumulate scores according to their weights. A dynamic negotiation mechanism is initiated when a rule conflict is triggered.

[0150] Numerical correction employs order-preserving scaling techniques to linearly map out-of-bounds values ​​to the valid range, thus preserving the data distribution characteristics.

[0151] Text correction forces direct replacement of assigned values, while pattern matching calls GAN to generate candidate values ​​for fields, which are then manually confirmed.

[0152] Record the complete correction trajectory, including the original value, correction strategy, final value, and operator (system / human). Establish a data lineage graph to support reverse tracing of the correction process.

[0153] The collaboration between rules and GANs has three unique features: Rule Feature Matrix: quantifies business rules into 4-dimensional features,

[0154] As an additional input condition for GAN; dynamic constraint injection: the generator network receives rule constraints, and the discriminator verifies the rule compliance;

[0155] Hybrid loss function: data distribution loss and rule compliance loss.

[0156] Discriminator verification formula: Compliant with the rules;

[0157] Here is an example of discriminator verification:

[0158] Generate sample 1: [0.25, "D Province", 0.82]; Accept (meets all rules);

[0159] Sample 2 was generated: [0.18, “Province A”, 0.75]; rejected (violating three rules).

[0160] Specifically, for each field, a 4-dimensional regular feature vector is calculated:

[0161] Range rule: [min_val, max_val, (max+min) / 2, log(max-min+1)];

[0162] Classification rules: [Number of categories, percentage of major categories, entropy value, mandatory flag];

[0163] The matrix dimension is 4 times the number of fields, and Min-Max normalization is performed.

[0164] In the generator input layer, a noise vector (100-dimensional) and a rule feature matrix (n×4-dimensional) are concatenated, and the rule weights are calculated using an attention mechanism. A rule verification module is added to the output layer to pre-filter obviously non-compliant generated results.

[0165] Authenticity determination branch: 3-layer CNN + 2-layer fully connected layer, outputs the probability of data authenticity;

[0166] Rule-based discrimination branch: Calculates the matching degree of rule features and outputs a [0, 1] conformity score;

[0167] Final judgment result = 0.6 × authenticity + 0.4 × rule compliance (adjustable parameter);

[0168] When the compliance rate of monitoring rules changes, and it is below the threshold for 3 consecutive epochs:

[0169] Increase the weight of the rule loss (α+=0.1);

[0170] Enhance rule attention (number of heads × 1.5);

[0171] Add rules to strengthen the samples.

[0172] Analyze high-quality samples generated by GANs to extract potential rule patterns: numerical distribution clustering - discover new value ranges; text pattern analysis - extract new regular expressions; association rule mining - discover cross-field constraints.

[0173] S140. The preliminary processing result and the encoding result are input into the dynamic GAN network model for cleaning to obtain the cleaning result; wherein, the dynamic GAN network model is trained by a rule-constrained generative adversarial architecture combined with real-time performance monitoring and adaptive adjustment mechanism.

[0174] In this embodiment, the dynamic GAN network model includes a rule-aware generator, a dual discriminator, a dynamic regulator, and a rule feature encoder;

[0175] The rule-aware generator is based on a deep residual network structure, integrates a rule attention mechanism, receives noise input and rule features, and outputs data repair results that meet the requirements.

[0176] The dual discriminator includes a data authenticity judgment channel and a rule compliance verification channel, and provides a comprehensive evaluation by fusing weights;

[0177] The dynamic regulator monitors performance indicators in real time, automatically triggers structural adjustment strategies, and optimizes network parameters.

[0178] The rule feature encoder is used to convert various business rules into numerical feature representations, supporting range and classification type encoding conversions.

[0179] In this embodiment, the dynamic GAN network model is trained using an innovative rule-constrained generative adversarial architecture combined with real-time performance monitoring and adaptive adjustment mechanisms. This model is designed to achieve the dual goals of data quality restoration and business rule compliance, and specifically consists of the following core modules:

[0180] The rule-aware generator is based on a deep residual network structure. This generator integrates a rule-attention mechanism, enabling it to receive noisy input and rule-based features, and output data restoration results that meet the requirements. The generator utilizes an improved residual block stacking structure. Within each residual block, local features are first extracted through three convolutional layers, and then a rule-attention sublayer calculates the importance weights of different rules, dynamically adjusting the feature map. Finally, the Tanh activation function is used to ensure that the generated data falls within a reasonable range.

[0181] The dual discriminator employs a dual-channel design, comprising a parallel data authenticity assessment channel and a rule compliance verification channel. The shared underlying feature extraction part is constructed using four convolutional layers, each followed by batch normalization and a LeakyReLU activation function. The data authenticity channel progressively compresses feature dimensions through fully connected layers, outputting a probability assessment of data authenticity. The rule compliance channel features a dedicated rule verification layer that directly compares the degree of matching between the generated data and business rules. The outputs of the two channels are integrated through a dynamic weight fusion module, automatically adjusting the weight coefficients based on the current training stage.

[0182] The dynamic regulator is responsible for monitoring network performance metrics in real time, such as gradient magnitude and rule compliance rate, and automatically triggering structural adjustment strategies to optimize network parameters. It adopts a closed-loop control mechanism of "monitoring-analysis-decision-execution" and selects the optimal adjustment scheme from predefined network adjustment schemes through a reinforcement learning-based policy selection algorithm, completing the network structure adjustment without interrupting the training process.

[0183] The rule-based feature encoder is used to convert various business rules into numerical feature representations, supporting multiple encoding conversion types such as range rules and classification rules. The encoder employs a hierarchical encoding architecture, including digital conversion methods for five types of business rules: numerical range rules, enumerated value rules, regular expression rules, logical combination rules, and temporal dependency rules. Furthermore, it includes a feature enhancement module that uses a self-attention mechanism to mine potential relationships between rules and generate a rule feature matrix with semantic information.

[0184] The training process of the dynamic GAN network model includes:

[0185] Obtain the preliminary processing results and corresponding encoding results from the historical data;

[0186] The preliminary historical processing results and corresponding encoding results are standardized, and the standardized data are fused into the same feature space through a fully connected layer. The feature weights are dynamically adjusted using an improved residual network architecture to obtain the generated data.

[0187] The generated data is processed using the Tanh activation function to ensure that the generated data falls within a reasonable range, thereby obtaining the generation result;

[0188] The total loss value is obtained by weighted summation of the difference between the generated results and the actual data distribution and the compliance with business rules. The Adam optimizer is used for backpropagation and gradient pruning is applied to ensure the stability of the training process. At the same time, multi-level features are extracted through convolutional layers, and batch normalization and LeakyReLU activation function are used to improve feature representation.

[0189] The encoded results are processed one by one, and the matching score between the generated data and the rules is calculated to ensure data compliance.

[0190] The fusion weights of the two independent channels, data authenticity judgment and rule compliance verification, are automatically adjusted during the training phase.

[0191] Real-time performance metrics are collected, and the optimal adjustment scheme is selected to maintain the model's best state, resulting in a dynamic GAN network model.

[0192] In this embodiment, pre-processed data is first collected as training samples. The historical preliminary processing results and corresponding encoding results are standardized to eliminate dimensional differences between different features. The standardized data is fused into the same feature space through a fully connected layer, and the feature weights are dynamically adjusted using an improved residual network architecture to generate new data. The generated data is processed using the Tanh activation function to ensure it falls within a reasonable range. The total loss value is obtained by weighted summation of the differences between the generated results and the real data distribution, as well as the compliance with business rules. Backpropagation is performed using the Adam optimizer, and gradient pruning is applied to ensure the stability of the training process. Multi-level features are extracted through convolutional layers, and batch normalization and LeakyReLU activation functions are used to improve feature representation capabilities. Encoding results are processed one by one, and the matching score between the generated data and the rules is calculated to ensure data compliance. The fusion weights of the two independent channels—data authenticity judgment and rule compliance verification—are automatically adjusted according to the training phase. Performance metrics are collected in real time, and the optimal adjustment scheme is selected to maintain the model's best state, thereby obtaining the optimal dynamic GAN network model.

[0193] The above steps ensure that the dynamic GAN network model can not only effectively repair data quality issues, but also strictly follow business rules and provide high-quality data cleaning services.

[0194] The dynamic GAN network layer employs an innovative rule-constrained generative adversarial architecture, achieving the dual goals of data quality restoration and business rule compliance. Through a unique dynamic dimensionality adaptation mechanism, the network automatically adjusts its depth and width based on the feature distribution of the input data and the complexity of the business rules, ensuring optimal processing capabilities for datasets of different types and sizes. In the network design, business rules are directly injected as hard constraints into the training process of the generator and discriminator. A multi-objective optimization algorithm balances the data distribution fit and rule compliance, ensuring that the generated data is natural and reasonable while strictly adhering to pre-set business specifications. The dynamic GAN network consists of five core modules: a rule-aware generator, a dual discriminator, a dynamic regulator, a rule feature encoder, and a training monitoring system.

[0195] The rule-aware generator, based on a deep residual network structure and integrating a rule attention mechanism, is responsible for receiving noisy input and rule features, and outputting data repair results that meet the requirements. The generator utilizes an improved residual block stacking structure. Within each residual block, local features are first extracted through three convolutional layers. Then, a rule attention sub-layer calculates the importance of different business rules in the current feature space and adjusts the feature weights accordingly. Finally, a Tanh activation function is used to ensure that the generated data falls within a reasonable range. Furthermore, the generator integrates gradient normalization and weight constraint techniques, effectively avoiding common problems such as mode collapse and gradient vanishing.

[0196] The dual discriminator employs a dual-channel architecture, comprising a parallel data authenticity discrimination channel and a rule compliance verification channel. The shared underlying feature extraction part is constructed using four convolutional layers, each followed by batch normalization and the LeakyReLU activation function. The data authenticity channel uses a conventional discriminative network structure, progressively compressing feature dimensions through three fully connected layers to ultimately output a probability assessment of data authenticity. The rule compliance channel features a dedicated rule verification layer that directly compares the generated data with the business rules, supporting verification of various constraint types such as range rules, enumeration rules, and regularization rules. The outputs of the two channels are integrated through a dynamic weight fusion module, with weight coefficients automatically adjusted according to the current training stage, emphasizing data authenticity in the early stages and strengthening rule compliance requirements in later stages.

[0197] The dynamic regulator is one of the core innovations in the network architecture. It monitors network performance metrics in real time, such as gradient magnitude, rule compliance rate, and pattern diversity index, and establishes a dynamic performance evaluation model. The regulation strategy library contains fifteen predefined network adjustment schemes, covering key parameters such as network depth, width, number of attention heads, and residual connection methods. The system employs a reinforcement learning-based strategy selection algorithm, which can automatically select the optimal adjustment scheme based on the current training state and complete the network structure adjustment without interrupting the training process. The adjustment process follows a gradual principle, with each adjustment not exceeding 30% of the original structure to ensure training stability.

[0198] The rule-based feature encoder converts various business rules into numerical feature representations, supporting multiple encoding conversion types such as range rules, classification rules, and complex logic rules. The encoder employs a hierarchical encoding architecture, including digital conversion methods for five types of business rules: numerical range rules, enumerated value rules, regular expression rules, logical combination rules, and temporal dependency rules. Furthermore, it includes a feature enhancement module that uses a self-attention mechanism to mine potential relationships between rules, generating a rule feature matrix with semantic information. The system maintains a rule-to-feature mapping cache, directly returning cached results for duplicate rules to improve encoding efficiency.

[0199] The training monitoring system provides comprehensive and multi-dimensional visualization and control of the training process, including real-time dashboards, anomaly detection, pattern analysis, rule tracking, resource monitoring, and intelligent diagnostics. Specifically, the real-time dashboard dynamically displays trend curves for twelve core indicators and supports comparative analysis of five historical versions; anomaly detection uses an LSTM prediction model to predict potential training anomalies three epochs in advance; pattern analysis uses t-SNE to visualize the distribution evolution of generated data and identify signs of pattern collapse; rule tracking records the compliance rate changes of each business rule and marks rules that consistently fail to meet standards; resource monitoring displays real-time hardware indicators such as GPU memory and utilization, providing early warnings of resource bottlenecks; and intelligent diagnostics automatically analyzes the root causes of training problems and provides feasibility assessments of adjustment suggestions. The system supports creating monitoring snapshots, allowing for the review of the complete state at any training moment, including network parameters, data samples, and evaluation results. The entire dynamic GAN network model achieves efficient data cleaning and rule adherence through the collaborative work of these modules.

[0200] Specifically, the generator network employs an improved residual network architecture, innovatively introducing a rule constraint layer and a dynamic attention mechanism. The network input consists of two parts: a 100-dimensional random noise vector and an encoded rule feature matrix. These two types of information are organically combined through a dedicated feature fusion layer. The main body of the network is composed of five stacked residual blocks, each containing three convolutional layers and one rule attention sub-layer. The rule attention sub-layer dynamically calculates the importance of different business rules in the current feature space and adjusts the feature weights accordingly. Finally, the network uses a Tanh activation function to constrain the output range, ensuring that the generated data falls within a preset reasonable range. The generator also integrates gradient normalization and weight constraint techniques, effectively avoiding common problems such as mode collapse and gradient vanishing.

[0201] In the input preprocessing stage, the system standardizes the 100-dimensional random noise vector to make it conform to a standard normal distribution, and simultaneously normalizes the regular feature matrix to eliminate the influence of dimensional differences between different rules. A fully connected layer maps the noise vector and the regular feature matrix to the same feature space, and feature fusion is performed using element-wise addition to preserve the semantic information of the original features. The fused features are processed through five cascaded residual blocks. Within each residual block, local features are first extracted through three convolutional layers, and then the attention weights for different rules are calculated by a regular attention sublayer to dynamically adjust the feature map. The output of the last residual block is then global average pooled, mapped to the target data dimension through a fully connected layer, and the Tanh activation function is used to constrain the output range.

[0202] During the loss calculation phase, the system simultaneously calculates the distribution difference loss between the generated data and the real data, as well as the conformity loss between the generated data and the business rules; the weighted sum of these two losses is the total loss. During the parameter update phase, the Adam optimizer is used for backpropagation to dynamically adjust the learning rate, and gradient pruning techniques are applied to ensure training stability.

[0203] Let the random noise vector be... The regular feature matrix is ​​R. Here, the random noise vector refers to the preliminary processing result;

[0204] Standardize the random noise vector: where and These are the mean and standard deviation of z, respectively. Normalize the regular feature matrix: ; through a fully connected layer and Map to the same feature space, then add element by element: ;in and It is the weight matrix of the fully connected layer. It is the bias vector.

[0205] Let the input of the i-th residual block be... The output is Each residual block contains 3 convolutional layers and 1 regular attention sublayer.

[0206] ;

[0207] The rule-based attention sublayer calculates attention weights and adjusts feature maps based on business rules.

[0208] Output of the last residual block Obtained through global average pooling Then, it is mapped to the target data dimension G through a fully connected layer and activated using the Tanh activation function. ;

[0209] Loss Calculation: Let the generated data be G, the real data be x, and the loss due to the distribution difference between the generated data and the real data be... The loss in the conformity between the generated data and the business rules is The total loss is ;in It is the weighting coefficient.

[0210] The discriminator network employs an innovative dual-channel architecture, separating a data authenticity discrimination channel and a rule compliance verification channel based on a shared underlying feature extraction network. The shared feature extraction part uses four convolutional layers, each followed by batch normalization and a LeakyReLU activation function. The data authenticity channel uses a conventional discriminator network structure, progressively compressing feature dimensions through three fully connected layers to ultimately output a probability assessment of data authenticity. The rule compliance channel features a dedicated rule verification layer that directly compares the generated data with business rules, supporting verification of various constraint types such as range rules, enumeration rules, and regularization rules. The outputs of the two channels are integrated through a dynamic weight fusion module, with weight coefficients automatically adjusted according to the current training stage, emphasizing data authenticity in the early stages and strengthening rule compliance requirements in later stages.

[0211] In the input preprocessing stage, the input data is standardized to eliminate dimensional differences between features, and the rule features are re-encoded to extract higher-level semantic representations. Multi-level features are extracted from the data through four convolutional layers, with each convolution followed by batch normalization and a LeakyReLU activation function to preserve negative half-axis information. High-level features are input into two independent channels: the data authenticity channel undergoes gradual dimensionality reduction through fully connected layers, while the rule compliance channel performs precise matching through a rule validation layer. In the rule compliance channel, the system processes each business rule sequentially, calculating the matching score between the generated data and the rule, supporting flexible strategies such as partial matching and fuzzy matching. The fusion weights of the two channels are dynamically adjusted based on the current training epochs. In the early training phase (first 50 epochs), a 7:3 weighting emphasizes data authenticity, gradually transitioning to a 5:5 balanced weighting later. The discriminator loss consists of three parts: real data discrimination loss, generated data discrimination loss, and rule validation loss, which are jointly optimized through a multi-task learning framework.

[0212] Specifically, X standardizes the input data: The rule features are encoded twice to obtain Multi-level features of the data are extracted through four convolutional layers: ;

[0213] Channel separation processing: Data authenticity channel: Probability assessment of the authenticity of output data .

[0214] Rule compliance channel: Output rule compliance score .

[0215] Channel separation processing: Dynamically adjust the fusion weights of the two channels based on the current training epoch number t. Early training phase (first 50 rounds) Later .

[0216] Discriminator loss It consists of three parts: real data discrimination loss. Generate data to determine loss Loss of rule validation ; .

[0217] The dynamic adjustment system is the core innovation of the network architecture, employing a closed-loop control mechanism of "monitoring-analysis-decision-execution". The system establishes a dynamic performance evaluation model by collecting performance metrics across 10 dimensions in real time (including gradient magnitude, rule compliance rate, and pattern diversity index). The adjustment strategy library contains 15 predefined network adjustment schemes, covering key parameters such as network depth, width, number of attention heads, and residual connection methods. The system uses a reinforcement learning-based strategy selection algorithm, which can automatically select the optimal adjustment scheme based on the current training state and complete the network structure adjustment without interrupting the training process. The adjustment process follows a gradual principle, with each adjustment not exceeding 30% of the original structure to ensure training stability.

[0218] During the performance monitoring phase, a complete set of metrics is collected every 50 training iterations, including core metrics such as generator loss, discriminator loss, gradient L2 norm, and rule compliance rate. The collected metrics are input into a pre-trained performance evaluation model, which outputs a current network state score (0-100) and a major bottleneck analysis. Based on the state evaluation results, three most likely adjustment schemes are selected from the policy library, and the expected benefits and execution risks of each scheme are calculated. A temporary network replica is built in memory to simulate the execution of the adjustment schemes and predict the performance changes after adjustment. After selecting the optimal scheme, the structural adjustment is completed in three small steps, with each step spaced 10 iterations apart, and the adjustment effect is continuously monitored during this period. After the adjustment is completed, the network structure is locked, and focused training is performed for 100 iterations to evaluate the final adjustment effect and update the policy library.

[0219] The rule-based feature encoder employs a hierarchical encoding architecture, supporting the digital transformation of five types of business rules: numerical range rules, enumeration value rules, regular expression rules, logical combination rules, and temporal dependency rules. Numerical range rules are encoded as four-dimensional vectors [min, max, mean, range] and standardized. Enumeration value rules use One-Hot encoding combined with TF-IDF weights to retain important category information. Regular expression rules are converted into 128-dimensional semantic vectors using a pre-trained Regex2Vec model. Logical combination rules first parse the AST syntax tree and then recursively encode the subtrees. Temporal dependency rules use an LSTM network to capture time series patterns and output fixed-dimensional temporal features. The encoder also includes a feature enhancement module that uses a self-attention mechanism to mine potential relationships between rules and generate a rule feature matrix with semantic information. The system maintains a mapping cache from rules to features, directly returning cached results for duplicate rules to improve encoding efficiency.

[0220] During the rule parsing phase, the ANTLR parser is used to convert business rules into a standard syntax tree, identifying rule types and parameters. Based on the rule type, rules are routed to the corresponding encoding submodule, loading a pre-trained feature transformation model. Numerical rules calculate descriptive statistics, enumerated rules undergo weighted One-Hot encoding, and regularized rules extract semantic features through a neural network. The basic features are input into a 3-layer Transformer encoder to generate context-aware rule features. An MD5 fingerprint is generated for each rule, and a rule-feature mapping cache is established with a 24-hour validity period. The encoding results are sampled and checked to ensure that the feature space retains the semantic relationships of the original rules.

[0221] Numerical range rules: For the `subscribed_contribution_ratio` field, let its value be `x`, encoded as a four-dimensional vector; `v = [min, max, max(x), range(x)]`, and perform standardization. Rules: `x ≥ 0.2`, `min = 0.2`, `max` can be set according to the actual situation, `mean(x)` is the mean of all values ​​in this field, and `range(x) = max - min`.

[0222] Enumeration value rules: For the `register_province` field, let its value be y, using One-Hot encoding combined with TF-IDF weights. Rule: y = "D province". If y is "D province", the corresponding position in the One-Hot encoding is 1, and the rest are 0, and the weights are calculated according to TF-IDF.

[0223] A complete set of metrics is collected every 50 training iterations, including core metrics such as generator loss, discriminator loss, gradient L2 norm, and rule compliance rate.

[0224] The training monitoring system enables comprehensive, multi-dimensional visualization and control of the training process. A real-time dashboard dynamically displays trend curves for 12 core metrics, supporting comparative analysis of five historical versions. An LSTM-based predictive model predicts potential training anomalies three epochs in advance. t-SNE is used to visualize the distribution evolution of generated data and identify signs of pattern collapse. The system records changes in the compliance rate of each business rule and marks rules that consistently fail to meet standards. Real-time display of hardware metrics such as GPU memory and utilization provides early warnings of resource bottlenecks. The system automatically analyzes the root causes of training problems and provides feasibility assessments for adjustment suggestions. The system supports creating monitoring snapshots, allowing users to revisit the complete state at any training moment, including network parameters, data samples, and evaluation results.

[0225] During the metric collection phase, lightweight metrics (loss value, gradient, etc.) are collected every 10 iterations, and complete metrics (distribution statistics, rule compliance rate, etc.) are collected every epoch. Outlier handling, moving average smoothing, and standardization transformation are applied to the collected metrics. Real-time dashboards are updated, and threshold alarms are triggered (e.g., gradient explosion, mode collapse). The current metrics are input into the prediction model to generate performance prediction curves for the next 3 epochs. Pre-set contingency measures (learning rate adjustment, training pause, etc.) are implemented for serious issues (e.g., NaN gradients). Daily training reports are automatically generated, including key metric statistics, problem analysis, and improvement suggestions. Complete monitoring data is compressed and stored, establishing a version-managed training archive.

[0226] During training, the data and rule matrix are sent to the dynamic GAN model for training. The model is trained iteratively over 150 epochs, with the generator and discriminator being optimized alternately. At the end of each training epoch, the loss values ​​(Gloss and Dloss) of the generator and discriminator are returned to evaluate the model performance.

[0227] S150. The cleaning results are processed to obtain the cleaned data.

[0228] In this embodiment, the cleaned data refers to the structured data obtained after verification, format conversion, optimization, and fragmentation.

[0229] In one embodiment, step S150 described above may include steps S151 to S156.

[0230] S151. Verify the consistency of the cleaning results and the integrity of the data.

[0231] In this embodiment, firstly, in step S151, the system validates the DataFrame object from the cleaning engine, checking its schema consistency and data integrity. This step includes checking more than 20 constraints, such as the number of fields, data types, and value ranges, to ensure that the input data meets expectations. This effectively prevents problems caused by data inconsistency or loss, laying a solid foundation for subsequent processing.

[0232] S152. Select the output format of the cleaning results according to the requirements and perform flattening processing on the nested structure of the cleaning results to generate a type metadata description file.

[0233] In this embodiment, the type metadata description file refers to the file that describes the type of each field after the complex data type (such as Array, Map) has been flattened.

[0234] Next, in step S152, a suitable output format is selected based on user configuration or the needs of the downstream system, automatically detected. For data with complex nested structures, the system performs flattening processing, transforming complex type fields such as Array and Map into an easily understandable form and generating corresponding type metadata description files. This not only facilitates understanding and use of the data but also provides necessary preparation for subsequent optimization processing.

[0235] S153. The type metadata description file is optimized through memory compression and type optimization to obtain the optimization result.

[0236] In this embodiment, the optimization result refers to the data and its type information obtained through memory compression and type optimization processes, in order to reduce storage space and improve processing efficiency.

[0237] In step S153, the system performs memory compression and type optimization operations, uniformly converting string fields to Category types and applying minimal data type inference (e.g., from int8 to int16) to numeric fields. Furthermore, a memory-mapped cache is established to reduce copy overhead during serialization. These measures help improve data processing efficiency, reduce storage space usage, and ensure data quality remains unaffected.

[0238] S154. Analyze the optimization results and calculate the optimal sharding strategy.

[0239] In this embodiment, subsequently, in step S154, based on the optimization results obtained in the previous step, the system analyzes data characteristics and calculates the optimal sharding strategy. Factors considered include the number of records in a single file (default setting is 1 million records), file size (target 256MB), and field access patterns (prioritizing hot fields). The result of this step is a detailed sharding plan that guides how to efficiently partition and organize data for subsequent reading and analysis.

[0240] S155. Based on the optimal sharding strategy, a multi-threaded asynchronous writing mode is adopted to optimize the Parquet format of the cleaning results using RowGroup to obtain optimized data.

[0241] In this embodiment, the optimized data refers to the final output data processed using a multi-threaded asynchronous write mode according to the optimal sharding strategy, and optimized using RowGroup on the Parquet format.

[0242] Step S155 involves processing the cleaning results using a multi-threaded asynchronous write mode according to the calculated optimal sharding strategy. Each shard is assigned an independent write thread. Especially for the Parquet format, a RowGroup optimization strategy is applied, automatically determining the optimal RowGroup size (typically between 64MB and 256MB) based on the field type. This approach not only significantly improves write speed but also enhances the read performance of downstream systems.

[0243] S156. Generate a checksum for the optimized data and create a manifest file, and perform a final consistency check to obtain the cleaned data.

[0244] In this embodiment, finally, in step S156, the system generates an SHA-256 checksum for each output file and creates a manifest file (manifest.json) containing all fragment metadata. Furthermore, a final consistency check is performed on the output directory to ensure that the number and size of files meet expectations. These measures ensure data integrity and reliability, allowing the cleaned data to be directly used for business analysis or other purposes.

[0245] In summary, step S150, through meticulous processing and optimization of the cleaning results, ensures the quality, security, and availability of the data, meeting the needs of large-scale data processing in modern data analysis.

[0246] The output layer employs a multi-channel output architecture, capable of simultaneously generating three core deliverables: structured data, quality reports, and operation logs. Through intelligent compression and hierarchical storage technologies, the system can efficiently process data output tasks involving tens of millions of records, while ensuring the integrity and traceability of the results. The output layer also features a specially designed dynamic data anonymization mechanism that automatically adapts the data's sensitivity level based on user permissions, meeting data security requirements in various scenarios.

[0247] The output layer consists of the following two main functional modules:

[0248] The data formatting engine module is responsible for converting and serializing the resulting data.

[0249] Quality Analyzer Module: Generates multi-dimensional data quality assessment reports.

[0250] The data formatting engine module supports six standard data formats, including JSON, CSV, Parquet, XML, Avro, and ORC. Employing columnar storage optimization technology, the engine improves Parquet write speed by more than three times through predictive prefetching and intelligent caching strategies. The system innovatively implements an adaptive sharding algorithm, dynamically adjusting the sharding strategy of output files based on data characteristics (such as record size, number of fields, and nesting depth), optimizing downstream system read efficiency while ensuring file manageability. The output process includes a complete post-processing pipeline, performing standardized operations such as secondary data type validation, unified null value representation (replacing NULL with ""), and encoding standardization (enforcing UTF-8), ensuring that delivered data can be directly used for business analysis without additional processing.

[0251] During the input validation phase, DataFrame objects from the cleaning engine are received, and schema consistency and data integrity are verified. The checks include over 20 constraints such as the number of fields, data types, and value ranges to ensure the input data meets expectations. In the format preprocessing phase, the target output format is selected based on user configuration or automatically detected downstream system requirements. Nested data structures are flattened, and type metadata description files are generated for complex types (such as Array / Map). In the resource optimization phase, memory compression and type optimization are performed, string fields are uniformly converted to Category types, numeric fields are subjected to minimal data type inference (e.g., upgrading from int8 to int16), and a memory-mapped cache is established to reduce copy overhead during serialization. In the intelligent sharding phase, based on data feature analysis results, the optimal sharding strategy is calculated, considering factors such as the number of records per file (default 1 million), file size (target 256MB), and field access patterns (hot fields prioritized), generating a sharding plan and establishing a global index. In the parallel write phase, a multi-threaded asynchronous write mode is adopted, with each shard allocated an independent write thread. A RowGroup optimization strategy is applied to the Parquet format, automatically determining the optimal RowGroup size (typically 64MB-256MB) based on the field type. During the integrity assurance phase, a SHA-256 checksum is generated for each output file, a manifest file (manifest.json) containing all fragment metadata is created, and an eventual consistency check is performed on the output directory to verify that the number and size of files meet expectations.

[0252] The quality report generator includes three main categories and nine subcategories of quality assessment indicators. The system employs a progressive report structure of "problem discovery - root cause analysis - improvement suggestions," automatically identifying data quality problem patterns through machine learning algorithms and providing actionable improvement recommendations. An innovative visualization engine supports dynamic interaction, allowing users to drill down into specific problem record examples by clicking on charts. Report content covers field-level integrity analysis (such as null value rate, uniqueness, etc.), record-level compliance checks (such as rule violation statistics), and dataset-level distribution detection (such as changes in statistical characteristics).

[0253] In the indicator calculation phase, batch calculations of over 50 quality indicators are performed, including basic statistics (such as mean and variance), information theory indicators (such as entropy and KL divergence), and business-specific indicators (such as ID card verification code accuracy). A distributed computing framework is used to accelerate the processing. In the problem classification phase, a two-stage classification method based on rules and machine learning is applied. First, clear problem types are identified through predefined rule templates, then clustering algorithms are used to discover potential abnormal patterns, ultimately forming a standardized problem classification system. In the root cause reasoning phase, a correlation graph between quality problems and processing processes is constructed, and causal reasoning algorithms are used to locate the root causes of problems. Analysis dimensions include data source characteristics, cleaning rule settings, and processing algorithm selection, and confidence scores and supporting evidence are generated. In the visualization design phase, visualization schemes are automatically selected based on report type. Macro reports use large-scale displays such as maps and heatmaps, while micro reports provide detailed record comparison views. All charts support interactive filtering and dimension switching. In the multi-format generation phase, three report formats are output simultaneously: interactive HTML (for web viewing), print-optimized PDF (for archiving), and structured JSON (for system integration), maintaining content consistency across different formats. During the version association phase, the report is precisely associated with the input data version, the cleaning rule version, and the processing code version, and a reproducible quality analysis environment description file (environment.yml) is generated.

[0254] In another embodiment, the cleaning result is inversely converted, decoded, and normalized to obtain the cleaned data.

[0255] During the data generation phase, the system first uses a trained model to generate cleaned data. Then, it performs inverse transformation operations on the generated data, including decoding and normalization, to ensure the data is restored to its original format. Finally, the system returns cleaned data that strictly conforms to all strong rule requirements, thus guaranteeing high data quality.

[0256] During the results delivery phase, the intelligent cleaning system generates a detailed cleaning report, recording the entire cleaning process and the final results. The cleaned data is exported as an Excel file named "Synthetic Data.xlsx" for users to download and use. Furthermore, the system injects new rules and features discovered during the cleaning process into the rule engine to continuously optimize system performance.

[0257] The entire workflow begins with users uploading data, followed by key stages such as rule encoding, model training, and data generation, ultimately producing high-quality cleaned data and delivering it to the user. Simultaneously, the system continuously learns and optimizes, gradually improving the effectiveness and efficiency of data cleaning.

[0258] S160. Output the cleaned data and a cleaning report, wherein the cleaning report includes the cleaning process.

[0259] Cleaned data is the result of a series of data cleaning processes, which may include, but are not limited to, removing duplicates, filling missing values, formatting, and type conversion. The ultimate goal is to ensure the quality of the dataset so that it meets the requirements of subsequent analysis or processing. When outputting, appropriate storage formats (such as CSV, JSON, Parquet, etc.) and locations (local file system, database, cloud storage service, etc.) can be selected according to needs. To facilitate subsequent processing or analysis, the data usually also needs to be properly organized and structured.

[0260] A data cleaning report is a document or file that documents all the operations performed during the entire data cleaning process and their results. Its main purpose is to provide users with a transparent and easy-to-understand data processing history, facilitating the verification of data quality and the identification of potential problems. The following are common components of a data cleaning report:

[0261] Initial data status: Describes the basic situation of the original data, such as size, format, and main problems.

[0262] Specific operation steps: List all operations to be performed and their order, such as deduplication, filling in missing values, format conversion, etc., and attach the specific parameters and implementation methods for each step.

[0263] Problems encountered and solutions: Document the challenges encountered during the cleaning process and how to solve these problems.

[0264] Data cleaning effectiveness evaluation: The effectiveness of data cleaning is evaluated by comparing changes in data before and after cleaning, including but not limited to changes in data volume and improvements in data quality.

[0265] Conclusions and Recommendations: Based on the above, we offer a summary of our findings and provide suggestions for future use of the data.

[0266] In practice, after data cleaning is completed, the cleaned data should first be saved to a designated location. Next, a cleaning report should be compiled based on the log information and statistical data generated during the cleaning process. This report should not only comprehensively reflect the entire cleaning work but also be readable and logically structured to facilitate understanding and use by users with different backgrounds. Furthermore, to enhance the report's practical value, charts or statistics can be included to visually demonstrate the differences before and after cleaning. Finally, ensure that all outputs (data and reports) conform to established standards or specifications to guarantee their validity and reliability.

[0267] Compared with the prior art, the method of this embodiment has the following advantages:

[0268] By deeply coupling the GAN network with the rule engine, adaptive execution of business rules is achieved. This design overcomes the lack of flexibility caused by the excessive rigidity of rules in traditional technologies, improving rule adaptation efficiency by 60% in government data cleaning scenarios.

[0269] The use of a generative adversarial network guided by a rule-based feature matrix ensures both business compliance and reasonable distribution in data repair. This improved the data quality pass rate from 85% to 99.5% and reduced manual review workload by 70%.

[0270] An innovative three-level conflict resolution mechanism (priority / weight / context analysis) is introduced, which, combined with reinforcement learning optimization, effectively solves the problem of rule conflicts in complex business scenarios.

[0271] By tracing the entire lineage and visualizing the decision-making process, it provides a complete audit trail from raw data to remediation results, meeting financial-grade compliance requirements and reducing compliance inspection time by 80%.

[0272] By quantifying business rules into a four-dimensional feature matrix and using it as input constraints for the GAN network, the organic integration of business rules and deep learning models is achieved. Specifically, a rule-based attention layer is designed to dynamically adjust the weight distribution of different business rules in the model based on the input data. The residual block depth (1-8 layers) and the number of attention heads (2-16) of the GAN can be automatically adjusted according to the characteristics of the input data, thereby improving the model's adaptability to different types of data. A three-level conflict handling system is constructed, including priority-based hard filtering, a weighted voting mechanism, and GAN-assisted contextual decision-making, providing solutions for different levels of conflict. The closed-loop learning system automatically transforms the data patterns discovered by the GAN into interpretable business rules. Furthermore, through a hybrid loss function (L_total=0.6L_adv+0.3L_rule+0.1L_dist) and an innovative gradient backpropagation algorithm, the collaborative optimization of business knowledge and model parameters is achieved.

[0273] These core innovations not only enhance the system's flexibility, accuracy, and processing speed, but also improve its interpretability and compliance, providing entirely new approaches to addressing challenges in existing technologies.

[0274] The aforementioned intelligent data cleaning method, through the collaborative optimization of a dynamic rule engine and a GAN network, achieves fully automated processing from the acquisition of data to be cleaned to the final output of cleaned data and reports. First, the data to be cleaned undergoes preliminary processing. Then, the processing results are transformed into a machine-understandable form through rule encoding and input into a dynamic GAN network model that integrates real-time performance monitoring and adaptive adjustment mechanisms for deep cleaning. This process not only ensures high accuracy and semantic consistency in data cleaning but also enhances the flexibility and real-time response capability for processing different types of data through adaptive adjustment mechanisms. The final output data and cleaning report provide a complete audit trail from raw data to cleaning results, significantly improving data quality and business response efficiency, and effectively addressing the shortcomings of existing technologies in terms of flexibility, real-time performance, data diversity processing, model robustness, and generalization ability.

[0275] Figure 3 This is a schematic block diagram of a data intelligent cleaning device 300 provided in an embodiment of the present invention. Figure 3 As shown, corresponding to the above-described intelligent data cleaning method, the present invention also provides a intelligent data cleaning apparatus 300. This intelligent data cleaning apparatus 300 includes a unit for performing the above-described intelligent data cleaning method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 3 The intelligent data cleaning device 300 includes an acquisition unit 301, a preliminary processing unit 302, an encoding unit 303, a cleaning unit 304, a post-processing unit 305, and an output unit 306.

[0276] The system comprises: an acquisition unit 301 for acquiring data to be cleaned; a preliminary processing unit 302 for performing preliminary processing on the data to be cleaned to obtain a preliminary processing result; an encoding unit 303 for performing rule encoding on the preliminary processing result to obtain an encoding result; a cleaning unit 304 for inputting the preliminary processing result and the encoding result into a dynamic GAN network model for cleaning to obtain a cleaned result; wherein the dynamic GAN network model is trained using a rule-constrained generative adversarial architecture combined with real-time performance monitoring and adaptive adjustment mechanisms; a post-processing unit 305 for processing the cleaning result to obtain cleaned data; and an output unit 306 for outputting the cleaned data and a cleaning report, wherein the cleaning report includes the cleaning process.

[0277] In one embodiment, the preliminary processing unit 302 is configured to:

[0278] The system intelligently selects a reading strategy based on the size of the data to be cleaned and automatically adapts to the encoding format to parse the data; it automatically extracts and standardizes the key metadata information of the data to be cleaned and stores it in the database; and it performs basic cleaning and format unification on the data to be cleaned to obtain preliminary processing results.

[0279] In one embodiment, the encoding unit 303 is used for:

[0280] The field names of the preliminary processing results are converted to Unicode format to obtain the converted field names; a mapping relationship is created between the converted field names and the data of the preliminary processing results; the business rules configured by the registered user are registered; a rule matrix is ​​generated according to the business rules and the mapping relationship to obtain the encoding result.

[0281] In one embodiment, the post-processing unit 305 is configured to:

[0282] Verify the consistency and integrity of the cleaning results; select the output format of the cleaning results according to requirements and perform flattening processing on the nested structure of the cleaning results to generate a type metadata description file; optimize the type metadata description file through memory compression and type optimization to obtain the optimized result; analyze the optimized result and calculate the optimal sharding strategy; optimize the Parquet format of the cleaning results using RowGroup in a multi-threaded asynchronous write mode according to the optimal sharding strategy to obtain the optimized data; generate a checksum for the optimized data and create a manifest file, and perform an eventual consistency check to obtain the cleaned data.

[0283] In one embodiment, the post-processing unit 305 is further configured to: perform inverse conversion, decoding, and normalization on the cleaning result to obtain cleaned data.

[0284] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned intelligent data cleaning device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0285] The aforementioned intelligent data cleaning device 300 can be implemented as a computer program, which can, for example... Figure 4 It runs on the computer device shown.

[0286] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0287] See Figure 4 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0288] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a data intelligent cleaning method.

[0289] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0290] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can perform a data intelligent cleaning method.

[0291] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0292] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the intelligent data cleaning method.

[0293] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0294] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0295] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the intelligent data cleaning method.

[0296] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0297] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0298] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0299] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0300] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0301] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data intelligent cleansing method, characterized in that, The method comprises the following steps: acquiring to-be-cleaned data; performing preliminary processing on the to-be-cleaned data to obtain a preliminary processing result; performing rule coding on the preliminary processing result to obtain a coding result; inputting the preliminary processing result and the coding result into a dynamic GAN network model for cleaning to obtain a cleaning result, wherein the dynamic GAN network model is trained by combining real-time performance monitoring and self-adaptive adjustment mechanism through a rule-constrained generative adversarial architecture; processing the cleaning result to obtain cleaned data; outputting the cleaned data and a cleaning report, wherein the cleaning report comprises a cleaning process; wherein the dynamic GAN network model comprises a rule-aware generator, a dual discriminator, a dynamic regulator and a rule feature encoder; the rule-aware generator is based on a deep residual network structure, integrates a rule attention mechanism, receives noise input and rule features, and outputs a data repair result meeting the requirements; the dual discriminator comprises a data authenticity discrimination channel and a rule compliance verification channel, and gives a comprehensive evaluation through a fusion weight; the dynamic regulator monitors performance indicators in real time, automatically triggers a structure adjustment strategy, and optimizes network parameters; the rule feature encoder is used to convert various business rules into numerical feature representations, supporting multi-type encoding conversion of ranges and classifications.

2. The data intelligent cleaning method of claim 1, wherein, The preliminary processing on the to-be-cleaned data to obtain a preliminary processing result comprises: intelligently selecting a reading strategy according to the size of the to-be-cleaned data and automatically adapting to the coding format to parse the data; automatically extracting and standardizing key metadata information of the to-be-cleaned data and storing it in a database; performing basic cleaning and format unification on the to-be-cleaned data to obtain a preliminary processing result.

3. The data intelligent cleansing method of claim 1, wherein, The rule coding on the preliminary processing result to obtain a coding result comprises: converting the field name of the preliminary processing result into Unicode format to obtain a converted field name; creating a mapping relationship between the converted field name and the data of the preliminary processing result; registering a user-configured business rule; generating a rule matrix according to the business rule and the mapping relationship to obtain a coding result.

4. The data intelligent cleansing method of claim 1, wherein, The training process of the dynamic GAN network model comprises: acquiring historical preliminary processing results and corresponding coding results; standardizing the historical preliminary processing results and corresponding coding results, and fusing the data obtained after standardization into the same feature space through a fully connected layer, dynamically adjusting the feature weight through an improved residual network architecture to obtain generated data; processing the generated data using a Tanh activation function to make the generated data fall within a reasonable interval to obtain a generated result; combining the generated result with the weighted sum of the difference between the real data distribution and the business rule compliance degree to obtain a total loss value, using an Adam optimizer to back-propagate and applying gradient clipping technology to ensure the stability of the training process, while extracting multi-level features through a convolution layer and using batch normalization and LeakyReLU activation function to improve feature representation; The encoding results are processed piece by piece to calculate a matching score between the generated data and the rules to ensure data compliance; The fusion weight of the two independent channels of data authenticity discrimination and rule compliance verification is automatically adjusted according to the training phase; Real-time performance indicators are collected, and the optimal adjustment scheme is selected to maintain the best state of the model to obtain a dynamic GAN network model.

5. The data intelligent cleansing method of claim 1, wherein, The cleaning results are processed to obtain cleaned data, including: The cleaning results are verified for consistency and data integrity; The output format of the cleaning results is selected according to the requirements, and the nested structure of the cleaning results is flattened to generate a type metadata description file; The type metadata description file is compressed in memory and optimized by type to obtain an optimization result; The optimization result is analyzed and the optimal sharding strategy is calculated; According to the optimal sharding strategy, a multi-thread asynchronous writing mode is adopted to optimize the RowGroup of the Parquet format of the cleaning results to obtain optimized data; A check code is generated for the optimized data, and a manifest file is created for final consistency checking to obtain cleaned data.

6. The data intelligence cleansing method of claim 1, wherein, The cleaning results are processed to obtain cleaned data, including: The cleaning results are processed to obtain cleaned data, including:

7. A data intelligent cleansing device, characterized by, The data intelligent cleaning method comprises the following steps: An acquisition unit is configured to acquire to-be-cleaned data; A preliminary processing unit is configured to perform preliminary processing on the to-be-cleaned data to obtain a preliminary processing result; An encoding unit is configured to perform rule encoding on the preliminary processing result to obtain an encoding result; A cleaning unit is configured to input the preliminary processing result and the encoding result into a dynamic GAN network model for cleaning to obtain a cleaning result, wherein the dynamic GAN network model is trained by combining a rule-constrained generative adversarial architecture, real-time performance monitoring, and an adaptive adjustment mechanism; A post-processing unit is configured to process the cleaning result to obtain cleaned data; An output unit is configured to output the cleaned data and a cleaning report, wherein the cleaning report includes a cleaning process.

8. A computer device, comprising: The computer device comprises a memory and a processor, and the memory stores a computer program, and the processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images