Data synchronization method and system for intelligent dirty data detection and restoration based on DataX

Abstract

The invention relates to a data synchronization method and system for intelligent dirty data detection and restoration based on DataX. According to the method, an intelligent dirty data processing engine is embedded between a Reader plug-in and a Writer plug-in of DataX, type matching, format verification, constraint conflict pre-detection and abnormal value recognition are completed online, and type conversion, format standardization, abnormal value replacement or flexible writing are carried out on dirty data according to a preset or self-adaptive strategy. And finally, the Writer executes insertion, updating, skipping or log recording only according to an instruction, and a traceable governance log is generated in the whole process. According to the method, the synchronization-cleaning capability is embedded into the DataX framework, so that the synchronization success rate and the data quality are remarkably improved, the external ETL dependence and the artificial script cost are reduced, and the method has the advantages of high intelligence, high throughput and flexible configuration.

Description

Technical Field

[0001] This application relates to the field of database synchronization and data migration technology, and in particular to a data synchronization method, system, computer-readable storage medium, and electronic device based on DataX for intelligent dirty data detection and repair. Background Technology

[0002] DataX is a high-efficiency, low-cost, and versatile tool for synchronizing massive amounts of data. It boasts significant capabilities in data extraction and loading, and is widely used in data processing scenarios such as ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform). Its core advantage lies in its scalable Reader and Writer plugin architecture, which supports connections to various heterogeneous data sources and target storage, and provides diverse data format adaptation capabilities, meeting the fundamental needs of cross-system data migration and synchronization.

[0003] To address common "dirty data" issues during data synchronization (typically data type mismatches, format errors, database constraint conflicts, missing fields, or outliers), DataX currently relies primarily on the following mechanisms for basic processing: 1. Built-in database validation capability: Relying on the native constraints of the source or target database (such as field type, unique index, etc.), database-level validation is triggered during the data writing phase to intercept dirty data that does not conform to the constraints of the target database. 2. User-defined SQL preprocessing: This requires users to manually write SQL scripts before data extraction (on the source database side) or after data writing (on the target database side) to achieve dirty data management through filtering, cleaning, or transformation operations. 3. Basic type conversion support: Some plugins have built-in simple type conversion functions (such as string to number, date format standardization, etc.), but the functions are relatively fixed and the flexibility and adaptability are limited.

[0004] Limitations of existing technology Although DataX can handle some basic dirty data issues under the above mechanism, its dirty data processing capabilities are still significantly insufficient, specifically manifested in the following ways: 1. Weak ability to process complex and dirty data: For complex and dirty data scenarios that are not explicit and have strong logic, such as fuzzy matching (e.g., address information similarity verification), semantic inconsistency (e.g., conflict of multiple sources for the same entity), and cross-field logic verification (e.g., the product of order amount and quantity does not match the total price), the existing mechanism lacks effective identification and repair capabilities. 2. High cost of manual intervention: Governing complex and dirty data requires users to develop a large number of SQL scripts or integrate external ETL tools, which leads to a longer development cycle, increased maintenance complexity, and script errors may introduce new data quality problems; 3. Significant risk of synchronization interruption: When dirty data triggers a write failure, the DataX task is prone to overall interruption, requiring manual troubleshooting, data repair, and task restart, which seriously affects the timeliness of data synchronization and business continuity. 4. Insufficient data quality assurance: Even if the synchronization is completed, dirty data that has not been effectively processed will still cause data distortion in the target database, thereby affecting the accuracy of downstream data analysis results and the reliability of decision-making. 5. Low level of intelligence: The existing processing methods are mainly driven by static rules (such as predefined type conversion rules or SQL filtering conditions), which makes it difficult to dynamically adapt to changes in data patterns (such as table structure modification, field meaning adjustment) or dirty data patterns of unknown types. 6. Synchronization efficiency is limited by preprocessing: Large-scale custom SQL preprocessing may consume additional computing resources, creating resource competition with data synchronization tasks and reducing overall synchronization performance.

[0005] In summary, existing data synchronization solutions based on DataX suffer from problems such as ambiguous capability boundaries, high reliance on manual intervention, and insufficient robustness in the dirty data detection and repair process. There is an urgent need for an intelligent and adaptive solution to improve the quality and efficiency of data synchronization. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this application proposes a novel data synchronization method and system based on DataX for intelligent dirty data detection and repair.

[0007] The main technical strategy employed in this invention is to provide a "data synchronization framework for intelligent dirty data perception and adaptive processing." This framework integrates an "Intelligent Dirty Data Processing Engine" between the Reader and Writer plugins of DataX. This engine can proactively perceive and analyze dirty data in the data stream, and intelligently detect, convert, format-correct, handle outliers, and perform flexible / strategic writing of dirty data according to preset or adaptive learning strategies. Based on the above strategy, this invention can transfer dirty data processing capabilities from the source / target database or independent ETL tools to the "embedded" dirty data processing process of DataX, achieving intelligent and automated processing of "synchronization as cleaning."

[0008] To achieve the above objectives, the present invention employs the following design: (1) Seamless integration of “synchronization and cleaning”: The intelligent dirty data detection and repair capabilities are embedded into the data synchronization process of DataX, breaking the traditional DataX model of only performing data migration and outsourcing the cleaning task. This invention achieves automation and efficiency improvement at the DataX framework level through this integrated processing.

[0009] (2) Multi-dimensional and intelligent dirty data processing engine: The dirty data processing engine integrates multiple detection and processing capabilities, including: a) Intelligent type automatic conversion: It can actively attempt to parse strings of various formats and convert them to the target type, rather than simply hard-coding the conversion.

[0010] b) Flexible constraint conflict handling: It provides multiple strategies such as SKIP, UPDATE, and LOG_ONLY to avoid interrupting the entire synchronization task due to a small number of constraint conflicts, thereby improving the robustness of the system.

[0011] c) Adaptive outlier handling: Supports outlier detection based on thresholds and business rules, and provides a variety of processing strategies.

[0012] (3) Strategic and configurable processing flow: It provides a highly configurable interface that allows users to flexibly define various dirty data detection rules and processing strategies through JSON configuration, so as to meet the application needs of different scenarios and have high flexibility and scalability.

[0013] (4) Enhanced DataX plugin design: By enhancing the synergy of the Reader and Writer plugins and introducing independent engine components, the DataX ecosystem has been effectively expanded without destroying the original core functions of DataX.

[0014] (5) Detailed logging and traceability: It realizes detailed recording of the dirty data processing process, which facilitates users to audit, analyze and subsequently manage data.

[0015] Specifically, this application provides the following technical solutions: The first aspect of this application provides a data synchronization method for intelligent dirty data detection and repair based on DataX, the method comprising: S1. During the DataX job startup phase, read the user-configured dirty data detection rule set and processing strategy set, and instantiate the intelligent dirty data processing engine. S2. Read source data records one by one through the enhanced Reader plugin, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine; S3. The intelligent dirty data processing engine performs online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and outputs dirty data tags and corresponding processing instructions. S4. The intelligent dirty data processing engine performs adaptive repair on the marked records based on the processing instructions, including at least intelligent type conversion, format standardization, outlier replacement or marking, and flexible handling of constraint conflicts, to obtain the repaired records. S5. Send the repaired record and accompanying instructions to the enhanced Writer plugin. The Writer plugin will perform normal writing, update writing, skip, or log recording operations according to the instructions, and provide real-time feedback on the writing results. S6. Record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

[0016] Furthermore, in the method of this application, the intelligent dirty data processing engine in step S1 includes the following functional modules: (1) Dirty data detection module: detects data type mismatch, format errors (date / time, number, string length / pattern), constraint conflicts (uniqueness, primary key), outliers (threshold, business rules) and missing values; (2) Automatic data type conversion module: Converts the source data type to the target schema type (such as string to number, date, and normalization of boolean values), and handles conversion failures; (3) Format standardization module: clean up strings (remove leading and trailing spaces, merge embedded spaces), and unify text case; (4) Flexible Constraint Conflict Handling Module: Configure SKIP, UPDATE, MERGE, and LOG_ONLY strategies to handle primary key or unique key conflicts; (5) Outlier handling module: Configure KEEP, REPLACE_WITH_NULL, REPLACE_WITH_STATS, and TAG strategies to handle outliers.

[0017] Furthermore, in the method of this application, the enhanced Reader plugin described in step S2 also performs the following operations: (1) Data format awareness: Identify the actual format of the field (such as date string format, number thousands separator); (2) Preliminary type verification: Determine the matching of field types with the target schema, and mark potentially mismatched data as "to be verified"; (3) Meta-information generation: Output records carrying meta-information, which includes at least the original type and format inference information.

[0018] Furthermore, in the method of this application, the type matching verification in step S3 compares the source type of the intermediate record with the target Schema field type. If they are inconsistent, the subsequent intelligent type conversion step is triggered. The format compliance verification includes: (1) Parse the date string according to the preset multi-format template. If the parsing fails, mark it as a format error. (2) Perform regular expression pattern matching on the numeric string to identify non-numeric interference characters; (3) Perform length and regular expression pattern validation on the string field; The constraint conflict pre-detection includes: (1) Maintain a set of processed primary keys or unique keys in the memory cache and perform fast conflict detection on the record to be written; (2) When a primary key conflict is predicted, output the SKIP, UPDATE, MERGE or LOG_ONLY policy instruction; The outlier identification employs a dual-mode approach using threshold rules and business rules. (1) Threshold rules include upper and lower limits, and standard deviation multiples; (2) Business rules describe field-level logical restrictions using user-defined expressions.

[0019] Furthermore, in the method of this application, the intelligent type conversion in step S4 includes: (1) Perform precision-preserving numeric type conversion on parsable numeric strings; (2) Standardize the recognizable date strings according to the target time type; (3) Perform semantic mapping transformation on the boolean string; The flexible handling of constraint conflicts includes: (1) When the instruction is UPDATE, the Writer plugin sends a UPSERT or UPDATE statement to the target. (2) When the instruction is SKIP, the Writer plugin skips the current record and updates the skip counter; (3) When the instruction is LOG_ONLY, the Writer plugin will only write the records to the exception table and will not update the official table.

[0020] Furthermore, in the method of this application, the enhanced Writer plugin described in step S5 also performs the following operations: capturing write errors, recording failure records and reasons, and outputting them to a log or a dedicated error data table to avoid interrupting the synchronization task.

[0021] Furthermore, in the method of this application, the dirty data governance log described in step S6 records at least: table name, field name, original value, repair value, processing strategy, timestamp, task instance ID, and whether the writing was successful.

[0022] A second aspect of this application provides a data synchronization system for intelligent dirty data detection and repair based on DataX, the system comprising: The configuration parsing module is used to read the user-configured dirty data detection rule set and processing strategy set during the DataX job startup phase, and instantiate the intelligent dirty data processing engine. An enhanced Reader plugin is used to read source data records one by one, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine. The intelligent dirty data processing engine is used to perform online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and output dirty data tags and corresponding processing instructions; then, based on the processing instructions, the marked records are adaptively repaired, including at least intelligent type conversion, format standardization, outlier replacement or tagging, and flexible constraint conflict processing, to obtain the repaired records; The enhanced Writer plugin is used to receive repaired records and accompanying instructions, and perform normal write, update write, skip or log-only operations according to the instructions, and provide real-time feedback on the write results; The logging module is used to record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

[0023] The system implements the steps of the aforementioned data synchronization method for intelligent dirty data detection and repair based on DataX during runtime.

[0024] Furthermore, in the system of this application, the intelligent dirty data processing engine includes: (1) Dirty data detection module: used to detect data type mismatch, format errors (date / time, number, string length / pattern), constraint conflicts (uniqueness, primary key), outliers (threshold, business rules) and missing values; (2) Automatic data type conversion module: used to convert the source data type to the target schema type (such as string to number, date, Boolean value normalization), and handle the case of conversion failure; (3) Format standardization module: used to clean up strings (remove leading and trailing spaces, merge embedded spaces) and unify text case; (4) Flexible Constraint Conflict Handling Module: Used to configure SKIP, UPDATE, MERGE, and LOG_ONLY strategies to handle primary key or unique key conflicts; (5) Exception handling module: used to configure KEEP, REPLACE_WITH_NULL, REPLACE_WITH_STATS, TAG policies and handle exception values.

[0025] Furthermore, in the system of this application, the intelligent dirty data processing engine is embedded in the Channel of DataX in the form of an interceptor or a plugin, maintaining thread-level pipeline parallelism with the Reader and Writer plugins, and without intruding on the native core of DataX.

[0026] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned data synchronization method for intelligent dirty data detection and repair based on DataX.

[0027] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned data synchronization method for intelligent dirty data detection and repair based on DataX.

[0028] In summary, the data synchronization method for intelligent dirty data detection and repair based on DataX proposed in this invention has the following technical advantages: (1) Significantly improves the success rate and robustness of data synchronization tasks. By embedding an "intelligent dirty data processing engine," this method proactively identifies and handles common dirty data issues (such as type mismatches, format errors, and simple logical conflicts) throughout the entire data synchronization process, reducing the risk of write failures due to data quality defects from the source. Compared to the traditional approach that relies on passive database verification or post-processing by external tools, this solution prioritizes quality control, effectively ensuring the stable execution of synchronization tasks and significantly improving task completion rates.

[0029] (2) Systematically optimize the quality of target data This method integrates intelligent format standardization, adaptive type conversion, and intelligent outlier cleaning capabilities, enabling multi-dimensional normalization of dirty data. The processed data shows significant improvements in format consistency, type accuracy, and value range rationality, resulting in higher quality data output to the target end. This directly meets the high-reliability data requirements of downstream data analysis and modeling scenarios, reducing subsequent data governance costs.

[0030] (3) Reduce user development and maintenance costs By leveraging built-in intelligent processing capabilities, this method reduces reliance on complex external ETL tools and eliminates the need for users to manually write numerous preprocessing / postprocessing SQL scripts, simplifying development and maintenance processes and significantly reducing manpower and time costs.

[0031] (4) Enhance the intelligent processing capabilities for complex and dirty data scenarios. Unlike traditional rule-driven fixed validation logic, this method introduces intelligent algorithms combined with adaptive strategies to effectively handle complex dirty data issues such as fuzzy matching, cross-field logical validation, and semantic inconsistencies. Simultaneously, the system possesses a certain degree of pattern adaptation capability, enabling it to cope with minor changes in data structure (such as adding fields or adjusting data types), thus improving the generalization ability of data synchronization.

[0032] (5) Improve synchronization efficiency and reduce redundant data processing In traditional solutions, complex and dirty data requires multiple rounds of extraction-transformation-loading (ETL) via external ETL, leading to redundant data processing and time consumption. This method deeply integrates dirty data detection and repair into the main DataX synchronization process, avoiding additional data transfer steps. While ensuring quality, it shortens the end-to-end synchronization cycle, resulting in a significant improvement in overall efficiency compared to solutions that rely on external tools.

[0033] (6) Supports flexible and configurable business adaptation This method provides multi-dimensional configuration options, allowing users to flexibly select or customize processing strategies according to specific business scenarios (such as strong consistency requirements for financial data and fault tolerance requirements for log data), balancing generality and scenario adaptability, and improving the scalability of the system.

[0034] (7) Strengthen the auditing and traceability of data processing This system has a built-in detailed dirty data processing log recording function, which fully records information such as the type of dirty data, detection location, repair strategy, and data status before and after processing. This log can be used for problem backtracking analysis, data quality assessment, and compliance auditing, which is beneficial for problem investigation and data governance, and improves the reliability and management efficiency of the data synchronization process.

[0035] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0036] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this solution will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the overall implementation of the data synchronization method for intelligent dirty data detection and repair based on DataX, as described in this application.

[0038] Figure 2 This is the overall design architecture diagram of this application.

[0039] Figure 3 This is a structural diagram of the data synchronization system for intelligent dirty data detection and repair based on DataX, as described in this application.

[0040] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0042] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0043] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0044] Figure 1 The diagram shows the overall implementation flow of the data synchronization method for intelligent dirty data detection and repair based on DataX provided in this application, including the following steps: S1. During the DataX job startup phase, read the user-configured dirty data detection rule set and processing strategy set, and instantiate the intelligent dirty data processing engine. S2. Read source data records one by one through the enhanced Reader plugin, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine; S3. The intelligent dirty data processing engine performs online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and outputs dirty data tags and corresponding processing instructions. S4. The intelligent dirty data processing engine performs adaptive repair on the marked records based on the processing instructions, including at least intelligent type conversion, format standardization, outlier replacement or marking, and flexible handling of constraint conflicts, to obtain the repaired records. S5. Send the repaired record and accompanying instructions to the enhanced Writer plugin. The Writer plugin will perform normal writing, update writing, skip, or log recording operations according to the instructions, and provide real-time feedback on the writing results. S6. Record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

[0045] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0046] The overall design architecture of this invention is as follows: Figure 2 As shown, its core functions and processes are described below: This invention enhances the DataX framework by introducing a core "intelligent dirty data processing engine," which can be integrated into the DataX workflow as a plugin or interceptor.

[0047] 1. Enhanced Reader plugin Data format awareness: When reading data, the Reader plugin can not only read the schema, but also try to detect the actual data format of the fields (for example, to identify different date string formats, thousands separators of numbers, etc.).

[0048] Preliminary type validation: Before passing the data to the engine, a preliminary type judgment is performed, and data that may not match is marked as "to be validated".

[0049] Generate enhanced metadata information: When outputting a record, the Reader plugin can carry more metadata information, such as the original type of the field and possible format inferences, for the engine to use.

[0050] 2. Intelligent dirty data processing engine This engine is the core of the invention. It receives the Record stream output by the Reader plugin and performs a series of dirty data processing according to the configuration.

[0051] (1) Dirty Data Detection Module Objective: To identify dirty data in a data stream.

[0052] accomplish: a) Data type mismatch detection: Compare the types parsed by the Reader with the types defined in the fields of the target Schema to identify mismatches.

[0053] b) Formatting error detection: Date / Time Format: Attempts to parse using several predefined common date formats (such as yyyy-MM-dd, MM / dd / yyyy, yyyy / MM / dd HH:mm:ss, etc.). If it cannot be parsed, it is marked as a format error.

[0054] Numeric format: Checks whether the numeric value contains non-numeric characters (such as the non-standard use of +, -, , and .).

[0055] String length / pattern: Checks whether the string is compliant based on preset length limits or regular expressions.

[0056] c) Constraint conflict early warning: Uniqueness detection (memory / cached): For small amounts of data, a set of written field values ​​can be maintained in memory to detect potential uniqueness constraint conflicts.

[0057] Primary key conflict prediction: Compare the primary key of the record to be written with the primary key of the existing record in memory to predict possible conflicts.

[0058] d) Outlier detection: Threshold rules: Users can configure the upper and lower limits (such as min_value, max_value) or standard deviation multiples for numeric fields.

[0059] Business logic rules: Users can define simple business rules (such as age cannot be negative, height cannot exceed 3 meters, etc.).

[0060] e) Missing value detection: Identify null values ​​(null, '').

[0061] (2) Automatic data type conversion module Objective: To automatically attempt to convert source data into the type of the target schema.

[0062] accomplish: a) Numerical conversion: If the source is a string, and the string can be converted to the target numeric type (e.g., "123.45" -> 123.45, "+789" -> 789), then the conversion is performed directly.

[0063] If the conversion fails, or if the result is converted to a floating-point number but the target is an integer, it will be handled according to a predefined strategy (mark, set to NULL, round down).

[0064] b) Date / Time Conversion: Try parsing the date string using the various formats identified by the detection module.

[0065] After successful parsing, it is converted into the date / time type defined in the target schema (such as java.sql.Timestamp).

[0066] If the conversion fails, mark it or set it to NULL.

[0067] c) Boolean value conversion: Convert "true", "1", "yes" etc. to Boolean true; convert "false", "0", "no" etc. to Boolean false.

[0068] (3) Format standardization module Objective: To standardize data formats and make them conform to common specifications.

[0069] accomplish: a) String cleanup: Remove leading and trailing spaces from fields (trim()), and replace consecutive embedded spaces with a single space.

[0070] b) Case consistent: For configured text fields, they can be converted to all uppercase or all lowercase.

[0071] (4) Flexible Constraint Conflict Handling Module Objective: To prevent the entire task from failing due to constraint conflicts and to provide strategic solutions.

[0072] accomplish: a) Strategy Configuration: Provides multiple processing strategies (based on primary key / unique key): SKIP (Ignore): When a conflict is detected, skip the current record and do not write it to the log.

[0073] UPDATE: When a primary key conflict is detected, update the existing record according to the field priority specified by the user.

[0074] MERGE: Combined with UPDATE, it performs more complex logical merges (such as array merging).

[0075] LOG_ONLY (Log Only): If a conflict is found, it is not written; the conflict record is only written to the log or a specific "anomaly table".

[0076] b) Execution: When the Writer plugin receives a Record, it performs the corresponding operation according to the processing instructions (such as SKIP, UPDATE, etc.) provided by the interceptor.

[0077] (5) Outlier handling module Objective: To handle outliers, with options to retain, mark, or replace them.

[0078] accomplish: a) Strategy configuration: KEEP: An outlier was detected, but it was still written to the original value (marked as an outlier).

[0079] REPLACE_WITH_NULL (replace with NULL): Replaces the exception value with NULL.

[0080] REPLACE_WITH_STATS (replace with statistics): Replace with the configured mean, median, mode, etc. (must be calculated or estimated in advance).

[0081] TAG: Add a field to the data to mark that the record contains an outlier, without modifying the original value.

[0082] b) Execution: The engine modifies abnormal values ​​or adds marker fields in the Record according to the configuration.

[0083] (6) Log recording module Objective: To document the details of all dirty data processing.

[0084] accomplish: Record detected dirty data types (type mismatch, format error, constraint conflict, outlier, missing value).

[0085] Record the table name, field name, and original value involved.

[0086] Record the processing strategies used (conversion, standardization, skipping, updating, replacement, etc.).

[0087] Record the processed value (if any).

[0088] Record failed writes and the reasons for them, and output them to a specified file or database table.

[0089] 3. Enhanced Writer plugin (1) Receive processed Record: The Writer plugin receives Records from the engine that may have been modified.

[0090] (2) Execution strategy writing: For records that are written normally, execute directly.

[0091] For records marked as "skipped", the Writer discards them directly and records them in the log.

[0092] For a Record that needs to be "updated", the Writer needs to interact with the target to perform the UPDATE operation.

[0093] For records marked as "outliers" but retained, the Writer writes normally.

[0094] (3) Graceful error handling: Even after processing, a very small number of dirty data may still cause write errors. In this case, the Writer plugin needs to capture these errors, record detailed information, and send the failed record to the log module or write it to a special "error data table" instead of interrupting the entire synchronization task.

[0095] Figure 3 The figure shown is a data synchronization system for intelligent dirty data detection and repair based on DataX, as proposed in this application. The system includes: The configuration parsing module is used to read the user-configured dirty data detection rule set and processing strategy set during the DataX job startup phase, and instantiate the intelligent dirty data processing engine. An enhanced Reader plugin is used to read source data records one by one, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine. The intelligent dirty data processing engine is used to perform online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and output dirty data tags and corresponding processing instructions; then, based on the processing instructions, the marked records are adaptively repaired, including at least intelligent type conversion, format standardization, outlier replacement or tagging, and flexible constraint conflict processing, to obtain the repaired records; The enhanced Writer plugin is used to receive repaired records and accompanying instructions, and perform normal write, update write, skip or log-only operations according to the instructions, and provide real-time feedback on the write results; The logging module is used to record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

[0096] The above system implements the data synchronization method for intelligent dirty data detection and repair based on DataX disclosed in this application during operation.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0098] like Figure 4 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned data synchronization method for intelligent dirty data detection and repair based on DataX.

[0099] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0100] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0101] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0102] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0103] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the data synchronization method for intelligent dirty data detection and repair based on DataX disclosed in this application.

[0104] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0105] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0106] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0107] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0108] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0109] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0110] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0111] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0112] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0113] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0114] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0115] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the data synchronization method for intelligent dirty data detection and repair based on DataX disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.

[0116] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0117] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data synchronization method for intelligent dirty data detection and repair based on DataX, characterized in that, The method includes: S1. During the DataX job startup phase, read the user-configured dirty data detection rule set and processing strategy set, and instantiate the intelligent dirty data processing engine. S2. Read source data records one by one through the enhanced Reader plugin, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine; S3. The intelligent dirty data processing engine performs online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and outputs dirty data tags and corresponding processing instructions. S4. The intelligent dirty data processing engine performs adaptive repair on the marked records based on the processing instructions, including at least intelligent type conversion, format standardization, outlier replacement or marking, and flexible handling of constraint conflicts, to obtain the repaired records. S5. The repaired record and accompanying instructions are sent to the enhanced Writer plugin, which performs normal writing, update writing, skip, or log recording operations according to the instructions, and provides real-time feedback on the writing results. S6. Record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

2. The method according to claim 1, characterized in that, The intelligent dirty data processing engine mentioned in step S1 includes the following functional modules: (1) Dirty data detection module: detects data type mismatch, format error, constraint conflict, outlier and missing values; (2) Automatic data type conversion module: Converts the source data type to the target schema type; (3) Format standardization module: clean up strings and unify text case; (4) Flexible Constraint Conflict Handling Module: Configure SKIP, UPDATE, MERGE, and LOG_ONLY strategies to handle primary key or unique key conflicts; (5) Outlier handling module: Configure KEEP, REPLACE_WITH_NULL, REPLACE_WITH_STATS, and TAG strategies to handle outliers.

3. The method according to claim 1, characterized in that, The enhanced Reader plugin described in step S2 also performs the following operations: (1) Data format awareness: Identify the actual format of a field; (2) Preliminary type verification: Determine the matching of field types with the target schema, and mark potentially mismatched data as "to be verified"; (3) Meta-information generation: Output records carrying meta-information, which includes at least the original type and format inference information.

4. The method according to claim 1, characterized in that, The type matching verification in step S3 compares the source type of the intermediate record with the target schema field type. If they are inconsistent, the subsequent intelligent type conversion step is triggered. The format compliance verification includes: (1) Parse the date string according to the preset multi-format template. If the parsing fails, mark it as a format error. (2) Perform regular expression pattern matching on the numeric string to identify non-numeric interference characters; (3) Perform length and regular expression pattern validation on the string field; The constraint conflict pre-detection includes: (1) Maintain a set of processed primary keys or unique keys in the memory cache and perform fast conflict detection on the record to be written; (2) When a primary key conflict is predicted, output the SKIP, UPDATE, MERGE or LOG_ONLY policy instruction; The outlier identification employs a dual-mode approach using threshold rules and business rules. (1) Threshold rules include upper and lower limits, and standard deviation multiples; (2) Business rules describe field-level logical restrictions using user-defined expressions.

5. The method according to claim 1, characterized in that, The intelligent type conversion in step S4 includes: (1) Perform precision-preserving numeric type conversion on parsable numeric strings; (2) Standardize the recognizable date strings according to the target time type; (3) Perform semantic mapping transformation on the boolean string; The flexible handling of constraint conflicts includes: (1) When the instruction is UPDATE, the Writer plugin sends a UPSERT or UPDATE statement to the target. (2) When the instruction is SKIP, the Writer plugin skips the current record and updates the skip counter; (3) When the instruction is LOG_ONLY, the Writer plugin will only write the records to the exception table and will not update the official table.

6. The method according to claim 1, characterized in that, The enhanced Writer plugin described in step S5 also performs the following operations: captures write errors, records failure records and reasons, and outputs them to a log or a dedicated error data table to avoid interrupting the synchronization task.

7. The method according to claim 1, characterized in that, The dirty data governance log described in step S6 shall record at least the following: table name, field name, original value, repair value, processing strategy, timestamp, task instance ID, and whether the writing was successful.

8. A data synchronization system for intelligent dirty data detection and repair based on DataX, characterized in that, The system includes: The configuration parsing module is used to read the user-configured dirty data detection rule set and processing strategy set during the DataX job startup phase, and instantiate the intelligent dirty data processing engine. An enhanced Reader plugin is used to read source data records one by one, generate intermediate records carrying the original values ​​and source type information, and push them to the intelligent dirty data processing engine. The intelligent dirty data processing engine is used to perform online dirty data detection on the intermediate records, including at least type matching verification, format compliance verification, constraint conflict pre-detection and outlier identification, and output dirty data tags and corresponding processing instructions; then, based on the processing instructions, the marked records are adaptively repaired, including at least intelligent type conversion, format standardization, outlier replacement or tagging, and flexible constraint conflict processing, to obtain the repaired records; The enhanced Writer plugin is used to receive repaired records and accompanying instructions, and perform normal write, update write, skip or log-only operations according to the instructions, and provide real-time feedback on the write results; The logging module is used to record detection and repair events throughout the entire process, including recording dirty data types, processing strategies, values ​​before and after processing, and failure write information, forming a traceable dirty data governance log.

9. The system according to claim 8, characterized in that, The intelligent dirty data processing engine includes: (1) Dirty data detection module: used to detect data type mismatch, format error, constraint conflict, outlier and missing values; (2) Automatic data type conversion module: used to convert the source data type to the target schema type; (3) Format standardization module: used to clean up strings and unify text case; (4) Flexible Constraint Conflict Handling Module: Used to configure SKIP, UPDATE, MERGE, and LOG_ONLY strategies to handle primary key or unique key conflicts; (5) Exception handling module: used to configure KEEP, REPLACE_WITH_NULL, REPLACE_WITH_STATS, TAG policies and handle exception values.

10. The system according to claim 8, characterized in that, The intelligent dirty data processing engine is embedded in DataX's Channel as an interceptor or plugin, maintaining thread-level pipeline parallelism with the Reader and Writer plugins, and without intruding on DataX's native core.

Images