Real-time data acquisition dirty data processing method and system based on Flink CDC
By introducing a pre-verification mechanism and metadata caching during the real-time data acquisition process of Flink CDC, dirty data is identified and isolated, solving the problem of low efficiency in dirty data identification in existing technologies. This enables early identification and interception, improves data write success rate and system stability, and provides accurate dirty data information and closed-loop processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ELECTRONICS CLOUD DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies suffer from low efficiency in identifying and processing dirty data during real-time data acquisition, delayed detection timing, ambiguous error information, easy concealment of anomalies, and limited capture range, resulting in poor data quality and processing timeliness.
We adopt a real-time data acquisition method based on Flink CDC, introduce a pre-validation mechanism to perform strict validation before writing data to the target end, identify dirty data through metadata caching and dynamic rule engine, and encapsulate it into structured messages and send them to Kafka isolation topics. Combined with an exception capture mechanism, we form a closed loop for dirty data processing.
It enables early identification and interception of dirty data, improves data writing success rate, reduces processing costs, enhances the stability of the target system, provides accurate dirty data information, facilitates rapid location and repair, and builds a closed loop for dirty data processing.
Smart Images
Figure CN121996646A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of real-time big data acquisition and processing technology, and in particular to a method, system, computer-readable storage medium, and electronic device for real-time data acquisition, dirty data identification, collection, and processing based on Flink CDC (Change Data Capture). Background Technology
[0002] The era of big data places higher demands on real-time data acquisition and processing capabilities. Apache Flink CDC, as a core component of the Flink ecosystem, can efficiently capture real-time changes from data sources such as databases and synchronize them to downstream systems. However, in actual acquisition scenarios, defects in source data quality or non-standard table structures on the target side (such as excessive field length, incompatible data types, missing non-null constraints, etc.) often lead to data write failures, resulting in so-called "dirty data."
[0003] Existing technologies primarily handle dirty data through write exception handling mechanisms, i.e., post-event interception when illegal data triggers a database write error. However, this approach has significant drawbacks:
[0004] 1. Delayed detection: The anomaly is only discovered after the data has already flowed through the transmission link and may even have been partially written successfully, which significantly increases the cost and complexity of data rollback, retry and repair.
[0005] 2. Vague error messages: Some general database exceptions only return broad write failure messages, making it impossible to pinpoint the specific details of the violation of validation rules (such as length, type, and constraints).
[0006] 3. Exceptions are easily masked: In complex ETL processes, low-level write exceptions may be captured, transformed, or encapsulated by high-level logic, resulting in the loss of the original dirty data context and making it difficult to trace the root cause.
[0007] 4. Limited capture range: Not all non-standard data situations will trigger explicit resolvable exceptions. Some problems manifest as silent data corruption, logical inconsistencies, etc., which cannot be directly identified by write failures.
[0008] The aforementioned limitations result in low efficiency in locating, analyzing, and repairing dirty data, severely impacting data quality and processing timeliness. There is an urgent need for a proactive and refined real-time dirty data identification and processing solution. Summary of the Invention
[0009] To overcome the aforementioned shortcomings of traditional solutions, this application proposes a novel dirty data processing method and system based on Flink CDC for real-time data acquisition. This invention introduces a dirty data processing mechanism in the Flink CDC real-time data acquisition process that prioritizes pre-verification and supplements it with anomaly capture. Strict verification is performed before data is written to the target end, and supplementary capture is performed when data writing fails. By isolating dirty data and its detailed information to Kafka, a complete dirty data processing closed loop is ultimately formed.
[0010] To achieve the above objectives, the present invention employs the following technical strategies:
[0011] (1) Pre-verification mechanism: Implement strict multi-dimensional verification before writing data to the target end, and advance the timing of dirty data identification from the traditional write failure to the data flow stage, thereby fundamentally reducing the cost of anomaly handling.
[0012] (2) Dynamic rule engine: Construct a real-time quality rule base based on target metadata, covering validation dimensions such as field length, data type, non-null constraint, field missing and name mapping, and accurately intercept non-standard data.
[0013] (3) Metadata hot loading and caching: Through metadata caching and timed refresh mechanism, the target table structure changes are dynamically synchronized, ensuring the real-time performance of the verification while avoiding performance bottlenecks caused by frequent access to the metadata service.
[0014] (4) Structured tagging and dead letter queue isolation: The original content of dirty data, the type of verification failure, error details and the target end specification requirements are encapsulated into a structured message and sent to the Kafka dead letter topic to achieve physical isolation between dirty data and normal data and ensure that the problem is traceable.
[0015] (5) Two-layer fault tolerance system: the pre-verification intercepts most format-related exceptions, and the exception capture mechanism serves as a fallback strategy to handle underlying write exceptions (such as network timeouts and transaction conflicts) that cannot be covered by the pre-verification, thus achieving full coverage of dirty data identification.
[0016] (6) End-to-end closed-loop management: Relying on Kafka to centrally collect dirty data, which is consumed and analyzed by independent Consumers, supporting automated repair or manual intervention, forming a complete data governance closed loop of "identification-isolation-analysis-repair".
[0017] Specifically, this application provides the following technical solutions:
[0018] The first aspect of this application provides a method for processing dirty data from real-time data acquisition based on Flink CDC, the method comprising:
[0019] S1. Use the Flink CDC connector to listen for and capture data change events in the source database in real time, and convert them into Flink DataStream;
[0020] S2. Obtain and cache metadata information of the target data table;
[0021] S3. Before writing data to the target database, pre-validate the data change event based on the metadata information, identify dirty data that does not conform to the target table structure specifications, and encapsulate the identified dirty data and its validation failure information into a structured message and send it to a preset Kafka isolation topic.
[0022] S4. Perform a target end write operation on the data that has passed the pre-verification, and capture exceptions during the write process. Encapsulate the failed write data and its exception information into a structured message and send it to the Kafka isolation topic.
[0023] S5. Consume dirty data messages in the Kafka isolated topic through an independent Kafka Consumer, analyze and repair the dirty data, forming a closed loop of dirty data processing of "identification-isolation-analysis-repair".
[0024] Furthermore, in the method of this application, the step S2 of obtaining and caching the metadata information of the target data table includes:
[0025] When the Flink application starts or the cache expires, the target database is queried for metadata information, which includes field name, field type, field length and non-null constraint.
[0026] The metadata information is cached in the application memory, and a cache expiration period is set to refresh the cache periodically.
[0027] Furthermore, in the method of this application, the setting of the cache validity period is used to balance the database query pressure and the timeliness of table structure change perception; the triggering conditions for periodically refreshing the cache include the cache reaching its validity period or the discovery of inconsistencies between metadata and the actual table structure during verification.
[0028] Furthermore, in the method of this application, the pre-verification in step S3 includes:
[0029] (1) Field length validation: Check whether the length of the field value to be written on the source side exceeds the maximum allowed length of the field on the target side;
[0030] (2) Field type validation: Check whether the data type of the field to be written on the source end is compatible with the data type of the field on the target end;
[0031] (3) Non-null constraint verification: Check whether the non-null constraint field on the target side is NULL in the source data;
[0032] (4) Field quantity verification: Check whether the number of fields in the source data change event is consistent with the number of fields defined in the target table structure.
[0033] Furthermore, in the method of this application, step S3 involves encapsulating the identified dirty data and its verification failure information into a structured message. The encapsulated content includes: the original source data change record, the verification failure type identifier, the target field specification requirements, the specific failure reason description, and the source table name and the target table name.
[0034] Furthermore, in the method of this application, the capture of anomalies in step S4 specifically includes:
[0035] Write exceptions are intercepted in the write logic of the Sink Function using an exception handling mechanism;
[0036] The exception information is parsed into structured data and then encapsulated together with the corresponding source data change record into a structured dirty data message.
[0037] Furthermore, in the method of this application, the abnormal information in step S4 includes abnormal types and abnormal messages, and the abnormal types include network timeout, transaction conflict, and database constraint violation.
[0038] Furthermore, in the method of this application, the analysis and repair processing of dirty data in step S5 specifically includes:
[0039] Locate the root cause of the problem based on dirty data messages;
[0040] Based on the failure type and reason in the dirty data message, repair the source data or adjust the target table structure;
[0041] The repaired data is then reinjected into the data collection process to achieve closed-loop management of dirty data.
[0042] A second aspect of this application provides a real-time data acquisition dirty data processing system based on Flink CDC, wherein the system implements the steps of the aforementioned real-time data acquisition dirty data processing method based on Flink CDC during operation, and the system includes:
[0043] The data capture module is used to listen for and capture data change events from the source database in real time through the Flink CDC connector, and convert them into Flink DataStream;
[0044] The metadata management module is used to acquire and cache metadata information of the target data table;
[0045] The pre-verification module is used to pre-verify the data change event based on the metadata information before the data is written to the target database, identify dirty data that does not conform to the target table structure specifications, and encapsulate the identified dirty data and its verification failure information into a structured message and send it to a preset Kafka isolation topic.
[0046] The write exception capture module is used to perform target end write operations on data that has passed pre-verification, capture exceptions during the write process, encapsulate the failed write data and its exception information into a structured message and send it to the Kafka isolation topic;
[0047] The dirty data processing module is used to consume dirty data messages in the Kafka isolated topic through an independent Kafka Consumer, analyze and repair the dirty data, and form a closed loop of dirty data processing of "identification-isolation-analysis-repair".
[0048] A third aspect of this application provides an electronic device, including: a memory and a processor;
[0049] Memory: Used to store computer programs;
[0050] Processor: Used to execute the computer program to implement the steps of the aforementioned Flink CDC-based real-time data acquisition dirty data processing method.
[0051] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned Flink CDC-based real-time data acquisition dirty data processing method.
[0052] In summary, compared with traditional solutions, the present invention has the following technical advantages:
[0053] (1) Achieve early identification and interception of dirty data: By pre-verifying before writing, the timing of dirty data discovery is significantly advanced, shortening the problem chain and preventing it from entering the target system.
[0054] (2) Improve data writing success rate and processing stability: Pre-verification excludes data with obvious mismatches in format, type, length, etc., reducing the probability of writing failure and enhancing overall processing stability.
[0055] (3) Reduce data processing costs: Early verification and interception avoid complex rollback, retries and downstream secondary processing, saving computing resources and manpower costs.
[0056] (4) Enhance the stability of the target system: prevent non-standard data from impacting the target database and avoid system crashes or unexpected behaviors caused by parsing errors or constraint violations.
[0057] (5) Provides accurate and timely dirty data information: pre-check generates detailed error types and reasons, which are more instructive than general write exceptions and facilitate quick location and problem solving.
[0058] (6) Construct a closed loop for dirty data processing: Collect dirty data and information centrally using Kafka, analyze and repair it through an independent Consumer, ensure no omissions and support continuous feedback and improvement, forming a virtuous cycle.
[0059] (7) Improve data quality observability: Centralized dirty data information provides a basis for quality monitoring and analysis, and helps to identify common problems in the data source or collection process.
[0060] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description
[0061] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.
[0062] Figure 1 This is a flowchart illustrating the overall implementation of the real-time data acquisition and dirty data processing method based on Flink CDC according to the present invention.
[0063] Figure 2 This is a diagram of the overall design architecture of the present invention.
[0064] Figure 3 This is a flowchart illustrating the dirty data collection process in an embodiment of this application.
[0065] Figure 4 This is a structural diagram of the real-time data acquisition and dirty data processing system based on Flink CDC according to the present invention.
[0066] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0067] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0068] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.
[0069] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."
[0070] Figure 1 The diagram shows the overall implementation flow of the real-time data acquisition dirty data processing method based on Flink CDC provided in this application, including the following steps:
[0071] S1. Use the Flink CDC connector to listen for and capture data change events in the source database in real time, and convert them into Flink DataStream;
[0072] S2. Obtain and cache metadata information of the target data table;
[0073] S3. Before writing data to the target database, pre-validate the data change event based on the metadata information, identify dirty data that does not conform to the target table structure specifications, and encapsulate the identified dirty data and its validation failure information into a structured message and send it to a preset Kafka isolation topic.
[0074] S4. Perform a target end write operation on the data that has passed the pre-verification, and capture exceptions during the write process. Encapsulate the failed write data and its exception information into a structured message and send it to the Kafka isolation topic.
[0075] S5. Consume dirty data messages in the Kafka isolated topic through an independent Kafka Consumer, analyze and repair the dirty data, forming a closed loop of dirty data processing of "identification-isolation-analysis-repair".
[0076] Example: A method for real-time data acquisition and dirty data processing based on Flink CDC
[0077] Figure 2 The diagram shows the overall design architecture of this invention. Figure 3 The diagram illustrates the dirty data collection process in this embodiment. As shown, this solution includes the following steps:
[0078] Step 1: Capture source data changes: Use the Flink CDC connector to monitor and capture data change events in the source database (such as MySQL, PostgreSQL, Oracle, etc.) in real time, including INSERT, UPDATE, and DELETE operations, and convert them into Flink DataStream.
[0079] Step 2, Metadata Management:
[0080] When the Flink CDC application starts for the first time, or when the cache expires, query the metadata information of the target data table (such as field name, field type, field length, whether it is nullable, etc.).
[0081] The acquired metadata information is cached in the application, with an appropriate cache expiration time (e.g., 24 hours). This is to balance database query pressure and promptly detect changes in the target table structure.
[0082] Before each verification, the target metadata is retrieved from the cache first; if the cache has expired, the target database is queried again to retrieve the latest metadata and then cached.
[0083] Step 3, Pre-verification (primarily pre-verification):
[0084] Before writing the captured source data changes to the target database, the data is pre-validated using a custom ProcessFunction or other Flink operator.
[0085] The verification content includes:
[0086] (1) Field length verification: Check whether the length of the field to be written in the source data exceeds the maximum allowed length of the corresponding field in the target (based on metadata).
[0087] (2) Field type validation: Check whether the data type of the field to be written in the source data is significantly different from the data type of the corresponding field in the target data. For example, try writing a string into a date type field, or a number into a boolean type field.
[0088] (3) Field Count Missing Validation: Check whether the number of fields in the source data changes is consistent with the number of fields defined in the target table structure. If the source has more fields than the target, then identify the missing fields in the target.
[0089] (4) Non-null constraint verification: Check whether the value of the non-null constraint field in the target table is NULL during the source data change.
[0090] The purpose of pre-validation:
[0091] 1) Improve write success rate: Detect and intercept data that is obviously non-compliant in advance to prevent it from reaching the target end, thereby reducing write failures and retries caused by format issues.
[0092] 2) Reduce processing costs: Validating data before it enters complex data pipelines or database write processes is far less costly than cleaning or repairing it after it has been written.
[0093] 3) Protect the stability of the target: Avoid injecting incorrectly formatted or abnormal data into the target to prevent the target system from crashing or behaving unexpectedly due to parsing errors or data abnormalities.
[0094] 4) Expose data quality issues as early as possible: Pre-validation failures can be immediately reported to the data source or upstream system, which helps to quickly locate and fix problems at the source of data production.
[0095] 5) Provide detailed error information: Pre-validation can accurately identify the specific validation failure type (such as "field 'email' length exceeds the limit, maximum allowed 100, actual 255"), and include the reason for the failure, providing precise clues for troubleshooting.
[0096] Step 4: Isolate dirty data (send to Kafka):
[0097] If dirty data is identified during the pre-verification process, do not attempt to write it directly to the target.
[0098] Encapsulate the dirty data (i.e., the original source data change record) along with detailed dirty data information. The encapsulation information should include at least:
[0099] 1) Original source data change records;
[0100] 2) Dirty data types (such as "field length mismatch", "field type mismatch", "target field missing", "target field NOT null constraint violated", etc.).
[0101] 3) Relevant target field information (such as length requirements, type requirements, and whether nullable);
[0102] 4) Specific reasons for failure (e.g., "Field 'age' is expected to be of type INT, but was actually received as STRING");
[0103] 5) Source table name and target table name.
[0104] The encapsulated dirty data information is sent to a predefined Kafka topic (e.g., flink-dirty-data).
[0105] Step 5: Target end write attempt (with exception handling as a secondary measure):
[0106] For data that passes the pre-validation, it is handed over to Flink's Sink Function (such as JDBC Sink) to attempt to write it to the target database.
[0107] Fault-tolerant writes: In the invoke() method of the Sink Function, use a try-catch block to catch any write exceptions that may occur.
[0108] Capture exception information: When data writes fail due to reasons that are not covered by pre-verification (such as network interruption, database deadlock, trigger exception, etc.), capture the resulting exception information.
[0109] Secondary dirty data processing: The data to be written that has an exception, the type of the exception captured (identified by parsing the exception information), and the detailed exception message are uniformly encapsulated and sent to the same preset Kafka topic (flink-dirty-data).
[0110] Step 6: Subsequent dirty data processing (forming a closed loop):
[0111] Create a Kafka Consumer: Deploy one or more Kafka Consumer programs independently and subscribe to the flink-dirty-data topic.
[0112] Data Analysis: The Consumer program reads dirty data information from Kafka, analyzes the dirty data based on the encapsulated details (dirty data type, failure reason, target requirements, etc.), and understands the root cause of the problem.
[0113] Manual repair:
[0114] (1) Based on the analysis results, make targeted adjustments and repairs to the source data.
[0115] (2) If the dirty data is generated due to the non-standard structure of the target table, then the target table should be adjusted.
[0116] (3) The repaired data can be re-collected or imported through other means.
[0117] Closed-loop formation: Through pre-verification, anomaly capture, Kafka isolation, and independent Consumer analysis and repair, a complete closed loop for dirty data processing is formed, ensuring timely data processing and continuous quality improvement.
[0118] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.
[0119] Suppose we need to synchronize user table data from a MySQL database to a PostgreSQL database user table in real time.
[0120] 1. Flink CDC configuration:
[0121] Configure the Flink CDC MySQL Connector to capture changes to the user_table.
[0122] The target is PostgreSQL, using the JDBC Sink Connector.
[0123] 2. Metadata Management Module:
[0124] When the Flink application starts, it queries the metadata of the user_table through the PostgreSQL JDBC driver, such as: id(INT, NOT NULL), name (VARCHAR(50), NOT NULL), age (SMALLINT), create_time(TIMESTAMP).
[0125] Cache this metadata and set a 24-hour expiration period.
[0126] 3. Pre-verification module (implemented as a ProcessFunction):
[0127] In the processElement() method of ProcessFunction, receive MySQL data change events (such as RowData) from Flink CDC.
[0128] Scenario 1: Field length exceeds limit
[0129] A record is received from the MySQL source: {"id": 1, "name": "张三李四王五赵六钱七孙八周九吴十", "age": 25, "create_time": "2023-10-27 10:00:00"}
[0130] During pre-verification, it is found that the length of the name field (20 Chinese characters, which may be longer after actual encoding) exceeds the maximum length (50 characters) of the name field in the PostgreSQL target table.
[0131] This record is encapsulated into a dirty data message, including: the original data, the dirty data type "field length mismatch", the length requirement "50" of the target field name, and the failure reason "the length of the field 'name' exceeds the limit, the actual length is greater than 50". Then it is sent to the Kafka topic flink-dirty-data.
[0132] Scenario 2: Field type mismatch
[0133] A record is received from the MySQL source: {"id": 2, "name": "李四", "age": "二十", "create_time": "2023-10-27 10:05:00"}
[0134] During pre-verification, it is found that the value "二十" of the age field cannot be converted to the SMALLINT type of the age field in the PostgreSQL target table.
[0135] This record is encapsulated into a dirty data message, including: the original data, the dirty data type "field type mismatch", the type requirement "SMALLINT" of the target field age, and the failure reason "the field 'age' expects the type SMALLINT, but actually receives the STRING value '二十'". Then it is sent to the Kafka topic flink-dirty-data.
[0136] Scenario 3: Violation of the not-null constraint
[0137] A record is received from the MySQL source: {"id": 3," name": null, "age": 30, "create_time": "2023-10-27 10:10:00"}
[0138] During pre-verification, it is found that the name field in the PostgreSQL target table requires not null (NOT NULL), while the value of the name field in the source data is null.
[0139] Package this record into a dirty data message, including: the original data, the dirty data type "violating the not-null constraint of the target field", the constraint "NOT NULL" of the target field name, and the failure reason "the field 'name' is NOT NULL, but a NULL value was received". Then send it to the Kafka topic flink-dirty-data.
[0140] Records passing the pre-verification:
[0141] {"id": 4, "name": "Zhao Liu", "age": 28, "create_time": "2023-10-27 10:15:00"}
[0142] Pass this record to the downstream JDBC Sink.
[0143] 4. Target-side writing module (JDBC Sink):
[0144] The JDBC Sink receives the record {"id": 4, "name": "Zhao Liu", "age": 28, "create_time": "2023-10-27 10:15:00"} that passed the pre-verification.
[0145] The invoke() method of the Sink Function attempts to execute the SQL: INSERT INTO user_table (id,name, age, create_time) VALUES (4, 'Zhao Liu', 28, '2023-10-27 10:15:00').
[0146] Scenario 4: Exception caused by a trigger
[0147] If there is a trigger on the PostgreSQL user table to check if age is less than 0 when inserting data, and throw an exception if it is less than 0.
[0148] Suppose there is a record at this time (this situation is not covered in the pre-verification): {"id": 5, "name": "Qian Qi", "age": -5, "create_time": "2023-10-27 10:20:00"}.
[0149] The invoke() method of the JDBC Sink captures the exceptions thrown by PostgreSQL, such as: "ERROR: newrow for relation "user_table" violates check constraint "age_check" (age >=0)".
[0150] Exception data processing module:
[0151] Package the exception record {"id": 5, "name": "Qian Qi", "age": -5, "create_time": "2023-10-27 10:20:00"} and the exception message "ERROR: new row for relation "user_table" violatescheck constraint "age_check" (age >= 0)". Then send it to the Kafka topic flink-dirty-data.
[0152] Figure 4 The following shows a real-time data acquisition dirty data processing system based on Flink CDC proposed by this application. The system includes:
[0153] Data capture module, used to listen for and capture data change events of the source database in real time through the Flink CDC connector, and convert them into Flink DataStream;
[0154] Metadata management module, used to obtain and cache metadata information of the target data table;
[0155] Pre-verification module, used to pre-verify the data change events based on the metadata information before writing the data to the target database, identify dirty data that does not conform to the target table structure specification, and encapsulate the identified dirty data and its verification failure information into a structured message, and send it to a preset Kafka isolation topic;
[0156] Write exception capture module, used to perform the target-side write operation on the data that passes the pre-verification, capture exceptions during the write process, encapsulate the data with write failures and their exception information into a structured message, and then send it to the Kafka isolation topic;
[0157] Dirty data processing module, used to consume the dirty data messages in the Kafka isolation topic through an independent Kafka Consumer, analyze and repair the dirty data, and form a closed loop of "identification - isolation - analysis - repair" for dirty data processing.
[0158] The above system implements the steps of the real-time data acquisition and dirty data processing method based on Flink CDC disclosed in this application when it is running.
[0159] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.
[0160] like Figure 5 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned Flink CDC-based real-time data acquisition dirty data processing method.
[0161] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).
[0162] The operations described in this application can be implemented by writing computer program code using one or more programming languages or a combination thereof. The programming languages include, but are not limited to, the following types:
[0163] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.
[0164] Conventional procedural programming languages, such as "C" or similar programming languages.
[0165] The execution methods of program code include, but are not limited to:
[0166] It runs entirely on the user's computer;
[0167] Part of it executes on the user's computer, and part of it executes on a remote computer;
[0168] Execute as a standalone software package;
[0169] It is executed entirely on a remote computer or server.
[0170] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.
[0171] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the real-time data acquisition dirty data processing method based on FlinkCDC disclosed in this application.
[0172] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:
[0173] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.
[0174] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.
[0175] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.
[0176] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.
[0177] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.
[0178] (6) Fiber optic storage devices: storage media based on fiber optic technology.
[0179] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.
[0180] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.
[0181] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.
[0182] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.
[0183] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.
[0184] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the Flink CDC-based real-time data acquisition dirty data processing method disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.
[0185] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.
[0186] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for real-time data acquisition and dirty data processing based on Flink CDC, characterized in that, The method includes: S1. Use the Flink CDC connector to listen for and capture data change events in the source database in real time, and convert them into Flink DataStream; S2. Obtain and cache metadata information of the target data table; S3. Before writing data to the target database, pre-validate the data change event based on the metadata information, identify dirty data that does not conform to the target table structure specifications, and encapsulate the identified dirty data and its validation failure information into a structured message and send it to a preset Kafka isolation topic. S4. Perform a target end write operation on the data that has passed the pre-verification, and capture exceptions during the write process. Encapsulate the failed write data and its exception information into a structured message and send it to the Kafka isolation topic. S5. Consume dirty data messages in the Kafka isolated topic through an independent Kafka Consumer, analyze and repair the dirty data, forming a closed loop of dirty data processing of "identification-isolation-analysis-repair".
2. The method according to claim 1, characterized in that, The step S2 of obtaining and caching the metadata information of the target data table includes: When the Flink application starts or the cache expires, the target database is queried for metadata information, which includes field name, field type, field length and non-null constraint. The metadata information is cached in the application memory, and a cache expiration period is set to refresh the cache periodically.
3. The method according to claim 2, characterized in that, The cache expiration period setting is used to balance database query pressure and the timeliness of table structure change awareness; the triggering conditions for periodically refreshing the cache include the cache reaching its expiration period or the discovery of inconsistencies between metadata and the actual table structure during verification.
4. The method according to claim 1, characterized in that, The pre-verification in step S3 includes: (1) Field length validation: Check whether the length of the field value to be written on the source side exceeds the maximum allowed length of the field on the target side; (2) Field type validation: Check whether the data type of the field to be written on the source end is compatible with the data type of the field on the target end; (3) Non-null constraint verification: Check whether the non-null constraint field on the target side is NULL in the source data; (4) Field quantity verification: Check whether the number of fields in the source data change event is consistent with the number of fields defined in the target table structure.
5. The method according to claim 1, characterized in that, In step S3, the identified dirty data and its verification failure information are encapsulated into a structured message. The encapsulated content includes: the original source data change record, the verification failure type identifier, the target field specification requirements, the specific failure reason description, and the source table name and the target table name.
6. The method according to claim 1, characterized in that, The exception capture described in step S4 specifically includes: Write exceptions are intercepted in the write logic of the Sink Function using an exception handling mechanism; The exception information is parsed into structured data and then encapsulated together with the corresponding source data change record into a structured dirty data message.
7. The method according to claim 1, characterized in that, The abnormal information mentioned in step S4 includes abnormal types and abnormal messages. The abnormal types include network timeout, transaction conflict, and database constraint violation.
8. The method according to claim 1, characterized in that, The analysis and repair of dirty data described in step S5 specifically includes: Locate the root cause of the problem based on dirty data messages; Based on the failure type and reason in the dirty data message, repair the source data or adjust the target table structure; The repaired data is then reinjected into the data collection process to achieve closed-loop management of dirty data.
9. A real-time data acquisition and dirty data processing system based on Flink CDC, characterized in that, The system implements the steps of the real-time data acquisition dirty data processing method based on Flink CDC as described in any one of claims 1-8 during operation, and the system includes: The data capture module is used to listen for and capture data change events from the source database in real time through the Flink CDC connector, and convert them into Flink DataStream; The metadata management module is used to acquire and cache metadata information of the target data table; The pre-verification module is used to pre-verify the data change event based on the metadata information before the data is written to the target database, identify dirty data that does not conform to the target table structure specifications, and encapsulate the identified dirty data and its verification failure information into a structured message and send it to a preset Kafka isolation topic. The write exception capture module is used to perform target end write operations on data that has passed pre-verification, capture exceptions during the write process, encapsulate the failed write data and its exception information into a structured message and send it to the Kafka isolation topic; The dirty data processing module is used to consume dirty data messages in the Kafka isolated topic through an independent Kafka Consumer, analyze and repair the dirty data, and form a closed loop of dirty data processing of "identification-isolation-analysis-repair".
10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the real-time data acquisition dirty data processing method based on FlinkCDC as described in any one of claims 1-8.