Configurable big data import method and system based on Flink cdc

Abstract

The present invention provides a configurable big data import method and system based on Flink cdc, which relates to the field of data synchronization technology, including: step S1: for the newly added import task of the system, setting configuration information, and performing task import process configuration; step S2: the user Flink program generates executable user program code according to the configuration information in step S1, and submits it to the Flink cluster to execute the user program; step S3: the Flink cluster calls the user program code to execute according to the specified execution class, and completes the data import operation. The present invention can improve the flexibility and scalability of the system and improve the data processing capability.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a configurable big data import method and system based on Flink cdc. Background Art

[0002] Flink cdc is based on Apache Flink, and has high-performance, high-throughput, and low-latency data processing capabilities. It can capture changes in data sources in real time to ensure that the imported data is the latest and most accurate. However, when the data source changes, the data processing logic changes, the target database changes, etc., developers need to consider redeveloping based on the Flink cdc framework for each change, be compatible with different change logic processing, and resubmit the flink task. This generates a lot of maintenance and testing workload, and customized compatible logic will also reduce the flexibility and scalability of the system.

[0003] In summary, the existing related technologies have the following defects: the traditional Flink cdc-based import system may lack flexibility and be difficult to adapt to the needs of specific business scenarios. When dealing with different data sources and targets, as well as complex data conversion and processing logic, additional development work may be required to customize the system, resulting in poor scalability of the traditional system and the inability to flexibly and quickly make changes and compatibility. The simple Flink cdc import system does not support complex configuration management, especially with the explosive growth of data volume and the further improvement of data real-time requirements. In complex scenarios, the efficiency of data conversion and cleaning needs to be improved accordingly to reduce duplication of work and difficulty in getting started, and reduce maintenance costs and management time costs. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a configurable big data import method and system based on Flink cdc.

[0005] According to a configurable big data import method and system based on Flink cdc provided by the present invention, the scheme is as follows:

[0006] In a first aspect, a configurable big data import method based on Flink cdc is provided, the method comprising:

[0007] Step S1: For the newly added import task in the system, set the configuration information and configure the process of task import;

[0008] Step S2: The user Flink program generates executable user program code according to the configuration information in step S1, and submits it to the Flink cluster to execute the user program;

[0009] Step S3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation;

[0010] Among them, step S3 first reads and parses the task import process configuration information, verifies whether the task process configuration is correct, then collects data in real time, parses, assembles and returns the data, and then filters and logically processes the data. Finally, the processed data is sunk to the specified storage medium, and the operation process is handled abnormally.

[0011] Preferably, the step S1 comprises:

[0012] Step S1.1: Configure the main task value acquisition logic, including: main table data source connection configuration, main table value acquisition logic configuration, main table filter condition configuration, main table data processing logic configuration, custom window configuration and target database configuration;

[0013] Step S1.2: Configure the sub-table value logic, including: sub-table data source configuration, mapping field configuration with the main table, data assembly configuration, and sub-table value SQL configuration. In addition, the association ID with the main task configuration table is also configured. The value logic configuration information of the corresponding sub-table can be queried through the main task ID;

[0014] Step S1.3: If the value logic of the sub-table is configured in the above step S1.2 process, and if the sub-table is changed, if the result data also needs to be changed, it is necessary to configure the incremental dynamic monitoring of the sub-table in the sub-table dynamic monitoring table, including: sub-table data source configuration, sub-table name, monitoring field and associated field configuration. In addition, the association ID with the main task configuration table is also configured, and the monitoring configuration information of the corresponding sub-table can be queried through the main task ID;

[0015] Step S1.4: Configure the task group, including the group ID and task ID configuration; wherein, the group ID configuration: configure the group ID in the task group configuration table, which is used as the input parameter for job startup, and search the corresponding task configuration list according to the group ID;

[0016] Task ID configuration: The task ID under the group in the task group configuration table specifies that all tasks under an import requirement are configured in one group.

[0017] Preferably, in step S1.1, configuring the main task value logic includes:

[0018] Main table data source connection configuration: Configure the data source connection information corresponding to the main table in the main task configuration table to implement the connection of the Source data source in the Flink program;

[0019] Main table value retrieval logic configuration: configure the fields to be queried, the listening fields, and the value retrieval path information in the main task configuration table;

[0020] Main table filter condition configuration: configure the filter conditions for the main table data in the main task configuration table;

[0021] Main table data processing logic configuration: configure the fields that need special processing in the main task configuration table, including: data format conversion, dictionary mapping value, Lua script, field encryption and decryption of the specified algorithm, and field name mapping with the target table;

[0022] Custom window configuration: The import of tasks is realized by reading the task configuration table through the Flink program, and generating executable code according to the configuration information to complete the import operation;

[0023] Target database configuration: Configure the target database information that the data needs to be imported into in the main task configuration table, mainly including: database connection information, import table name and import fields.

[0024] Preferably, configuring the sub-table value logic in step S1.2 includes:

[0025] Subtable data source configuration: Configure the subtable data source connection information in the subtask configuration table to connect to the subtable data source for subsequent SQL operations;

[0026] Mapping field configuration with the main table: Configure the mapping field with the main table in the subtask configuration table to read the corresponding data of the main table as the input parameter of the SQL for subtable value retrieval;

[0027] Data assembly configuration: Configure the data assembly method with the main table in the subtask configuration table, that is, directly nest it in the main data or flatten it;

[0028] Sub-table value SQL configuration: Configure the sub-table value SQL in the sub-task configuration table to obtain the sub-table data corresponding to the main table and supplement the sub-table import logic.

[0029] Preferably, the sub-table dynamic monitoring configuration in step S1.3 includes:

[0030] Sub-table data source configuration: configure the sub-table data source connection information in the sub-table dynamic monitoring table to connect to the sub-table data source and provide connection parameter configuration for the sub-table's cdc connector;

[0031] Sub-table name: configure the table name of the sub-table that needs to be dynamically monitored for changes in the sub-table dynamic monitoring table;

[0032] Monitoring fields: configure the fields that need to dynamically monitor data changes in the sub-table dynamic monitoring table;

[0033] Related field configuration: Configure the related fields with the main table in the sub-table dynamic monitoring table for related queries with the main table.

[0034] Preferably, the step S2 comprises:

[0035] Step S2.1: Upload the user program jar package and specify the execution main class; including: the user Flink program generates executable program code according to the configuration information in step S1, and then builds the user program into a jar file format; upload the jar package in the Flink client-task submission page by uploading the jar package; after the upload is successful, find the uploaded file in the corresponding jar file list, and specify the user program execution main class Entry Class in the corresponding job submission form;

[0036] Step S2.2: Specify the group ID to be started; in the job submission form of step S2.1, specify the task group ID to be started, which is used to find the corresponding task list, and then perform the task operation, which is executed through the parameter GroupId in the start command;

[0037] Step S2.3: Specify the data extraction mode: full or incremental; In the job submission form of step S2.1, specify the task extraction mode: full or incremental. The first startup or refresh data is full, and other cases are incremental. It is specified in the startup command through the parameter startupOptions;

[0038] Step S2.4: Set the job parallelism. In the job submission form of step S2.1, specify the task parallelism.

[0039] Step S2.5: Start the execution of the user program through a command; in the job submission form of step S2.1, specify the start command, start it through the flink run command, and execute the user program.

[0040] Preferably, the step S3 comprises:

[0041] Step S3.1: Check the configuration information. If it is correct, execute step S3.2. If it is incorrect, throw an exception directly and print the detailed error information in the log file. After the exception retry fails for the specified number of times, the Flink job stops directly.

[0042] Step S3.2: Get the Source data source configuration, build the Source, connect to the specified data source, and collect data in real time;

[0043] Step S3.3: Get the configuration information of Source, build FlatMap, parse, assemble and return the data;

[0044] Step S3.4: If the task is configured with a filter condition expression, build a Filter operator to filter the Source data stream;

[0045] Step S3.5: Get the configured window parameters, build keyBy, window, and trigger operators by implementing the abstract classes and interfaces within Flink, and generate a custom window.

[0046] Step S3.6: Get the data logic processing configuration, including the value logic configuration of the sub-table and the data processing logic configuration of the main table. Build the processFunction function by implementing the ProcessWindowFunction abstract class inside Flink, and perform logical processing on the data in the window function.

[0047] Step S3.7: Get the Sink related configuration information, build the Sink, connect the Sink data source, and sink the processed data to the specified storage medium;

[0048] Step S3.8: Handle job process exceptions. During the Flink job construction process, each process is exception captured. If the current process construction is abnormal, the current exception is first captured to obtain the location of the abnormal process and the detailed information of the exception before throwing the exception.

[0049] Step S3.9: Retry if the job runs abnormally; if an exception occurs in the user program in step S3.8, it will continue to be thrown, causing the entire Flink Job to run abnormally. When the Flink Job is abnormal, the abnormal retry mechanism is triggered;

[0050] During the Flink program submission process, the Job operation retry strategy is specified through the setRestartStrategy method, including: fixed interval restart strategy, failure failure rate restart strategy, and no restart. The default is the fixed interval restart strategy.

[0051] Preferably, the step S3.3 comprises:

[0052] Step S3.3.1: Get SourceConfig configuration information, and parse and obtain data stream according to the configuration path;

[0053] Step S3.3.2: According to the configured monitoring field, obtain the information of the change value of the monitoring field before and after;

[0054] Step S3.3.3: Filter and return the data source fields according to the configured return fields;

[0055] Step S3.3.4: For the physical deletion scenario, set the logical deletion identifier Flag=1 for downstream business logic processing.

[0056] Preferably, the step S3.6 comprises:

[0057] Step S3.6.1: Obtain the logical value processing of the sub-table data; obtain the sub-table value logic SQL from the sub-table value logic configuration table, fill in the parameters according to the associated field value, connect to the sub-table data source, execute the SQL statement, obtain the result, assemble it into the source table data and pass it down for data processing;

[0058] Step S3.6.2: Obtain data formatting configuration and complete data format conversion for different configurations;

[0059] Step S3.6.3: Get Lua script configuration, parse the script through LuaValue and execute it;

[0060] Step S3.6.4: Get the field mapping configuration, get the corresponding attribute value according to the source table field, assign it to the mapped field, and then fill the mapped field into the data stream;

[0061] Step S3.6.5: Get the configuration of the newly added field, get the corresponding value from the data stream and assign it to the newly added field according to the value logic of the newly added field, and then fill the newly added field into the data stream.

[0062] In a second aspect, a configurable big data import system based on Flink cdc is provided, and the system includes:

[0063] Module M1: For newly added import tasks in the system, set configuration information and configure the process of task import;

[0064] Module M2: The user Flink program generates executable user program code based on the configuration information in module M1 and submits it to the Flink cluster to execute the user program;

[0065] Module M3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation;

[0066] Among them, the module M3 first reads and parses the task import process configuration information, verifies whether the task process configuration is correct, then collects data in real time, parses, assembles and returns data, and then filters and logically processes the data. Finally, the processed data is sunk to the specified storage medium and the operation process is handled abnormally.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention provides more flexible configuration options, allowing users to adjust the data import process according to specific needs without repeated development. For example, you can customize data conversion logic, loading strategy, error handling, etc. It is more compatible with changes in data processing logic for various heterogeneous data sources, can adapt to different data sources and different scales of data import requirements, and improve the flexibility and scalability of the system.

[0069] 2. The present invention can make better use of Flink's dynamic scaling characteristics, automatically adjust resource allocation according to actual load, and through precise timestamps and transaction IDs, the configurable system can ensure data consistency and reliability.

[0070] 3. The present invention can automatically switch to the configured backup path or take other recovery measures when a failure occurs, thereby reducing downtime and providing a more complete abnormal data processing flow and analysis solution.

[0071] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0073] Figure 1 It is a schematic diagram of the operation flow of the present invention;

[0074] Figure 2 Execute the process for platform user programs;

[0075] Figure 3 Acquire the process of multiple data sources for user programs;

[0076] Figure 4 Incremental monitoring process for user program subtable;

[0077] Figure 5 It is the logic flow of obtaining values ​​from the user program subtable. DETAILED DESCRIPTION

[0078] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0079] The embodiment of the present invention provides a configurable big data import method based on Flink cdc, referring to Figure 1 As shown, the method specifically includes:

[0080] Step S1: For the newly added import task in the system, set the configuration information and configure the process of task import.

[0081] The import logic of the task mainly covers the process of extracting data from different tables, assembling data, and writing data into the target database. In this process, the processing of the main table and sub-tables is involved.

[0082] For the newly added import task, its configuration process mainly includes the following key links: configuration of the value logic of the main table (main task), configuration of the association mapping between the main and sub-tables, configuration of the dynamic monitoring of the sub-table, and configuration of the task group. All these configuration information are saved in the main task configuration table, sub-task configuration table, sub-table dynamic monitoring configuration, and task group table of MySQL for subsequent use and management.

[0083] The detailed steps are as follows:

[0084] Step S1.1: Configure the main task value logic;

[0085] Main table data source connection configuration: Configure the data source connection information corresponding to the main table in the main task configuration table, for example: mysql: / / userName:password@ip:port / database / table name. The main purpose is to realize the connection of Source data source in Flink program.

[0086] Specifically include: main table data source connection configuration, main table value logic configuration, main table filtering condition configuration, main table data processing logic configuration, custom window configuration and target database configuration.

[0087] Among them, main table data source connection configuration: configure the data source connection information corresponding to the main table in the main task configuration table to realize the connection of the Source data source in the Flink program;

[0088] Main table value retrieval logic configuration: configure the fields to be queried, the listening fields, and the value retrieval path information in the main task configuration table (applicable to Kafka);

[0089] Main table filter condition configuration: Configure the filter conditions for the main table data in the main task configuration table; for example, to filter the data named john whose age is less than 12 or greater than 22, you can configure it as: [name] == 'john'&&([age]<12 ||[age]>22);

[0090] Main table data processing logic configuration: configure the fields that need special processing in the main task configuration table. Mainly includes: data format conversion (such as timestamp conversion to a specified format date field), dictionary mapping value, Lua script, field encryption and decryption of the specified algorithm, and field name mapping with the target table.

[0091] Custom window configuration: The import of tasks is realized by reading the task configuration table through the Flink program, and generating executable code based on the configuration information to complete the import operation. The Flink execution process needs to select appropriate windows and functions for different import tasks, so the main task configuration table provides custom window configurations for different main tasks. The main configurations include: window type window, size, grouping field keyBy, window function ProcessWindowFunction, and trigger configuration.

[0092] Target database configuration: Configure the target database information for data import in the main task configuration table. Mainly includes: database connection information, import table name, import field; in addition to database related information, also configure the number of import failure retries and the file path for writing failed data.

[0093] Step S1.2: configure the sub-table value logic;

[0094] If the import task's value-taking logic involves the value-taking logic of the sub-table in addition to the value-taking logic of the main table, you need to configure the sub-table extraction logic in the sub-task configuration table. The sub-table configuration mainly includes: sub-table data source configuration, mapping fields with the main table, data assembly method, and sub-table value-taking SQL.

[0095] In addition to configuring the value logic of the sub-table, the sub-task configuration table also configures the association ID with the main task configuration table. The main purpose is to query the value logic configuration information of the corresponding sub-table through the main task ID.

[0096] Specifically, sub-table data source configuration: configure the sub-table data source connection information in the sub-task configuration table to connect to the sub-table data source for subsequent SQL operations;

[0097] Mapping field configuration with the main table: Configure the mapping field with the main table in the subtask configuration table to read the corresponding data of the main table as the input parameter of the SQL for subtable value retrieval;

[0098] Data assembly configuration: Configure the data assembly method with the main table in the subtask configuration table, that is, directly nest it in the main data or flatten it;

[0099] Subtable value SQL configuration: Configure the subtable value SQL in the subtask configuration table, for example: SELECT field 1, field 2 FROM subtable WHERE associated field in (the associated field corresponds to the attribute value of the main table), which is used to obtain the subtable data corresponding to the main table and supplement the import logic of the subtable.

[0100] Step S1.3: If the sub-table value logic is configured in the above step S1.2 process, and if the sub-table changes, if the result data also needs to be changed, it is necessary to configure the incremental dynamic monitoring of the sub-table in the sub-table dynamic monitoring table, including: sub-table data source configuration, sub-table name, monitoring field and associated field configuration. In addition to configuring the sub-table monitoring configuration, the sub-table dynamic monitoring configuration table also configures the association ID with the main task configuration table. The main purpose is to query the corresponding sub-table monitoring configuration information through the main task ID.

[0101] Among them, sub-table data source configuration: configure the sub-table data source connection information in the sub-table dynamic monitoring table, which is used to connect to the sub-table data source and provide connection parameter configuration for the sub-table cdc connector;

[0102] Sub-table name: configure the table name of the sub-table that needs to be dynamically monitored for changes in the sub-table dynamic monitoring table;

[0103] Monitoring fields: configure the fields that need to dynamically monitor data changes in the sub-table dynamic monitoring table;

[0104] Related field configuration: Configure the related fields with the main table in the sub-table dynamic monitoring table for related queries with the main table.

[0105] Step S1.4: Configure the task group, including group ID and task ID configuration.

[0106] After the above process configuration is completed, the basic configuration of the import task process is completed. The above configuration process is that one master table corresponds to one Flink job, but if a task involves the import of multiple master tables, multiple Flink jobs need to be started, which is not easy to manage; therefore, the concept of group is proposed to manage tasks in groups. One group corresponds to one Flink job, and one or more master tasks can be configured under one group.

[0107] Among them, group ID configuration: configure the group ID (custom generated, unique) in the task group configuration table, which is used as the input parameter for job startup, and find the corresponding task configuration list according to the group ID;

[0108] Task ID configuration: The task ID of the group in the task group configuration table (corresponding to the main task configuration table ID). Generally, all tasks under an import requirement are configured in one group.

[0109] Step S2: The user Flink program generates executable user program code based on the configuration information in step S1 and submits it to the Flink cluster to execute the user program.

[0110] Step S2.1: Upload the user program jar package and specify the execution main class; including: the user Flink program generates executable program code according to the configuration information in step S1, and then builds the user program into a jar file format; upload the jar package in the Flink client-task submission page by uploading the jar package; after the upload is successful, find the uploaded file in the corresponding jar file list, and specify the user program execution main class Entry Class in the corresponding job submission form;

[0111] Step S2.2: Specify the group ID to be started; in the job submission form of step S2.1, specify the task group ID to be started, which is used to find the corresponding task list, and then perform the task operation, which is executed through the parameter GroupId in the start command;

[0112] Step S2.3: Specify the data extraction mode: full or incremental; In the job submission form of step S2.1, specify the task extraction mode: full or incremental. The first startup or refresh data is full, and other cases are incremental. It is specified in the startup command through the parameter startupOptions;

[0113] Step S2.4: Set the job parallelism. In the job submission form of step S2.1, specify the task parallelism.

[0114] Step S2.5: Start the execution of the user program through a command; in the job submission form of step S2.1, specify the start command, start it through the flink run command, and execute the user program.

[0115] Step S3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation.

[0116] The step S3 specifically includes:

[0117] Step S3.1: Check the configuration information. If it is correct, execute step S3.2. If it is incorrect, throw an exception directly and print the detailed error information in the log file. After the exception retry fails for the specified number of times, the Flink job stops directly.

[0118] The user program obtains the corresponding task configuration according to the startup parameter group ID, parses different configurations, and operates to check whether the process reports an error. For example, when parsing the data source configuration and connecting, if there is an error in parsing or connection, the table name configuration information is abnormal, and an exception is directly thrown. After the exception retry fails, the Flink job will stop directly. The user can view the specific error information according to the log and repair the configuration; if there are no problems with all configuration parsing and operations, and the Flink program can run normally, it means that the task process configuration is correct.

[0119] Step S3.2: Get the Source data source configuration, build the Source, connect to the specified data source, and collect data in real time.

[0120] Step S3.2.1: Source supports data sources: Kafka, MySQL, SQLServer, MongoDB;

[0121] Step S3.2.2: Support multiple data source configurations. This can be achieved by configuring multiple tables or data sources in the Source data source configuration.

[0122] Step S3.2.3: Support real-time update of sub-table increments. By configuring sub-table monitoring information, sub-table increment data can be obtained in real time.

[0123] Step S3.3: Get the configuration information of Source, build FlatMap, parse, assemble and return the data.

[0124] Step S3.3 includes:

[0125] Step S3.3.1: Get SourceConfig configuration information, and parse and obtain data stream according to the configuration path;

[0126] Step S3.3.2: According to the configured monitoring field, obtain the information of the change value of the monitoring field before and after;

[0127] Step S3.3.3: Filter and return the data source fields according to the configured return fields;

[0128] Step S3.3.4: For the physical deletion scenario, set the logical deletion identifier Flag=1 for downstream business logic processing.

[0129] Step S3.4: If the task is configured with a filter condition expression, build a Filter operator to filter the Source data stream;

[0130] The Flink program implements the internal FilterFunction abstract class to build the Filter operator to filter the data stream. In the FilterFunction implementation class, the configured filter condition expression [name] == 'john'&&([age]<12 || [age]>22) is obtained, the condition attribute field is extracted, and the value of the variable (obtained from the Source data stream) is provided through the MapContext of JexlExpression to execute the expression and complete the filtering of the data stream.

[0131] Step S3.5: Get the configured window parameters, build keyBy, window, and trigger operators by implementing the abstract classes and interfaces within Flink, and generate a custom window.

[0132] Step S3.5.1: Get the configured keyBy field for grouping processing.

[0133] The Flink program obtains the keyBy field configured in the custom window configuration column in the main task configuration table, and builds the keyBy operator by implementing the Flink internal KeySelector interface. Flink groups data internally according to the KeyBy field, and groups data with the same keyBy attribute value into one group for subsequent operations.

[0134] Step S3.5.2: Get the window configuration and generate a custom window.

[0135] The window types supported are: count window and time window. The window allocator can further allocate the following types: rolling window, sliding window and session window.

[0136] The time window supports the configuration of process time and event time. The event time window can be configured with watermarks, late data processing, etc. It supports the configuration of window triggers and supports triggering of quantity and time.

[0137] Step S3.6: Get the data logic processing configuration, including the value logic configuration of the sub-table and the data processing logic configuration of the main table. Build the processFunction function by implementing the ProcessWindowFunction abstract class inside Flink, and perform logical processing on the data within the window function.

[0138] This step S3.6 includes:

[0139] Step S3.6.1: Obtain the logical value processing of the sub-table data; obtain the sub-table value logic SQL from the sub-table value logic configuration table, fill in the parameters according to the associated field value, connect to the sub-table data source, execute the SQL statement, obtain the results, assemble them into the source table data and pass them down for data processing.

[0140] Step S3.6.2: Get the data formatting configuration and complete the data format conversion for different configurations; for example: date format conversion, use DateTimeFormatter to achieve data format conversion.

[0141] Step S3.6.3: Get the Lua script configuration, parse the script through LuaValue and execute it.

[0142] Step S3.6.4: Get the field mapping configuration, obtain the corresponding attribute value according to the source table field, assign it to the mapped field, and then fill the mapped field into the data flow.

[0143] Step S3.6.5: Get the configuration of the newly added field, get the corresponding value from the data stream and assign it to the newly added field according to the value logic of the newly added field, and then fill the newly added field into the data stream.

[0144] Step S3.7: Get the relevant configuration information of Sink, build Sink, connect Sink data source, and sink the processed data to the specified storage medium. Sink supports data sources: MySQL, SQLServer, MongoDB, Doris, ClickHouse, ElasticSearch, etc. Sink writes to the target table in two modes: add and update.

[0145] Step S3.8: Handle job process exceptions; During the Flink job construction process, exceptions will be captured for each process. If the current process construction is abnormal, the current exception will be captured first, and the location of the abnormal process and the detailed information of the exception will be obtained before the exception is thrown. The purpose is to facilitate users to find the specific process location where the exception occurred.

[0146] Step S3.9: Retry if the job runs abnormally; if an exception occurs in the user program in step S3.8, it will continue to be thrown, causing the entire Flink Job to run abnormally. When the Flink Job is abnormal, the abnormal retry mechanism is triggered;

[0147] During the Flink program submission process, the Job operation retry strategy is specified through the setRestartStrategy method, including: fixed interval restart strategy, failure failure rate restart strategy, and no restart. The default is the fixed interval restart strategy.

[0148] The present invention also provides a configurable big data import system based on Flink cdc, which can be implemented by executing the process steps of the configurable big data import method based on Flink cdc, that is, those skilled in the art can understand the configurable big data import method based on Flink cdc as a preferred implementation of the configurable big data import system based on Flink cdc. The system specifically includes:

[0149] Module M1: For newly added import tasks in the system, set configuration information and configure the process of task import;

[0150] Module M2: The user Flink program generates executable user program code based on the configuration information in module M1 and submits it to the Flink cluster to execute the user program;

[0151] Module M3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation.

[0152] Next, the present invention will be described in more detail.

[0153] A configurable big data import method based on Flink cdc, such as Figure 1 As shown, the following steps are included:

[0154] 1) Task import process configuration; proceed to step 2);

[0155] 2) Submit the user program to the Flink cluster for execution and start the task through commands; proceed to step 3);

[0156] 3) Read and parse the task import process configuration information, fill in the task process, and execute the user program.

[0157] The core of this method is to read and parse the configuration to fill the task flow, such as Figure 2 As shown, the following steps are included:

[0158] 3.1) Read and parse the task import process configuration information; verify whether the task process configuration is correct; if the configuration is incorrect, directly raise the corresponding error message; if the configuration is correct, execute step 3.2).

[0159] 3.2) Get the Source data source configuration, build the Source, connect to the specified data source, and collect data in real time.

[0160] 3.2.1) Log types are collected using Flink and support Kafka; Binlog and cdc data sources are collected using Flinkcdc, which is convenient for collecting real-time incremental data. Supported data sources include MySQL, SQLServer, and MongoDB.

[0161] 3.2.2) Support multiple data source configuration. Source data source supports multiple table configuration. You can directly configure multiple tables in the same database. For different databases, configure multiple data sources and use separators to splice them. For multiple data source configuration, the Flink program first splits the multiple data sources, builds different Source streams, and then uses the Union operator to merge the data streams. The specific operation process is as follows: Figure 3 shown.

[0162] 3.2.3) Support real-time update of sub-table increments. By configuring sub-table monitoring information, sub-table incremental data can be obtained in real time. The detailed implementation process is as follows Figure 4 shown.

[0163] Flink CDC monitors changes in multiple sub-tables, merges data streams from different sub-tables, customizes windows, and associates the main table with the main table in the window based on the associated fields of the main table to obtain the main table information; splits the acquired data stream, flows it into the main task Source data stream branch, and performs Union merge with the main table Source data stream for subsequent logical processing.

[0164] 3.3) Get the configuration information of Source, build FlatMap, parse, assemble and return the data.

[0165] 3.3.1) Get SourceConfig configuration information and parse the data stream according to the configuration path.

[0166] 3.3.2) According to the configuration of the monitoring field, obtain the value information of the monitoring field before and after change.

[0167] 3.3.3) Filter and return data source fields according to the configured return fields.

[0168] 3.3.4) For physical deletion scenarios, the logical deletion identifier Flag=1 will be set for downstream business logic processing.

[0169] 3.4) Get the configured filter condition expression, build a Filter, and filter the Source data stream.

[0170] Use the JEXL plug-in to parse and execute expressions. Data with a false result will be filtered out; data with a true result will continue to be executed.

[0171] 3.5) Get the configured window parameters, build the keyBy, window, and trigger operators, and generate a custom window.

[0172] 3.5.1) Get the configured keyBy field for grouping processing.

[0173] 3.5.2) Get window configuration and generate custom window.

[0174] The window types supported are: count window and time window. The window allocator can further allocate the following types: rolling window, sliding window and session window.

[0175] The time window supports the configuration of process time and event time. The event time window can be configured with watermarks, late data processing, etc.

[0176] Supports configuration of window triggers, and supports triggering by quantity and time.

[0177] 3.6) Get the data logic processing configuration process, build processFunction, and perform logical processing on the data in the window function.

[0178] 3.6.1) Support logical value processing of sub-table data;

[0179] Get the sub-table value-taking logic SQL from the sub-table value-taking logic configuration table, fill in the parameters according to the associated field values, connect to the sub-table data source, execute the SQL statement, get the results, assemble them into the source table data, and pass them down for data processing. The detailed operation process is as follows: Figure 5 shown.

[0180] 3.6.2) Support data formatting, date, integer and other related type fields;

[0181] 3.6.3) Support dictionary mapping;

[0182] 3.6.4) Support custom Lua scripts, use LuaValue to parse and execute scripts;

[0183] 3.6.5) Support field mapping;

[0184] 3.6.6) Support adding new fields.

[0185] 3.7) Get the relevant configuration information of Sink and build Sink. Connect Sink data source and sink the processed data to the specified storage medium.

[0186] Sink supports data sources, mainly including: MySQL, SQLServer, MongoDB, Doris, ClickHouse, ElasticSearch, etc. Sink writes to the target table in two modes: adding and updating.

[0187] 3.8) Operation process exception handling mechanism;

[0188] During the Flink job construction process, if an exception occurs, a corresponding exception handling mechanism will be implemented for the exceptions in different locations.

[0189] 3.8.1) Supports Flink operator parameter configuration verification, and directly prompts the corresponding configuration exception information if the configuration is abnormal.

[0190] 3.8.2) Supports Sink write exception retry mechanism, the retry time and number of retries are configurable, and if multiple retries still fail, the abnormal data will be written to the file without affecting the operation of the overall process.

[0191] The job runs abnormally and retry mechanism; the Flink program turns on CheckPoint, and sets the number of abnormal retries and the job restart interval. The number of retries is the maximum number of job restarts after a problem occurs. After the maximum number of restarts is reached, the current job will be judged as failed and closed; the interval between each restart is the waiting time from the failure of the job to the next restart.

[0192] The embodiment of the present invention provides a configurable big data import method and system based on Flink cdc, which can improve flexibility and scalability. The traditional import system based on Flink cdc may lack flexibility and be difficult to adapt to the needs of specific business scenarios. When processing different data sources and targets, as well as complex data conversion and processing logic, additional development work may be required to customize the system, resulting in poor scalability of traditional systems and inability to flexibly and quickly make changes and compatibility. Secondly, the present invention has faster data processing capabilities. The simple Flink cdc import system does not support complex configuration management. In particular, with the explosive growth of data volume and the further improvement of data real-time requirements, in complex scenarios, the efficiency of data conversion and cleaning needs to be improved accordingly to reduce duplication of work and difficulty in getting started, and reduce maintenance costs and management time costs.

[0193] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0194] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A configurable big data import method based on Flink cdc, characterized in that: include: Step S1: For the newly added import task in the system, set the configuration information and configure the process of task import; Step S2: The user Flink program generates executable user program code according to the configuration information in step S1, and submits it to the Flink cluster to execute the user program; Step S3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation; Among them, step S3 first reads and parses the task import process configuration information, verifies whether the task process configuration is correct, then collects data in real time, parses, assembles and returns the data, then filters and logically processes the data, and finally sinks the processed data to the specified storage medium, and performs abnormal processing of the operation process; The step S1 comprises: Step S1.1: Configure the main task value acquisition logic, including: main table data source connection configuration, main table value acquisition logic configuration, main table filter condition configuration, main table data processing logic configuration, custom window configuration and target database configuration; Step S1.2: Configure the sub-table value logic, including: sub-table data source configuration, mapping field configuration with the main table, data assembly configuration, and sub-table value SQL configuration. In addition, the association ID with the main task configuration table is also configured. The value logic configuration information of the corresponding sub-table can be queried through the main task ID; Step S1.3: If the value logic of the sub-table is configured in the above step S1.2 process, and if the sub-table is changed, if the result data also needs to be changed, it is necessary to configure the incremental dynamic monitoring of the sub-table in the sub-table dynamic monitoring table, including: sub-table data source configuration, sub-table name, monitoring field and associated field configuration. In addition, the association ID with the main task configuration table is also configured, and the monitoring configuration information of the corresponding sub-table can be queried through the main task ID; Step S1.4: Configure the task group, including the group ID and task ID configuration; wherein, the group ID configuration: configure the group ID in the task group configuration table, which is used as the input parameter for job startup, and search the corresponding task configuration list according to the group ID; Task ID configuration: The task ID of the group in the task group configuration table specifies that all tasks under an import requirement are configured in one group; Step S3 specifically includes: Step S3.1: Check the configuration information. If it is correct, execute step S3.

2. If it is incorrect, throw an exception directly and print the detailed error information in the log file. After the exception retry fails for the specified number of times, the Flink job stops directly. Step S3.2: Get the Source data source configuration, build the Source, connect to the specified data source, and collect data in real time; Step S3.3: Get the configuration information of Source, build FlatMap, parse, assemble and return the data; Step S3.4: If the task is configured with a filter condition expression, build a Filter operator to filter the Source data stream; Step S3.5: Get the configured window parameters, build keyBy, window, and trigger operators by implementing the abstract classes and interfaces within Flink, and generate a custom window. Step S3.6: Get the data logic processing configuration, including the value logic configuration of the sub-table and the data processing logic configuration of the main table. Build the processFunction function by implementing the ProcessWindowFunction abstract class inside Flink, and perform logical processing on the data in the window function. Step S3.7: Get the Sink related configuration information, build the Sink, connect the Sink data source, and sink the processed data to the specified storage medium; Step S3.8: Handle job process exceptions. During the Flink job construction process, each process is exception captured. If the current process construction is abnormal, the current exception is first captured to obtain the location of the abnormal process and the detailed information of the exception before throwing the exception. Step S3.9: Retry if the job runs abnormally; if an exception occurs in the user program in step S3.8, it will continue to be thrown, causing the entire Flink Job to run abnormally. When the Flink Job is abnormal, the abnormal retry mechanism is triggered; During the Flink program submission process, the Job operation retry strategy is specified through the setRestartStrategy method, including: fixed interval restart strategy, failure failure rate restart strategy, and no restart. The default is the fixed interval restart strategy.

2. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: In step S1.1, configuring the main task value logic includes: Main table data source connection configuration: Configure the data source connection information corresponding to the main table in the main task configuration table to implement the connection of the Source data source in the Flink program; Main table value retrieval logic configuration: configure the fields to be queried, the listening fields, and the value retrieval path information in the main task configuration table; Main table filter condition configuration: configure the filter conditions for the main table data in the main task configuration table; Main table data processing logic configuration: configure the fields that need special processing in the main task configuration table, including: data format conversion, dictionary mapping value, Lua script, field encryption and decryption of the specified algorithm, and field name mapping with the target table; Custom window configuration: The import of tasks is realized by reading the task configuration table through the Flink program, and generating executable code according to the configuration information to complete the import operation; Target database configuration: Configure the target database information that the data needs to be imported into in the main task configuration table, mainly including: database connection information, import table name and import fields.

3. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: The sub-table value logic is configured in step S1.2, including: Subtable data source configuration: Configure the subtable data source connection information in the subtask configuration table to connect to the subtable data source for subsequent SQL operations; Mapping field configuration with the main table: Configure the mapping field with the main table in the subtask configuration table to read the corresponding data of the main table as the input parameter of the SQL for subtable value retrieval; Data assembly configuration: Configure the data assembly method with the main table in the subtask configuration table, that is, directly nest it in the main data or flatten it; Sub-table value SQL configuration: Configure the sub-table value SQL in the sub-task configuration table to obtain the sub-table data corresponding to the main table and supplement the sub-table import logic.

4. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: The sub-table dynamic monitoring configuration in step S1.3 includes: Sub-table data source configuration: configure the sub-table data source connection information in the sub-table dynamic monitoring table to connect to the sub-table data source and provide connection parameter configuration for the sub-table's cdc connector; Sub-table name: configure the table name of the sub-table that needs to be dynamically monitored for changes in the sub-table dynamic monitoring table; Monitoring fields: configure the fields that need to dynamically monitor data changes in the sub-table dynamic monitoring table; Related field configuration: Configure the related fields with the main table in the sub-table dynamic monitoring table for related queries with the main table.

5. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: The step S2 comprises: Step S2.1: Upload the user program jar package and specify the execution main class; including: the user Flink program generates executable program code according to the configuration information in step S1, and then builds the user program into a jar file format; upload the jar package in the Flink client-task submission page by uploading the jar package; after the upload is successful, find the uploaded file in the corresponding jar file list, and specify the user program execution main class Entry Class in the corresponding job submission form; Step S2.2: Specify the group ID to be started; in the job submission form of step S2.1, specify the task group ID to be started, which is used to find the corresponding task list, and then perform the task operation, which is executed through the parameter GroupId in the start command; Step S2.3: Specify the data extraction mode: full or incremental; In the job submission form of step S2.1, specify the task extraction mode: full or incremental. The first startup or refresh data is full, and other cases are incremental. It is specified in the startup command through the parameter startupOptions; Step S2.4: Set the job parallelism. In the job submission form of step S2.1, specify the task parallelism. Step S2.5: Start the execution of the user program through a command; in the job submission form of step S2.1, specify the start command, start it through the flink run command, and execute the user program.

6. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: The step S3.3 comprises: Step S3.3.1: Get SourceConfig configuration information, and parse and obtain data stream according to the configuration path; Step S3.3.2: According to the configured monitoring field, obtain the information of the change value of the monitoring field before and after; Step S3.3.3: Filter and return the data source fields according to the configured return fields; Step S3.3.4: For the physical deletion scenario, set the logical deletion identifier Flag=1 for downstream business logic processing.

7. The configurable big data import method based on Flink cdc according to claim 1 is characterized in that: The step S3.6 comprises: Step S3.6.1: Obtain the logical value processing of the sub-table data; obtain the sub-table value logic SQL from the sub-table value logic configuration table, fill in the parameters according to the associated field value, connect to the sub-table data source, execute the SQL statement, obtain the result, assemble it into the source table data and pass it down for data processing; Step S3.6.2: Obtain data formatting configuration and complete data format conversion for different configurations; Step S3.6.3: Get Lua script configuration, parse the script through LuaValue and execute it; Step S3.6.4: Get the field mapping configuration, get the corresponding attribute value according to the source table field, assign it to the mapped field, and then fill the mapped field into the data stream; Step S3.6.5: Get the configuration of the newly added field, get the corresponding value from the data stream and assign it to the newly added field according to the value logic of the newly added field, and then fill the newly added field into the data stream.

8. A configurable big data import system based on Flink cdc, characterized in that: include: Module M1: For newly added import tasks in the system, set configuration information and configure the process of task import; Module M2: The user Flink program generates executable user program code based on the configuration information in step S1 and submits it to the Flink cluster to execute the user program; Module M3: The Flink cluster calls the user program code according to the specified execution class to complete the data import operation; Among them, the module M3 first reads and parses the task import process configuration information, verifies whether the task process configuration is correct, then collects data in real time, parses, assembles and returns data, then filters and logically processes the data, and finally sinks the processed data to the specified storage medium, and performs abnormal processing of the operation process; The step S1 comprises: Step S1.1: Configure the main task value acquisition logic, including: main table data source connection configuration, main table value acquisition logic configuration, main table filter condition configuration, main table data processing logic configuration, custom window configuration and target database configuration; Step S1.2: Configure the sub-table value logic, including: sub-table data source configuration, mapping field configuration with the main table, data assembly configuration, and sub-table value SQL configuration. In addition, the association ID with the main task configuration table is also configured. The value logic configuration information of the corresponding sub-table can be queried through the main task ID; Step S1.3: If the value logic of the sub-table is configured in the above step S1.2 process, and if the sub-table is changed, if the result data also needs to be changed, it is necessary to configure the incremental dynamic monitoring of the sub-table in the sub-table dynamic monitoring table, including: sub-table data source configuration, sub-table name, monitoring field and associated field configuration. In addition, the association ID with the main task configuration table is also configured, and the monitoring configuration information of the corresponding sub-table can be queried through the main task ID; Step S1.4: Configure the task group, including the group ID and task ID configuration; wherein, the group ID configuration: configure the group ID in the task group configuration table, which is used as the input parameter for job startup, and search the corresponding task configuration list according to the group ID; Task ID configuration: The task ID of the group in the task group configuration table specifies that all tasks under an import requirement are configured in one group; Module M3 specifically includes: Module M3.1: Verify the configuration information. If it is correct, execute module M3.

2. If it is incorrect, throw an exception directly and print the detailed error information in the log file. After the exception retry fails for the specified number of times, the Flink job stops directly. Module M3.2: Get the Source data source configuration, build the Source, connect to the specified data source, and collect data in real time; Module M3.3: Get the configuration information of Source, build FlatMap, parse, assemble and return the data; Module M3.4: If the task is configured with a filter condition expression, build a Filter operator to filter the Source data stream; Module M3.5: Gets the configured window parameters, builds keyBy, window, and trigger operators by implementing the abstract classes and interfaces within Flink, and generates custom windows. Module M3.6: Obtain data logic processing configuration, including the value logic configuration of the sub-table and the data processing logic configuration of the main table. By implementing the ProcessWindowFunction abstract class inside Flink to build the processFunction function, the data is logically processed in the window function. Module M3.7: Get Sink-related configuration information, build Sink, connect Sink data source, and sink the processed data to the specified storage medium; Module M3.8: Handle job process exceptions. During the Flink job construction process, each process is exception captured. If the current process construction is abnormal, the current exception is first captured, the location of the abnormal process and the detailed information of the exception are obtained, and then the exception is thrown. Module M3.9: Retry when job runs abnormally; in module M3.8, if an exception occurs in the user program, it will continue to be thrown, causing the entire Flink Job to run abnormally. When the Flink Job is abnormal, the abnormal retry mechanism is triggered; During the Flink program submission process, the Job operation retry strategy is specified through the setRestartStrategy method, including: fixed interval restart strategy, failure failure rate restart strategy, and no restart. The default is the fixed interval restart strategy.

Images