Data synchronization method and device based on FlinkCDC task and storage medium

Abstract

The invention relates to the field of data synchronization, and discloses a data synchronization method and device based on a FlinkCDC task and a storage medium. The method comprises the following steps of: deploying a FlinkCDC task in a StreamPark service; receiving a configuration file, and connecting a MySQL database corresponding to the configuration file to obtain a schema meta-information table; according to the configuration file, adding a table prefix to the schema meta-information table, and generating a detail model table and a primary key model table of the StarRocks database; reading a Datax task correspondingly generated by the configuration file, loading data corresponding to the Datax task into the primary key model table, and generating an updated primary key model table; according to the FlinkCDC task, a binlog event is obtained from the MySQL database; according to the binlog event, modifying the updated primary key model table to obtain a changed primary key model table; and according to a preset synchronous updating period, loading the Datax task and the change primary key model table into the detail model table, and generating a synchronous detail model table. According to the embodiment of the invention, high accuracy of data synchronization when a large amount of data is processed is realized, and data consistency is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization, and in particular to a data synchronization method, device and storage medium based on a FlinkCDC task. Background Art

[0002] With the development of technologies such as the Internet of Things, 5G, and artificial intelligence, the amount of data is growing rapidly. According to statistics from Statista, an international authoritative organization, the amount of data generated globally is expected to reach 47ZB, and is expected to reach 2142ZB by 2035, showing a trend of large-scale data explosion. Big data technology has developed into a vast technical system, including basic technologies for data storage and processing, as well as peripheral technologies such as data governance, data analysis applications, and data security circulation. Technical trends include the introduction of concepts such as storage and computing separation, Serverless, and cloud native, as well as the application of artificial intelligence in data asset management, which has improved the automation and intelligence level of data management.

[0003] However, as the amount of data continues to grow, the existing data analysis has problems with data synchronization accuracy in the use scenario of multi-parallel synchronization. Therefore, in view of the technical problem that the data system has insufficient accuracy of data synchronization when processing large amounts of data, a new technology is needed to solve the current problem. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem of insufficient accuracy of data synchronization when a data system processes a large amount of data.

[0005] A first aspect of the present invention provides a data synchronization method based on a FlinkCDC task, comprising the steps of:

[0006] Deploy the FlinkCDC task in the pre-configured StreamPark service;

[0007] Receive the configuration file, connect to the MySQL database corresponding to the configuration file, and obtain the schema meta information table;

[0008] According to the configuration file, add a table prefix to the schema meta-information table to generate a detail model table and a primary key model table of the StarRocks database;

[0009] Read the Datax task generated corresponding to the configuration file, load the data corresponding to the Datax task into the primary key model table, and generate an updated primary key model table;

[0010] According to the FlinkCDC task, obtain binlog events from the MySQL database;

[0011] According to the binlog event, modify the updated primary key model table to obtain a changed primary key model table;

[0012] According to the preset synchronization update cycle, the Datax task and the changed primary key model table are loaded into the detail model table to generate a synchronization detail model table.

[0013] Optionally, in a first implementation of the first aspect of the present invention, modifying the updated primary key model table according to the binlog event to obtain the changed primary key model table includes:

[0014] Determine whether the binlog event is a table structure change event;

[0015] When it is a table structure change event, the acquisition of data from the MySQL database is stopped, and based on the data in the preset buffer, the table structure of the updated primary key model table in the StarRocks database is adjusted to obtain the changed primary key model table;

[0016] If it is not a table structure change event, determine whether the binlog event is a flush event;

[0017] When it is a flush event, the data in the preset buffer is flushed to the updated primary key model table to obtain a changed primary key model table;

[0018] When it is not a flush event, the data corresponding to the binlog event is written into a preset buffer.

[0019] Optionally, in a second implementation of the first aspect of the present invention, before writing the data corresponding to the binlog event into a preset buffer, it also includes:

[0020] Based on the primary key number of the updated primary key model table, the preset buffer is partitioned to obtain a buffer corresponding to the primary key number.

[0021] Optionally, in a third implementation of the first aspect of the present invention, writing the data corresponding to the binlog event into a preset buffer includes:

[0022] Write the data corresponding to the binlog event into the buffer corresponding to the primary key number.

[0023] Optionally, in a fourth implementation of the first aspect of the present invention, obtaining a binlog event from the MySQL database according to the FlinkCDC task includes:

[0024] Based on the Debezium platform, information simulation processing is performed on the FlinkCDC task to generate a simulated MySQLslave component;

[0025] Based on the simulated MySQL slave component, binlog events are obtained from the MySQL database.

[0026] Optionally, in a fifth implementation of the first aspect of the present invention, deploying the FlinkCDC task in a pre-set StreamPark service includes:

[0027] Upload the FlinkCDC task to the Kubernetes cluster and generate a Docker container;

[0028] Integrate the Docker container into the StreamPark service.

[0029] Optionally, in a sixth implementation of the first aspect of the present invention, the receiving the configuration file, connecting to a MySQL database corresponding to the configuration file, and obtaining the schema meta information table includes:

[0030] Receive configuration files;

[0031] Based on the JDBC interface, connect to the MySQL database corresponding to the configuration file to obtain the schema meta information table.

[0032] Optionally, in a seventh implementation of the first aspect of the present invention, the step of loading the Datax task and the changed primary key model table into the detail model table according to a preset synchronization update cycle, and generating the synchronization detail model table includes:

[0033] According to the preset synchronization update cycle, the Datax task and the changed primary key model table are loaded into the detail model table by streaming loading to generate a synchronized detail model table.

[0034] A second aspect of the present invention provides a data synchronization device based on a FlinkCDC task, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected through a line; the at least one processor calling the instructions in the memory so that the data synchronization device based on the FlinkCDC task executes the above-mentioned data synchronization method based on the FlinkCDC task.

[0035] A third aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned data synchronization method based on FlinkCDC tasks.

[0036] In the embodiment of the present invention, by constructing the primary key model table and the detail model table at the same time, the two tables adopt different synchronization strategies to synchronize data, and the historical data and the real-time data are synchronized separately, so as to ensure the flexibility of data synchronization and the effective use of resources and avoid wasting resources. Based on the source table, the data is uniformly directed to the StarRocks database according to the configured rules, so as to achieve high accuracy of data synchronization when processing a large amount of data and ensure data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of an embodiment of a data synchronization method based on a FlinkCDC task in an embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of a specific embodiment of step 105 of the data synchronization method based on the FlinkCDC task in an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of a specific embodiment of step 106 of the data synchronization method based on the FlinkCDC task in an embodiment of the present invention;

[0040] Figure 4 The figure is a schematic diagram of an embodiment of a data synchronization device based on a FlinkCDC task in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Embodiments of the present invention provide a data synchronization method, device, and storage medium based on a FlinkCDC task.

[0042] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0043] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0044] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the data synchronization method based on the FlinkCDC task in an embodiment of the present invention includes:

[0045] 101. Deploy the FlinkCDC task in the pre-configured StreamPark service;

[0046] In this embodiment, FlinkCDC is a stream-based data integration tool used to capture change data from different databases. FlinkCDC is deeply integrated and driven by Apache Flink, providing an end-to-end data integration framework, API, support for processing multiple tables, whole database synchronization, and automatic synchronization of table structure changes.

[0047] StreamPark is a stream processing service. StreamPark provides a development framework for writing stream processing applications using Apache Flink and Apache Spark. It can perform application development, debugging, interactive query, deployment, operation and maintenance, real-time data warehouse and other operations.

[0048] Therefore, the FlinkCDC task on the FlinkCDC integration tool is deployed in the pre-configured StreamPark service so that the data operations set by the FlinkCDC task can be performed later.

[0049] Specifically, step 101 includes the following specific implementation methods:

[0050] 1011. Upload the FlinkCDC task to the Kubernetes cluster and generate a Docker container;

[0051] 1012. Integrate the Docker container into the StreamPark service.

[0052] In steps 1011-1012, Kubernetes is an open source container orchestration platform. A Kubernetes cluster is a group of nodes, which can be virtual machines or physical machines configured to run containerized applications. First, upload the FlinkCDC task to be executed to the Kubernetes cluster, package the FlinkCDC task, build an image, push it to the remote warehouse, and generate a Docker container.

[0053] Configure the basic environment in StreamPark, including Kubernetes and Docker repository information. Configure Docker account information and Flink client information directly in the StreamPark settings interface, and integrate the packaged Docker container into the StreamPark service.

[0054] 102. Receive a configuration file, connect to a MySQL database corresponding to the configuration file, and obtain a schema meta-information table;

[0055] In this embodiment, the configuration file is a pre-written database designated for monitoring, and a MySQL database is used as an operation database object, and a schema meta information table is obtained from the MySQL database.

[0056] The schema meta-information table refers to a collection of information that defines data structure and data type. In a relational database, the table structure defines the table's column name, data type, primary key, foreign key, index and other information. Data integrity constraints can be set, such as NOT NULL, UNIQUE, PRIMARY KEY, FOREIGN KEY, etc.

[0057] Specifically, step 102 includes the following specific implementations:

[0058] 1021. Receive a configuration file;

[0059] 1022. Based on the JDBC interface, connect to the MySQL database corresponding to the configuration file to obtain a schema meta information table.

[0060] In steps 1021-1022, the configuration file data is received, and then the MySQL database corresponding to the configuration file is accessed through the JDBC interface. The JDBC interface is a Java API that allows Java programs to connect to the database and execute SQL statements. JDBC provides a unified access method for different types of databases, whether they are relational databases such as MySQL, Oracle, PostgreSQL, or non-relational databases such as MongoDB. Therefore, using the JDBC interface to obtain the schema meta-information table from the MySQL database has good versatility and does not require additional configuration of adaptive components.

[0061] 103. According to the configuration file, add a table prefix to the schema meta-information table to generate a detail model table and a primary key model table of the StarRocks database;

[0062] In this embodiment, according to the configuration file, the schema meta information is read according to the set date rule, the table prefix is ​​automatically added to generate the detail model table in the StarRocks database, and the snapshot time field is additionally added to the detail table for partitioning according to the 24-hour rule. Similarly, the schema meta information is read to automatically add the table prefix to generate the primary key model table in the StarRocks database.

[0063] It should be noted that the StarRocks database is a high-performance analytical data warehouse that uses technologies such as vectorization, MPP (Massively Parallel Processing) architecture, CBO (Cost Based Optimizer), intelligent materialized views, and a columnar storage engine that can be updated in real time to achieve multi-dimensional, real-time, and highly concurrent data analysis.

[0064] 104. Read the Datax task generated corresponding to the configuration file, load the data corresponding to the Datax task into the primary key model table, and generate an updated primary key model table;

[0065] In this embodiment, DataX is an open source offline synchronization tool for heterogeneous data sources. Multiple tasks can be set on the DataX tool. The configuration file is read to automatically generate the Datax (mysql-starrocks data synchronization) task, and the historical data is initialized to the StarRocks primary key model table through the Stream Load method to generate an updated primary key model table.

[0066] It should be noted that Stream Load is a data import method, which is usually used to synchronously import local files or data streams into a database that supports this function.

[0067] 105. According to the FlinkCDC task, obtain binlog events from the MySQL database;

[0068] In this embodiment, the MySQL database is accessed based on the previously set FlinkCDC task, and the binlog event data is obtained from the MySQL database. It should be noted that the Binlog (Binary Log) event is a log file of the MySQL database server, which is used to record all operations that change data in the database.

[0069] See also Figure 2 , Figure 2 The following is a schematic diagram of a specific embodiment of step 105 of the data synchronization method based on the FlinkCDC task in an embodiment of the present invention. Step 105 includes the following specific implementation methods:

[0070] 1051. Based on the Debezium platform, perform information simulation processing on the FlinkCDC task to generate a simulated MySQL slave component;

[0071] 1052. Based on the simulated MySQL slave component, obtain binlog events from the MySQL database.

[0072] In steps 1051-1052, Debezium is an open source distributed platform for capturing database change data (Change Data Capture, CDC), built on Apache Kafka, and provides connectors compatible with Kafka Connect for monitoring specific database management systems.

[0073] In the Debezium platform, the FlinkCDC task is simulated to obtain a simulated MySQL slave component. The simulated MySQL slave component can request data in the binary log (binlog) from the MySQL database to obtain binlog events.

[0074] 106. According to the binlog event, modify the updated primary key model table to obtain a changed primary key model table;

[0075] In this embodiment, based on the binlog event, the binlog event sends data events, table structure change events, and flush events to the downstream operator, and the updated primary key model table is modified to obtain the changed primary key model table.

[0076] See also Figure 3 , Figure 3 The following is a schematic diagram of a specific embodiment of step 106 of the data synchronization method based on the FlinkCDC task in an embodiment of the present invention. Step 106 includes the following specific implementation methods:

[0077] 1061. Determine whether the binlog event is a table structure change event;

[0078] 1062. When it is a table structure change event, the acquisition of data from the MySQL database is stopped, and based on the data in the preset buffer, the table structure of the updated primary key model table in the StarRocks database is adjusted to obtain the changed primary key model table;

[0079] 1063. When it is not a table structure change event, determine whether the binlog event is a flush event;

[0080] 1064. When it is a flush event, the data in the preset buffer is flushed to the updated primary key model table to obtain a changed primary key model table;

[0081] 1065. When it is not a flush event, the data corresponding to the binlog event is written into a preset buffer.

[0082] In steps 1061-1062, it is first determined whether the binlog event is a table structure change event. If it is a table structure change event, the acquisition of data from the MySQL database is stopped, and the buffer data based on the primary key model table is flushed to the updated primary key model table corresponding to the StarRocks database through the StreamLoad method. After the flushing is completed, the updated primary key model table corresponding to the StarRocks database is updated to generate a changed primary key model table. After the table change is completed, the binlog event is received from the MySQL database again.

[0083] When it is not a table structure change event, it is necessary to further determine whether the binlog event is a flush event. If it is a flush event, the data in the preset buffer is flushed to the update primary key model table to obtain the change primary key model table. If it is not a flush event, the valid business data in the binlog event is written to the buffer, waiting for the flush instruction to enter the update primary key model table.

[0084] Specifically, before step 1065, the following specific implementation methods are also included:

[0085] 106X. Based on the primary key number of the updated primary key model table, partition the preset buffer to obtain a buffer corresponding to the primary key number.

[0086] In step 106X, based on the primary key number (id number) of the updated primary key model table, the buffer is partitioned to ensure data accuracy.

[0087] Specifically, step 1065 includes the following specific implementation methods:

[0088] 10651. Write the data corresponding to the binlog event into the buffer corresponding to the primary key number.

[0089] In step 10651, the table is partitioned by the primary key id, and the valid binlog business data is cached in the buffer Buffer, waiting for the opportunity to flush.

[0090] 107. According to a preset synchronization update cycle, the Datax task and the changed primary key model table are loaded into the detail model table to generate a synchronization detail model table.

[0091] In this embodiment, the synchronization update cycle can be set to 24 hours, and the configuration table is read regularly at dawn every day to automatically generate a Datax (mysql-starrocks data synchronization) task. The snapshot data of the primary key model table changed on that day is loaded into the corresponding detail model table through streamload to generate a synchronized detail model table.

[0092] In the embodiment of the present invention, by constructing the primary key model table and the detail model table at the same time, the two tables adopt different synchronization strategies to synchronize data, and the historical data and the real-time data are synchronized separately, so as to ensure the flexibility of data synchronization and the effective use of resources and avoid wasting resources. Based on the source table, the data is uniformly directed to the StarRocks database according to the configured rules, so as to achieve high accuracy of data synchronization when processing a large amount of data and ensure data consistency.

[0093] Figure 4 4 is a structural diagram of a data synchronization device based on a FlinkCDC task provided by an embodiment of the present invention. The data synchronization device 400 based on the FlinkCDC task may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more massive storage devices) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 can be short-term storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data synchronization device 400 based on the FlinkCDC task. Furthermore, the processor 410 can be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the data synchronization device 400 based on the FlinkCDC task.

[0094] The data synchronization device 400 based on the FlinkCDC task may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that Figure 4 The illustrated structure of the data synchronization device based on the FlinkCDC task does not constitute a limitation on the data synchronization device based on the FlinkCDC task, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0095] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the data synchronization method based on the FlinkCDC task.

[0096] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0098] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data synchronization method based on FlinkCDC task, characterized in that: Includes steps: Deploy the FlinkCDC task in the pre-configured StreamPark service; Receive the configuration file, connect to the MySQL database corresponding to the configuration file, and obtain the schema meta information table; According to the configuration file, add a table prefix to the schema meta-information table to generate a detail model table and a primary key model table of the StarRocks database; Read the Datax task generated corresponding to the configuration file, load the data corresponding to the Datax task into the primary key model table, and generate an updated primary key model table; According to the FlinkCDC task, obtain binlog events from the MySQL database; According to the binlog event, modify the updated primary key model table to obtain a changed primary key model table; According to the preset synchronization update cycle, the Datax task and the changed primary key model table are loaded into the detail model table to generate a synchronization detail model table.

2. The data synchronization method based on FlinkCDC task according to claim 1 is characterized in that: The updating primary key model table is modified according to the binlog event to obtain the changed primary key model table, including: Determine whether the binlog event is a table structure change event; When it is a table structure change event, the acquisition of data from the MySQL database is stopped, and based on the data in the preset buffer, the table structure of the updated primary key model table in the StarRocks database is adjusted to obtain the changed primary key model table; If it is not a table structure change event, determine whether the binlog event is a flush event; When it is a flush event, the data in the preset buffer is flushed to the updated primary key model table to obtain a changed primary key model table; When it is not a flush event, the data corresponding to the binlog event is written into a preset buffer.

3. The data synchronization method based on FlinkCDC task according to claim 2 is characterized in that: Before writing the data corresponding to the binlog event into a preset buffer, the method further includes: Based on the primary key number of the updated primary key model table, the preset buffer is partitioned to obtain a buffer corresponding to the primary key number.

4. The data synchronization method based on FlinkCDC task according to claim 3 is characterized in that: Writing the data corresponding to the binlog event into a preset buffer includes: Write the data corresponding to the binlog event into the buffer corresponding to the primary key number.

5. The data synchronization method based on FlinkCDC task according to claim 1 is characterized in that: According to the FlinkCDC task, obtaining the binlog event from the MySQL database includes: Based on the Debezium platform, perform information simulation processing on the FlinkCDC task to generate a simulated MySQL slave component; Based on the simulated MySQL slave component, binlog events are obtained from the MySQL database.

6. The data synchronization method based on FlinkCDC task according to claim 1, characterized in that: Deploying the FlinkCDC task in the pre-configured StreamPark service includes: Upload the FlinkCDC task to the Kubernetes cluster and generate a Docker container; Integrate the Docker container into the StreamPark service.

7. The data synchronization method based on FlinkCDC task according to claim 1 is characterized in that: The receiving configuration file, connecting to the MySQL database corresponding to the configuration file, and obtaining the schema meta information table includes: Receive configuration files; Based on the JDBC interface, connect to the MySQL database corresponding to the configuration file to obtain the schema meta information table.

8. The data synchronization method based on FlinkCDC task according to claim 1, characterized in that: The step of loading the Datax task and the changed primary key model table into the detailed model table according to the preset synchronization update cycle, and generating the synchronized detailed model table comprises: According to the preset synchronization update cycle, the Datax task and the changed primary key model table are loaded into the detail model table by streaming loading to generate a synchronized detail model table.

9. A data synchronization device based on FlinkCDC task, characterized in that: The data synchronization device based on the FlinkCDC task includes: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected through a line; The at least one processor calls the instruction in the memory so that the FlinkCDC task-based data synchronization device executes the FlinkCDC task-based data synchronization method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data synchronization method based on the FlinkCDC task according to any one of claims 1 to 8 is implemented.

Images