Data synchronization method, system and computer readable medium

Abstract

The present invention provides a data synchronization method, system and computer-readable medium. The method includes: in the full data synchronization stage, extracting data from the first database through the first streaming computing program corresponding to the first computing framework to form a first data storage message queue; in the incremental data synchronization stage after the full data synchronization stage is completed, determining the data starting point of the incremental data synchronization through the first streaming computing program, extracting data from the data starting point, and forming a second data storage message queue; extracting data from the first data storage message queue or the second data storage message queue through the second streaming computing program; performing data screening and format conversion operations on the data extracted from the first data storage message queue or the second data storage message queue to form processed data; storing the processed data in the second database. The incremental synchronization and full synchronization of the present invention are performed in the same program, which reduces maintenance costs.

Description

Technical Field

[0001] The present invention mainly relates to the field of data processing, and in particular to a data synchronization method, system and computer-readable medium. Background Art

[0002] When multiple copies of a piece of data need to be stored, consistency issues will arise, so synchronization is required. There are two types of synchronization: full synchronization and incremental synchronization. Full synchronization refers to storing all data in the target system on a regular or periodic basis. Incremental synchronization is based on full synchronization, capturing the difference data after a certain moment or checkpoint and synchronizing it to the target system. The moment or checkpoint that determines the triggering of the incremental synchronization process is called the update point.

[0003] The commonly used synchronization method is to synchronize through Spark in the incremental synchronization stage, while the incremental synchronization stage is synchronized by reading the Binlog log of the MySQL database through canal. The two are executed separately. After the full synchronization is completed, the incremental synchronization stage cannot determine the update point of the Binlog, and then a separate canal client is required to maintain the log, which increases the maintenance cost. At the same time, canal also needs to build a cluster, which increases the development cost. Moreover, canal cannot guarantee that data will not be lost in terms of semantics.

[0004] Therefore, there is an urgent need for a data synchronization method, system and computer-readable medium with low maintenance cost and development cost. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a data synchronization method, system and computer-readable medium to solve the problem of high maintenance cost and development cost of the data synchronization method.

[0006] In order to solve the above technical problems, the present invention provides a data synchronization method. It includes the following steps: in the full data synchronization stage, extract data from the first database through the first streaming computing program corresponding to the first computing framework to form a first data storage message queue; in the incremental data synchronization stage after the full data synchronization stage is completed, determine the data starting point of the incremental data synchronization through the first streaming computing program corresponding to the first computing framework, extract data from the data starting point, and form a second data storage message queue; in the full data synchronization stage and the incremental data synchronization stage, extract data from the first data storage message queue or the second data storage message queue through the second streaming computing program corresponding to the first computing framework; perform data screening and format conversion operations on the data extracted from the first data storage message queue or the second data storage message queue to form processed data; store the processed data in the second database.

[0007] In one embodiment of the present invention, the method also includes: monitoring the operation time of extracting data from the first database through the first streaming computing program corresponding to the first computing framework to obtain an operation delay value; comparing the operation delay value with a set first threshold to obtain a judgment result; and determining whether to request new operating resources for the data extraction operation based on the judgment result.

[0008] In one embodiment of the present invention, the operation time of extracting data from a first database through a first streaming computing program corresponding to a first computing framework is monitored to obtain an operation delay value, including: obtaining the number of data corresponding to the extraction operation in units of a set first time interval; obtaining an average value of the extraction operation of each data based on the number of data and the first time interval; and using the average value of the extraction operation of each data as the operation delay value.

[0009] In an embodiment of the present invention, the operating resources include processing resources and storage resources.

[0010] In one embodiment of the present invention, the first computing framework includes a flink computing framework, the first streaming computing program includes a flink-cdc streaming computing program, and the second streaming computing program includes a flink streaming computing program.

[0011] In one embodiment of the present invention, the extracted data includes Binlog data.

[0012] In one embodiment of the present invention, the first data storage message queue and the second data storage message queue include Kafka message queues.

[0013] In one embodiment of the present invention, the first database includes a MySQL database, and the second database includes a KUDU database.

[0014] In order to solve the above technical problems, the present invention provides a data synchronization system, comprising: a memory, used to store instructions executable by a processor; a processor, used to execute the instructions to implement the above data synchronization method.

[0015] In order to solve the above technical problem, the present invention provides a computer-readable medium storing a computer program code, and the computer program code implements the above data synchronization method when executed by a processor.

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] The data synchronization method of the present invention performs incremental synchronization and full synchronization in the same program, has few dependent components, and reduces maintenance costs and development costs; the present invention can seamlessly switch to the incremental data synchronization stage after the full data synchronization stage is completed, and does not require a separate canal client to maintain the synchronization log, thereby reducing maintenance costs; the present invention integrates the binlog delay indicator on the basis of flink-cdc to determine whether to request new running resources, thereby ensuring the real-time performance of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are provided to provide a further understanding of the present application. They are included and constitute a part of the present application. The accompanying drawings illustrate embodiments of the present application and together with the present specification serve to explain the principles of the present application. In the accompanying drawings:

[0019] Figure 1 is an exemplary flow chart of a data synchronization method according to an embodiment of the present invention;

[0020] Figure 2 is an exemplary timing diagram of a data synchronization method according to an embodiment of the present invention;

[0021] Figure 3 is a system block diagram of a data synchronization system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0025] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0026] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0027] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0028] Figure 1 FIG. 1 is an exemplary flow chart of a data synchronization method according to an embodiment of the present invention. Figure 1 As shown, the data synchronization method 10 of this embodiment includes the following steps:

[0029] Step S11: In the full data synchronization stage, extract data from the first database through the first streaming computing program corresponding to the first computing framework to form a first data storage message queue;

[0030] Step S12: in the incremental data synchronization phase after the full data synchronization phase is completed, the data starting point of the incremental data synchronization is determined by the first streaming computing program corresponding to the first computing framework, and data is extracted from the data starting point to form a second data storage message queue;

[0031] Step S13: In the full data synchronization phase and the incremental data synchronization phase, extracting data from the first data storage message queue or the second data storage message queue through the second streaming computing program corresponding to the first computing framework;

[0032] Step S14: performing data screening and format conversion operations on the data extracted from the first data storage message queue or the second data storage message queue to form processed data;

[0033] Step S15: Store the processed data in the second database.

[0034] Combine the following Figure 1-Figure 2 The above steps S11 to S15 are described in detail.

[0035] In step S11, in the full data synchronization stage, data is extracted from the first database through the first streaming computing program corresponding to the first computing framework to form a first data storage message queue. In some embodiments, the first computing framework includes a flink computing framework, and the first streaming computing program includes a flink-cdc streaming computing program. Flink is a new generation of distributed streaming data processing framework, and its unified processing engine can process both batch data and streaming data. Flink-cdc is a component of flink. Among them, cdc is the abbreviation of Change Data Capture, which is the abbreviation of change data capture. It can synchronize the incremental change records of the source database (Source) to one or more data target pools (Sink). In other words, flink-cdc is a Source component that can directly read full data and incremental change data from the database.

[0036] In some embodiments, the first database may be a MySQL database, or a PostgreSQL, Oracle, MongoDB, etc. database, which is not limited in this application. When the first database is a MySQL database, the extracted data may be Binlog data.

[0037] When extracting full data from a MySQL database through flink-cdc, the database's historical full data will be read first, and then smoothly switched to Binlog reading to ensure that no more or less data is read. Even if a failure occurs, data can be processed using Exactly Once semantics. After the reading is completed, the first data storage message queue is formed.

[0038] In some embodiments, the first data storage message queue comprises a kafka message queue.

[0039] Figure 2 FIG. 1 is an exemplary timing diagram of a data synchronization method according to an embodiment of the present invention. Figure 2 As shown, the components for running the data synchronization method include: MySQL database 21, flink-cdc streaming computing program 22, kafka cluster 23, flink streaming computing program 24, and KUDU database 25. The flink-cdc streaming computing program 22 extracts the full amount of data from the MySQL database 21, and controls the flow of each database data in the program to determine which kafka topic the data is sent to, thereby forming a first kafka message queue, and then inputting the first kafka message queue into the kafka cluster 23. After the first kafka message queue is formed, it means that the full synchronization phase is completed, and the flink-cdc streaming computing program 22 will seamlessly switch to the incremental data synchronization phase.

[0040] In step S12, in the incremental data synchronization phase after the full data synchronization phase is completed, the data starting point of the incremental data synchronization is determined by the first streaming computing program corresponding to the first computing framework, and data is extracted from the data starting point to form a second data storage message queue.

[0041] In some embodiments, the second data storage message queue can be a kafka message queue, which is not limited in this application. Specifically, flink-cdc first divides the data table of the full data into snapshot chunks (SnapshotChunk) by primary key, and then distributes the Snapshot Chunk to multiple SourceReaders. When reading each Snapshot Chunk, the algorithm is used to achieve consistent reading under lock-free conditions. When reading, SourceReader supports chunk-granular checkpoints, which are the data starting points for incremental data synchronization. After all Snapshot Chunks are read, a binlog chunk is sent to read the incremental Binlog data from the data starting point checkpoint to form a second data storage message queue.

[0042] like Figure 2 As shown, the flink-cdc streaming computing program 22 reads the incremental Binlog data from the data starting point checkpoint based on the full data extracted from the MySQL database 21, forms a second Kafka message queue, and then inputs the second Kafka message queue into the Kafka cluster 23.

[0043] In step S13, during the full data synchronization phase and the incremental data synchronization phase, data is extracted from the first data storage message queue or the second data storage message queue through the second streaming computing program corresponding to the first computing framework.

[0044] In some embodiments, the second stream computing program includes a flink stream computing program. flink provides a dedicated Kafka connector to read or write data to a Kafka topic. Specifically, Figure 2 As shown, the flink stream computing program 24 connects to the kafka cluster 23 through the Kafka connector. First, set Kafka related parameters in the flink stream computing program 24, such as the topic name of the data source, the port number, etc. After the setting is completed, the flink stream computing program 24 reads the data with the topic name as the set name in the kafka message queue to generate the original target data stream.

[0045] In step S14, data extracted from the first data storage message queue or the second data storage message queue is subjected to data screening and format conversion operations to form processed data. Specifically, the original target data stream is subjected to data screening through the flink operator, and the data screening method may be grouping (GROUP BY), multi-table association (JOIN), etc., which is not limited in this application. The Data Stream-level API provided by flink converts the format of the original target data stream after data screening according to the requirements of the second database through the flink API to generate the target data stream.

[0046] In step S15, the target data stream stores the processed data in the second database. In some embodiments, the second database may be a KUDU database, which is not limited in the present application. Figure 2 As shown, the target data stream generated by the flink streaming computing program 24 is stored in the KUDU database 25. The target data stream stored in the KUDU database can be used as a data source for the data operation layer of the data warehouse to speed up data extraction.

[0047] In some embodiments, the data synchronization method also includes: monitoring the operation time of extracting data from the first database through the first streaming computing program corresponding to the first computing framework to obtain an operation delay value; comparing the operation delay value with a set first threshold to obtain a judgment result; and determining whether to request new operating resources for data extraction operations based on the judgment result.

[0048] For example, the solution of the present invention can integrate the Binlog delay indicator based on flink-cdc, and integrate the Binlog delay indicator into flink's metrics, and finally monitor the data delay time through the grafana component to adjust the corresponding resources according to the indicator. Here, Binlog delay refers to the time consumed from the data being generated in MySQL to the flink task pulling the data to the program for processing. Figure 2 As shown, the operation time of the flink-cdc streaming computing program 22 extracting data from the MySQL database 21 is monitored by the monitoring component grafana. The time when the flink-cdc streaming computing program 22 starts to extract data from the MySQL database 21 is the start time, and the end time is when the extracted data is stored in the kafka cluster 23. The end time minus the start time is used to obtain the delay value, and the operation delay value is compared with the set first threshold. If the delay value is greater than the first threshold, it means that the current operating resources are tight and cannot meet the real-time nature of data synchronization, so new operating resources are requested to extract data. In some embodiments, the operating resources include processing resources and storage resources.

[0049] In some embodiments, the operation time of extracting data from the first database by the first streaming computing program corresponding to the first computing framework is monitored to obtain the operation delay value, including: obtaining the number of data corresponding to the extraction operation in units of a set first time interval; obtaining the average value of the extraction operation of each data based on the number of data and the first time interval; and using the average value of the extraction operation of each data as the operation delay value. For example, Figure 2 As shown, a first time interval is set, and the number of data extracted by the flink-cdc streaming computing program 22 from the MySQL database 21 within the first time interval is counted. The first time interval is divided by the total number of data to obtain the average time of the extraction operation of each data, and this average value is used as the delay value. By comparing the delay value with the first threshold, it is determined whether to request a new running resource.

[0050] In one embodiment, maintaining the synchronization log includes, for example, maintaining data starting point information of incremental data synchronization.

[0051] The data synchronization method of the present invention performs incremental synchronization and full synchronization in the same program, has few dependent components, and reduces maintenance costs and development costs; the present invention can seamlessly switch to the incremental data synchronization stage after the full data synchronization stage is completed, and does not require a separate canal client to maintain the synchronization log, thereby reducing maintenance costs; the present invention integrates the binlog delay indicator on the basis of flink-cdc to determine whether to request new running resources, thereby ensuring the real-time performance of data synchronization.

[0052] The present invention also includes a data synchronization system, including a memory and a processor, wherein the memory is used to store instructions executable by the processor; and the processor is used to execute the instructions to implement the above data synchronization method.

[0053] Figure 3 is a system block diagram of a data synchronization system according to an embodiment of the present invention. Figure 3 As shown, the data synchronization system 300 may include an internal communication bus 301, a processor 302, a read-only memory (ROM) 303, a random access memory (RAM) 304, and a communication port 305. When applied on a personal computer, the operating device 300 may also include a hard disk 306. The internal communication bus 301 can realize data communication between the components of the operating device 300. The processor 302 can make judgments and issue prompts. In some embodiments, the processor 302 can be composed of one or more processors. The communication port 305 can realize data communication between the operating device 300 and the outside. In some embodiments, the operating device 300 can send and receive information and data from the network through the communication port 305. The operating device 300 may also include different forms of program storage units and data storage units, such as a hard disk 306, a read-only memory (ROM) 303 and a random access memory (RAM) 304, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 302. The processor executes these instructions to implement the main part of the method. The processing result of the processor is transmitted to the user device through the communication port and displayed on the user interface.

[0054] The above-mentioned data synchronization method can be implemented as a computer program, stored in the hard disk 306, and loaded into the processor 302 for execution to implement the data synchronization of the present application.

[0055] The present invention also includes a computer-readable medium storing a computer program code, which implements the data synchronization method described above when executed by a processor.

[0056] When the data synchronization method is implemented as a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., an electrically erasable programmable read-only memory (EPROM), a card, a stick, a key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain and / or carry code and / or instructions and / or data.

[0057] It should be understood that the embodiments described above are only illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or a combination thereof.

[0058] Some aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes ...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs ...), smart cards, and flash memory devices (e.g., cards, sticks, key drives ...).

[0059] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above mediums.

[0060] The basic concepts have been described above. Obviously, for those skilled in the art, the above invention disclosure is only used as an example and does not constitute a limitation of the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements and amendments to the present application. Such modifications, improvements and amendments are suggested in the present application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of the present application.

[0061] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0062] Similarly, it should be noted that in order to simplify the description of the disclosure of this application and thus help understand one or more application embodiments, in the above description of the embodiments of this application, multiple features are sometimes merged into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the object of this application are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0063] Although the present application has been described with reference to the current specific embodiments, ordinary technicians in this technical field should recognize that the above embodiments are only used to illustrate the present application, and various equivalent changes or substitutions may be made without departing from the spirit of the present application. Therefore, as long as the changes and modifications to the above embodiments are within the essential spirit of the present application, they will fall within the scope of the claims of the present application.

Claims

1. A data synchronization method, comprising the following steps: In the full data synchronization stage, data is extracted from the first database through a first streaming computing program corresponding to the first computing framework to form a first data storage message queue; In the incremental data synchronization phase after the full data synchronization phase is completed, the data starting point of the incremental data synchronization is determined by the first streaming computing program corresponding to the first computing framework, and data is extracted from the data starting point to form a second data storage message queue; In the full data synchronization phase and the incremental data synchronization phase, extracting data from the first data storage message queue or the second data storage message queue through a second streaming computing program corresponding to the first computing framework; Performing data screening and format conversion operations on the data extracted from the first data storage message queue or the second data storage message queue to form processed data; storing the processed data in a second database; as well as The operation time of extracting data from the first database through the first streaming computing program corresponding to the first computing framework is monitored to obtain an operation delay value; including: obtaining the number of data corresponding to the extraction operation in units of a set first time interval; obtaining an average value of the extraction operation of each data based on the number of data and the first time interval; and using the average value of the extraction operation of each data as the operation delay value; Compare the operation delay value with a set first threshold value to obtain a judgment result; Based on the judgment result, it is determined whether to request new operating resources for the data extraction operation.

2. The data synchronization method according to claim 1, characterized in that: The operating resources include processing resources and storage resources.

3. The data synchronization method according to claim 1, characterized in that: The first computing framework includes a flink computing framework, the first streaming computing program includes a flink-cdc streaming computing program, and the second streaming computing program includes a flink streaming computing program.

4. The data synchronization method according to claim 1, characterized in that: The extracted data includes Binlog data.

5. The data synchronization method according to claim 1, characterized in that: The first data storage message queue and the second data storage message queue include a Kafka message queue.

6. The data synchronization method according to claim 1, characterized in that: The first database includes a MySQL database, and the second database includes a KUDU database.

7. A data synchronization system, comprising: a memory for storing instructions executable by a processor; as well as A processor, configured to execute the instructions to implement the method according to any one of claims 1 to 6.

8. A computer readable medium storing computer program code, wherein the computer program code, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images