Method, device, electronic equipment and readable storage medium for database synchronization

Abstract

Embodiments of the present application provide a database synchronization method and device, electronic equipment and computer readable storage medium, and relate to the technical field of data processing. The method comprises: obtaining at least one target database to be synchronized to a kafka cluster; the target database comprises at least one target data table; the kafka cluster comprises at least one kafka topic; each kafka topic can store one kind of data table; determining a target kafka topic in the kafka cluster corresponding to the kind of the target data table; and synchronizing each target data table to the corresponding target kafka topic respectively. Embodiments of the present application realize that after the target database is synchronized to the kafka cluster, each target data table in the kafka cluster has a clear classification, so that when a consumer reads a data table, no additional classification work is required, and without creating a thread corresponding to each data table, each data table in the kafka cluster has a clear classification, which is not prone to human error and reduces resource waste.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, electronic device, and readable storage medium for database synchronization. Background Technology

[0002] Kafka cluster is a high-throughput distributed publish-subscribe messaging system. In a Kafka cluster, a category attribute is used to classify the data. This class is called a Kafka topic, and the data stored in a Kafka topic is considered to be data of the same class.

[0003] When synchronizing data from a database to a Kafka cluster, the complete data flow is as follows: producers write data from various databases to Kafka topics, and consumers read data from the Kafka cluster and process it in other data processing systems. During the producer's data writing process, all tables from a database are typically written to a single Kafka topic, which is relatively simple. However, after writing to a Kafka topic, because the Kafka topic stores rows of data from various tables without a clear categorization, consumers need to perform additional categorization when reading data from the Kafka topic, increasing the burden on consumers. As the number of data tables increases, the data in a Kafka topic becomes increasingly mixed, and the above problems become more severe.

[0004] The existing solution addresses the above problems by creating a thread for each data table in the database during the synchronization process. Each thread has a one-to-one relationship with each data table, and each thread is used to synchronize one data table to the corresponding Kafka topic. Multiple threads can synchronize their respective data tables to their respective Kafka topics. Although this approach solves the aforementioned problems, the excessive number of threads created due to the need to set up a thread for each data table makes them difficult to manage, prone to errors, and wasteful of resources. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for database synchronization, which are used to solve at least one technical problem in the background art.

[0006] According to a first aspect of the embodiments of this application, a database synchronization method is provided, the method comprising:

[0007] Obtain at least one target database to be synchronized to the Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store one type of data table;

[0008] For each target data table, determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table;

[0009] Each target data table is synchronized to its corresponding target Kafka topic.

[0010] In one possible implementation, the Kafka topic name includes the table name of the corresponding data table and the database name to which the corresponding data table belongs;

[0011] Determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table, including:

[0012] Configure the target data table; the configuration information includes the table name of the target data table and the database name of the target database to which the target data table belongs;

[0013] If it is determined that a Kafka topic exists in the Kafka cluster that corresponds to the configuration information of the target data table, then that Kafka topic will be used as the target Kafka topic for the target data table.

[0014] In one possible implementation, after setting the configuration information for the target data table, the following is also included:

[0015] If it is determined that there is no Kafka topic in the Kafka cluster that corresponds to the configuration information of the target data table, then create the target Kafka topic corresponding to the target data table in the Kafka cluster.

[0016] In one possible implementation, the characteristic is that the configuration information of the target data table is set, including:

[0017] Obtain the initial regular expression. The variables in the initial regular expression include a first variable and a second variable. The first variable represents the table name of the data table, and the second variable represents the database name to which the data table belongs.

[0018] In the initial regular expression, the name of the target data table is assigned to the first variable, and the name of the target database is assigned to the second variable, resulting in the target regular expression. This target regular expression is then used as the configuration information for the target data table.

[0019] In one possible implementation, the target data table includes at least one data row;

[0020] Each target data table will be synchronized to its corresponding target Kafka topic, including:

[0021] For each target data table, at least one data row in the target data table is encoded to obtain the encoding information corresponding to each data row;

[0022] The encoded information corresponding to each data row is merged to obtain the Kafka message corresponding to the target data table;

[0023] Store the Kafka messages corresponding to each target data table into the corresponding target Kafka topic.

[0024] In one possible implementation, the encoded information corresponding to each data row is merged, including:

[0025] Determine the order of the data rows and the length of each data row corresponding to its encoding information;

[0026] Each data length is encoded separately, and the header of the Kafka message is generated based on the encoding information corresponding to each data length.

[0027] The encoded information of each data row is sequentially linked to the header of the Kafka message based on the order of data transmission.

[0028] In one possible implementation, there is a mapping relationship between each data table and its corresponding target Kafka topic;

[0029] Each target data table is synchronized to its corresponding target Kafka topic, including:

[0030] Create and run a target thread, which is used to synchronize each target data table to the corresponding target Kafka topic based on the mapping relationship.

[0031] According to a second aspect of the embodiments of this application, a database synchronization apparatus is provided, the apparatus comprising:

[0032] The acquisition module is used to acquire at least one target database to be synchronized to the Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store one type of data table;

[0033] The target Kafka topic determination module is used to determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table for each target data table.

[0034] The synchronization module is used to synchronize each target data table to its corresponding target Kafka topic.

[0035] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device including a memory, a processor and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the method provided in the first aspect.

[0036] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method provided in the first aspect.

[0037] According to a fifth aspect of the present application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein when a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform steps implementing the method provided in the first aspect.

[0038] The beneficial effects of the technical solution provided in this application are as follows: By determining the target Kafka topic in the Kafka cluster corresponding to the type of the target data table, this application achieves a clear classification of each data table in the Kafka cluster after synchronizing the target database to the Kafka cluster. This eliminates the need for additional classification work when consumers read data tables from the Kafka cluster. Furthermore, this application achieves a clear classification of each data table in the Kafka cluster without creating a thread for each data table, reducing the likelihood of human error and minimizing resource waste. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0040] Figure 1 This is a schematic diagram of the system architecture for a database synchronization method provided in an embodiment of this application;

[0041] Figure 2 A flowchart illustrating a database synchronization method provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram illustrating an application scenario of a database synchronization method provided in an embodiment of this application.

[0043] Figure 4 This is a schematic diagram of the structure of a database synchronization device provided in an embodiment of this application;

[0044] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] First, let's introduce and explain several terms used in this application:

[0049] A Kafka cluster possesses three key functions for end-to-end event streaming: 1. Publishing (writing) and subscribing (reading) event streams. 2. Storing event streams and providing good fault tolerance. 3. Processing events as they are generated.

[0050] Producer: An application that publishes (writes) events to the Kafka cluster; Consumer: An application that subscribes to (reads and processes) events in the Kafka cluster.

[0051] Kafka topics: The types of Kafka messages are called Kafka topics. A Kafka topic stores a type of Kafka message, which is equivalent to classifying Kafka messages. Producers can publish data to the selected Kafka topic.

[0052] This application provides methods, apparatus, electronic devices, computer-readable storage media, and computer program products for database synchronization, aiming to solve the above-mentioned technical problems of the prior art.

[0053] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0054] Figure 1 The system architecture diagram for the database synchronization method provided in this application embodiment includes a producer 110, a Kafka cluster 120, and a consumer 130. The producer 110 is any application that can write (publish) data or event streams to the Kafka cluster 120, and the consumer 130 is any application that can read (subscribe) data or event streams from the Kafka cluster 120. The Kafka cluster 120 can store event streams and has good fault tolerance.

[0055] The entity implementing a database synchronization method in this application embodiment can be the aforementioned producer.

[0056] This application provides a method for database synchronization, such as... Figure 2 As shown, the method includes:

[0057] Step S201: Obtain at least one target database to be synchronized to the Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store a type of data table.

[0058] The embodiments of this application occur during the data synchronization process, which synchronizes the target database to the Kafka cluster. The target database includes at least one target data table. During the data synchronization process, data is generally synchronized on a table-by-table basis, but in reality, at least one data row in the data table is synchronized.

[0059] Kafka topics are the core of a Kafka cluster, decoupling producers and consumers. Consumers read messages from Kafka topics, while producers write messages to Kafka topics. A Kafka topic can contain multiple producers and multiple consumers.

[0060] The Kafka cluster in this application includes at least one Kafka topic. Each Kafka topic can store a data table of one type. One data table represents one type, meaning that each Kafka topic stores one data table.

[0061] Step S202: For each target data table, determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table.

[0062] The types of data tables in this application embodiment are determined based on the name of the data table and the database name of the data table database, and the type of each data table is unique.

[0063] In a Kafka cluster, each Kafka topic can store a type of data table. When synchronizing data, it is necessary to determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table. The target Kafka topic can then store the target data table. There is a one-to-one mapping relationship between the target data table and the target Kafka topic, which enables the classification of the target data table.

[0064] Specifically, during this synchronization process, if a Kafka topic corresponding to the type of the target data table can be found directly in the Kafka cluster, then that Kafka topic will be used as the target Kafka topic. If a Kafka topic corresponding to the type of the target data table cannot be found in the Kafka cluster, then a new Kafka topic corresponding to the type of the target data table will be created in the Kafka cluster.

[0065] Step S203: Synchronize each target data table to the corresponding target Kafka topic.

[0066] In this embodiment of the application, after determining the target Kafka topic corresponding to each data table, each target data table is synchronized to the corresponding target Kafka topic. Specifically, a target thread can be created, which can synchronize each target data table to the corresponding target Kafka topic based on the mapping relationship between the target data table and the corresponding target Kafka topic.

[0067] This application embodiment achieves a clear classification of data tables in the Kafka cluster after synchronizing the target database to the Kafka cluster by determining the target Kafka topic corresponding to the type of the target data table. This eliminates the need for additional classification work when consumers read data tables from the Kafka cluster. Furthermore, this application embodiment achieves a clear classification of data tables in the Kafka cluster without creating a thread for each data table, reducing the likelihood of human error and minimizing resource waste.

[0068] This application provides a possible implementation method in which the topic name of the Kafka topic includes the table name of the corresponding data table and the database name to which the corresponding data table belongs.

[0069] In this embodiment of the application, each Kafka topic can store a type of data table. The topic name of the Kafka topic includes the table name of the corresponding data table and the database name to which the corresponding data table belongs. For example, if the topic name of a Kafka topic is tidb_mydb_test1, where tidb is a fixed field, "test1" is the table name, and "mydb" is the database name to which the corresponding data table belongs, the Kafka topic with the topic name tidb_mydb_test1 can store the "test1" data table in the "mydb" database.

[0070] Determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table, including:

[0071] Configure the target data table; the configuration information includes the table name of the target data table and the database name of the target database to which the target data table belongs;

[0072] If it is determined that a Kafka topic exists in the Kafka cluster that corresponds to the configuration information of the target data table, then that Kafka topic will be used as the target Kafka topic for the target data table.

[0073] If it is determined that there is no Kafka topic in the Kafka cluster that corresponds to the configuration information of the target data table, then a target Kafka topic corresponding to the target data table is created in the Kafka cluster.

[0074] To determine the target Kafka topic in the Kafka cluster that corresponds to the type of target data table, configuration information needs to be set for each target data table. The configuration information includes the table name of the target data table and the database name of the target database to which the target data table belongs.

[0075] Specifically, the target database "mydb" contains 100 target data tables, which are named test1, test2, ..., test100. These data tables need to be synchronized from their respective target databases to the Kafka cluster. The configuration information set for each data table is tidb_mydb_test1, tidb_mydb_test2, ..., tidb_mydb_test100.

[0076] After setting the configuration information for each target data table, it is necessary to check in the Kafka cluster whether there is a Kafka topic with the same configuration information as the target data table. If there is a Kafka topic with the same configuration information as the target data table, it means that there is a Kafka topic in the Kafka cluster that corresponds to the configuration information of the target data table. This target data table has been synchronized before, and in this synchronization process, the Kafka topic can be directly used as the target Kafka topic corresponding to the target data table.

[0077] If no Kafka topic with the same configuration information as the target data table exists, it means that there is no Kafka topic in the Kafka cluster corresponding to the configuration information of the target data table. This indicates that the target data table has not been synchronized. During this synchronization process, a new target Kafka topic corresponding to the target data table needs to be created in the Kafka cluster. The topic name of this target Kafka topic is the configuration information of the target data table.

[0078] This application provides a possible implementation method for setting the configuration information of the target data table, including:

[0079] Obtain the initial regular expression. The variables in the initial regular expression include a first variable and a second variable. The first variable represents the table name of the data table, and the second variable represents the database name to which the data table belongs.

[0080] In the initial regular expression, the table name of the target data table is assigned to the first variable, and the database name of the target database is assigned to the second variable, resulting in the target regular expression. This target regular expression is then used as the configuration information for the target data table.

[0081] This application embodiment can set configuration information for each target data table through regular expression matching. Specifically, an initial regular expression is obtained, which includes a first variable and a second variable. The first variable represents the table name of the data table, and the second variable represents the database name to which the data table belongs.

[0082] For example, the initial regular expression is: tidb_{schema}_{table}, where "tidb" is a fixed field, "table" is the first variable representing the table name, and "schema" is the second variable representing the database name to which the table belongs.

[0083] If the target database "mydb1" contains 10 target data tables, namely "test1", "test2", ..., "test100", then after assigning the table name of each target data table and the database name of the target database to the first variable and the second variable respectively, the resulting configuration information will be "tidb_mydb_test1", "tidb_mydb_test2", ..., "tidb_mydb_test10".

[0084] This application provides a possible implementation method in which the target data table includes at least one data row;

[0085] Each target data table will be synchronized to its corresponding target Kafka topic, including:

[0086] For each target data table, at least one data row in the target data table is encoded to obtain the encoding information corresponding to each data row;

[0087] The encoded information corresponding to each data row is merged to obtain the Kafka message corresponding to the target data table;

[0088] Store the Kafka messages corresponding to each target data table into the target Kafka topic corresponding to the target data table.

[0089] The target data table consists of at least one data row. In existing solutions, data row is typically synchronized as one message during data synchronization. However, this synchronization is slow. To improve the synchronization speed, this application embodiment encodes at least one data row in the target data table to obtain the encoding information corresponding to each data row. The encoding information corresponding to each data row is merged to obtain the Kafka message corresponding to the target data table. The Kafka message corresponding to each target data table is stored in the target Kafka topic corresponding to the target data table.

[0090] After synchronizing Kafka messages to a Kafka topic, the Kafka topic decodes the Kafka messages to obtain the target data table.

[0091] This application provides a possible implementation method for merging the encoding information corresponding to each data row, including:

[0092] Determine the order of the data rows and the length of each data row corresponding to its encoding information;

[0093] Each data length is encoded separately, and the header of the Kafka message is generated based on the encoding information corresponding to each data length.

[0094] The encoded information of each data row is sequentially linked to the header of the Kafka message based on the order of data transmission.

[0095] During the generation of a Kafka message, after encoding the data lines, it is necessary to determine the length of each data line corresponding to the encoding information, encode each data length separately, generate the Kafka message header based on the encoding information corresponding to each data length, and then concatenate the encoding information of each data line to the Kafka message header according to the order of the data lines.

[0096] Specifically, assuming the target data table includes three data rows: row1:1, row2:2, and row3:3, these three data rows are encoded, resulting in encoded information as row1_1, row2_2, and row3_3, respectively. The header of a Kafka message is generated based on the encoded information corresponding to each data length. The encoded information of each data row is then sequentially linked to the header of the Kafka message, resulting in the Kafka message header_row1_1_row2_2_row3_3, where the header includes the encoded information corresponding to each data length.

[0097] A Kafka message obtained by merging encoded data rows can be directly synchronized to the Kafka cluster.

[0098] This application provides a possible implementation method in which there is a mapping relationship between each data table and the corresponding target Kafka topic;

[0099] Each target data table is synchronized to its corresponding target Kafka topic, including:

[0100] Create and run a target thread, which is used to synchronize each target data table to the corresponding target Kafka topic based on the mapping relationship.

[0101] In this embodiment, after determining the target Kafka topic corresponding to each data table, each target data table is synchronized to the corresponding target Kafka topic. Specifically, a target thread can be created, which can synchronize each target data table to the corresponding target Kafka topic based on the mapping relationship between the target data table and the corresponding target Kafka topic. Compared with the prior art of creating multiple threads (threads corresponding to each data table), this embodiment only needs to create one target thread to synchronize each target data table to the corresponding target Kafka topic.

[0102] like Figure 3As shown, it exemplarily illustrates an application scenario diagram of a database synchronization method provided in this application embodiment. The target database "mydb" includes 100 target data tables, and the naming rule for these data tables is test1, test2, ..., test100. These data tables all need to be synchronized from the current target database to the Kafka cluster.

[0103] The configuration information set for each target data table using the initial regular expression tidb_{schema}_{table} is: "tidb_mydb_test1", "tidb_mydb_test2", ..., "tidb_mydb_test10".

[0104] In the Kafka cluster, synchronize test1 to the target Kafka topic named "tidb_mydb_test1", synchronize test2 to the target Kafka topic named "tidb_mydb_test2", ..., synchronize test100 to the target Kafka topic named "tidb_mydb_test100".

[0105] By identifying the target Kafka topics in the Kafka cluster that correspond to the types of target data tables, a clear classification of each data table in the Kafka cluster is achieved after synchronizing the target database to the Kafka cluster. This eliminates the need for consumers to perform additional classification work when reading data tables from the Kafka cluster.

[0106] This application provides a database synchronization apparatus, such as... Figure 4 As shown, the database synchronization device 40 may include:

[0107] The acquisition module 410 is used to acquire at least one target database to be synchronized to the Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store a type of data table;

[0108] The target Kafka topic determination module 420 is used to determine the target Kafka topic in the Kafka cluster that corresponds to the type of the target data table for each target data table.

[0109] The synchronization module 430 is used to synchronize each target data table to its corresponding target Kafka topic.

[0110] This application embodiment achieves a clear classification of data tables in the Kafka cluster after synchronizing the target database to the Kafka cluster by determining the target Kafka topic corresponding to the type of the target data table. This eliminates the need for additional classification work when consumers read data tables from the Kafka cluster. Furthermore, this application embodiment achieves a clear classification of data tables in the Kafka cluster without creating a thread for each data table, reducing the likelihood of human error and minimizing resource waste.

[0111] This application provides a possible implementation method in which the topic name of the Kafka topic includes the table name of the corresponding data table and the database name of the database to which the corresponding data table belongs;

[0112] The target Kafka topic determination module includes:

[0113] The configuration information setting submodule is used to set the configuration information of the target data table; the configuration information includes the table name of the target data table and the database name of the target database to which the target data table belongs;

[0114] The target Kafka topic determination submodule is used to determine the target Kafka topic if a Kafka topic corresponding to the configuration information of the target data table exists in the Kafka cluster.

[0115] This application provides a possible implementation method in which the target Kafka topic determination submodule is further configured to create a target Kafka topic corresponding to the target data table in the Kafka cluster if it is determined that there is no Kafka topic in the Kafka cluster that corresponds to the configuration information of the target data table.

[0116] This application embodiment provides a possible implementation, wherein the configuration information setting submodule includes:

[0117] The initial regular expression retrieval unit is used to retrieve the initial regular expression. The variables in the initial regular expression include a first variable and a second variable; the first variable represents the table name of the data table, and the second variable represents the database name to which the data table belongs.

[0118] The assignment unit is used to assign the name of the target data table to the first variable and the name of the target database to the second variable in the initial regular expression, thereby obtaining the target regular expression, which is then used as the configuration information for the target data table.

[0119] This application provides a possible implementation method in which the target data table includes at least one data row;

[0120] The synchronization module includes:

[0121] The first encoding submodule is used to encode at least one data row in each target data table to obtain the encoding information corresponding to each data row.

[0122] The merging submodule is used to merge the encoded information corresponding to each data row to obtain the Kafka message corresponding to the target data table.

[0123] The synchronization submodule is used to store the Kafka messages corresponding to each target data table into the target Kafka topic corresponding to the target data table.

[0124] This application embodiment provides a possible implementation method, in which the merged sub-modules include:

[0125] The sub-unit is used to determine the order of data rows and the length of each data row corresponding to its encoding information.

[0126] The second encoding subunit is used to encode each data length separately and generate the header of the Kafka message based on the encoding information corresponding to each data length.

[0127] The linking subunit is used to sequentially link the encoded information of each data row to the header of the Kafka message based on their order.

[0128] This application provides a possible implementation method in which there is a mapping relationship between each data table and the corresponding target Kafka topic;

[0129] The synchronization module also includes:

[0130] The target thread creation submodule is used to create and run a target thread, which is used to synchronize each target data table to the corresponding target Kafka topic based on the mapping relationship.

[0131] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0132] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a database synchronization method. Compared with related technologies, this method can achieve: obtaining at least one target database to be synchronized to a Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store a type of data table; for each target data table, determining the target Kafka topic in the Kafka cluster corresponding to the type of the target data table; and synchronizing each target data table to the corresponding target Kafka topic. This achieves a clear classification of the data tables in the Kafka cluster after synchronizing the target database to the Kafka cluster, so that consumers do not need to perform additional classification work when reading data tables from the Kafka cluster. Furthermore, this application embodiment does not require creating a thread corresponding to each data table to achieve a clear classification of the data tables in the Kafka cluster, reducing the risk of human error and minimizing resource waste.

[0133] In one alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0134] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0135] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0136] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0137] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0138] The electronic device package may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0139] This application provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program can implement the steps and corresponding content of the aforementioned method embodiments. Compared with the prior art, it can achieve: obtaining at least one target database to be synchronized to a Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store a type of data table; for each target data table, determining the target Kafka topic in the Kafka cluster corresponding to the type of the target data table; synchronizing each target data table to the corresponding target Kafka topic, thus achieving a clear classification of the data tables in the Kafka cluster after synchronizing the target database to the Kafka cluster, so that consumers do not need to perform additional classification work when reading data tables from the Kafka cluster. In addition, this application embodiment does not require creating a thread corresponding to each data table to achieve a clear classification of the data tables in the Kafka cluster, which reduces the introduction of human error and reduces resource waste.

[0140] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0141] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments. Compared with the prior art, it can achieve: obtaining at least one target database to be synchronized to a Kafka cluster; the target database includes at least one target data table; the Kafka cluster includes at least one Kafka topic; each Kafka topic can store a type of data table; for each target data table, determining the target Kafka topic in the Kafka cluster corresponding to the type of the target data table; and synchronizing each target data table to the corresponding target Kafka topic. This achieves a clear classification of the data tables in the Kafka cluster after synchronizing the target database to the Kafka cluster, so that consumers do not need to perform additional classification work when reading data tables from the Kafka cluster. In addition, this application embodiment does not require creating a thread corresponding to each data table to achieve a clear classification of the data tables in the Kafka cluster, which reduces the introduction of human error and reduces resource waste.

[0142] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0143] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0144] The above are only optional implementation methods for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A method of database synchronization, characterized by, The application relates to a method for synchronizing target databases to a kafka cluster. The method comprises the following steps: acquiring at least one target database to be synchronized to a kafka cluster; the target database comprises at least one target data table; the kafka cluster comprises at least one kafka topic; each kafka topic can store one kind of data table; the kind is used for representing the table name of the data table and the library name of the database; for each target data table, determining a target kafka topic corresponding to the kind of the target data table in the kafka cluster based on configuration information of the target data table; the configuration information comprises the table name of the target data table and the library name of the target database to which the target data table belongs; synchronizing each target data table to the corresponding target kafka topic respectively; wherein a mapping relationship exists between each data table and the corresponding target kafka topic; the step of synchronizing each target data table to the corresponding target kafka topic respectively further comprises:

2. The method of claim 1, wherein, creating and running a target thread, which is used for synchronizing each target data table to the corresponding target kafka topic based on the mapping relationship. The topic name of the kafka topic comprises the table name of the corresponding data table and the library name of the database to which the corresponding data table belongs; the step of determining the target kafka topic corresponding to the kind of the target data table in the kafka cluster comprises: setting the configuration information of the target data table; 3. The method of claim 2, wherein, if it is determined that a kafka topic corresponding to the configuration information of the target data table exists in the kafka cluster, the kafka topic is taken as the target kafka topic corresponding to the target data table. after the step of setting the configuration information of the target data table, the method further comprises:

4. The method according to any one of claims 2-3, characterized in that, if it is determined that a kafka topic corresponding to the configuration information of the target data table does not exist in the kafka cluster, a target kafka topic corresponding to the target data table is created in the kafka cluster. the step of setting the configuration information of the target data table comprises: acquiring an initial regular expression; variables in the initial regular expression comprise a first variable and a second variable; the first variable represents the table name of a data table, and the second variable represents the library name of the database to which the data table belongs; 5. The method of claim 1, wherein, in the initial regular expression, the table name of the target data table is assigned to the first variable, and the library name of the target database is assigned to the second variable, so as to obtain a target regular expression; the target regular expression is taken as the configuration information of the target data table. the target data table comprises at least one data row; the step of synchronizing each target data table to the corresponding target kafka topic respectively comprises: for each target data table, encoding at least one data row in the target data table to obtain corresponding encoding information of each data row; merging the corresponding encoding information of each data row to obtain a kafka message corresponding to the target data table. Store the corresponding kafka message of each target data table to the corresponding target kafka topic respectively.

6. The method of claim 5, wherein, The merging of the encoding information corresponding to each data row comprises: determining the sequence of each data row and the data length corresponding to the encoding information of each data row; encoding each data length, and generating the header of the kafka message according to the encoding information corresponding to each data length; linking the encoding information of each data row to the header of the kafka message in sequence based on the sequence.

7. An apparatus for database synchronization, the apparatus comprising: Applied to a sending end, comprising: an acquisition module, configured to acquire at least one target database to be synchronized to a kafka cluster; the target database comprises at least one target data table; the kafka cluster comprises at least one kafka topic; each kafka topic can store one kind of data table; the kind is used to represent the table name of the data table and the library name of the database to which the data table belongs; a target kafka topic determination module, configured to, for each target data table, determine the target kafka topic corresponding to the kind of the target data table in the kafka cluster based on the configuration information of the target data table; the configuration information comprises the table name of the target data table and the library name of the target database to which the target data table belongs; a synchronization module, configured to synchronize each target data table to the corresponding target kafka topic respectively; wherein, there is a mapping relationship between each data table and the corresponding target kafka topic; the synchronization module can also be configured to: create and run a target thread, and the target thread is configured to synchronize each target data table to the corresponding target kafka topic based on the mapping relationship.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 7. The processor executes the computer program to implement the steps of the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-6.

Images