Database master-slave synchronization method, device, equipment and readable storage medium

Abstract

The present application provides a database master-slave synchronization method, device, equipment and readable storage medium, the method comprising: from the multiple transaction data returned from the main library to the pseudo-slave library, the transaction data are written into the message queue one by one from the earliest to the latest in the return order, and after each message is written, the latest executed transaction number of the pseudo-slave library is updated to the transaction number of the transaction data in the message, wherein the pseudo-slave library is connected to the main library, and the latest executed transaction number of the pseudo-slave library is used to request the main library to return the transaction data, and its initial value is the latest executed transaction number of the main library obtained when the main library is first configured; for each slave library connected to the pseudo-main library, the backlog of messages is consumed from the message queue according to the consumer group ID of the slave library, and the transaction data in the message is returned to the slave library through the pseudo-main library, wherein the consumer group IDs of different slave libraries are different. Through this application, the consumption of IO resources and network bandwidth is greatly reduced, and the performance of the main library is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database master-slave synchronization method, device, equipment and readable storage medium. Background Art

[0002] Master-slave replication is an asynchronous replication process. At least one slave database copies the transaction log from the master database, parses the log and applies it to itself, and finally synchronizes the data between the master and the slave database. At present, the synchronization method of a database system with one master and multiple slaves is that multiple slave databases are directly connected to the master database to replicate the transaction log. For the master database, each additional slave database will occupy the IO resources and network bandwidth of the master database, affecting the performance of the master database. Summary of the invention

[0003] The present application provides a database master-slave synchronization method, device, equipment and readable storage medium, which can solve the technical problem in the prior art that the increase in the number of slave libraries affects the performance of the master library.

[0004] In a first aspect, an embodiment of the present application provides a database master-slave synchronization method, the database master-slave synchronization method comprising:

[0005] Among the multiple transaction data returned from the master database to the pseudo-slave database, write the transaction data into the message queue one by one from the earliest to the latest in the return order. After each message is written, update the latest executed transaction number of the pseudo-slave database to the transaction number of the transaction data in the message. The pseudo-slave database is connected to the master database, and the latest executed transaction number of the pseudo-slave database is used to request the master database to return the transaction data. Its initial value is the latest executed transaction number of the master database obtained when the master database is first configured;

[0006] For each slave library connected to the pseudo master library, the backlog of messages is consumed from the message queue according to the slave library's consumer group ID, and the transaction data in the message is returned to the slave library through the pseudo master library, where different slave libraries have different consumer group IDs.

[0007] Furthermore, in one embodiment, the database master-slave synchronization method further includes:

[0008] After the first database as the master database goes down, the write traffic of the first database is cut off, and the connection between the pseudo slave database and the first database is cut off;

[0009] Select a second database from the normal database;

[0010] After the message backlog of the consumer group corresponding to the second database is zero, the connection between the second database and the pseudo master database is cut off, the second database is switched to the master database mode, the pseudo slave database is connected to the second database, the read traffic of the second database is cut off, and the write traffic is connected to the second database.

[0011] Furthermore, in one embodiment, after the step of cutting off the connection between the pseudo slave database and the first database, the following step is further included:

[0012] After the first database returns to normal, the first database is switched to slave database mode, and the first database is connected to the pseudo master database;

[0013] When the message backlog of the consumer group corresponding to the first database is less than a preset threshold, the read traffic is connected to the first database.

[0014] Furthermore, in one embodiment, the step of selecting the second database from the normal database includes:

[0015] Use the slave database that is in normal state and has the smallest message backlog for the corresponding consumer group as the second database; or

[0016] Monitor the message backlog of the consumer group corresponding to the slave database in normal state, and use the slave database corresponding to the consumer group whose message backlog becomes zero first as the second database.

[0017] Furthermore, in one embodiment, the database master-slave synchronization method further includes:

[0018] After the third database as the slave database goes down, cut off the read traffic of the third database;

[0019] When the third database returns to normal and the message backlog of the corresponding consumer group is less than the preset threshold, the read traffic is connected to the third database.

[0020] Furthermore, in one embodiment, before the step of consuming the backlogged messages from the message queue according to the consumer group ID of the slave library for each slave library connected to the pseudo master library, and returning the transaction data in the message to the slave library through the pseudo master library, the step further includes:

[0021] For each slave library, when the slave library is connected to the pseudo master library, the corresponding consumer group ID is generated according to the server ID of the slave library.

[0022] Furthermore, in one embodiment, the master database and the slave database are one of MySQL or PostgreSQL, and the message queue is one of Kafka, RabbitMQ or RocketMQ.

[0023] In a second aspect, an embodiment of the present application further provides a database master-slave synchronization device, the database master-slave synchronization device comprising:

[0024] A write update module is used to write the transaction data from the master database to the pseudo slave database one by one in the message queue in the order of return from the earliest to the latest. After each message is written, the latest executed transaction number of the pseudo slave database is updated to the transaction number of the transaction data in the message. The pseudo slave database is connected to the master database, and the latest executed transaction number of the pseudo slave database is used to request the master database to return the transaction data. Its initial value is the latest executed transaction number of the master database obtained when the master database is first configured.

[0025] The consumption return module is used to consume the backlog of messages from the message queue according to the consumer group ID of each slave library connected to the pseudo master library, and return the transaction data in the message to the slave library through the pseudo master library, where the consumer group IDs of different slave libraries are different.

[0026] In a third aspect, an embodiment of the present application also provides a database master-slave synchronization device, which includes a processor, a memory, and a database master-slave synchronization program stored in the memory and executable by the processor, wherein when the database master-slave synchronization program is executed by the processor, the steps of the above-mentioned database master-slave synchronization method are implemented.

[0027] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, on which a database master-slave synchronization program is stored, wherein when the database master-slave synchronization program is executed by a processor, the steps of the above-mentioned database master-slave synchronization method are implemented.

[0028] In this application, after receiving the transaction data returned by the master library, the pseudo slave library writes it into the message queue one by one to achieve data persistence. The pseudo master library consumes the backlogged messages from the message queue in order according to the consumer group ID of the slave library and returns the transaction data therein to the slave library. Through this application, the message queue has the ability to have multiple copies of data, ensuring the real-time, stability and fault tolerance of data synchronization from the master library to each slave library, and no matter how many slave libraries there are, the master library only needs to maintain the connection with the pseudo slave library, thereby greatly reducing the consumption of IO resources and network bandwidth and ensuring the performance of the master library. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of a database master-slave synchronization method in one embodiment of the present application;

[0030] Figure 2 This is a connection diagram of a database master-slave synchronization system in an embodiment of the present application;

[0031] Figure 3 for Figure 2 The connection diagram of the database master-slave synchronization system after the master database failure is handled;

[0032] Figure 4This is a schematic diagram of the hardware structure of a database master-slave synchronization device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0035] In a first aspect, an embodiment of the present application provides a database master-slave synchronization method.

[0036] Figure 1 A schematic diagram of the process of a master-slave synchronization method of a database in an embodiment of the present application is shown; Figure 2 A connection diagram of a database master-slave synchronization system in an embodiment of the present application is shown.

[0037] Reference Figure 1 and Figure 2 In one embodiment, the database master-slave synchronization method includes the following steps:

[0038] S11. Among the multiple transaction data returned from the main database to the pseudo-slave database, write the transaction data into the message queue one by one from the earliest to the latest in the return order. After each message is written, update the latest executed transaction number of the pseudo-slave database to the transaction number of the transaction data in the message. The pseudo-slave database is connected to the main database, and the latest executed transaction number of the pseudo-slave database is used to request the main database to return the transaction data. Its initial value is the latest executed transaction number of the main database obtained when the main database is first configured.

[0039] In this embodiment, the pseudo slave database is connected to the master database in the same way as a normal slave database, that is, the slave database mode is enabled and the master database information is configured, such as the IP address of the master database, the user name and password for data synchronization, etc. Figure 2In the embodiment shown, the IP address in the main library information of the pseudo slave library configuration is IP1, that is, the IP address of the real main library. After each startup (including normal startup, fault restart, etc.) of the pseudo slave library, it is necessary to send a request to the main library, and the request contains the latest executed transaction number of the pseudo slave library. After receiving the request, the main library continues to obtain the transaction data after the transaction number from the transaction log, and returns the transaction data to the pseudo slave library from front to back according to the arrangement order in the transaction log. Wherein, the transaction number is included in the transaction data, and has a one-to-one correspondence with the transaction data. When the main library returns the transaction data to the pseudo slave library, it can be returned one by one or several at a time.

[0040] Unlike ordinary slaves, the transaction data received by the pseudo-slave is not persisted locally, but written to the message queue to achieve data persistence, and multiple copies of the message queue data are used to achieve high availability. Transaction data is written to the message queue one by one from early to late in the order in which the transaction data is returned, so that the order of each message in the message queue is consistent with the order of the corresponding transaction data in the transaction log of the master. By updating the latest executed transaction number of the pseudo-slave after the message is written, the transaction number can feedback the latest recorded transaction data in the pseudo-slave, and then the master can determine the starting point of returning transaction data each time a request is sent.

[0041] It can be understood that when the master library is configured for the first time, no message has been written to the message queue. The initial value of the latest executed transaction number of the pseudo slave library is configured as the latest executed transaction number of the master library at that time. After that, the transaction number will be automatically updated as the message is written.

[0042] S12. For each slave library connected to the pseudo master library, consume the backlogged messages from the message queue according to the consumer group ID of the slave library, and return the transaction data in the message to the slave library through the pseudo master library, wherein the consumer group IDs of different slave libraries are different.

[0043] In this embodiment, the way the slave database connects to the pseudo master database is the same as that of the ordinary master database, that is, the slave database mode is enabled and the master database information is configured. For the slave database, the pseudo master database is the master database, so the IP address in the master database information is the IP address of the pseudo master database, and the user name and password are consistent with the information configured when the pseudo slave database is connected to the master database. Figure 2 In the embodiment shown, the IP address in the master library information configured by each slave library is IP2, that is, the IP address of the pseudo master library. After each startup, the slave library will also send a request to the pseudo master library, and the request includes the latest executed transaction number of the slave library.

[0044] Unlike an ordinary master library, a pseudo master library does not obtain the transaction data to be returned from the transaction log of the local disk, but consumes the backlog of messages from the message queue to obtain the corresponding transaction data from the message. Each slave library has its own consumer group ID, and each consumer group has its own offset, which is used to record the consumption progress of the consumer group. Therefore, according to the consumer group ID of the slave library, it can be determined in which message the next transaction data to be returned is, and the backlog of messages is consumed from front to back according to the arrangement order in the message queue, and the transaction data therein is returned to the slave library. The transaction data received by the slave library is normally persisted in the form of a transaction log. When the pseudo master library returns transaction data to the slave library, it can return one by one or several at a time.

[0045] In addition, for the slave database, although the request sent to the pseudo master database contains the latest executed transaction number at that time, the pseudo master database does not actually use the transaction number when returning transaction data, but uses the slave database's consumer group ID. The latest executed transaction number sent by the slave database to the pseudo master database is redundant information in this solution and can be used to verify whether the offset of the consumer group is accurate.

[0046] Therefore, in this embodiment, after receiving the transaction data returned by the master library, the pseudo slave library writes it into the message queue one by one to achieve data persistence. The pseudo master library consumes the backlogged messages from the message queue in order according to the consumer group ID of the slave library and returns the transaction data therein to the slave library. Through this embodiment, the message queue has the ability to have multiple copies of data, ensuring the real-time, stability and fault tolerance of data synchronization from the master library to each slave library, and no matter how many slave libraries there are, the master library only needs to maintain the connection with the pseudo slave library, thereby greatly reducing the consumption of IO resources and network bandwidth, and ensuring the performance of the master library.

[0047] Optionally, the master database and the slave database are one of MySQL or PostgreSQL, and the message queue is one of Kafka, RabbitMQ or RocketMQ. It should be noted that the master database and the slave database need to be the same model of database, that is, the master database and the slave database are both MySQL, or the master database and the slave database are both PostgreSQL.

[0048] Specifically, when the master and slave databases are MySQL, the transaction log is binlog, the transaction number is GTID (GlobalTransaction ID), and the command to query the latest executed transaction number is "show masterstatus". When the master and slave databases are PostgreSQL, the transaction log is xlog, the transaction number is LSN (Log Sequence Number), and the command to query the latest executed transaction number is "SELECT pg_current_wal_lsn()".

[0049] Optionally, when writing transaction data to a message queue, the transaction number may be used as a key or tag of the transaction data for easy query.

[0050] Specifically, when the message queue is Kafka, the transaction number is used as the key; when the message queue is RabbitMQ, the transaction number is used as the routekey; when the message queue is RocketMQ, the transaction number is used as the tag.

[0051] Furthermore, in one embodiment, before the step of consuming the backlogged messages from the message queue according to the consumer group ID of the slave library for each slave library connected to the pseudo master library, and returning the transaction data in the message to the slave library through the pseudo master library, the step further includes:

[0052] For each slave library, when the slave library is connected to the pseudo master library, the corresponding consumer group ID is generated according to the server ID of the slave library.

[0053] In this embodiment, different server IDs of the slave libraries are used to facilitate the generation of different consumer group IDs. Of course, in other embodiments, the consumer group ID can also be generated in other ways, such as random generation, generation based on connection time, etc., to ensure that the consumer group IDs of different slave libraries are different and can be queried based on the unique identification information of the slave library (such as server ID).

[0054] Figure 3 Shows Figure 2 Connection diagram of the database master-slave synchronization system after the master database failure is handled.

[0055] Reference Figure 2 and Figure 3 Further, in one embodiment, the database master-slave synchronization method further includes:

[0056] After the first database as the master database goes down, the write traffic of the first database is cut off, and the connection between the pseudo slave database and the first database is cut off;

[0057] Select a second database from the normal database;

[0058] After the message backlog of the consumer group corresponding to the second database is zero, the connection between the second database and the pseudo master database is cut off, the second database is switched to the master database mode, the pseudo slave database is connected to the second database, the read traffic of the second database is cut off, and the write traffic is connected to the second database.

[0059] In this embodiment, the master-slave database adopts a read-write separation solution, the master database is used to receive write traffic, and each slave database is used to receive read traffic. The transaction logs of the master database and the slave database are completely consistent. After the master database goes down, the slave database in normal state can run as the new master database to ensure the normal operation of the read and write services.

[0060] It should be noted that the prerequisite for the second database to replace the first database as the new primary database is that the transaction data in the first database is written to the message queue, and the data in the message queue is written to the second database. The former can be guaranteed by configuring enhanced semi-synchronization, and the latter is determined by whether the message backlog of the consumer group is zero.

[0061] Specific to Figure 2 and Figure 3 In the illustrated embodiment, the IP address of the first database is IP1, the IP address of the second database is IP3, and the specific operation of connecting the pseudo slave database to the second database includes changing the IP address in the master database information configured by the pseudo slave database from IP1 to IP3.

[0062] Furthermore, in one embodiment, after the step of cutting off the connection between the pseudo slave database and the first database, the following step is further included:

[0063] After the first database returns to normal, the first database is switched to slave database mode, and the first database is connected to the pseudo master database;

[0064] When the message backlog of the consumer group corresponding to the first database is less than a preset threshold, the read traffic is connected to the first database.

[0065] In this embodiment, after the first database is restored to normal, it continues to run as a new slave database, further ensuring the normal operation of the read service. During the downtime of the first database, the transaction log of the first database lacks the transaction data added during the downtime compared to the message queue. Therefore, the corresponding consumer group will have a certain backlog of messages. If the message backlog is too large, it will cause the difference in the read services provided by different slave databases to be too large, so it is necessary to wait until the message backlog is less than the preset threshold before resuming the read service.

[0066] Specific to Figure 2 and Figure 3In the illustrated embodiment, the specific operation of connecting the first database to the pseudo master database includes setting the IP address in the master database information configured for the first database to IP2. In addition, if a consumer group has not been created for the first database in the message queue, a new consumer group needs to be created and the consumer group ID needs to be determined, and the offset of the consumer group needs to be determined according to the latest executed transaction number of the first database.

[0067] As an optional implementation, the step of selecting the second database from the slave database in normal state includes: selecting the slave database in normal state and corresponding to the smallest message backlog of the consumer group as the second database. In this way, the second database can be quickly selected. Since the message backlog of the consumer group corresponding to the second database is the smallest, the second database is most likely to consume all messages in the message queue first, and then quickly start to execute the subsequent master database switching operation.

[0068] As another optional implementation, the step of selecting the second database from the slave database in normal state includes: monitoring the message backlog of the consumer group corresponding to the slave database in normal state, and selecting the slave database corresponding to the consumer group whose message backlog becomes zero earliest as the second database. In this way, it can be ensured that the database that completes consumption first is selected as the second database.

[0069] It can be understood that the former solution only needs to obtain the message backlog of all normal slaves once before selection, and only needs to monitor the message backlog of the second database after selection, while the latter solution needs to continuously monitor the message backlog of all normal slaves before selection. The latter solution requires more monitoring overhead, but it can ensure that the master database switching operation is executed as soon as possible.

[0070] Furthermore, in one embodiment, the database master-slave synchronization method further includes:

[0071] After the third database as the slave database goes down, cut off the read traffic of the third database;

[0072] When the third database returns to normal and the message backlog of the corresponding consumer group is less than the preset threshold, the read traffic is connected to the third database.

[0073] In this embodiment, the read traffic of the third database is cut off after the third database goes down to ensure the normal operation of the read service. Similarly, for the recovery operation of the first database, the third database also needs to wait until the message backlog is less than the preset threshold before resuming the read service to avoid too large differences in the read services provided by different slaves.

[0074] In a second aspect, an embodiment of the present application also provides a database master-slave synchronization device.

[0075] In one embodiment, the database master-slave synchronization device includes:

[0076] A write update module is used to write the transaction data from the master database to the pseudo slave database one by one in the message queue in the order of return from the earliest to the latest. After each message is written, the latest executed transaction number of the pseudo slave database is updated to the transaction number of the transaction data in the message. The pseudo slave database is connected to the master database, and the latest executed transaction number of the pseudo slave database is used to request the master database to return the transaction data. Its initial value is the latest executed transaction number of the master database obtained when the master database is first configured.

[0077] The consumption return module is used to consume the backlog of messages from the message queue according to the consumer group ID of each slave library connected to the pseudo master library, and return the transaction data in the message to the slave library through the pseudo master library, where the consumer group IDs of different slave libraries are different.

[0078] Furthermore, in one embodiment, the database master-slave synchronization device further includes a fault processing module, which is used to:

[0079] After the first database as the master database goes down, the write traffic of the first database is cut off, and the connection between the pseudo slave database and the first database is cut off;

[0080] Select a second database from the normal database;

[0081] After the message backlog of the consumer group corresponding to the second database is zero, the connection between the second database and the pseudo master database is cut off, the second database is switched to the master database mode, the pseudo slave database is connected to the second database, the read traffic of the second database is cut off, and the write traffic is connected to the second database.

[0082] Furthermore, in one embodiment, the fault processing module is also used to:

[0083] After the first database returns to normal, the first database is switched to slave database mode, and the first database is connected to the pseudo master database;

[0084] When the message backlog of the consumer group corresponding to the first database is less than a preset threshold, the read traffic is connected to the first database.

[0085] Furthermore, in one embodiment, the fault handling module is used to:

[0086] Use the slave database that is in normal state and has the smallest message backlog for the corresponding consumer group as the second database; or

[0087] Monitor the message backlog of the consumer group corresponding to the slave database in normal state, and use the slave database corresponding to the consumer group whose message backlog becomes zero first as the second database.

[0088] Furthermore, in one embodiment, the database master-slave synchronization device further includes a fault processing module, which is used to:

[0089] After the third database as the slave database goes down, cut off the read traffic of the third database;

[0090] When the third database returns to normal and the message backlog of the corresponding consumer group is less than the preset threshold, the read traffic is connected to the third database.

[0091] Furthermore, in one embodiment, the database master-slave synchronization device further includes a consumer group creation module, which is used to:

[0092] For each slave library, when the slave library is connected to the pseudo master library, the corresponding consumer group ID is generated according to the server ID of the slave library.

[0093] Furthermore, in one embodiment, the master database and the slave database are one of MySQL or PostgreSQL, and the message queue is one of Kafka, RabbitMQ or RocketMQ.

[0094] Among them, the functional implementation of each module in the above-mentioned database master-slave synchronization device corresponds to the steps in the above-mentioned database master-slave synchronization method embodiment, and its functions and implementation processes are no longer repeated here.

[0095] In a third aspect, an embodiment of the present application provides a database master-slave synchronization device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0096] Figure 4 A schematic diagram of the hardware structure of a database master-slave synchronization device involved in an embodiment of the present application is shown.

[0097] Reference Figure 4 In an embodiment of the present application, the database master-slave synchronization device may include a processor, a memory, a communication interface, and a communication bus.

[0098] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0099] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the database master-slave synchronization device, and the interface used to realize the interconnection between the database master-slave synchronization device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0100] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0101] The processor may be a general-purpose processor, and the general-purpose processor may call the database master-slave synchronization program stored in the memory and execute the database master-slave synchronization method provided by the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the database master-slave synchronization program is called may refer to the various embodiments of the database master-slave synchronization method of the present application, and will not be repeated here.

[0102] Those skilled in the art will understand that Figure 4 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0103] In a fourth aspect, an embodiment of the present application also provides a readable storage medium.

[0104] The readable storage medium of the present application stores a database master-slave synchronization program, wherein when the database master-slave synchronization program is executed by a processor, the steps of the above-mentioned database master-slave synchronization method are implemented.

[0105] Among them, the method implemented when the database master-slave synchronization program is executed can refer to the various embodiments of the database master-slave synchronization method of the present application, and will not be repeated here.

[0106] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0107] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0108] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0109] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0110] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0112] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A database master-slave synchronization method, characterized in that: The database master-slave synchronization method comprises: Among the multiple transaction data returned from the master database to the pseudo slave database, the transaction data are written into the message queue one by one from the earliest to the latest in the return order. After each message is written, the latest executed transaction number of the pseudo slave database is updated to the transaction number of the transaction data in the message. The pseudo slave database enables the slave database mode and configures the master database information to the information corresponding to the master database to connect to the master database. The latest executed transaction number of the pseudo slave database is used to request the master database to return the transaction data, and its initial value is the latest executed transaction number of the master database obtained when the master database is first configured; For each slave library connected to the pseudo master library, the backlog of messages is consumed from the message queue according to the slave library's consumer group ID, and the transaction data in the message is returned to the slave library through the pseudo master library, so that the slave library can persist the data in the form of a transaction log after receiving the transaction data. The master library and the slave library are databases of the same model. The slave library enables the slave library mode and configures the master library information to the information corresponding to the pseudo master library to connect to the pseudo master library. Different slave libraries have different consumer group IDs.

2. The database master-slave synchronization method according to claim 1, characterized in that: The database master-slave synchronization method also includes: After the first database as the master database goes down, the write traffic of the first database is cut off, and the connection between the pseudo slave database and the first database is cut off; Select a second database from the normal database; After the message backlog of the consumer group corresponding to the second database is zero, the connection between the second database and the pseudo master database is cut off, the second database is switched to the master database mode, the pseudo slave database is connected to the second database, the read traffic of the second database is cut off, and the write traffic is connected to the second database.

3. The database master-slave synchronization method according to claim 2, characterized in that: After the step of cutting off the connection between the pseudo slave database and the first database, the method further includes: After the first database returns to normal, the first database is switched to slave database mode, and the first database is connected to the pseudo master database; When the message backlog of the consumer group corresponding to the first database is less than a preset threshold, the read traffic is connected to the first database.

4. The database master-slave synchronization method according to claim 2, characterized in that: The step of selecting the second database from the normal state database includes: Use the slave database that is in normal state and has the smallest message backlog for the corresponding consumer group as the second database; or Monitor the message backlog of the consumer group corresponding to the slave database in normal state, and use the slave database corresponding to the consumer group whose message backlog becomes zero first as the second database.

5. The database master-slave synchronization method according to claim 1, characterized in that: The database master-slave synchronization method also includes: After the third database as the slave database goes down, cut off the read traffic of the third database; When the third database returns to normal and the message backlog of the corresponding consumer group is less than the preset threshold, the read traffic is connected to the third database.

6. The database master-slave synchronization method according to any one of claims 1 to 5, characterized in that: Before the step of consuming the backlogged messages from the message queue according to the consumer group ID of the slave library for each slave library connected to the pseudo master library, and returning the transaction data in the message to the slave library through the pseudo master library, the method further includes: For each slave library, when the slave library is connected to the pseudo master library, the corresponding consumer group ID is generated according to the server ID of the slave library.

7. The database master-slave synchronization method according to any one of claims 1 to 5, characterized in that: The master and slave databases are either MySQL or PostgreSQL, and the message queue is either Kafka, RabbitMQ, or RocketMQ.

8. A database master-slave synchronization device, characterized in that: The database master-slave synchronization device comprises: A write update module is used to write the transaction data from the master database to the pseudo slave database one by one in the message queue according to the return order from the earliest to the latest. After each message is written, the latest executed transaction number of the pseudo slave database is updated to the transaction number of the transaction data in the message. The pseudo slave database enables the slave database mode and configures the master database information to the information corresponding to the master database to connect to the master database. The latest executed transaction number of the pseudo slave database is used to request the master database to return the transaction data. Its initial value is the latest executed transaction number of the master database obtained when the master database is first configured. The consumption return module is used to consume the backlog of messages from the message queue according to the consumer group ID of each slave library connected to the pseudo master library, and return the transaction data in the message to the slave library through the pseudo master library, so that the slave library can persist the data in the form of transaction log after receiving the transaction data. The master library and the slave library are databases of the same model. The slave library enables the slave library mode and configures the master library information to the information corresponding to the pseudo master library to connect to the pseudo master library. Different slave libraries have different consumer group IDs.

9. A database master-slave synchronization device, characterized in that: The database master-slave synchronization device includes a processor, a memory, and a database master-slave synchronization program stored in the memory and executable by the processor, wherein when the database master-slave synchronization program is executed by the processor, the steps of the database master-slave synchronization method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a database master-slave synchronization program, wherein when the database master-slave synchronization program is executed by a processor, the steps of the database master-slave synchronization method according to any one of claims 1 to 7 are implemented.

Images