A method and apparatus for wide area network data replication transmission

Abstract

The application provides a wide area network data replication transmission method and device, comprising the following steps: creating a transmission channel between a master database and a standby database, and establishing a transmission heartbeat mechanism for ensuring transmission connection based on the transmission channel; establishing a check heartbeat mechanism for ensuring data integrity between the master database and the standby database; and transmitting a log file from the master database to the standby database. The transmission method can solve the problems of frequent establishment and disconnection of communication between databases, resulting in waste of time and resources, and inability to guarantee the integrity and effectiveness of transmitted data, and can continuously guarantee effective transmission of data and reliability of data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network communication, in particular to a wide area network data replication transmission method and device. BACKGROUND

[0002] With the digital transformation of financial services, data security and reliability become crucial, and regulatory authorities require financial institutions to develop and implement disaster recovery and backup plans to address risks and threats, ensuring business continuity and data security. The specific measures are to back up data to another device or data center in a different geographic location to quickly recover systems and data when the local data center is damaged or unavailable.

[0003] However, in practical applications, when replicating and synchronizing database data between multiple servers in different geographic locations, there are problems of data transmission loss and transmission channel abnormalities, resulting in low data transmission efficiency, and the reliability and integrity of data transmission cannot be guaranteed.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] Therefore, the present application discloses a wide area network data replication transmission method and device to solve the problem of frequent communication between databases, resulting in waste of time and resources, and the integrity and effectiveness of transmitted data cannot be guaranteed. The transmission channel can be replaced in time when the transmission channel is abnormal, and the effective transmission of data and the reliability of data can be continuously guaranteed.

[0006] Specifically, the present application is realized by the following technical solutions:

[0007] In a first aspect, the present application discloses a wide area network data replication transmission method, comprising the following steps:

[0008] A transmission channel is created between the primary database and the standby database, and a transmission heartbeat mechanism for ensuring transmission connection is established based on the transmission channel;

[0009] A check heartbeat mechanism for ensuring data integrity is established between the primary database and the standby database;

[0010] The primary database transmits log files to the standby database.

[0011] The application adopts two levels of heartbeat mechanisms when data replication is performed, one is a handshake and communication at the transmission and receiving level, i.e. a transmission heartbeat mechanism, which guarantees the persistent establishment of communication connection between databases, can reduce the time and resource consumption of frequent connection establishment and disconnection, thereby improving the data transmission efficiency; the other level is a check heartbeat mechanism established between the primary database and the standby database, which is used for listening to heartbeat information, and through checking the heartbeat and displacement relationship between the primary database and the standby database, the integrity and effectiveness of the transmitted data are double guaranteed.

[0012] Preferably, before the primary database transmits the log file to the standby database, the primary database and the standby database are full-synchronized, and the standby database records the last displacement point synchronized. After the primary database sends all the data to the standby database to complete the full-synchronization, the last displacement point recorded by the standby database can facilitate the synchronization starting from the displacement point when the incremental synchronization is performed later.

[0013] Preferably, the check heartbeat mechanism comprises that after the standby database receives the log file, feedback data is sent to the primary database, and the primary database ensures the data integrity and effectiveness by checking the feedback data.

[0014] The feedback data comprises the serial number and displacement information of the log file.

[0015] Preferably, when the primary database checks that the feedback data is wrong, the log file is retransmitted to the standby database, and after the standby database receives the log file, the file replacement is realized according to the serial number and displacement information of the log file, and the feedback data after the replacement is sent to the primary database.

[0016] The standby database only needs to return the serial number and displacement information of the received log file through the transmission channel to enable the primary database to check, and the data volume of the communication package is very small, which does not occupy the bandwidth. Even when the standby database receives the data incorrectly, the incorrect log file can be replaced according to the serial number and displacement information of the retransmitted log file, and the whole log file after the error occurs does not need to be retransmitted.

[0017] Preferably, the transmission heartbeat mechanism comprises that the primary database sends a heartbeat package to the standby database at a regular time, and receives the heartbeat package returned by the standby database to confirm that the transmission channel connection is normal.

[0018] The transmission channel created between the primary database and the standby database is several, when the primary database detects that the transmission channel connection is abnormal, the primary database switches to a new transmission channel for transmission, and the standby database remains normal communication.

[0019] When the transmission channel is abnormal, the master database can discover the abnormality in time through the heartbeat packet in the transmission heartbeat mechanism and switch to a new transmission channel, while the standby database continues to communicate when the channel is abnormal. In this way, when the channel is not connected for a few times, the database will perform multiple retry operations at the TCP protocol level and will not directly replace the transmission channel. Even if the transmission channel is abnormal for a short time, since the standby database is insensitive to communication, it can ensure that the received log files are not missing. After replacing the new transmission channel, the log files retransmitted by the master database contain relevant sequence numbers and displacement information, which can ensure that the log files received by the standby database are not repeated or misaligned, thereby ensuring the completeness and reliability of the transmitted data. In addition, the above mechanism can reduce the time and resource consumption of frequent connection establishment and disconnection, and improve the data transmission efficiency.

[0020] Preferably, the step of transmitting the log file from the master database to the standby database comprises: the master database asynchronously buffers the log file to a local port when writing the log file, and the local port transmits the log file to a remote port of the standby database.

[0021] Preferably, the local port transmits the log file to the remote port after compression processing, and the remote port transmits the log file to the standby database in a streaming manner after decompression.

[0022] The compression processing adopts a compression algorithm based on the TCP protocol, and the compression algorithm and the length of the compressed data are added to the header of the protocol.

[0023] In the prior art, network transmission equipment manufacturers often provide log compression and transmission schemes suitable for their own database types, which rely too much on the database bottom layer in use. The compression capabilities of databases produced by different manufacturers are different, making such log compression and transmission schemes lack universality. In addition, the network layer byte-level transmission compression technology used in the prior art is application-independent, which indiscriminately compresses any traffic, affects normal communication, and has unpredictable effects on the upper layer application.

[0024] The present application adopts a universal log compression and transmission scheme based on open source algorithms and protocols, which supports database independence and can seamlessly support cross-data center disaster recovery for most database products. On the other hand, the present application adopts an application layer-based compression and transmission scheme, which transmits data between ports and can avoid affecting the upper layer application, thereby improving the stability and performance of the application.

[0025] In a second aspect, the present application discloses a transmission device of a wide area network data replication transmission method, comprising:

[0026] Transmission heartbeat mechanism module: used to create a transmission channel between the primary database and the backup database, and to establish a transmission heartbeat mechanism based on the transmission channel to ensure transmission connection;

[0027] Verification heartbeat mechanism module: Used to establish a verification heartbeat mechanism between the primary database and the backup database to ensure data integrity;

[0028] Transmission module: Used to transfer log files from the primary database to the backup database.

[0029] Thirdly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the transmission method as described in the first aspect.

[0030] Fourthly, the present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the transmission method as described in the first aspect.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] (1) A dual heartbeat mechanism is adopted, which can reduce the time and resource consumption of frequent connection establishment and disconnection, improve data transmission efficiency, ensure the integrity and reliability of data transmission, and promptly detect and make effective adjustments when errors and anomalies occur.

[0033] (2) This invention adopts a universal log compression and transmission scheme based on open-source algorithms and protocols. It is not limited by the compression capabilities of different database vendors, nor is it limited by database brand or model. It can seamlessly support cross-datacenter disaster recovery for the vast majority of database products, combining versatility and practicality. Furthermore, it can achieve compression and transmission at the application layer, thereby avoiding impact on upper-layer applications.

[0034] (3) The data synchronization scheme of the present invention, through the combination of incremental synchronization, streaming transmission and compression algorithm, has a lower dependence on bandwidth and can support large-scale off-site disaster recovery backup and synchronization of TB-level data, which greatly reduces the network cost of customers. Attached Figure Description

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0036] Figure 1 This is a schematic flowchart of a data copying and transmission method provided in an embodiment of the present invention;

[0037] Figure 2 A flowchart of a process for establishing a transmission heartbeat mechanism is provided for an embodiment of the present application.

[0038] Figure 3 A flowchart of a computer device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0039] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description is not intended to represent all embodiments in accordance with the present disclosure. Rather, they merely represent exemplary devices and methods in accordance with some aspects of the present disclosure, as detailed in the appended claims.

[0040] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0041] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one type of information from another. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present disclosure. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0042] The present disclosure discloses a wide area network data replication transmission method and device, which can be used for disaster recovery construction of a database.

[0043] The transmission method of the present disclosure refers to Figure 1 As shown in the figure, before data replication, a full synchronization is first performed, the main database sends the used data to the backup database, and the backup database records the last displacement point of the synchronization, so as to facilitate subsequent incremental synchronization.

[0044] After full synchronization, the main database can start business operation and continuously write log files. When the main database writes a new log file, the log file is asynchronously buffered to the local port, the local port records the serial number and displacement information of the log file, and compression processing is performed.

[0045] Then, asFigure 2 As shown, a plurality of transmission channels are created between the primary database and the standby database, and a transmission heartbeat mechanism for ensuring transmission connection is established based on the transmission channels, and a check heartbeat mechanism for ensuring data integrity is also established between the primary database and the standby database.

[0046] The check heartbeat mechanism comprises that the standby database sends feedback data to the primary database after receiving the log file, and the primary database ensures data integrity and validity by checking the feedback data; the feedback data comprises serial number and displacement information of the log file, so that the database is facilitated to check and the standby database is facilitated to replace, and the data amount of the feedback data is very small, so that bandwidth is not occupied.

[0047] The check heartbeat mechanism comprises that the standby database sends feedback data to the primary database after receiving the log file, and the primary database ensures data integrity and validity by checking the feedback data; the feedback data comprises serial number and displacement information of the log file, so that the database is facilitated to check and the standby database is facilitated to replace, and the data amount of the feedback data is very small, so that bandwidth is not occupied.

[0048] Then the local port transmits the compressed log file to the remote port of the standby database, and the remote port receives the log file, decompresses the log file and transmits the decompressed log file to the standby database in a streaming manner.

[0049] The compression algorithm is based on TCP protocol, and the compression algorithm and the length of compressed data are added in the header of the protocol.

[0050] The standby database continuously receives the log file from the remote port, and writes the log file into the corresponding position of the standby database according to the last displacement point, the serial number and the displacement information of the log file, so that data replication is realized.

[0051] Finally, the standby database applies the log file which is checked to be complete to redo, the log file which is not complete is notified to the upper application to be retransmitted, the received log file is returned to the primary database through the transmission channel to ensure data integrity and reliability, and the asynchronous replication state with the primary database is continuously maintained.

[0052] The device of the application mainly comprises the three modules, and the system is built to realize the purpose of parallel operation by mounting the same file system.

[0053] In practice, each of the above modules can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each of the above units can refer to the method embodiments described above, which will not be described here.

[0054] Figure 3 A structural diagram of a computer device is disclosed. Referring to FIG. 4, the computer device 400 at least includes a memory 402 and a processor 401; the memory 402 is connected with the processor through a communication bus 403, and is used to store computer instructions executable by the processor 401; and the processor 301 is used to read the computer instructions from the memory 402 to implement the steps of the transmission method described in any of the above embodiments. Figure 3

[0055] For the above device embodiments, since they basically correspond to the method embodiments, the related parts can refer to the part of the method embodiments. The above described device embodiments are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present disclosure. Those skilled in the art can understand and implement without creative labor.

[0056] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including semiconductor memory devices (e.g. EPROM, EEPROM, and flash memory devices), magnetic disks (e.g. internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by or incorporated in special logic circuitry.

[0057] Finally, it should be noted that: although this specification contains many specific implementation details, these should not be interpreted as limiting the scope or claimed scope of any invention, but mainly for describing the features of the specific embodiments of the particular invention. Some features described in this specification in multiple embodiments can also be implemented in a single embodiment. On the other hand, various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable subcombination. In addition, although the features can function as described above in some combinations and even originally claimed as such, one or more features from the claimed combination can be removed in some cases, and the claimed combination can refer to a subcombination or a variation of a subcombination. ​

[0058] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order nor limiting it to the particular order in which these operations are depicted in the drawings. One will appreciate that many other operations could be performed or the presented operations could be performed in a different order. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0059] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims could be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures could be implemented in a different order or concurrently. Additionally, certain actions could be performed concurrently or in different orders than those depicted in the figures. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0060] The above-described embodiments are merely possible implementations of the present disclosure, and do not limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the present disclosure.

Claims

1. A method for wide area network data replication and transmission, characterized in that, Includes the following steps: Create a transport channel between the primary database and the backup database, and establish a transport heartbeat mechanism based on the transport channel to ensure transport connection; Establish a verification heartbeat mechanism between the primary and backup databases to ensure data integrity. The primary database transfers log files to the secondary database; Before the primary database transmits the log file to the backup database, the primary database and the backup database are fully synchronized, and the backup database records the last offset point synchronized. The verification heartbeat mechanism includes the backup database sending feedback data to the master database after receiving the log file, and the master database verifying the feedback data to ensure the data is complete and valid. The feedback data includes the sequence number and offset information of the log file; The transmission heartbeat mechanism includes the primary database periodically sending heartbeat packets to the backup database and receiving heartbeat packets returned by the backup database to confirm that the transmission channel connection is normal. Several transmission channels are created between the primary database and the backup database. When the primary database detects an abnormal connection of a transmission channel, it switches to a new transmission channel for transmission, while the backup database maintains normal communication.

2. The transmission method according to claim 1, characterized in that, When the primary database verifies that the feedback data is incorrect, it retransmits the log file to the backup database. After receiving the log file, the backup database replaces the file according to the sequence number and offset information of the log file, and sends the replaced feedback data to the primary database.

3. The transmission method according to claim 1, characterized in that, The steps for the primary database to transfer log files to the backup database include: when the primary database writes the log file, it asynchronously buffers the log file to a local port, and the local port transfers the log file to the remote port of the backup database.

4. The transmission method according to claim 3, characterized in that, The local port compresses the log file and transmits it to the remote port. The remote port then decompresses the log file and transmits it to the backup database in a streaming manner. The compression process employs a compression algorithm based on the TCP protocol, and the compression algorithm and the length of the compressed data are added to the header of the protocol.

5. A transmission apparatus employing the wide area network data copying and transmission method according to any one of claims 1-4, characterized in that, include: Transmission heartbeat mechanism module: used to create a transmission channel between the primary database and the backup database, and to establish a transmission heartbeat mechanism based on the transmission channel to ensure transmission connection; Verification heartbeat mechanism module: Used to establish a verification heartbeat mechanism between the primary database and the backup database to ensure data integrity; Transmission module: Used to transfer log files from the primary database to the backup database.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the steps of the transmission method according to any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the transmission method as described in any one of claims 1-4.

Images