Database stream replication method and apparatus

Abstract

The application relates to a database stream replication method and device, in particular to the technical field of network communication. The method comprises the following steps: a prewrite log is generated; an interface function of an accelerated network protocol stack DPDK-ANS based on a data plane development kit is loaded in a master node database; the DPDK-ANS is called through the interface function of the DPDK-ANS, the prewrite log is transmitted to a backup node device by the DPDK-ANS, and data updating is performed on a backup node database; the backup node device is loaded with the backup node database; and the backup node database is loaded with the interface function of the DPDK-ANS. According to the above scheme, the prewrite log is sent to the backup node database through the DPDK-ANS, message processing is not required through a complex kernel network of Linux, the delay of a stream replication process is reduced, and the possibility of cluster failure caused by data asynchronization is reduced as much as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network communication, in particular to a database stream replication method and device. BACKGROUND

[0002] A distributed database system is composed of various computers, each of which can be placed separately in a place, and each computer has its own local database. Many computers located in different places are connected to each other through a network to form a complete, global, logically centralized, and physically distributed large database.

[0003] Taking a database cluster with MPP architecture as an example, the MPP database cluster is mainly composed of Master, Standby, and Segment three parts. The Master node is responsible for processing client requests, distributing execution plans to each Segment node and summarizing the results, and synchronizing all xlog logs of the Master to the Standby node through stream replication to ensure that the data of the Standby is synchronized with the data of the Master.

[0004] However, in the above scheme, when a large amount of data insertion query or a large number of client connections are performed, the network performance will become a bottleneck, and the high delay will cause the stream replication between the Master / Standby to fail, resulting in data asynchronization, and further causing cluster failure. SUMMARY

[0005] The present application provides a database stream replication method and device, which reduces the possibility of cluster failure caused by data asynchronization. The technical scheme is as follows.

[0006] On the one hand, a database stream replication method is provided, which is used for a master node device, and the method comprises:

[0007] Generating a pre-write log; the pre-write log is used to indicate the update of a master node database on the master node device; the master node database loads an interface function of an accelerated network protocol stack DPDK-ANS based on a data plane development kit;

[0008] Through the interface function of the DPDK-ANS, the DPDK-ANS is called to transmit the pre-write log to a backup node device, so that the backup node database performs data update; the backup node device loads a backup node database; the backup node database loads the interface function of the DPDK-ANS.

[0009] In another aspect, a database stream replication method is provided, which is used for a backup node device, wherein a backup node database is loaded on the backup node device; the method comprises:

[0010] receiving a pre-write log sent by a master node device through an accelerated network protocol stack based on a data plane development kit (DPDK-ANS); the pre-write log is used to indicate an update of a master node database; an interface function of the DPDK-ANS is loaded in the master node database;

[0011] importing the pre-write log into a backup node database; an interface function of the DPDK-ANS is loaded in the backup node database.

[0012] In another aspect, a database stream replication apparatus is provided, which is applied to a master node device, and comprises:

[0013] a log generation module configured to generate a pre-write log; the pre-write log is used to indicate an update of a master node database on the master node device; an interface function of an accelerated network protocol stack based on a data plane development kit (DPDK-ANS) is loaded in the master node database;

[0014] a log transmission module configured to call the DPDK-ANS to transmit the pre-write log to a backup node device through the interface function of the DPDK-ANS, so that a backup node database performs data update; a backup node database is loaded on the backup node device; the interface function of the DPDK-ANS is loaded in the backup node database.

[0015] In a possible implementation, the apparatus further comprises:

[0016] a first file acquisition module configured to acquire a first source code file;

[0017] a first header file replacement module configured to replace a linux interface header file in the first source code file with an adaptation layer header file to obtain a first adaptation code file; the adaptation layer header file contains the interface function of the DPDK-ANS;

[0018] a master node database loading module configured to compile the first adaptation code file, generate and load a first executable program, and start the master node database.

[0019] In a possible implementation, the apparatus further comprises:

[0020] an update data acquisition module configured to acquire first update data generated by a first master application program;

[0021] The updating data transmission module is configured to transmit the first updating data to the backup node device by invoking a kernel network through a Linux interface function on the master node device, so as to realize data updating of the first backup application on the backup node device.

[0022] In another aspect, a database stream replication apparatus is provided, which is configured for a backup node device loaded with a backup node database; the apparatus comprises:

[0023] The log receiving module is configured to receive a prewrite log sent by the master node device through an acceleration network protocol stack based on a data plane development kit (DPDK-ANS); the prewrite log is configured to indicate updating of the master node database; and the master node database is loaded with an interface function of the DPDK-ANS.

[0024] The log importing module is configured to import the prewrite log into the backup node database; and the backup node database is loaded with the interface function of the DPDK-ANS.

[0025] In a possible implementation, the apparatus further comprises:

[0026] The second file obtaining module is configured to obtain a second source code file.

[0027] The second header file replacing module is configured to replace a Linux interface header file in the second source code file with an adaptation layer header file to obtain a second adaptation code file; and the adaptation layer header file comprises the interface function of the DPDK-ANS.

[0028] The backup node database loading module is configured to compile the second adaptation code file, generate and load a second executable program, and start the backup node database.

[0029] In a possible implementation, the updating data receiving module is configured to receive first updating data sent by the master node device through a kernel network; and the first updating data is generated by a first master application.

[0030] The updating data importing module is configured to import the first updating data into the first backup application through a Linux interface function in the first backup application.

[0031] In another aspect, a computer device is provided, which comprises a processor and a memory; the memory stores at least one instruction, at least one program, a code set or an instruction set; and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned database stream replication method.

[0032] In yet another aspect, a computer readable storage medium is provided, the storage medium having stored therein at least one instruction, the at least one instruction being loadable by a processor and executable by the processor to implement the database stream replication method described above.

[0033] In yet another aspect, a computer program product is provided, the computer program product comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the database stream replication method described above.

[0034] The technical solutions provided in the present application can include the following beneficial effects:

[0035] In a distributed database system having a master node database and a backup node database, data update synchronization can be achieved through stream replication, and in a stream assistance process, an interface function of DPDK-ANS can be loaded in the master node database, so that prewrite logs are sent to the backup node database through DPDK-ANS, without the need for complex kernel network processing of messages in Linux, thereby reducing the delay of the stream replication process and minimizing the possibility of cluster failure caused by data asynchronization. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0037] Figure 1 is a structural schematic diagram of a database stream replication system according to an exemplary embodiment.

[0038] Figure 2 is a method flowchart of a database stream replication method according to an exemplary embodiment.

[0039] Figure 3 shows a stream replication process based on a kernel network related to an embodiment of the present application.

[0040] Figure 4 shows a stream replication process based on a data plane development kit related to an embodiment of the present application.

[0041] Figure 5A method flowchart of a database stream replication method is shown according to an example embodiment.

[0042] Figure 6 A database code modification schematic diagram is shown.

[0043] Figure 7 A data transmission flow schematic diagram is shown.

[0044] Figure 8 An MPP cluster stream replication schematic diagram is shown.

[0045] Figure 9 A software loading schematic diagram is shown.

[0046] Figure 10 A structure block diagram of a database stream replication apparatus is shown according to an example embodiment.

[0047] Figure 11 A structure block diagram of a database stream replication apparatus is shown according to an example embodiment.

[0048] Figure 12 A computer device schematic diagram is provided according to an example embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] It should be understood that the "indication" mentioned in the embodiments of the present application can be direct indication, indirect indication, or can be an indication of an associated relationship. For example, A indicates B, which can mean that B can be obtained through A; or it can mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; or it can mean that A and B have an associated relationship.

[0051] In the description of the embodiments of the present application, the term "corresponding" can mean that there is a direct or indirect corresponding relationship between the two, or it can mean that there is an associated relationship between the two, or it can mean an indication and being indicated, configuration and being configured, etc.

[0052] In the embodiments of the present application, the "predefinition" can be realized by pre-storing corresponding codes, tables or other means for indicating relevant information in devices (for example, including terminal devices and network devices), and the specific implementation manner is not limited in the present application.

[0053] Before the various embodiments shown in the present application are described, several concepts related to the present application are first introduced.

[0054] 1) MPP (Massively Parallel Processing, large-scale parallel analysis)

[0055] The MPP architecture is to distribute the task in parallel to multiple servers and nodes, and after the calculation is completed on each node, the results of each part are summarized together to obtain the final result. The database using the MPP architecture is called MPP database. In the database non-shared cluster, each node has an independent disk storage system and memory system, and the business data is divided into each node according to the database model and application characteristics, each data node is connected to each other through a dedicated network or a commercial general-purpose network, and each node is calculated cooperatively to provide database services as a whole. The non-shared database cluster has the advantages of complete scalability, high availability, high performance, excellent cost performance, resource sharing, etc. In short, MPP is to distribute the task in parallel to multiple servers and nodes, and after the calculation is completed on each node, the results of each part are summarized together to obtain the final result. The MPP database cluster is mainly composed of Master (master node), Standby (backup node) and Segment (segment node), wherein the Master node is responsible for processing the requests of the client, distributing the execution plan to each Segment node and summarizing the results, and on the other hand, the xlog log of the Master is synchronized to the Standby node through stream replication to ensure that the data of the Standby is synchronized with the data of the Master.

[0056] 2) DPDK

[0057] The data plane development kit (DPDK, Data Plane Development Kit) is developed by multiple companies, mainly based on Linux system, and is a function library and driver set for fast packet processing, which can greatly improve the data processing performance and throughput, and improve the working efficiency of data plane application program. DPDK uses polling instead of interrupt to process packets. When a packet is received, the network card driver loaded by DPDK will not notify the CPU through an interrupt, but will directly store the packet in the memory and deliver it to the application layer software for direct processing through the interface provided by DPDK, thereby saving a large amount of CPU interrupt time and memory copy time.

[0058] Figure 1 Fig. 1 is a structural schematic diagram of a database stream replication system according to an exemplary embodiment. The database stream replication system comprises a server 110 and a server 120.

[0059] Optionally, the server 110 comprises a master node database in an MPP database, and the server 110 can be referred to as a master node device in the database stream replication system. The server 120 comprises a backup node database in an MPP database, and the server 120 can be referred to as a master node device in the database stream replication system.

[0060] Optionally, the server 110 and the server 120 are servers loaded with a Linux operating system. When the server 110 and the server 120 are connected in communication, the Linux interface function can be called to process messages by using the network kernel of the Linux, and a socket connection can be established to realize the communication connection between the server 110 and the server 120.

[0061] Optionally, the server 110 can also be installed with a DPDK-ANS module. The ANS (Accelerated Network Stack) in the DPDK-ANS module is a DPDK-based TCP / IP stack, which can run in the linux userspace. Users can freely add, delete ether interfaces and IP addresses, and add, delete static routes. The server 110 can establish a communication connection with the server 120 by using the DPDK-ANS running in the linux userspace.

[0062] Optionally, the server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms.

[0063] Optionally, the system can further comprise a management device for managing the system (such as managing the connection state between each module and the server). The management device and the server are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0064] Optionally, the wireless or wired networks described above use standard communications technologies and / or protocols. The networks typically are wired networks using, for example, copper and / or fiber optics, but also can be wireless data transmission technologies, e.g., radio frequency (RF), BLUETOOTH, infrared, etc. The networks are typically the Internet, but can also be any other publicly or

[0065] Figure 2 is a method flow chart of a database stream replication method according to an exemplary embodiment. The method is performed by a master node device and a backup node device together, the master node device can be the server 110 in the database stream replication system as shown in Figure 1 , and the backup node device can be the server 120 in the database stream replication system as shown in Figure 1 . As shown in Figure 2 , the database stream replication method can include the following steps:

[0066] Step 201, the master node device generates a pre-write log.

[0067] The pre-write log is used to indicate the update of the master node database on the master node device; the master node database loads an interface function of an accelerated network protocol stack DPDK-ANS based on a data plane development kit.

[0068] The pre-write log is a WAL (Write Ahead logging) log, which is a kind of efficient log algorithm in a database, that is, when modifying the database, the modified data is written into the log and synchronized to the disk, so that other read processes have no effect. If the database crashes, scan the log file after restarting, and then update to the database.

[0069] The principle of stream replication is that the backup continuously synchronizes the corresponding data from the master, and loads each WAL record in the backup. The transmission unit of stream replication here is the record of the WAL log each time.

[0070] And in a possible implementation manner of the embodiment of the application, the master node database loads the interface function of the DPDK-ANS, at this time, when the master node database communicates with the backup node database, it can choose to call the network kernel of Linux to realize the communication connection, or choose to call the DPDK-ANS to realize the communication connection.

[0071] In step 202, the master node device calls the DPDK-ANS to transmit the pre-write log to the backup node device through the interface function of the DPDK-ANS based on the data plane development kit, so as to update the data of the backup node database.

[0072] Correspondingly, the backup node device receives the pre-write log sent by the master node device through the DPDK-ANS; the pre-write log is used to indicate the update of the master node database.

[0073] The interface function of the DPDK-ANS is loaded in the master node database. The backup node database is loaded in the backup node device, and the interface function of the DPDK-ANS is loaded in the backup node database.

[0074] Since the interface function of the DPDK-ANS is loaded in the master node database, the master node device can call the interface function of the DPDK-ANS to transmit the WAL log generated by the master node database to the backup node device through the DPDK-ANS, so that the backup node database updates the data according to the WAL log.

[0075] And the interface function of the DPDK-ANS is also loaded in the backup node database in the backup node device, so that the backup node database and the master node database can establish a communication connection through the DPDK-ANS, bypassing the complex message forwarding system in the Linux network kernel, thereby improving the speed of WAL log transmission and reducing the delay of WAL log transmission.

[0076] In step 203, the backup node device imports the pre-write log into the backup node database.

[0077] Since the interface function of the DPDK-ANS is loaded in the backup node database, after receiving the message transmitted by the master node device, the backup node database can decode the protocol in the DPDK-ANS, so as to be directly transmitted to the application layer and imported into the backup node database. Similarly, without going through the complex mechanism in the Linux network kernel, the direct transmission from the network layer to the application layer can be realized, thereby improving the rate of importing the WAL log into the backup node database.

[0078] Please refer to Figure 3 which shows a kernel network based flow replication process according to an embodiment of the present application. As shown in Figure 3As shown, when the two Linux-based devices interact with each other, the data generated by the application APP in the user space (Linux user space) needs to be processed through the connection module (socket) and the driver module (driver) in the kernel network (Linux kernel space), and frequent memory copying and system calls are required in the process. After being processed by each module, the generated message can be transmitted to other devices through hardware hardware, thereby completing the transmission of data.

[0079] Please refer to Figure 4 which shows a flow replication process based on a data plane development kit according to an embodiment of the present application. As Figure 4 shown, by loading the application APP with DPDK-ANS, the data generated in the APP can be directly transmitted to other devices through hardware hardware, thereby completing the transmission of data.

[0080] Therefore, by using DPDK-ANS, the complex message processing process of the Linux kernel can be bypassed, and the message can be directly sent from the hardware to the user space to import the backup node database, thereby realizing the flow replication process in the MMP database, reducing the time delay of the flow replication process, and thereby minimizing the possibility of cluster failure caused by data asynchronization.

[0081] In summary, in a distributed database system having a master node database and a backup node database, data update synchronization can be realized by a flow replication method, and in the flow auxiliary process, the interface function of DPDK-ANS can be loaded in the master node database, thereby realizing the sending of the pre-write log to the backup node database through DPDK-ANS, without the need for complex kernel network message processing in Linux, reducing the delay of the flow replication process, and thereby minimizing the possibility of cluster failure caused by data asynchronization.

[0082] Figure 5 is a method flowchart of a database flow replication method according to an exemplary embodiment. The method is jointly executed by a master node device and a backup node device, the master node device can be a server 110 in the database flow replication system as Figure 1 shown, and the backup node device can be a server 120 in the database flow replication system as Figure 1 shown. As Figure 5 shown, the database flow replication method can include the following steps:

[0083] Step 501, obtaining a first source code file.

[0084] The first source code file can be stored in a server. When the server needs to load a node database, the source code file can be compiled to generate an executable program of the database. At this time, the server loads the application program of the database to start the node database.

[0085] In step 502, the Linux interface header file in the first source code file is replaced by an adaptation layer header file to obtain a first adaptation code file.

[0086] The adaptation layer header file contains interface functions of DPDK-ANS.

[0087] Please refer to Figure 6 , which shows a database code modification schematic diagram related to an embodiment of the present application. As shown in Figure 6 Before a node device creates a node database MPP, the node device can replace the Linux-related header file in the source code file of the node database with the related header file of the pre-set adaptation layer (Adap layer), and retain other header files and other codes in the source code file of the node database.

[0088] At this time, the interface functions of DPDK-ANS exist in the related header file of the adaptation layer in the modified adaptation code file, so that the node database obtained by compiling the application program of the node database according to the adaptation code file is set as a master node database. When the master node device where the master node database is located can directly call DPDK-ANS through the interface functions of DPDK-ANS to send the WAL log in the master node database to the backup node device.

[0089] In step 503, the first adaptation code file is compiled to generate and load a first executable program to load the master node database.

[0090] After the first adaptation code file is obtained by replacing the header file in the first source code file, the first adaptation code file can be compiled to obtain an executable program of the node database, that is, a node database application program with interface functions of DPDK-ANS. At this time, the server (that is, the master node device) loads the first executable program to start the master node database.

[0091] It should be noted that in another possible implementation, the process of replacing the Linux header file in the first source code file with the adaptation layer header file in step 501 and step 502, and compiling the first adaptation code file in step 503, can also be performed in other computer devices.

[0092] For example, in a certain terminal, the Linux header file in the first source code file can be replaced by the adaptation layer header file, and a first adaptation code file is generated by compiling, and a first executable program is generated, and the terminal sends the first executable program to the master node device, so that the master node device loads the first executable program and starts the master node database.

[0093] In step 504, the pre-write log is generated.

[0094] During the running of the master node database, the pre-write log related to the running process is generated and sent to the backup node database in the form of stream replication, so as to realize the backup of the database state.

[0095] In step 505, the DPDK-ANS is called through the interface function of the DPDK-ANS to transmit the pre-write log to the backup node device.

[0096] Since the adaptation layer header file with the interface function of the DPDK-ANS is contained in the master node database code at this time, the master node database can directly call the DPDK-ANS to realize the transmission of the WAL log through the DPDK-ANS interface function when performing stream replication.

[0097] In step 506, the second source code file is obtained.

[0098] The second source code file can be saved in the server, and when the server needs to load the node database, the source code file can be compiled to generate the application program of the database, and the server loads the application program of the database at this time, so as to realize the start of the database.

[0099] In step 507, the Linux interface header file in the second source code file is replaced by the adaptation layer header file to obtain a second adaptation code file.

[0100] The adaptation layer header file contains the interface function of the DPDK-ANS.

[0101] Similarly, Figure 6 The backup node database can also be instructed to load the code modification process of the source code file corresponding thereto before. For example, before a node device creates a node database MPP, the node device can replace the Linux related header file in the source code file of the node database with the pre-set adaptation layer related header file (Adap layer), and retain other header files and other codes in the source code file of the node database.

[0102] At this time, in the modified adaptation code file, the interface function of the DPDK-ANS exists in the header file of the adaptation layer, so when the node database application program is compiled according to the adaptation code file, the node database obtained by the application program is set as a backup node database, and the backup node device where the backup node database is located can directly call the DPDK-ANS to receive the WAL log sent by the master node device through the DPDK-ANS.

[0103] In step 508, the second adaptation code file is compiled to generate and load a second executable program to start the backup node database.

[0104] When the second adaptation code file is obtained by replacing the header file in the second source code file, the second adaptation code file can be compiled to obtain a second executable program, that is, a node database application program with the interface function of the DPDK-ANS. At this time, the server loads the second executable program to start the backup node database.

[0105] It should be noted that in another possible implementation, the process of replacing the Linux header file in the second source code file with the adaptation layer header file in step 506 and step 507, and the process of compiling the second adaptation code file in step 508, can also be executed in other computer devices.

[0106] For example, in a terminal, the Linux header file in the second source code file can be replaced with the adaptation layer header file, and the second adaptation code file can be compiled to generate a second executable program. At this time, the terminal sends the second executable program to the backup node device, so that the backup node device loads the second executable program to start the master node database.

[0107] In step 509, the pre-write log sent by the master node device is received through the DPDK-ANS.

[0108] Please refer to Figure 7 , which shows a data transmission process diagram related to an embodiment of the present application. As Figure 7 shown, in the embodiment of the present application, when the master node database and the backup node database are started, the Walsender process is run on the master node device, that is, when the specified condition is met, the Wal log file generated on the master node device is sent to the backup node database.

[0109] The backup node device also runs a WalReceiver process to receive the Wal log file sent by the master node device. During the transmission of the Wal log, the master node device and the backup node device establish a socket link based on the DPDK-ANS protocol stack, and can realize data interaction through the TPC / IP address configured in the socket.

[0110] At step 510, the pre-write log is imported into the backup node database.

[0111] When the backup node device receives the pre-write log, the pre-write log can be directly transmitted to the application layer of the backup node device through DPDK-ANS to import the backup node database, thereby reducing the time delay of the backup node database receiving the pre-write log.

[0112] In a possible implementation, the master node device also loads a first master application program, and the first update data generated by the first master application program is acquired at this time; the first update data is transmitted to the backup node device through the kernel network of the linux interface function of the master node device, so as to realize data update of the first backup application program on the backup node device.

[0113] Correspondingly, the backup node device loads a first backup application program, and the first update data sent by the master node device is received through the kernel network at this time; the first update data is generated by the first master application program; and the first update data is imported into the first backup application program through the linux interface function in the first backup application program.

[0114] That is, in the embodiment of the application, the master node device and the backup node device can realize data transmission through the network kernel of Linux itself in addition to realizing communication connection through DPDK-ANS. Since data transmission through DPDK-ANS needs to call the interface function of DPDK-ANS, and for most software, compatibility with DPDK-ANS needs to adjust the code. Therefore, for most computer devices, the program does not pursue time delay, and the data transmission can be realized through the network kernel of Linux itself, and only a small part of the application program or module needs to be adjusted, and the workload of adjusting the application program is reduced on the premise of ensuring the normal operation of the program.

[0115] Please refer to Figure 8 which shows a MPP cluster stream replication schematic diagram related to the embodiment of the application. As shown in Figure 8As shown, the Master Host starts a WalSender process, which continuously sends data to the Standby Master Host through DPDK-ANS, while the Standby Master Host starts a WalReceiver process, which accepts data through DPDK-ANS. Other Apps or MPP other modules keep the original mechanism, with minor changes to solve the data stream replication problem of Master Host and Standby Master Host.

[0116] Optionally, in the embodiments of the present application, DPDK-ANS needs to be additionally installed on the Master Host and the Standby Master Host as an application program.

[0117] Please refer to Figure 9 , which shows a software loading schematic diagram related to the embodiments of the present application. As shown in Figure 9 .

[0118] 1. When installing MPP, install DPDK-ANS on the Master Host and the Standby Master Host.

[0119] 2. After installation, initialize and configure DPDK, such as installing UIO / VFIO / KNI / modules, creating hugepage mapping, binding network cards / interfaces to DPDK, and creating DPDK forwarding threads.

[0120] 3. Configure network cards / interfaces on the ANS protocol stack, such as ip and acl.

[0121] Through the above steps, the environment of DPDK-ANS is already available, which can provide services for the upper layer.

[0122] It should be noted that in the embodiments shown in Figure 2 or Figure 5 , the database with MPP architecture is taken as an example to describe the specific steps of executing the solution of the present application to realize the stream replication process, and each step in the embodiments shown in Figure 2 or Figure 5 may also be applicable to the distributed database system with a Master database and a Standby database.

[0123] In summary, in the distributed database system with the master node database and the backup node database, the data update synchronization can be realized through the stream replication mode, and in the stream assistance process, the interface function of the DPDK-ANS can be loaded in the master node database, so that the pre-write log is sent to the backup node database through the DPDK-ANS, without the message processing through the complex Linux kernel network, the delay of the stream replication process is reduced, and the possibility of the cluster failure caused by the data asynchronization is reduced as much as possible.

[0124] Figure 10 Fig. 1 is a structural block diagram of a database stream replication device according to an exemplary embodiment. The device is used for a backup node device, and the backup node device is loaded with a backup node database. The device comprises:

[0125] A log generation module 1001 is configured to generate a pre-write log. The pre-write log is used to indicate the update of a master node database on a master node device. The master node database is loaded with an interface function of a data plane development kit-based acceleration network protocol stack (DPDK-ANS).

[0126] A log transmission module 1002 is configured to call the DPDK-ANS to transmit the pre-write log to a backup node device through the interface function of the DPDK-ANS, so that the backup node database performs data update. The backup node device is loaded with a backup node database. The backup node database is loaded with the interface function of the DPDK-ANS.

[0127] In a possible implementation, the device further comprises:

[0128] A first file acquisition module is configured to acquire a first source code file.

[0129] A first header file replacement module is configured to replace a Linux interface header file in the first source code file with an adaptation layer header file to obtain a first adaptation code file. The adaptation layer header file comprises the interface function of the DPDK-ANS.

[0130] A master node database loading module is configured to compile the first adaptation code file, generate and load a first executable program, and start the master node database.

[0131] In a possible implementation, the device further comprises:

[0132] An update data acquisition module is configured to acquire first update data generated by a first master application program.

[0133] The updating data transmission module is configured to transmit the first updating data to the backup node device by invoking a kernel network through a Linux interface function on the master node device, so as to realize data updating of the first backup application on the backup node device.

[0134] In summary, in the distributed database system with the master node database and the backup node database, the data updating synchronization can be realized by the stream replication mode, and in the stream assistance process, the interface function of the DPDK-ANS can be loaded in the master node database, so that the prewrite log is sent to the backup node database through the DPDK-ANS, without the message processing through the Linux complex kernel network, thereby reducing the delay of the stream replication process, and minimizing the possibility of cluster failure caused by data asynchronization.

[0135] Figure 11 Fig. 1 is a structural block diagram of a database stream replication device according to an exemplary embodiment. The device is used in a backup node device, and the backup node device loads a backup node database. The device comprises:

[0136] The log receiving module 1101 is configured to receive the prewrite log sent by the master node device through the DPDK-ANS. The prewrite log is used to indicate the updating of the master node database, and the interface function of the DPDK-ANS is loaded in the master node database.

[0137] The log importing module 1102 is configured to import the prewrite log into the backup node database, and the interface function of the DPDK-ANS is loaded in the backup node database.

[0138] In a possible implementation, the device further comprises:

[0139] The second file obtaining module is configured to obtain a second source code file.

[0140] The second header file replacing module is configured to replace the Linux interface header file in the second source code file with an adaptation layer header file to obtain a second adaptation code file, and the adaptation layer header file contains the interface function of the DPDK-ANS.

[0141] The backup node database loading module is configured to compile the second adaptation code file, generate and load a second executable program, and start the backup node database.

[0142] In a possible implementation, the updating data receiving module is configured to receive the first updating data sent by the master node device through a kernel network, and the first updating data is generated by the first master application.

[0143] The update data import module is configured to import the first update data into the first backup application through a Linux interface function in the first backup application.

[0144] In summary, in the distributed database system with the master node database and the backup node database, the data update synchronization can be achieved through the stream replication mode, and in the stream assistance process, the interface function of the DPDK-ANS can be loaded in the master node database, so that the prewrite log is sent to the backup node database through the DPDK-ANS, without the need of processing the message through the complex kernel network of Linux, thereby reducing the delay of the stream replication process, and minimizing the possibility of the cluster failure caused by the data asynchronization.

[0145] Please refer to Figure 12 According to an exemplary embodiment of the present application, a computer device is provided, and a schematic diagram of the computer device is shown in FIG. 1. The computer device includes a memory and a processor. The memory is configured to store a computer program. When the computer program is executed by the processor, the database stream replication method described above is implemented.

[0146] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0147] The memory is a non-transitory computer readable storage medium, which can be configured to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiments of the present application. The processor executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the method in the method embodiments described above.

[0148] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required by at least one function, and the like, and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some embodiments, the memory can optionally include a memory that is remotely located with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] In an exemplary embodiment, a computer readable storage medium storing at least one computer program is also provided, the at least one computer program is loaded and executed by the processor to implement all or part of the steps of the above method. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0150] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, which will be readily appreciated by those skilled in the art. The specification and examples are to be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0151] It is to be understood that the application is not limited to the precise construction described and as shown in the attached figures, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is to be determined only by the appended claims.

Claims

1. A database stream replication method, characterized by, The method is used for a master node device, and the method comprises: generating a prewrite log; the prewrite log is used to indicate an update of a master node database on the master node device; an interface function of a data plane development kit (DPDK-ANS) based accelerated network protocol stack is loaded in the master node database; transmitting, by the interface function of the DPDK-ANS, the prewrite log to a backup node device by the DPDK-ANS, so that a backup node database performs data update; the backup node database is loaded on the backup node device; the interface function of the DPDK-ANS is loaded in the backup node database; before the prewrite log is generated, the method further comprises: obtaining a first source code file; replacing a linux interface header file in the first source code file with an adaptation layer header file to obtain a first adaptation code file; the adaptation layer header file contains the interface function of the DPDK-ANS; compiling the first adaptation code file to generate and load a first executable program to start the master node database.

2. The method of claim 1, wherein, The master node device further loads a first master application program, and the method further comprises: obtaining first update data generated by the first master application program; transmitting, by a linux interface function on the master node device, the first update data to the backup node device by a kernel network to implement data update of a first backup application program on the backup node device.

3. A database stream replication method, characterized by, The method is used for a backup node device, and the backup node device loads a backup node database; the method comprises: receiving, by a data plane development kit (DPDK-ANS) based accelerated network protocol stack, a prewrite log sent by a master node device; the prewrite log is used to indicate an update of the master node database; an interface function of the DPDK-ANS is loaded in the master node database; importing the prewrite log into the backup node database; the interface function of the DPDK-ANS is loaded in the backup node database; before the prewrite log sent by the master node device is received by the DPDK-ANS, the method further comprises: obtaining a second source code file; replacing a linux interface header file in the second source code file with an adaptation layer header file to obtain a second adaptation code file; the adaptation layer header file contains the interface function of the DPDK-ANS; compiling the second adaptation code file to generate and load a second executable program to start the backup node database.

4. The method of claim 3, wherein, The backup node device further loads a first backup application program, and the method further comprises: receiving, by a kernel network, first update data sent by the master node device; the first update data is generated by a first master application program; importing the first update data into the first backup application program by a linux interface function in the first backup application program.

5. A database stream replication apparatus characterized by comprising: The device is applied to a master node device, and the device comprises: The log generation module is configured to generate a prewrite log, wherein the prewrite log is used to indicate an update of a master node database on the master node device, and the master node database is loaded with an interface function of a data plane development kit (DPDK)-accelerated network stack (ANS). The log transmission module is configured to transmit, by the interface function of the DPDK-ANS, the prewrite log to a backup node device via the DPDK-ANS, so that a backup node database on the backup node device updates data, wherein the backup node database is loaded with the interface function of the DPDK-ANS. The first file acquisition module is configured to acquire a first source code file. The first header file replacement module is configured to replace a linux interface header file in the first source code file with an adaptation layer header file to obtain a first adaptation code file, and the adaptation layer header file contains the interface function of the DPDK-ANS. The master node database loading module is configured to compile the first adaptation code file, generate and load a first executable program, and start the master node database.

6. The apparatus of claim 5, wherein, The device further comprises: The update data acquisition module is configured to acquire first update data generated by a first master application program. The update data transmission module is configured to transmit, by a linux interface function on the master node device, the first update data to the backup node device via a kernel network, so as to update data of a first backup application program on the backup node device.

7. A database stream replication apparatus characterized by comprising: The device is used for the backup node device, and the backup node device is loaded with a backup node database. The log receiving module is configured to receive, by a data plane development kit (DPDK)-accelerated network stack (ANS), a prewrite log sent by a master node device, wherein the prewrite log is used to indicate an update of a master node database, and the master node database is loaded with an interface function of the DPDK-ANS. The log import module is configured to import the prewrite log into a backup node database, wherein the backup node database is loaded with the interface function of the DPDK-ANS. The second file acquisition module is configured to acquire a second source code file. The second header file replacement module is configured to replace a linux interface header file in the second source code file with an adaptation layer header file to obtain a second adaptation code file, and the adaptation layer header file contains the interface function of the DPDK-ANS. The backup node database loading module is configured to compile the second adaptation code file, generate and load a second executable program, and start the backup node database.

8. The apparatus of claim 7, wherein, The device further comprises: The update data receiving module is configured to receive, by a kernel network, first update data sent by a master node device, wherein the first update data is generated by a first master application program. The update data import module is configured to import, by a linux interface function in a first backup application program, the first update data into the first backup application program.

Images