Database system, method and device for database log transfer
By introducing a programmable router as a relay device into the distributed database system, the problem of wasted bandwidth and computing resources caused by the master node repeatedly replicating database logs is solved, and efficient use of resources is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
- Filing Date
- 2024-12-06
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173467A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of database technology, and in particular to a database system, a method for transmitting database logs, and an apparatus. Background Technology
[0002] With the development of big data, database systems are being used more and more widely. A typical database system includes a primary cluster, a disaster recovery cluster, and a log storage device. The primary cluster consists of a primary node and at least one backup node. The backup node is used to back up the data stored on the primary node, the disaster recovery cluster is used to back up the data stored on the primary cluster, and the log storage device is used to back up and archive the database logs generated by the primary node. To achieve data consistency, after performing operations on data, the primary node generates a database log to record this operation and sends the database log to the backup node, the disaster recovery cluster, and the log storage device.
[0003] In the above distributed database system scheme, after the master node generates database logs each time, it needs to send the database logs to the backup node, the disaster recovery cluster and the log storage device respectively. The amount of data in the database logs is usually large, which will cause a significant waste of the master node's bandwidth resources. Summary of the Invention
[0004] This disclosure provides a database system, a method for database log transmission, and an apparatus that can reduce the bandwidth resource consumption of the master node during database log synchronization. The technical solution is as follows:
[0005] Firstly, a database system is provided, comprising a master node of a main cluster, a forwarding device group, and at least one log synchronization device for the master node. The forwarding device group includes at least one programmable router. The log synchronization device is a backup node of the main cluster, a disaster recovery cluster of the main cluster, or a log storage device. The master node is used to: perform data operations on the database and generate database logs, wherein the database logs record operation information for each data operation; send forwarding control messages to a target programmable router, wherein the forwarding control messages instruct the target programmable router to forward the database log transmission logic to at least one log synchronization device, and the target programmable router is a programmable router in the forwarding device group; and send the database logs to the target programmable router. The target programmable router is used to: forward the database logs to at least one log synchronization device based on the transmission logic.
[0006] In the above scheme, after the master node performs data operations on the database and generates database logs, it only needs to send the database logs to the target programmable router. The target programmable router then forwards the database logs to the log synchronization devices. In this way, the master node does not need to replicate the database logs multiple times, and only needs to send the database logs once to one device (the target programmable router), effectively reducing the bandwidth and computing resource consumption of the master node.
[0007] Furthermore, in the above scheme, the database log forwarding device is a programmable router, which can be flexibly configured by the master node to send database logs according to actual needs.
[0008] In one possible implementation, the sending logic is either multicast or unicast.
[0009] In one possible implementation, the above-mentioned at least one log synchronization device is used for:
[0010] Upon receiving database logs, a reception acknowledgment message is sent to the target programmable router. Correspondingly, the target programmable router is also configured to: based on the received reception acknowledgment message, determine whether the reception of database logs by at least one log synchronization device meets specified conditions; and if it is determined that the reception of database logs by at least one log synchronization device meets the specified conditions, send a specified message to the master node. Correspondingly, the master node is also configured to: upon receiving the specified message, send a data operation completion message to the client.
[0011] In the above scheme, the determination of the log synchronization device's reception of database logs is implemented by the target programmable router. The target programmable router only needs to feed back the determination result to the master node, which can further save the master node's computing resources.
[0012] In one possible implementation, the above-mentioned at least one log synchronization device is used for:
[0013] Upon receiving database logs, a reception acknowledgment message is sent to the target programmable router. Correspondingly, the target programmable router is also configured to: forward the received reception acknowledgment message to the master node. Correspondingly, the master node is also configured to: determine, based on the received reception acknowledgment message, whether the reception of database logs by at least one log synchronization device meets specified conditions; and if the reception of database logs by at least one log synchronization device meets the specified conditions, send a data operation completion message to the client.
[0014] In the above scheme, the target programmable router can act as a relay device for receiving confirmation messages without judging the reception status, thus saving the target programmable router's computing resources.
[0015] In one possible implementation, the database system also includes master nodes of other master clusters and log synchronization devices for those master nodes. Correspondingly, the target programmable router is also used to: determine the target port for receiving forwarding control messages; add the target port and the sending logic indicated by the forwarding control message to a mapping between ports and sending logic; after receiving database logs, determine the target port for receiving the database logs; and determine the sending logic corresponding to the target port based on the aforementioned mapping. Accordingly, the target programmable router is used to: forward database logs to at least one log synchronization device based on the sending logic corresponding to the target port.
[0016] In the above scheme, the forwarding device group can support the forwarding of database logs from multiple master nodes, and database logs from different master nodes can be configured with different sending logic, thus enabling more flexible forwarding of database logs.
[0017] In one possible implementation, the master node is also used for:
[0018] The target programmable router is determined from the forwarding device group based on the load balancing mechanism.
[0019] In the above scheme, the forwarding device group can include two or more programmable routers. In this case, the master node can determine the target programmable router in the forwarding device group according to the load balancing mechanism to perform database log forwarding.
[0020] In one possible implementation, the log storage device is a black box.
[0021] A black box is a data storage device characterized by its resistance to strong impacts, water and water pressure, corrosion, seawater immersion, and high temperatures. Black boxes are primarily used in aircraft to store critical data such as flight data and cockpit audio. In the above solution, the black box serves as a log storage device, used to store database logs.
[0022] Secondly, a method for transmitting database logs is provided. The method is applied to a database system, which includes a master node of a master cluster, a forwarding device group, and at least one log synchronization device of the master node. The forwarding device group includes at least one programmable router. The log synchronization device is a backup node of the master cluster, a disaster recovery cluster of the master cluster, or a log storage device.
[0023] The method includes:
[0024] The master node performs data operations on the database and generates database logs, which are used to record the operation information of this data operation.
[0025] The master node sends a forwarding control message to the target programmable router, wherein the forwarding control message is used to instruct the target programmable router to forward the sending logic of the database log to the at least one log synchronization device, and the target programmable router is a programmable router in the forwarding device group;
[0026] The master node sends the database log to the target programmable router;
[0027] The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic.
[0028] In one possible implementation, the sending logic is either multicast or unicast.
[0029] In one possible implementation, the method also includes:
[0030] Upon receiving the database logs, the at least one log synchronization device sends a receipt confirmation message to the target programmable router.
[0031] The target programmable router determines, based on the received reception confirmation message, whether the reception status of the database logs by the at least one log synchronization device meets the specified conditions. If it is determined that the reception status of the database logs by the at least one log synchronization device meets the specified conditions, it sends a specified message to the master node.
[0032] Upon receiving the specified message, the master node sends a data operation completion message to the client.
[0033] In one possible implementation, the database system further includes the master nodes of other master clusters and log synchronization devices for the master nodes of the other master clusters;
[0034] The method further includes:
[0035] The target programmable router determines the target port for receiving the forwarding control message;
[0036] The target programmable router adds the target port and the transmission logic indicated by the forwarding control message to the correspondence between the port and the transmission logic.
[0037] After receiving the database log, the target programmable router determines the target port for receiving the database log; and determines the transmission logic corresponding to the target port according to the correspondence.
[0038] The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic, including:
[0039] The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic corresponding to the target port.
[0040] In one possible implementation, the method further includes:
[0041] The master node determines the target programmable router from the forwarding device group according to the load balancing mechanism.
[0042] In one possible implementation, the log storage device is a black box.
[0043] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the methods provided in the second aspect and its possible implementations described above.
[0044] Fourthly, a computer-readable storage medium is provided, comprising computer program instructions that, when executed by a cluster of computing devices, perform the methods provided in the second aspect and its possible implementations.
[0045] Fifthly, a computer program product containing instructions is provided that, when executed by a cluster of computing devices, causes the cluster of computing devices to perform the method provided in the second aspect and its possible implementations. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of the architecture of a database system provided in an embodiment of this disclosure;
[0047] Figure 2 This is a schematic diagram of the architecture of a database system provided in an embodiment of this disclosure;
[0048] Figure 3 This is a schematic diagram of the architecture of a database system provided in an embodiment of this disclosure;
[0049] Figure 4 This is a schematic flowchart of a database log transmission method provided in an embodiment of this disclosure;
[0050] Figure 5 This is a schematic flowchart of a method for configuring sending logic provided in an embodiment of this disclosure;
[0051] Figure 6 This is a schematic flowchart of a method for changing forwarding behavior provided in an embodiment of this disclosure;
[0052] Figure 7 This is a schematic flowchart of a method for receiving and transmitting an acknowledgment message provided in an embodiment of this disclosure;
[0053] Figure 8 This is a schematic diagram of the architecture of a database system provided in an embodiment of this disclosure;
[0054] Figure 9 This is a flowchart of a method for receiving and transmitting an acknowledgment message provided in an embodiment of this disclosure;
[0055] Figure 10 This is a schematic diagram of the architecture of a database system provided in an embodiment of this disclosure;
[0056] Figure 11 This is a schematic diagram of a device structure for transmitting database logs according to an embodiment of the present disclosure;
[0057] Figure 12 This is a schematic diagram of the structure of a forwarding device provided in an embodiment of this disclosure;
[0058] Figure 13 This is a schematic diagram of a forwarding device cluster provided in an embodiment of this disclosure;
[0059] Figure 14 This is a schematic diagram of a forwarding device cluster provided in an embodiment of this disclosure. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.
[0061] The concepts involved in this disclosure are explained below:
[0062] Database logs
[0063] Database logs record information about operations performed on the database, including operation type, data processed, and operation time.
[0064] multicast
[0065] Multicast is a point-to-multipoint communication method. In multicast transmission, data packets are sent from the sender to a specific group of receivers who belong to a multicast group and share a multicast address.
[0066] unicast
[0067] Unicast is a point-to-point communication method. In unicast transmission, a message is sent from a sender to a specific receiver.
[0068] Programmable Router
[0069] A programmable router is a network layer device that enables flexible deployment of new services through configuration and programming. In this disclosure, the programmable router serves as a forwarding device for database logs, and its log transmission logic can be flexibly configured by the master node of the database system according to requirements.
[0070] Black box
[0071] A black box is a data storage device with properties such as resistance to strong impacts, water and water pressure, corrosion, seawater immersion, and high temperatures. Black boxes are primarily used in aircraft to store critical data such as flight data and cockpit audio. In this disclosure, the black box is used as a log storage device to store database logs.
[0072] Database systems in related technologies, such as Figure 1 As shown, a typical system includes the master node of the primary cluster and at least one log synchronization device for the master node. The log synchronization device can be a backup node, disaster recovery cluster, or log storage device within the primary cluster. After generating database logs, the master node sends the database logs to at least one log synchronization device via its network interface card (NIC) to back up both the data and the database logs. Therefore, in the aforementioned related technologies, assuming that at least one log synchronization device includes N backup nodes, a disaster recovery cluster, and a log storage device, the master node needs to perform N+1+1 replications for each database log, resulting in N+1+1 database logs, which are then sent to the N backup nodes, the disaster recovery cluster, and the log storage device respectively. This results in a significant waste of the master node's bandwidth and computing resources.
[0073] In the database system provided in the embodiments of this disclosure, such as Figure 2 As shown, in Figure 1The distributed database system shown introduces a forwarding device group, which includes at least one programmable router. The forwarding device group can communicate with the master node and the log synchronization device, respectively. The master node can send a forwarding control message to the target programmable router in the forwarding device group. This forwarding control message instructs the target programmable router to forward the database log sending logic to at least one log synchronization device. After the master node performs data operations on the database and generates database logs, it only needs to send the database logs to the target programmable router. Then, the target programmable router forwards the database logs to at least one log synchronization device of the master node using the aforementioned sending logic. In the technical solution provided by this disclosure embodiment, after the master node generates database logs, it does not need to perform multiple replications and only needs to send the database logs once to one device (the target programmable router), which can effectively reduce the bandwidth and computing resource consumption of the master node.
[0074] In one possible implementation, the technical solution provided in this disclosure embodiment may include multiple master clusters, each master cluster including one master node, and the database system may further include at least one log synchronization device for each master node. A forwarding device group can support the forwarding of database logs from multiple master nodes. For example... Figure 3 As shown, master node 1 corresponds to at least one log synchronization device 1, master node 2 corresponds to at least one log synchronization device 2, and master node 3 corresponds to at least one log synchronization device 3. The forwarding device group can support forwarding the database logs of master node 1 to log synchronization device 1, and can also support forwarding the database logs of master node 2 to log synchronization device 2, and can also support forwarding the database logs of master node 3 to log synchronization device 3.
[0075] In one possible implementation, the log storage device can be a device with storage capabilities, such as a black box.
[0076] The method for transmitting database logs provided in the embodiments of this disclosure will be described below.
[0077] The database log transmission method provided in this disclosure can be applied to database systems. See [link to documentation]. Figure 4 The method may include the following steps:
[0078] Step 601: The master node performs data operations on the database and generates database logs.
[0079] Users can send data operation requests to the master node through the client. The data operation requested can include creation, deletion, and update. After receiving the data operation request, the master node performs the data operation on the database based on the request and generates a database log to record the data operation after completion.
[0080] Step 602: The master node sends the database logs to the target programmable router.
[0081] After generating the database log, the master node sends the database log to the target programmable router using a specified communication method. This specified communication method can be wired, wireless, or other methods.
[0082] Here, when there is only one programmable router in the forwarding device group, that programmable router is the target programmable router. When there are two or more programmable routers in the forwarding device group, the target programmable router is one of the programmable routers in the forwarding device group. There are several ways to determine which programmable router it is. Several examples are listed below for illustration:
[0083] Method 1:
[0084] The master node determines the target programmable router from the forwarding device group according to the specified load balancing mechanism.
[0085] The specified load balancing mechanism can include Round Robin, Weighted Round Robin, etc. The following examples illustrate Round Robin and Weighted Round Robin respectively:
[0086] polling:
[0087] Assume there are three programmable routers: programmable router 1, programmable router 2, and programmable router 3. When the master node sends database logs to the target programmable routers, it can send them in a round-robin fashion, targeting programmable router 1, programmable router 2, and then programmable router 3. When sending the first database log, the master node targets programmable router 1 and sends the log to it; when sending the second database log, it targets programmable router 2 and sends the log to it; when sending the third database log, it targets programmable router 3 and sends the log to it; when sending the fourth database log, it targets programmable router 1 and sends the log to it, and so on.
[0088] Round-robin based on weights:
[0089] Assume there are three programmable routers: Programmable Router 1, Programmable Router 2, and Programmable Router 3. Based on their hardware performance, network conditions, and other information, each programmable router is assigned a weight. For example, Programmable Router 1 has a weight of 0.5, Programmable Router 2 has a weight of 0.3, and Programmable Router 3 has a weight of 0.2. When the master node sends the first database log to the fifth database log, it uses Programmable Router 1 as the target programmable router and sends the database log to it. When sending the sixth to the eighth database log, it uses Programmable Router 2 as the target programmable router and sends the database log to it. When sending the ninth and tenth database logs, it uses Programmable Router 3 as the target programmable router and sends the database log to it. When sending the eleventh to the fifteenth database log, it uses Programmable Router 1 as the target programmable router and sends the database log to it, and so on.
[0090] Method 2:
[0091] The forwarding device group is deployed using a primary / standby mechanism. Specifically, this mechanism can be implemented based on protocols such as Virtual Router Redundancy Protocol (VRRP) and Hot Standby Router Protocol (HSRP). The following explains the VRRP-based primary / standby mechanism:
[0092] Multiple programmable routers form a virtual forwarding device group, collectively functioning as a single logical forwarding device with a unique IP address. When the primary programmable router fails, backup programmable routers can quickly take over, ensuring the normal forwarding of database logs. VRRP defines active and backup states. The active programmable router is the primary programmable router, while the others are in backup state, acting as standby programmable routers. Among the multiple programmable routers, a primary programmable router is elected based on its configured priority. The programmable router with the highest priority is set to the active state and becomes the primary programmable router. The primary programmable router periodically sends VRRP advertisement messages to each backup programmable router, informing them of its operational status. If the primary programmable router fails, the backup programmable routers can re-elect a new primary programmable router. The primary programmable router is the target programmable router.
[0093] Step 603: The target programmable router forwards database logs to at least one log synchronization device of the master node.
[0094] After receiving the database logs sent by the master node, the target programmable router forwards the database logs to at least one log synchronization device of the master node according to the configured sending logic. The sending logic can be multicast or unicast. These two sending logics are explained below:
[0095] Sending logic one (multicast):
[0096] The target programmable router, based on the ingress port of the received database log, queries the multicast forwarding table to determine the corresponding target multicast address and target egress port, where the target multicast address is the destination address. Then, it encapsulates the database log into a multicast packet, which carries the database log and the target multicast address. Finally, it forwards the multicast packet through the target egress port. The multicast forwarding table can be shown in Table 1 below:
[0097] Table 1
[0098] Ingress port Destination address Output port Port 1 Multicast address 1 Port 5, Port 6, Port 7, Port 8 Port 2 Multicast address 2 Port 10, Port 11, Port 12, Port 13 …… …… ……
[0099] The following example illustrates the sending logic one, using Table 1 as an example:
[0100] After receiving the database logs from master node 1 via port 1, the system queries Table 1 above to determine that the destination address is multicast address 1 and the outgoing ports are port 5, port 6, port 7, and port 8. The database logs are then encapsulated into multicast packets, each carrying the database logs and multicast address 1. These multicast packets are then forwarded via ports 5, 6, 7, and 8. Ports 5, 6, 7, and 8 are connected to various log synchronization devices on master node 1, such as backup nodes, disaster recovery clusters, and log storage devices. These connections can be direct or indirect.
[0101] Sending logic two (unicast):
[0102] The target programmable router, based on the ingress port of the received database log, queries the unicast forwarding table to determine the corresponding multiple target IP addresses and the target egress port for each target IP address, where the target IP address is the destination address. Then, it encapsulates the database log into multiple unicast packets. Each unicast packet carries the database log and one of the multiple target IP addresses. Finally, it forwards the corresponding unicast packet through the target egress port corresponding to the target IP address. The unicast forwarding table is shown in Table 2 below:
[0103] Table 2
[0104]
[0105] The following example illustrates the second sending logic with reference to Table 2:
[0106] After receiving the database log from master node 3 via port 3, the system queries Table 2 above to determine the destination addresses as IP address 1, IP address 2, IP address 3, and IP address 4. The outgoing port for IP address 1 is port 20, for IP address 2 it is port 21, for IP address 3 it is port 22, and for IP address 4 it is port 23. Then, the database log is encapsulated into unicast packets 1, 2, 3, and 4. Unicast packet 1 carries the database log and IP address 1; unicast packet 2 carries the database log and IP address 2; unicast packet 3 carries the database log and IP address 3; and unicast packet 4 carries the database log and IP address 4. Finally, unicast packet 1 is forwarded via port 20, unicast packet 1 via port 21, unicast packet 3 via port 22, and unicast packet 4 via port 23. Ports 20, 21, 22, and 23 are connected to the log synchronization devices of the master node 3, which can be backup nodes, disaster recovery clusters, log storage devices, etc. These connections can be direct or indirect.
[0107] In one possible implementation, when a programmable router in the forwarding device group fails, the master node can switch to a network interface card (NIC) forwarding mode where the master node's own NIC forwards database packets to the log synchronization device. This switch can be performed manually by relevant personnel or automatically by the master node. The method for automatic master node switching is described below:
[0108] After forwarding the database log, the target programmable router can return a forwarding completion message to the master node. If the master node does not receive a forwarding completion message from the target programmable router within the first time interval after sending the database log, it will resend the database log to the target programmable router. If, after N retries, the master node still cannot receive a forwarding completion message from the target programmable router within the target time interval after sending the database log, it can be determined that the target programmable router is faulty, and the forwarding mode of the database log will be switched to network card forwarding mode. The target time interval and the retry threshold N can both be configured by relevant personnel according to actual needs; for example, the first time interval can be 1 second, and N can be 3.
[0109] Furthermore, after switching to NIC forwarding for the second duration, the master node can re-switch to the target programmable router for forwarding. If the forwarding device still fails, it can switch back to NIC forwarding. The second duration can be configured by relevant personnel according to actual needs; for example, the second duration could be 1 hour.
[0110] The sending logic used by the target programmable router to forward database logs can be configured directly by relevant personnel or by the master node. The following explains the master node's configuration of the target programmable router's database log sending logic; see [link to documentation]. Figure 5 The configuration process may include the following steps:
[0111] Step 701: The master node sends a forwarding control message to the target programmable router.
[0112] Relevant personnel input the sending logic configuration command to the master node. The forwarding configuration command carries the sending logic identifier. For example, the sending logic identifier is 1 to indicate multicast, and the sending logic identifier is 0 to indicate unicast.
[0113] The master node generates a forwarding control message based on the sending logical identifier. Specifically, if the sending logical identifier indicates multicast, the master node selects a multicast address from the available multicast addresses and generates a forwarding control message. This message carries the selected multicast address and multicast forwarding indication information. All log synchronization devices of this master node belong to the same multicast group and share this multicast address. If the sending logical identifier indicates unicast, the master node obtains the IP addresses of each log synchronization device, such as the IP addresses of each backup node, the disaster recovery cluster, and the log storage device, and generates a forwarding control message. This message carries the IP addresses of each log synchronization device and unicast forwarding indication information. Here, the IP address of the disaster recovery cluster refers to the IP address of a specified node within the disaster recovery cluster; this specified node can be any node in the disaster recovery cluster.
[0114] After generating the forwarding control message, the master node sends the forwarding control message to the target programmable router.
[0115] Step 702: The forwarding device determines the sending logic based on the received forwarding control message.
[0116] After receiving a forwarding control message, the forwarding device parses it. If the forwarding control message carries multicast forwarding indication information, the sending logic is determined to be multicast, and a multicast forwarding table is established according to the multicast protocol. If the forwarding control message carries unicast forwarding indication information, the sending logic is determined to be unicast, and a unicast forwarding table is established.
[0117] In one possible implementation, Figure 3In the database system architecture shown, the forwarding device group needs to support the forwarding of database logs from multiple master nodes. Database logs from different master nodes can use different sending logics. In this case, the target programmable router can record the sending logic required for database logs from different master nodes. The recording method is explained below:
[0118] Specifically, after receiving a forwarding control message from the master node, the target programmable router, if it determines that the forwarding control message carries multicast forwarding indication information, records the correspondence between the ingress port of the forwarding control message and multicast forwarding; if it determines that the forwarding control message carries unicast forwarding indication information, it records the correspondence between the ingress port of the forwarding control message and unicast forwarding. The above correspondence can be shown in Table 3 below:
[0119] Table 3
[0120] Ingress port Sending logic Port 1 multicast Port 2 multicast Port 3 unicast …… ……
[0121] Port 1, port 2, port 3, etc., connect to the master nodes of different master clusters. Table 1 is illustrated using an example where port 1, port 2, and port 3 are connected to master node 1, master node 2, and master node 3 respectively:
[0122] The database log sent by master node 1 is received by the target programmable router via port 1. After receiving the database log from port 1, the target programmable router determines, by consulting Table 3 above, that the database log should be forwarded via multicast, and then executes the first sending logic described above. The database log sent by master node 2 is received by the target programmable router via port 2. After receiving the database log from port 2, the target programmable router determines, by consulting Table 3 above, that the database log should be forwarded via multicast, and then executes the first sending logic described above. The database log sent by master node 3 is received by the target programmable router via port 3. After receiving the database log from port 3, the target programmable router determines, by consulting Table 3 above, that the database log should be forwarded via unicast, and then executes the second sending logic described above.
[0123] The master node can dynamically change its forwarding behavior based on changes in the deployment of log synchronization devices. When any log synchronization device on the master node fails and is removed, or when a new log synchronization device is added, the master node can send a forwarding change message to alter the forwarding behavior of the target programmable router. For example... Figure 6 As shown, the process may include the following steps:
[0124] Step 801: In the case of adding a log synchronization device, the master node sends a first forwarding change message to the target programmable router. The first forwarding change message carries the IP address of the added log synchronization device and the indication information of the added log synchronization device.
[0125] When a new log synchronization device is added to the database system, relevant personnel can send a message to the master node, which includes the IP address of the new log synchronization device. Then, the master node sends a first forwarding change message to the target programmable router, which includes the IP address of the new log synchronization device and indication information about the new log synchronization device.
[0126] Step 802: The target programmable router updates its forwarding table based on the first forwarding change message.
[0127] When the transmission logic is multicast forwarding, the target programmable router, upon receiving the first forwarding change message, parses the message. If it determines that the message carries an indication of a newly added log synchronization device, it adds an entry corresponding to the newly added log synchronization device to the multicast forwarding table. The following example illustrates this:
[0128] Assuming a backup node is added to primary node 2, primary node 2 sends a first forwarding change message to the new log synchronization device. This message carries the IP address of the new backup node. The target programmable router receives the first forwarding change message via port 2. The forwarding device then queries Table 3 to determine that the corresponding sending logic is multicast. Next, the target programmable router determines the corresponding outgoing port as port 15 based on the unicast routing information of the new backup node's IP address. Finally, the target programmable router adds port 15 to the outgoing entry corresponding to multicast address 2 in Table 1. Table 1 is then updated to Table 4 as follows:
[0129] Table 4
[0130]
[0131] When the transmission logic is unicast forwarding, after receiving the first forwarding change message, the target programmable router adds an entry corresponding to the newly added log synchronization device to the unicast forwarding table. The following example illustrates this:
[0132] Suppose a backup node is added to primary node 3. Primary node 3 sends a first forwarding change message to primary node 2, carrying the IP address 5 of the newly added backup node. The target programmable router receives the first forwarding change message via port 3. The forwarding device then queries Table 3 to determine that the corresponding sending logic is unicast. Based on the unicast routing information for IP address 5, the target programmable router determines the corresponding outgoing port to be port 24. Finally, the target programmable router adds the mapping between IP address 5 and port 24 to the entry for port 3 in Table 2. Table 2 is then updated to Table 5 as follows:
[0133] Table 5
[0134]
[0135] Step 803: In the case of removing the log synchronization device, the master node sends a second forwarding change message to the target programmable router. The second forwarding change message carries the IP address of the removed log synchronization device and the indication information for removing the log synchronization device.
[0136] When removing one or more log synchronization devices from a master node, personnel can remove the log synchronization devices from the master node, with the removal message carrying the IP address of the removed device. Then, the master node sends a second forwarding change message to the target programmable router, carrying the IP address of the removed log synchronization device and removal indication information.
[0137] Step 804: The target programmable router updates its forwarding table based on the second forwarding change message.
[0138] When the transmission logic is multicast forwarding, the target programmable router, upon receiving the second forwarding change message, parses the message. If it determines that the message carries an indication to remove the log synchronization device, it deletes the entry corresponding to the removal of the log synchronization device from the multicast forwarding table. The following example illustrates this:
[0139] Suppose that a backup node of node 1 fails and needs to be removed. The primary node 1 sends a second forwarding change message to the target programmable router. This message carries the IP address of the failed backup node. The target programmable router receives the message via port 1. It then queries Table 3 to determine that the corresponding transmission logic is multicast. Next, based on the unicast routing information of the removed backup node's IP address, it determines the corresponding outgoing port to be port 5. Finally, the forwarding device removes port 5 from the multicast forwarding table, updating Table 1 as shown in Table 6 below.
[0140] Table 6
[0141] Ingress port Destination address Output port Port 1 Multicast address 1 Port 6, Port 7, Port 8 Port 2 Multicast address 2 Port 10, Port 11, Port 12, Port 13 …… …… ……
[0142] When the transmission logic is unicast forwarding, after receiving the second forwarding change message, the target programmable router deletes the entry corresponding to the removed log synchronization device from its unicast forwarding table. The following example illustrates this:
[0143] Suppose that a backup node of primary node 4 fails and needs to be removed. Primary node 4 sends a second forwarding change message to the target programmable router. This message carries the IP address 7 of the removed backup node. The target programmable router receives the second forwarding change message via port 4. By consulting Table 3 above, the target programmable router determines that the corresponding transmission logic is unicast. Then, the target programmable router deletes the mapping between IP address 7 and port 30 from Table 2 above, updating Table 2 to Table 7 as follows:
[0144] Table 7
[0145]
[0146] After receiving database logs, the log synchronization device returns a receipt confirmation message to the master node. This confirmation message can be sent directly to the master node or forwarded by the forwarding device. The following explains the forwarding scenario using the forwarding device. Figure 7 and 8 As shown, the process may include the following steps:
[0147] Step 901: Upon receiving the database logs, the log synchronization device sends a receipt confirmation message to the target programmable router.
[0148] Step 902: The target programmable router forwards the received reception confirmation message to the master node.
[0149] The destination address carried in the receive confirmation message sent by the log synchronization device can be the IP address of the master node. After receiving the receive confirmation message, the target programmable router forwards it to the master node according to the destination address carried in it.
[0150] Step 903: Based on the received confirmation message, the master node determines whether the log synchronization device's reception of the database logs meets the specified conditions. If the master node determines that the log synchronization device's reception of the database logs meets the specified conditions, it sends a data operation completion message to the client.
[0151] The specified conditions can be configured according to actual needs. In the case that the log synchronization device includes a backup node, a disaster recovery cluster, and a log storage device, the following are some examples of specified conditions:
[0152] Specified condition one:
[0153] Received a receive confirmation message from at least one backup node of the primary node.
[0154] Specified condition two:
[0155] It receives a receipt confirmation message from at least one backup node and the disaster recovery cluster, respectively, from the primary node.
[0156] Specified condition three:
[0157] Receives acceptance confirmation messages from at least one backup node and the log storage device, respectively, from the primary node.
[0158] Specified condition four:
[0159] It receives confirmation messages from at least one backup node of the primary node, the disaster recovery cluster, and the log storage device.
[0160] Specified condition five:
[0161] It receives confirmation messages from all backup nodes, disaster recovery clusters, and log storage devices of the primary node.
[0162] To further conserve the master node's bandwidth and computing resources, the target programmable router can avoid forwarding the receive confirmation messages sent by the log synchronization device to the master node. Instead, it can perform the aforementioned condition checks independently and inform the master node of the results. Correspondingly, such as Figure 9 and 10 As shown, the process may include the following steps:
[0163] Step 1101: Upon receiving the database logs, the log synchronization device sends a receipt confirmation message to the target programmable router.
[0164] Step 1102: The target programmable router determines whether the log synchronization device's reception of database logs meets the specified conditions based on the received reception confirmation message. If the log synchronization device's reception of database logs meets the specified conditions, it sends the first message to the master node.
[0165] When the target programmable router determines that the log synchronization device's reception of database logs meets specified conditions, it generates a first message. This first message carries first indication information, which indicates that the log synchronization device's reception of database logs meets the specified conditions. Specifically, the first indication information can be a target field of 1 in the first message.
[0166] When the target programmable router determines that the log synchronization device's reception of database logs meets specified conditions, it generates a second message. This second message carries second indication information, which indicates that the log synchronization device's reception of database logs does not meet the specified conditions. The second indication information can be a target field of 0 from the first message.
[0167] Step 1103: Upon receiving the first message, the master node sends a data operation completion message to the client.
[0168] After receiving the first message from the target programmable router, if the master node parses out the first indication information carried in the first message, it determines that the log synchronization device's reception of the data log meets the specified conditions, and then sends a data operation completion message to the client.
[0169] This disclosure also provides a database system, such as Figure 11 As shown, the database system includes the master node of the main cluster, a forwarding device group, and at least one log synchronization device for the master node. The forwarding device group includes at least one programmable router. The log synchronization device is a backup node of the main cluster, a disaster recovery cluster of the main cluster, or a log storage device.
[0170] The master node is used to: perform data operations on the database and generate database logs, wherein the database logs are used to record the operation information of this data operation; specifically, it can implement the processing of step 601 and its implicit steps as described above.
[0171] The master node is also used to: send a forwarding control message to the target programmable router, wherein the forwarding control message is used to instruct the target programmable router to forward the sending logic of the database log to the at least one log synchronization device, and the target programmable router is a programmable router in the forwarding device group; specifically, the processing of the above step 701 and its implicit steps can be implemented.
[0172] The master node is also used to: send the database logs to the target programmable router; specifically, it can implement the processing of step 602 and its implicit steps.
[0173] The target programmable router is used to: forward the database logs to the at least one log synchronization device based on the sending logic. Specifically, it can implement the forwarding processing of step 603 and its implicit steps described above.
[0174] In one possible implementation, the sending logic is either multicast or unicast.
[0175] In one possible implementation, the at least one log synchronization device is used for:
[0176] Upon receiving the database log, a receipt confirmation message is sent to the target programmable router;
[0177] The target programmable router is also used for:
[0178] Based on the received confirmation message, determine whether the reception status of the database logs by the at least one log synchronization device meets the specified conditions. If it is determined that the reception status of the database logs by the at least one log synchronization device meets the specified conditions, send a specified message to the master node.
[0179] The master node is also used for:
[0180] Upon receiving the specified message, a data operation completion message is sent to the client.
[0181] In one possible implementation, the database system further includes the master nodes of other master clusters and log synchronization devices for the master nodes of the other master clusters;
[0182] The target programmable router is also used for:
[0183] Determine the target port for receiving the forwarding control message;
[0184] Add the target port and the sending logic indicated by the forwarding control message to the correspondence between the port and the sending logic;
[0185] After receiving the database log, determine the target port for receiving the database log;
[0186] Based on the correspondence, determine the sending logic corresponding to the target port;
[0187] The target programmable router is used for:
[0188] Based on the sending logic corresponding to the target port, the database logs are forwarded to the at least one log synchronization device.
[0189] In one possible implementation, the master node is also used for:
[0190] The target programmable router is determined from the group of forwarding devices according to the load balancing mechanism.
[0191] In one possible implementation, the log storage device is a black box.
[0192] Both the master node and the programmable router can be implemented in software or hardware. For example, the implementation of the master node will be described below. Similarly, the implementation of the programmable router can be referenced from that of the master node.
[0193] As an example of a software functional unit, a master node can include code running on a compute instance. This compute instance can be at least one of a physical host (compute device), a virtual machine, a container, or other compute devices. Furthermore, the aforementioned compute devices can be one or more. For example, a master node can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.
[0194] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0195] As an example of a hardware functional unit, a master node can include at least one computing device, such as a server. Alternatively, a master node can also be a device implemented using an ASIC or a PLD. The aforementioned PLD can be implemented using a CPLD, FPGA, GAL, or any combination thereof.
[0196] The master node comprises multiple computing devices that can be distributed within the same region or in different regions. Similarly, the YY device comprises multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the YY device comprises multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0197] In the database system provided in this disclosure, a forwarding device group is introduced, which includes at least one programmable router. The forwarding device group can communicate with the master node and the log synchronization device, respectively. The master node can send a forwarding control message to the target programmable router in the forwarding device group. The forwarding control message instructs the target programmable router to forward the database log sending logic to at least one log synchronization device. After the master node performs data operations on the database and generates database logs, it only needs to send the database logs to the target programmable router. Then, the target programmable router forwards the database logs to at least one log synchronization device of the master node using the aforementioned sending logic. In the technical solution provided in this disclosure, after the master node generates database logs, it does not need to perform multiple replications and only needs to send the database logs once to one device (the target programmable router), which can effectively reduce the bandwidth and computing resource consumption of the master node.
[0198] This disclosure also provides a computing device 100. For example... Figure 12 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, memory 106, and communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server, router, switch, programmable router, or other similar device. A server may be a central server, an edge server, or a local server in a local data center. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 100.
[0199] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).
[0200] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0201] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0202] The memory 106 stores executable code, which the processor 104 executes to implement the functions of the master node or programmable router in the above embodiments, thereby realizing the method of database log transmission. That is, the memory 106 stores instructions for executing the method of database log transmission.
[0203] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.
[0204] This disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device can be a server, router, switch, programmable router, or similar device. The server can be, for example, a central server, an edge server, or a local server in a local data center.
[0205] like Figure 13 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for performing a method of database log transmission.
[0206] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for performing the database log transmission method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for performing the database log transmission method.
[0207] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are used to execute part of the functions of the database log transmission method.
[0208] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for performing the database log transmission method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for performing the database log transmission method.
[0209] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing parts of the database log transmission method. That is, the instructions stored in the memory 106 of different computing devices 100 can implement different functions of the master node or different functions of the programmable router.
[0210] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14 As shown, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for performing a portion of the functions of the master node. Simultaneously, the memory 106 in computing device 100B stores instructions for performing another portion of the functions of the master node. It should be understood that... Figure 14 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.
[0211] This disclosure also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 13 and Figure 14 The connection method of the computing device cluster is different in that the memory 106 of one or more computing devices 100 in the computing device cluster can store the same instructions for performing the database log transmission method.
[0212] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for performing the database log transmission method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for performing the database log transmission method.
[0213] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing some functions of the database log transmission device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement different functions of the master node or different functions of the programmable router.
[0214] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on a computing device, it causes the computing device to perform a method for database log transmission.
[0215] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disk (DVD)), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to perform a method for database log transmission.
[0216] In this disclosure, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first" and "second," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first message can be referred to as a second message, and similarly, a second message can be referred to as a first message. Both first and second messages can be collectively referred to as messages, and in some cases, they can be separate and distinct messages.
[0217] In this disclosure, the term "at least one" means one or more, and the term "multiple" means two or more.
[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this disclosure.
Claims
1. A database system, characterized in that, The database system includes a master node of the master cluster, a forwarding device group, and at least one log synchronization device of the master node. The forwarding device group includes at least one programmable router. The log synchronization device is a backup node of the master cluster, a disaster recovery cluster of the master cluster, or a log storage device. The master node is used for: Perform data operations on the database and generate database logs, wherein the database logs are used to record the operation information of this data operation; Send a forwarding control message to the target programmable router, wherein the forwarding control message is used to instruct the target programmable router to forward the sending logic of the database log to the at least one log synchronization device, and the target programmable router is a programmable router in the forwarding device group; Send the database log to the target programmable router; The target programmable router is used for: Based on the sending logic, the database logs are forwarded to the at least one log synchronization device.
2. The database system according to claim 1, characterized in that, The sending logic is either multicast or unicast.
3. The database system according to claim 1 or 2, characterized in that, The at least one log synchronization device is used for: Upon receiving the database log, a receipt confirmation message is sent to the target programmable router; The target programmable router is also used for: Based on the received confirmation message, determine whether the reception status of the database logs by the at least one log synchronization device meets the specified conditions. If it is determined that the reception status of the database logs by the at least one log synchronization device meets the specified conditions, send a specified message to the master node. The master node is also used for: Upon receiving the specified message, a data operation completion message is sent to the client.
4. The database system according to any one of claims 1-3, characterized in that, The database system also includes the master nodes of other master clusters and the log synchronization devices of the master nodes of the other master clusters; The target programmable router is also used for: Determine the target port for receiving the forwarding control message; Add the target port and the sending logic indicated by the forwarding control message to the correspondence between the port and the sending logic; After receiving the database log, determine the target port for receiving the database log; Based on the correspondence, determine the sending logic corresponding to the target port; The target programmable router is used for: Based on the sending logic corresponding to the target port, the database logs are forwarded to the at least one log synchronization device.
5. The database system according to any one of claims 1-4, characterized in that, The master node is also used for: The target programmable router is determined from the group of forwarding devices according to the load balancing mechanism.
6. The database system according to any one of claims 1-5, characterized in that, The log storage device is a black box.
7. A method for transmitting database logs, characterized in that, The method is applied to a database system, which includes a master node of a master cluster, a forwarding device group, and at least one log synchronization device of the master node. The forwarding device group includes at least one programmable router. The log synchronization device is a backup node of the master cluster, a disaster recovery cluster of the master cluster, or a log storage device. The method includes: The master node performs data operations on the database and generates database logs, which are used to record the operation information of this data operation. The master node sends a forwarding control message to the target programmable router, wherein the forwarding control message is used to instruct the target programmable router to forward the sending logic of the database log to the at least one log synchronization device, and the target programmable router is a programmable router in the forwarding device group; The master node sends the database log to the target programmable router; The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic.
8. The method according to claim 7, characterized in that, The sending logic is either multicast or unicast.
9. The method according to claim 7 or 8, characterized in that, The method also includes: Upon receiving the database logs, the at least one log synchronization device sends a receipt confirmation message to the target programmable router. The target programmable router determines, based on the received reception confirmation message, whether the reception status of the database logs by the at least one log synchronization device meets the specified conditions. If it is determined that the reception status of the database logs by the at least one log synchronization device meets the specified conditions, it sends a specified message to the master node. Upon receiving the specified message, the master node sends a data operation completion message to the client.
10. The method according to any one of claims 7-9, characterized in that, The database system also includes the master nodes of other master clusters and the log synchronization devices of the master nodes of the other master clusters; The method further includes: The target programmable router determines the target port for receiving the forwarding control message; The target programmable router adds the target port and the transmission logic indicated by the forwarding control message to the correspondence between the port and the transmission logic. After receiving the database log, the target programmable router determines the target port for receiving the database log; and determines the transmission logic corresponding to the target port according to the correspondence. The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic, including: The target programmable router forwards the database logs to the at least one log synchronization device based on the sending logic corresponding to the target port.
11. The method according to any one of claims 7-10, characterized in that, The method further includes: The master node determines the target programmable router from the forwarding device group according to the load balancing mechanism.
12. The method according to any one of claims 7-11, characterized in that, The log storage device is a black box.
13. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, and each computing device includes a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 7-12.
14. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of forwarding devices, enable the cluster of computing devices to perform the method as described in any one of claims 7-12.
15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 7-12.