Log synchronization system, method and equipment of database and medium

Abstract

The invention provides a log synchronization system, method and device of a database and a medium, and relates to the technical field of databases, in order to relieve the technical problem that logs are not synchronized after main and standby switching in the prior art, the method comprises the steps that a pre-written log directory of an old host is obtained; determining the latest log position of the old host based on the pre-write log directory of the old host; based on the latest log position, determining a log bifurcation position between the new host and the old host, determining a to-be-synchronized transaction set from the old host to the new host, and determining start log serial numbers of all transactions in the to-be-synchronized transaction set; performing logic decoding on the to-be-synchronized transaction set to obtain structured change data; on the basis of the structured change data, the new host is triggered to execute log synchronization operation, so that log synchronization of the new host and the original host is achieved after the host and the standby are switched, and consistency and reliability of data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular, to a log synchronization system, method, device, and medium for a database. Background Art

[0002] In the prior art, during the use of the master-slave architecture in a database system, when an exception occurs in the master (such as the master node crashing), the asynchronous slave is switched with the master, that is, the asynchronous slave becomes the new master. There will be a problem of log asynchronization between the new master after the switch and the master with the exception. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a log synchronization system, method, device, and medium for a database to alleviate the technical problem of log asynchronization that occurs after the master-slave switch in the prior art.

[0004] In a first aspect, the present application provides a log synchronization system for a database. The log synchronization system is respectively connected to a new master and an old master; wherein, the new master is the original asynchronous backup node, and the old master is the original master node. The log synchronization system includes: A log position acquisition component, configured to determine the latest log position of the old master based on the write-ahead log directory of the old master; A fork point location component, configured to determine the log fork position between the new master and the old master and determine the set of transactions to be synchronized from the old master to the new master based on the latest log position, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; A decoded synchronization transaction component, configured to perform logical decoding on the set of transactions to be synchronized to obtain structured change data; and trigger the new master to perform a log synchronization operation based on the structured change data.

[0005] Optionally, the log position acquisition component is configured to determine the latest write-ahead log file corresponding to the write-ahead log directory based on the write-ahead log directory; and determine the latest log position corresponding to the latest write-ahead log file based on the latest write-ahead log file; wherein, the latest write-ahead log file includes multiple transaction log record segments.

[0006] Optionally, the fork point location component is configured to sequentially read the transaction log record segments in the latest write-ahead log file from the back forward based on the latest log position, and confirm whether the transaction log record segments are recorded in the new master. If not, mark the transactions corresponding to the transaction log record segments as needing to be decoded. If recorded, determine the position of the transaction log record segment as the log fork position.

[0007] Optionally, locate the fork point component to sequentially read transaction log record segments forward based on the log fork position and confirm whether the transaction log record segments are committed to the new host. If not, update the start log sequence number of the transaction corresponding to the transaction log record segment until the start log sequence numbers of all transactions in the transaction set to be synchronized are determined.

[0008] Optionally, the decoded synchronization transaction component is used to determine the secure restart log sequence number corresponding to the transaction set to be synchronized based on the log fork position; and based on the secure restart log sequence number, perform logical replication and pull changes on the transaction log record segments in the transaction set to be synchronized through a preset decoding policy to obtain a file in sql or json format as structured change data.

[0009] Optionally, the decoded synchronization transaction component is used to decode the transaction log record segments in the transaction set to be synchronized through region decoding to obtain a file in sql or json format as structured change data when the secure restart log sequence number does not exist in the transaction set to be synchronized; where region decoding is to decode the write-ahead log file between the start log sequence number and the end log sequence number of the transaction.

[0010] Optionally, the log synchronization system of the database in this application further includes an initialization configuration component for initializing the decoding plug-in to the default value; initializing the parameters that require password authentication and forced range decoding to negative or disabled; and obtaining the parameter values configured by the user for the initialization parameters and updating the current values of the initialization parameters based on the parameter values.

[0011] In a second aspect, this application provides a method for synchronizing the log of a database, including: Obtain the write-ahead log directory of the old host; Based on the write-ahead log directory of the old host, determine the latest log position of the old host; Based on the latest log position, determine the log fork position between the new host and the old host and determine the transaction set to be synchronized from the old host to the new host, and determine the start log sequence numbers of all transactions in the transaction set to be synchronized; Perform logical decoding on the transaction set to be synchronized to obtain structured change data; Based on the structured change data, trigger the new host to perform a log synchronization operation.

[0012] In a third aspect, this application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned method for synchronizing the log of the database.

[0013] Fourthly, the present application also provides a computer-readable storage medium storing computer instructions which, when executed by a processor, implement the above-mentioned log synchronization method for the database.

[0014] A log synchronization system, method, device and medium for a database provided by an embodiment of the present invention obtain the write-ahead log directory of an old host; determine the latest log position of the old host based on the write-ahead log directory of the old host; determine the log fork position between the new host and the old host and determine the set of transactions to be synchronized from the old host to the new host based on the latest log position, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; perform logical decoding on the set of transactions to be synchronized to obtain structured change data; trigger the new host to perform a log synchronization operation based on the structured change data, so as to realize log synchronization between the new host and the original host after the master-slave switch, ensuring data consistency and reliability.

[0015] To make the above objects, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 Shows a schematic structural diagram of a log synchronization system for a database provided by an embodiment of the present invention; Figure 2 Shows a schematic diagram of the intersection of the old host and the new host provided by an embodiment of the present invention; Figure 3 Shows a schematic diagram of determining the position of the log fork point provided by an embodiment of the present invention; Figure 4 Shows a schematic flowchart of determining the safe restart log sequence number provided by an embodiment of the present invention; Figure 5 Shows a schematic flowchart of the decoding synchronization transaction component performing logical decoding on the set of transactions to be synchronized provided by an embodiment of the present invention; Figure 6 Shows a schematic interaction flowchart of the log synchronization method for a database provided by an embodiment of the present invention; Figure 7 Shows a schematic flowchart of a log synchronization method for a database provided by an embodiment of the present invention; Figure 8 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of the embodiments. Usually, the components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0019] Currently, in a database system with a primary and standby architecture, according to the log synchronization policy between the standby database and the primary database, the standby database can be divided into a synchronous standby and an asynchronous standby. Among them, the synchronous standby refers to a node where the primary node needs to wait for the standby node to confirm when submitting a transaction to ensure data consistency. Therefore, the synchronous standby is consistent with the primary node; the asynchronous standby refers to a node where the primary node does not need to wait for the standby node to confirm when submitting a transaction. Therefore, the logs of the asynchronous standby are often inconsistent with the primary node. When using the primary and standby architecture, when the primary host may have an exception (such as the primary host crashing), a primary-standby switchover occurs, that is, the primary host is switched to an asynchronous standby host, and there will be an out-of-sync problem between the logs of the switched primary host and the original primary host.

[0020] Figure 1 The present application provides a schematic structural diagram of a log synchronization system for a database, as Figure 1 shown. The log synchronization system for a database provided by an embodiment of the present application is respectively connected to a new host and an old host; among them, the new host is the original asynchronous backup node, and the old host is the original primary node. The system provided by the embodiment of the present application at least includes: a log position acquisition component 120, a fork point positioning component 130, and a decoded synchronization transaction component 140; among them, the log position acquisition component 120 is used to determine the latest log position of the old host based on the write-ahead log directory of the old host; the fork point positioning component 130 is used to determine the log fork position between the new host and the old host and determine the set of transactions to be synchronized from the old host to the new host based on the latest log position, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; the decoded synchronization transaction component 140 is used to perform logical decoding on the set of transactions to be synchronized to obtain structured change data; and trigger the new host to perform a log synchronization operation based on the structured change data.

[0021] In the embodiments of the present application, the fork position that appears in the old and new host logs is determined by the log sequence number and the transaction status, and the target format log is obtained through transaction decoding by the write-ahead log. The new host restores the log by executing the target format to achieve the synchronization of the old and new host log information.

[0022] In an alternative embodiment, the log position acquisition component 120 is used to determine the latest write-ahead log file corresponding to the write-ahead log directory based on the write-ahead log directory; and determine the latest log position corresponding to the latest write-ahead log file based on the latest write-ahead log file, where the latest write-ahead log file includes multiple transaction log record segments.

[0023] In the embodiments of the present application, the log position acquisition component 120 is used to determine the latest (i.e., the largest numbered) and readable valid record WAL file in the WAL (write-ahead log) directory of the specified old host, and determine the maximum LSN (Log Sequence Number), the maximum LSN value, and relevant information such as WAL file information, time information, and distributed environment information; where the WAL file information is the WAL file name and the exact offset position; the time information is the last checkpoint time and the transaction commit timestamp; the distributed environment information is the LSN synchronization difference and replication delay amount between the master and slave nodes.

[0024] In a specific embodiment, the log position acquisition component 120 first obtains the log directory of the old host, and accesses all file names in the log directory of the old host by traversing (such as in chronological order), and selects the latest WAL log file from them. The WAL log file name is a 24-bit hexadecimal character, where the first 8 bits are the timeline (time line ID) used for recovery and point-in-time recovery, the middle 8 bits are the logid (logical ID) used to identify the continuous log segment range, and the last 8 bits are the segno (segment sequence number) used for the sequential number within the logical ID. The latest WAL log file can be obtained by comparing the string lexicographical order of the WAL log file names, that is, the combination value of the logical ID and the segment sequence number with the largest value is the latest WAL log file; Then, traverse the files in the log directory in reverse order from the latest WAL log file to determine the first readable log segment, and use the first readable log segment as the starting point; where the readable log segment needs to meet the following conditions: the log file physically exists on the disk, has not been deleted or moved; the current process has the permission to read the log file; the content of the log file has not been lost and can be normally parsed (such as passing the checksum verification); the log file format meets the expectations (such as the header identifier of the WAL log file is correct); further, the first readable log segment can be determined by programmatically attempting to open the log file and read the content. Finally, traverse all the log records backward from the starting point (the first readable log segment) to determine the maximum LSN, that is, the latest log position. Among them, the following traversal method can be used to determine the maximum LSN: at the position of the first readable log segment, maintain two pointers pointing to the start and end of the currently read record. Each time, move ReadRecPtr (the start position of the read record) to the start of the current record, and move EndRecPtr (the end position of the read record) to the start of the next record. Loop until NULL (null pointer) is returned. At this time, ReadRecPtr saves the end position of the last successfully read record, which is the maximum LSN.

[0025] In the log position acquisition component 120 provided in the embodiment of the present application, by determining the latest and readable valid record WAL file in the WAL directory of the specified old host, and determining the maximum LSN, that is, the latest log position, the parsing of the entire log segment and the determination of the log position are completed.

[0026] In an alternative embodiment, the fork point positioning component 130 is used to sequentially read the transaction log record segments in the latest write-ahead log file from back to front based on the latest log position, and confirm whether the transaction log record segments are recorded in the new host. If not, mark the transaction corresponding to the transaction log record segment as needing to be decoded. If recorded, determine the position of the transaction log record segment as the log fork position.

[0027] In the embodiment of the present application, the fork point positioning component 130 is used to determine the fork point between the logs of the old host and the new host, that is, the positions of the different log files between the logs of the old host and the new host, and at the same time collect all the transaction IDs and positions (such as LSN) in the log files of the old host that have been committed to the new host but not synchronized to the remote end (that is, the new host). As Figure 2 shown in the schematic diagram of the intersection of the old host and the new host, 0 / 75720E8 in the figure is the last synchronized LSN between the old host and the new host, that is, the intersection of the old host and the new host.

[0028] In a specific embodiment, transaction log record segments in the latest write-ahead log file are read backward in chronological order from the latest log position. For each read transaction log record segment, check the content of the transaction log record segment to determine whether it contains the identifier of the new host or the newly recorded host data. If the newly recorded host content in the transaction log record segment does not match the current target host, it indicates that the transaction log record segment has not been recorded in the new host. If the transaction log record segment has not been recorded in the current new host, mark the transaction log record segment as needing to be decoded. If the transaction log record segment is recorded in the new host for the first time and it is confirmed that all subsequent transaction log record segments are recorded in the new host, the latest recording time point where the transaction log record segment is located is the log fork position. The difference between the logs of the new host and the old host is determined through the log fork position, thereby facilitating operations such as segmentation and archiving during data processing and error recovery, and facilitating log data analysis and management.

[0029] Further, the fork point locating component 130 is used to sequentially read transaction log record segments forward based on the log fork position, and confirm whether the transaction log record segments are committed to the new host. If not committed, mark the transaction corresponding to the transaction log record segment as needing to be decoded; until the start log sequence numbers of all transactions in the set of transactions to be synchronized are determined (i.e., the minimum LSN that needs to be decoded).

[0030] In an embodiment of the present application, the fork point locating component 130 is further used to maintain a set of transactions to be synchronized that need to be decoded. The set of transactions to be synchronized that need to be decoded records all transaction IDs that have been committed but not synchronized to the new host, for subsequent decoding to obtain structured change data, and records the minimum LSN that needs to be decoded, recording the minimum LSN of all transactions that need to be decoded in the WAL, for subsequent positioning and reading of complete transaction log record segments.

[0031] In a specific embodiment, the fork point locating component 130 sequentially reads transaction log record segments forward starting from the maximum LSN to reduce the query volume across nodes. Among them, according to the recorded LSN and the Cyclic Redundancy Check (CRC) cyclic redundancy check code, determine whether the transaction corresponding to the transaction log record segment is recorded in the new host. If not recorded in the new host, record the transaction ID corresponding to the transaction log record segment in the set of transactions to be synchronized that need to be decoded, and update the minimum LSN that needs to be decoded to the currently recorded LSN; if recorded in the new host, the transaction ID corresponding to the transaction log record segment is the log fork position, end the verification, and obtain the final set of transactions to be synchronized.

[0032] In order to ensure the accuracy of the minimum required decoded LSN, in this application, it is necessary to continue searching forward at the log fork position, continue to verify whether the transaction is marked as needing to be decoded, and continuously update the minimum required decoded LSN until all log files in the old host are searched or the earliest start log sequence number (minimum required decoded LSN) of all transactions in the set of transactions to be synchronized is found.

[0033] As Figure 3 shown, assume that there is a transaction ABC on the old host. During the process of finding the fork point from back to front, the transaction IDs of transactions B and C are in the set of transactions to be synchronized. After finding the fork point position between the new host and the old host, continue to search forward until the search ends (i.e., determine the minimum LSN of the transaction), and the position of the minimum required decoded LSN is the starting position of transaction B.

[0034] In the positioning fork point component 130 provided in the embodiment of this application, by determining the log fork point position between the new host and the old host, the minimum LSN of all transactions that need to be decoded is determined based on the log fork position, so as to facilitate subsequent log file positioning and reading of complete logs.

[0035] In an alternative embodiment, the decoded synchronization transaction component 140 is used to determine the safe restart log sequence number corresponding to the set of transactions to be synchronized based on the log fork position; based on the safe restart log sequence number, the transaction log record segments in the set of transactions to be synchronized are logically replicated and the changes are pulled through a preset decoding policy to obtain a file in sql or json format as structured change data.

[0036] Furthermore, the decoded synchronization transaction component 140 is used to, when the set of transactions to be synchronized does not have a safe restart log sequence number, decode the transaction log record segments in the set of transactions to be synchronized through region decoding to obtain a file in sql or json format as structured change data; where region decoding is to decode the write-ahead log file between the start log sequence number and the end log sequence number of the transaction.

[0037] In the embodiment of this application, the decoded synchronization transaction component 140 decodes the different logs between the new and old hosts into sql or json format through a decoding plugin, so as to facilitate the new host to synchronize the different logs.

[0038] As Figure 4 shown, the process of determining the safe restart log sequence number by the decoded synchronization transaction component 140 during the logical decoding of the set of transactions to be synchronized is as follows: Based on the log fork position, scan the transactions in the set of transactions to be synchronized from the back to the front. If the current transaction was committed before the minimum LSN that needs to be decoded and is not an XLOG_RUNNING_XACTS record, then judge the next transaction until the transaction was committed before the minimum LSN that needs to be decoded and is an XLOG_RUNNING_XACTS. Then start decoding after this log file, and this log file can be used as the (restart LSN) safe restart log sequence number. By determining the point of the safe restart log sequence number to start decoding, neither the intermediate state of any transaction will be missed, nor will the committed transactions be repeated or skipped.

[0039] As Figure 5 shown, the decoding synchronization transaction component 140 performs the logical decoding process on the set of transactions to be synchronized as follows: First, based on the log fork position, scan the transactions in the set of transactions to be synchronized from the back to the front. Determine the safe restart log sequence number in the above manner, and based on the safe restart log sequence number, pull the changes through logical replication. Use the plugin (decoding plugin) parameter of the preset decoding plugin to create a logical replication slot. If the plugin parameter value is not set, the default parameter value is used. Starting from the restart LSN (restart log sequence number), the way to create a logical replication slot is to concatenate an SQL string: select * from pg_create_logical_replication_slot(name, plugin, restart_lsn, confirmed_flush_lsn), where restart_lsn is the safe restart log sequence number obtained in the previous step, and confirmed_flush_lsn is the log fork position. Execute this SQL. Then, loop to pull the logical changes. Similarly, spell out an SQL string: select * from pg_logical_slot_get_changes(slot_name, upto_lsn, count). Here, upto_lsn is set to NULL, which means to obtain all the subsequent changes. Count is set to 1000, and 1000 records are fetched at a time to avoid fetching too many at once. Loop to execute this statement and output the results to the file specified by the file (output file) parameter. Until the number of tuples in the returned PQresult is 0, which means the execution is over, jump out of the loop and delete the replication slot. If, according to the log fork position, transactions in the set of transactions to be synchronized are scanned from the back to the front, and the safe restart log sequence number is not determined in the above manner, the decoding synchronization transaction component 140 can perform regional decoding on the logs between the minimum LSN to be decoded and the maximum LSN. It is possible to directly read the local WAL file segments according to the LSN interval. The interval of each regional decoding is within the same physical WAL log, avoiding the situation of out-of-memory caused by reading multiple large files simultaneously. According to this design, when the start LSN and the end LSN are not in the same file segment, the decode_lsn for each batch is set to the beginning of the next file segment, and an SQL statement is assembled: select * from pg_logical_get_area_changes(start_lsn, decode_lsn,NULL, plugin, NULL); where the slot name and the termination LSN are NULL, indicating no slot and obtaining until the end, and the result is output to a file specified by file.

[0040] In an alternative embodiment, the log synchronization system of the database further includes an initialization configuration component for initializing the decoding plugin to the default value; initializing the parameters that require password authentication and forced range decoding to negative or disabled; and obtaining the parameter values configured by the user for the initialization parameters, and updating the current values of the initialization parameters based on the parameter values.

[0041] In the embodiment of the present application, the initialization configuration component is used to parse the parameters passed in by the user and the environment variables of the log synchronization system, and perform simple verification. The required parameters and their meanings are shown in Table 1: Table 1

[0042] Furthermore, the initialization configuration component initializes the above parameters into global variables. If the user does not input the parameter values corresponding to the parameters, the plugin (decoding plugin) will be initialized to the default value, and password (requiring password authentication) and force (forcing range decoding) will be set to false (negative or disabled); the initialization configuration component 110 is also used to obtain the data directory through the environment variable; and the user can obtain the parameters through --help or -?.

[0043] Based on the same inventive concept, the embodiment of the present application also provides a method for log synchronization of a database. Refer to Figure 6 As shown, the interaction process of the method for log synchronization of the database provided by the embodiment of the present application is as follows: Step 210, obtain the write-ahead log directory of the old host; Step 220, determine the latest log position of the old host based on the write-ahead log directory of the old host; Step 230: Based on the latest log position, determine the log fork position between the new host and the old host, determine the set of transactions to be synchronized from the old host to the new host, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; Step 240: Perform logical decoding on the set of transactions to be synchronized to obtain structured change data; Step 250: Based on the structured change data, trigger the new host to perform a log synchronization operation.

[0044] Next, the log synchronization method for the database mentioned in the embodiments of the present application will be described in detail. Refer to Figure 7 As shown, the general process of the log synchronization method for the database provided by the embodiments of the present application is as follows: Step 310: Set the values of the parameters in the log synchronization system through the initialization configuration component 110 in the log synchronization system. Among them, when there is no user input, initialize the decoding plugin to the default value; initialize the parameters that require password authentication and forced decoding by range to negative or disabled; if there is user input, configure the values of the parameters according to the parameter values input by the user; Step 320: The log position acquisition component in the log synchronization system acquires the write-ahead log directory of the old host; and determines the latest log position of the old host based on the write-ahead log directory of the old host; Step 330: The fork point positioning component in the log synchronization system determines the log fork position between the new host and the old host based on the latest log position and determines the set of transactions to be synchronized from the old host to the new host; among them, based on the latest log position, sequentially read the transaction log record segments in the latest write-ahead log file from back to front, and confirm whether the transaction log record segments are recorded in the new host. If not, mark the transactions corresponding to the transaction log record segments as needing to be decoded. If recorded, determine the position of the transaction log record segment as the log fork position; while determining the log fork position between the new host and the old host, the set of transactions to be synchronized from the old host to the new host can be determined; Step 340: Based on the log fork position, sequentially read the transaction log record segments forward from the log fork position, and confirm whether the transaction log record segments are committed to the new host. If not, update the start log sequence number of the transaction corresponding to the transaction log record segment; until the earliest start log sequence numbers of all transactions in the set of transactions to be synchronized are determined; Step 340: The decoding and synchronization transaction component in the log synchronization system performs logical decoding on the set of transactions to be synchronized to obtain structured change data. Specifically, based on the log fork position, determine the secure restart log sequence number corresponding to the set of transactions to be synchronized. Based on the secure restart log sequence number, perform logical replication and pull changes on the transaction log record segments in the set of transactions to be synchronized through a preset decoding policy, and obtain a file in sql or json format as the structured change data. And when there is no secure restart log sequence number in the set of transactions to be synchronized, decode the transaction log record segments in the set of transactions to be synchronized through region decoding, and obtain a file in sql or json format as the structured change data. Herein, region decoding is to decode the write-ahead log file between the start log sequence number and the end log sequence number of the transaction. Step 350: Trigger the new host to perform log synchronization operations based on the structured change data.

[0045] It should be noted that the principle of the database log synchronization method provided in the embodiments of this application to solve technical problems is similar to that of the database log synchronization system provided in the embodiments of this application. Therefore, the implementation of the database log synchronization method provided in the embodiments of this application can refer to the implementation of the database log synchronization system provided in the embodiments of this application, and the repeated parts will not be elaborated.

[0046] After introducing the database log synchronization system and device provided in the embodiments of this application, next, a brief introduction to the electronic device provided in the embodiments of this application will be given.

[0047] Refer to Figure 8 As shown, the electronic device 500 provided in the embodiments of this application includes at least a processor 501, a memory 502, and a computer program stored on the memory 502 and executable on the processor 501. When the processor 501 executes the computer program, it implements the database log synchronization method provided in the embodiments of this application.

[0048] The electronic device 500 provided in the embodiments of this application may further include a bus 503 connecting different components (including the processor 501 and the memory 502). Among them, the bus 503 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0049] The memory 502 may include a readable storage medium in the form of volatile memory, such as a random access memory (RAM) 5021 and / or a cache memory 5022, and may further include a read-only memory (ROM) 5023. The memory 502 may also include a program tool 5025 having a set (at least one) of program modules 5024, and the program modules 5024 include, but are not limited to, an operating subsystem, one or more application programs, other program modules, and program data, and the implementation of a network environment may be included in each or some combination of these examples.

[0050] The processor 501 may be a processing element or a collective term for multiple processing elements. For example, the processor 501 may be a central processing unit (CPU), or one or more integrated circuits configured to implement the log synchronization method of the database provided in the embodiments of the present application. Specifically, the processor 501 may be a general-purpose processor, including but not limited to a CPU, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0051] The electronic device 500 may communicate with one or more external devices 504 (such as a keyboard, a remote control, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500 (such as a mobile phone, a computer, etc.), and / or communicate with a device that enables the electronic device 500 to communicate with one or more other electronic devices 500 (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 505. And, the electronic device 500 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 506. As Figure 8 shown, the network adapter 506 communicates with other modules of the electronic device 500 through a bus 503. It should be understood that although Figure 8Although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, redundant arrays of independent disks (RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0052] It should be noted that Figure 8 The illustrated electronic device 500 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0053] Next, the computer-readable storage medium provided by the embodiments of the present application will be introduced. The computer-readable storage medium provided by the embodiments of the present application stores computer instructions, and when the computer instructions are executed by a processor, the log synchronization method of the database provided by the embodiments of the present application is implemented. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the log synchronization method of the database provided by the embodiments of the present application by executing the built-in or installed computer instructions.

[0054] In addition, the log synchronization method of the database provided by the embodiments of the present application can also be implemented as a computer program product. The computer program product includes program code, and when the program code runs on a processor, the log synchronization method of the database provided by the embodiments of the present application is implemented.

[0055] The computer program product provided by the embodiments of the present application can adopt one or more computer-readable storage media, and the computer-readable storage media can be, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any suitable combination of the above. Specifically, more specific examples (non-exhaustive list) of computer-readable storage media include electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0056] The computer program product provided by the embodiments of the present application may adopt a CD-ROM and include program codes, and may also run on an electronic device such as a computer. However, the computer program product provided by the embodiments of the present application is not limited thereto. In the embodiments of the present application, the computer-readable storage medium may be any tangible medium that contains or stores program codes, and the program codes may be used by or in combination with an instruction execution system, apparatus, or device.

[0057] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more units described above may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided and embodied by multiple units.

[0058] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0059] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0060] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A log synchronization system for a database, characterized in that, The log synchronization system is respectively connected to the new host and the old host; wherein, the new host is the original asynchronous backup node, and the old host is the original primary node. The log synchronization system includes: A log position acquisition component, configured to determine the latest log position of the old host based on the write-ahead log directory of the old host; A fork point positioning component, configured to determine the log fork position between the new host and the old host and determine the set of transactions to be synchronized from the old host to the new host based on the latest log position, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; A decoded synchronization transaction component, configured to perform logical decoding on the set of transactions to be synchronized to obtain structured change data; and trigger the new host to perform a log synchronization operation based on the structured change data.

2. The log synchronization system of the database according to claim 1, characterized in that, The log position acquisition component is configured to determine the latest write-ahead log file corresponding to the write-ahead log directory based on the write-ahead log directory, and determine the latest log position corresponding to the latest write-ahead log file based on the latest write-ahead log file; wherein, the latest write-ahead log file includes multiple transaction log record segments.

3. The log synchronization system of the database according to claim 2, characterized in that, The fork point positioning component is configured to sequentially read the transaction log record segments in the latest write-ahead log file from the back forward based on the latest log position, and confirm whether the transaction log record segments are recorded in the new host. If not, mark the transaction corresponding to the transaction log record segment as needing to be decoded. If recorded, determine the position of the transaction log record segment as the log fork position.

4. The log synchronization system of the database according to claim 3, characterized in that, The fork point positioning component is configured to sequentially read the transaction log record segments forward from the log fork position based on the log fork position, and confirm whether the transaction log record segments are committed to the new host. If not, update the start log sequence number of the transaction corresponding to the transaction log record segment; until the start log sequence numbers of all transactions in the set of transactions to be synchronized are determined.

5. The log synchronization system of the database according to claim 2, characterized in that, The decoded synchronization transaction component is configured to determine the safe restart log sequence number corresponding to the set of transactions to be synchronized based on the log fork position; Based on the safe restart log sequence number, perform logical replication and pull changes on the transaction log record segments in the set of transactions to be synchronized through a preset decoding strategy to obtain a file in sql or json format as the structured change data.

6. The log synchronization system of the database according to claim 5, wherein, The decoded synchronization transaction component is configured to, when the safe restart log sequence number does not exist in the set of transactions to be synchronized, perform decoding on the transaction log record segments in the set of transactions to be synchronized through region decoding to obtain a file in sql or json format as the structured change data; wherein, the region decoding is to decode the write-ahead log file between the start log sequence number and the end log sequence number of the transaction.

7. The log synchronization system of the database according to claim 1, characterized in that, It further includes an initialization configuration component, which is used to initialize the decoding plug-in to default values; initialize the parameters that require password authentication and forced range decoding to negative or disabled; and obtain the parameter values configured by the user for the initialization parameters, and update the current values of the initialization parameters based on the parameter values.

8. A method for log synchronization of a database, characterized in that, It includes: Obtain the write-ahead log directory of the old host; Based on the write-ahead log directory of the old host, determine the latest log position of the old host; Based on the latest log position, determine the log fork position between the new host and the old host and determine the set of transactions to be synchronized from the old host to the new host, and determine the start log sequence numbers of all transactions in the set of transactions to be synchronized; Perform logical decoding on the set of transactions to be synchronized to obtain structured change data; Based on the structured change data, trigger the new host to perform a log synchronization operation.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the log synchronization method of the database as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by the processor, it implements the log synchronization method of the database as described in claim 8.

Images