A log synchronization system, method, device and medium for a database

Abstract

The application provides a log synchronization system, method, device and medium of a database, and relates to the technical field of databases, to alleviate the technical problem of log asynchronization after master-slave switching in the prior art, by obtaining a pre-write log directory of an old host; determining a latest log position of the old host based on the pre-write log directory of the old host; determining a log fork position between a new host and the old host based on the latest log position, determining a to-be-synchronized transaction set from the old host to the new host, and determining a starting log sequence number of all transactions in the to-be-synchronized transaction set; logically decoding the to-be-synchronized transaction set to obtain structured change data; and triggering the new host to perform a log synchronization operation based on the structured change data, to realize log synchronization between the new host and the original host after master-slave switching, and ensure data consistency and reliability.

Description

Technical Field

[0001] This invention relates to the field of database technology, and more specifically, to a database log synchronization system, method, device, and medium. Background Technology

[0002] In existing technologies, when a database system uses a primary-standby architecture, if the primary node fails (such as the primary node crashes), the asynchronous standby node is switched to the primary node, meaning the asynchronous standby node becomes the new primary node. However, the logs of the new primary node after the switchover will be out of sync with those of the primary node that failed. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a database log synchronization system, method, device and medium to alleviate the technical problem of log inconsistency after master-slave switchover in the prior art.

[0004] In a first aspect, this application provides a database log synchronization system, which is connected to both a new host and an old host; wherein the new host is the original asynchronous backup node, and the old host is the original master node, and the log synchronization system includes:

[0005] The log location acquisition component is used to determine the latest log location on the old host based on the write-ahead log directory.

[0006] The fork point location component is used to determine the log fork location between the new host and the old host based on the latest log location, determine the set of transactions to be synchronized from the old host to the new host, and determine the start log sequence number of all transactions in the set of transactions to be synchronized.

[0007] The decoding synchronization transaction component is used to logically decode the set of transactions to be synchronized to obtain structured change data; based on the structured change data, it triggers the new host to perform log synchronization operations.

[0008] Optionally, the log location acquisition component is used to determine the latest write-ahead log file corresponding to the write-ahead log directory based on the write-ahead log directory; and to determine the latest log location corresponding to the latest write-ahead log file based on the latest write-ahead log file; wherein the latest write-ahead log file includes multiple transaction log record segments.

[0009] Optionally, a fork point location component is used to read the transaction log record segments in the latest write-ahead log file sequentially from back to front based on the latest log position, and to confirm whether the transaction log record segment is recorded in the new host. If it is not recorded, the transaction corresponding to the transaction log record segment is marked as needing to be decoded. If it is recorded, the position of the transaction log record segment is determined as the log fork position.

[0010] Optionally, a fork point location component is used to read transaction log segments sequentially forward from the log fork position, and to confirm whether the transaction log segments have been committed to the new host. If not committed, the start log sequence number of the transaction corresponding to the transaction segment is updated; until the start log sequence numbers of all transactions in the transaction set to be synchronized are determined.

[0011] Optionally, the decoding synchronization transaction component is used to determine the safe restart log sequence number corresponding to the set of transactions to be synchronized based on the log fork position; based on the safe restart log sequence number, the transaction log record segments in the set of transactions to be synchronized are logically copied and pulled for changes using a preset decoding strategy to obtain a file in SQL or JSON format as structured change data.

[0012] Optionally, the decoding synchronization transaction component is used to decode the transaction log record segments in the transaction set to be synchronized through region decoding when there is no safe restart log sequence number in the transaction set to be synchronized, and obtain a file in SQL or JSON format as structured change data; wherein, region decoding is to decode the write-ahead log file between the start log sequence number of the transaction and the end log sequence number of the transaction.

[0013] Optionally, the log synchronization system of the database in this application also includes an initialization configuration component, which is used to initialize the decoding plugin to the default value; initialize parameters that require password authentication and forced range decoding to negative or disabled; and obtain the parameter values ​​configured by the user for the initialization parameters, and update the current value of the initialization parameters based on the parameter values.

[0014] Secondly, this application provides a database log synchronization method, including:

[0015] Get the write-ahead log directory of the old host;

[0016] Determine the location of the latest logs on the old host based on the old host's write-ahead log directory;

[0017] Based on the latest log position, determine the log fork position between the new host and the old host, determine the set of transactions to be synchronized from the old host to the new host, and determine the start log sequence number of all transactions in the set of transactions to be synchronized.

[0018] Logical decoding of the set of transactions to be synchronized yields structured change data;

[0019] Based on the structured change data, the new host is triggered to perform log synchronization operations.

[0020] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned database log synchronization method.

[0021] Fourthly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned database log synchronization method.

[0022] This invention provides a database log synchronization system, method, device, and medium. The system obtains the write-ahead log directory of the old host; determines the latest log position of the old host based on the write-ahead log directory; determines the log fork position between the new host and the old host based on the latest log position, determines the set of transactions to be synchronized from the old host to the new host, and determines the start log sequence number of all transactions in the set of transactions to be synchronized; performs logical decoding on the set of transactions to be synchronized to obtain structured change data; and triggers the new host to perform log synchronization operations based on the structured change data. This achieves log synchronization between the new host and the original host after a master-slave switchover, ensuring data consistency and reliability.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This diagram illustrates the structure of a database log synchronization system provided in an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of the intersection point between the old host and the new host provided in an embodiment of the present invention is shown;

[0027] Figure 3 This diagram illustrates the method for determining the location of log fork points according to an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of the process for determining the secure reboot log sequence number provided in an embodiment of the present invention is shown;

[0029] Figure 5The diagram illustrates the logical decoding process of the decoding synchronization transaction component provided in this embodiment of the invention for a set of transactions to be synchronized.

[0030] Figure 6 A schematic diagram of the interaction flow of the database log synchronization method provided in an embodiment of the present invention is shown;

[0031] Figure 7 A schematic flowchart of a database log synchronization method provided in an embodiment of the present invention is shown;

[0032] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0034] Currently, in a master-slave database system, based on the log synchronization strategy between the slave and master databases, slave databases can be divided into synchronous slaves and asynchronous slaves. A synchronous slave refers to a node where the master node needs to wait for the slave node's confirmation when committing a transaction to ensure data consistency; therefore, a synchronous slave is consistent with the master node. An asynchronous slave refers to a node where the master node does not need to wait for the slave node's confirmation when committing a transaction; therefore, the logs of an asynchronous slave are often inconsistent with those of the master node. When using a master-slave architecture, if the master node experiences an anomaly (such as a master crash), a master-slave failover occurs, where the master switches to an asynchronous slave. The logs of the switched master will be out of sync with those of the original master.

[0035] Figure 1 This application provides a schematic diagram of the structure of a database log synchronization system, as shown in the embodiment. Figure 1As shown in the figure, the database log synchronization system provided in this application embodiment is connected to a new host and an old host respectively; wherein, the new host is the original asynchronous backup node, and the old host is the original master node. The system provided in this application embodiment includes at least: a log position acquisition component 120, a fork point location component 130, and a synchronization transaction decoding component 140; wherein, the log position acquisition component 120 is used to determine the latest log position of the old host based on the write-ahead log directory of the old host; the fork point location component 130 is used to determine the log fork position between the new host and the old host and determine the set of transactions to be synchronized from the old host to the new host based on the latest log position, and determine the start log sequence number of all transactions in the set of transactions to be synchronized; the synchronization transaction decoding component 140 is used to logically decode the set of transactions to be synchronized to obtain structured change data; and based on the structured change data, trigger the new host to perform log synchronization operation.

[0036] In this embodiment, the fork position in the logs of the old and new hosts is determined by the log sequence number and transaction status, and the target format log is obtained by decoding the transaction through write-ahead log. The new host performs log recovery by executing the target format to achieve synchronization of log information between the old and new hosts.

[0037] In one optional embodiment, the log location acquisition component 120 is used to determine the latest write-ahead log file corresponding to the write-ahead log directory based on the write-ahead log directory; and to determine the latest log location corresponding to the latest write-ahead log file based on the latest write-ahead log file; wherein the latest write-ahead log file includes multiple transaction log record segments.

[0038] In this embodiment, the log location acquisition component 120 is used to determine the latest (i.e., the largest number) and readable valid WAL file in the WAL (Write-Ahead Log) directory of the specified old host, as well as the maximum LSN (Log Sequence Number) and the maximum LSN value, along with relevant information such as WAL file information, time information, and distributed environment information; wherein, the WAL file information is the WAL file name and precise offset position; the time information is the last checkpoint time and transaction commit timestamp; and the distributed environment information is the LSN synchronization difference and replication latency between the master and slave nodes.

[0039] In a specific embodiment, the log location acquisition component 120 first acquires the log directory of the old host, and accesses all filenames in the log directory of the old host by traversing (e.g., in chronological order) to select the latest WAL log file. The WAL log filename is a 24-bit hexadecimal character. The first 8 bits are the timeline (timeline ID) used for recovery and point-in-time restoration, the middle 8 bits are the logid (logical ID) used to identify the range of consecutive log segments, and the last 8 bits are the segno (segment number) used for sequential numbering within the logical ID. The latest WAL log file can be obtained by comparing the lexicographical order of the WAL log filename strings. That is, the latest WAL log file is the one with the largest combination value of logical ID and segment number.

[0040] Then, starting from the latest WAL log file, the files in the log directory are traversed sequentially from back to front to determine the first readable log segment, which is then used as the starting point. A readable log segment must meet the following conditions: the log file physically exists on the disk and has not been deleted or moved; the current process has permission to read the log file; the log file content is intact and can be parsed normally (e.g., checksum verification passes); the log file format meets expectations (e.g., the WAL log file header is correct). Furthermore, the first readable log segment can be determined programmatically by attempting to open the log file and read its contents.

[0041] Finally, traverse all log records backward from the starting point (the first readable log segment) to determine the maximum LSN, i.e., the latest log position. The maximum LSN can be determined by the following traversal method: At the first readable log segment position, maintain two pointers pointing to the beginning and end of the currently read record. Each time, move ReadRecPtr (the start position of the read record) to the beginning of the current record and move EndRecPtr (the end position of the read record) to the beginning of the next record. Repeat this process until NULL (null pointer) is returned. At this point, ReadRecPtr has stored the end position of the last successfully read record, which is the maximum LSN.

[0042] In the log location acquisition component 120 provided in this application embodiment, the parsing of the entire log segment and the determination of the log location are completed by determining the latest and readable valid record WAL file in the WAL directory of the specified old host, and determining the maximum LSN, i.e. the latest log location.

[0043] In an optional embodiment, the fork point location component 130 is used to read the transaction log record segments in the latest write-ahead log file sequentially from back to front based on the latest log position, and to confirm whether the transaction log record segment is recorded in the new host. If it is not recorded, the transaction corresponding to the transaction log record segment is marked as needing to be decoded. If it is recorded, the position of the transaction log record segment is determined as the log fork position.

[0044] In this embodiment, the fork point location component 130 is used to determine the fork point between the logs of the old host and the new host, that is, the location of the different log files between the old host and the new host logs. Simultaneously, it collects all log files from the old host that have been committed to the new host but have not been synchronized to the remote end (i.e., the new host), along with their positions (e.g., LSNs) within the log files. Figure 2 The diagram shows the intersection of the old host and the new host. In the diagram, 0 / 75720E8 is the last synchronized LSN between the old host and the new host, which is the intersection of the old host and the new host.

[0045] In a specific embodiment, transaction log segments in the latest write-ahead log file are read sequentially from the latest log position backwards. For each read transaction log segment, its content is checked to determine whether it contains the identifier of the new host or recorded new host data. If the new host content recorded in the transaction log segment does not match the current target host, it means that the transaction log segment has not yet been recorded in the new host. If the transaction log segment is not recorded in the current new host, it is marked as needing to be decoded. If the transaction log segment is recorded in the new host for the first time, and it is confirmed that all subsequent transaction log segments are recorded in the new host, then the latest recording time point of the transaction log segment is the log fork position. The log fork position is used to determine the difference between the logs of the new host and the old host, thereby facilitating segmentation and archiving operations in the data processing and error recovery process, and making log data analysis and management easier.

[0046] Furthermore, the fork point location component 130 is used to read transaction log record segments sequentially forward from the log fork position based on the log fork position, and to confirm whether the transaction log record segment has been committed to the new host. If it has not been committed, the transaction corresponding to the transaction log record segment is marked as needing to be decoded; until the start log sequence number (i.e. the minimum LSN to be decoded) of all transactions in the transaction set to be synchronized is determined.

[0047] In this embodiment of the application, the fork point location component 130 is also used to maintain a set of transactions to be decoded and synchronized. The set of transactions to be decoded and synchronized records all transaction IDs that have been committed but not synchronized to the new host, which are used to subsequently decode and obtain structured change data. It also records the minimum LSN to be decoded and the minimum LSN of all transactions to be decoded in the WAL, which are used to subsequently locate and read complete transaction log record segments.

[0048] In a specific embodiment, the fork point location component 130 reads transaction log segments sequentially backward from the largest LSN to reduce the amount of cross-node queries. Specifically, based on the recorded LSN and CRC (Cyclic Redundancy Check) code, it determines whether the transaction corresponding to the transaction log segment is recorded on the new host. If it is not recorded on the new host, the transaction ID corresponding to the transaction log segment is recorded in the set of transactions to be decoded and synchronized, and the minimum LSN to be decoded is updated to the currently recorded LSN. If it is recorded on the new host, the transaction ID corresponding to the transaction log segment is the log fork position, the verification ends, and the final set of transactions to be synchronized is obtained.

[0049] To ensure the accuracy of the minimum required decoding LSN, this application requires continuing the search forward from the log fork position, verifying whether the transaction is marked as needing decoding, and continuously updating the minimum required decoding LSN until all log files in the old host are searched or the earliest start log sequence number (minimum required decoding LSN) of all transactions in the set of transactions to be synchronized is found.

[0050] like Figure 3 As shown, assuming there are transactions ABC on the old host, during the process of finding the fork point from back to front, there are transaction IDs of transactions B and C in the set of transactions to be synchronized. After finding the fork point position between the new host and the old host, the search continues to move forward until the search ends (i.e., the minimum LSN of the transaction is determined). The position of the minimum LSN that needs to be decoded is the starting position of transaction B.

[0051] The fork point location component 130 provided in this application embodiment determines the log fork point location of the new host and the old host, and determines the minimum LSN of all transactions that need to be decoded in the WAL based on the log fork location, so as to facilitate the subsequent location of log files and reading of complete logs.

[0052] In one optional embodiment, the decoding synchronization transaction component 140 is used to determine the safe restart log sequence number corresponding to the set of transactions to be synchronized based on the log fork position; based on the safe restart log sequence number, the transaction log record segments in the set of transactions to be synchronized are logically copied and pulled for changes through a preset decoding strategy to obtain a file in SQL or JSON format as structured change data.

[0053] Furthermore, the decoding synchronization transaction component 140 is used to decode the transaction log record segments in the transaction set to be synchronized through region decoding when there is no safe restart log sequence number in the transaction set to be synchronized, and obtain a file in SQL or JSON format as structured change data; wherein, region decoding is to decode the write-ahead log file between the start log sequence number of the transaction and the end log sequence number of the transaction.

[0054] In this embodiment of the application, the decoding synchronization transaction component 140 decodes the logs of the differences between the old and new hosts into SQL or JSON format through a decoding plugin, so that the new host can synchronize the logs of the differences.

[0055] like Figure 4 As shown, the process by which the decoding synchronization transaction component 140 determines the safe restart log sequence number during the logical decoding of the set of transactions to be synchronized is as follows:

[0056] Based on the log fork position, the transactions in the set of transactions to be synchronized are scanned from back to front. If the current transaction commits before the minimum required LSN for decoding and is not an XLOG_RUNNING_XACTS record, the next transaction is checked. This process continues until a transaction commits before the minimum required LSN for decoding and is an XLOG_RUNNING_XACTS record. Decoding then begins from the log file following that record, which serves as the safe restart LSN sequence number. By determining this safe restart sequence number as the starting point for decoding, no intermediate states of any transaction are missed, and committed transactions are neither repeated nor skipped.

[0057] like Figure 5 As shown, the logical decoding process of the synchronization transaction component 140 on the set of transactions to be synchronized is as follows:

[0058] First, based on the log fork position, the transactions in the set of transactions to be synchronized are scanned from back to front. The safe restart log sequence number is determined using the method described above. Based on the safe restart log sequence number, logical replication is performed and changes are pulled. Logical replication slots are created using the preset decoding plugin's plugin parameters. If no plugin parameter value is set, the default parameter value is used. Starting from the restart LSN (restart log sequence number), the logical replication slot is created by concatenating an SQL string: `select * from pg_create_logical_replication_slot(name,plugin,restart_lsn,confirmed_flush_lsn)`, where `restart_lsn` is the safe restart log sequence number obtained in the previous step, and `confirmed_flush_lsn` is the log fork position. This SQL is then executed.

[0059] Then, iteratively retrieves logical changes, again by constructing an SQL string: `select * from pg_logical_slot_get_changes(slot_name, upto_lsn, count)`. Here, `upto_lsn` is set to NULL, representing all changes after that, and `count` is set to 1000, retrieving 1000 records at a time to avoid retrieving too many at once. This statement is executed repeatedly, and the results are output to the file specified by the set `file` parameter. The execution ends when the number of tuples in the returned `PQresult` is 0, indicating the end of the execution. The loop is then exited and the copy slot is deleted.

[0060] If the transaction sequence number for safe restart is not determined by scanning the transactions in the set of transactions to be synchronized from back to front according to the log fork position, the decoding synchronization transaction component 140 performs area decoding on the log between the minimum LSN to be decoded and the maximum LSN. It can directly read the local WAL file segment by LSN interval. Each area decoding interval is within the same physical WAL log, avoiding the situation of memory exhaustion caused by reading multiple large files at the same time. According to this design, if the starting LSN and the ending LSN are not in the same file segment, the decode_lsn of each batch is set to the beginning of the next file segment, and an SQL statement is constructed: select * from pg_logical_get_area_changes(start_lsn, decode_lsn,NULL, plugin, NULL); where NULL for slot name and ending LSN indicates that there is no slot and the acquisition continues until the end, and the result is output to the specified file.

[0061] In one alternative embodiment, the database log synchronization system further includes an initialization configuration component for initializing the decoding plugin to its default value; initializing parameters requiring password authentication and forced range decoding to negative or disabled; and obtaining the parameter values ​​configured by the user for the initialization parameters, and updating the current values ​​of the initialization parameters based on the parameter values.

[0062] In this embodiment, the initialization configuration component is used to parse the parameters passed by the user and the environment variables of the log synchronization system, and to perform simple validation. The required parameters and their meanings are shown in Table 1:

[0063] Table 1

[0064]

[0065] Furthermore, the initialization configuration component initializes the above parameters into global variables. If the user does not input the corresponding parameter value, the plugin (decoding plugin) will be initialized to the default value, and the password (password authentication required) and force (force range decoding) will be set to false (negation or disabling). The initialization configuration component 110 is also used to obtain the data directory through environment variables; and the user can obtain parameters through --help or -?.

[0066] Based on the same inventive concept, embodiments of this application also provide a database log synchronization method, see below. Figure 6 As shown in the embodiments of this application, the interaction flow of the database log synchronization method is as follows:

[0067] Step 210: Obtain the write-ahead log directory of the old host;

[0068] Step 220: Determine the latest log location on the old host based on the old host's write-ahead log directory;

[0069] Step 230: Based on the latest log position, determine the log fork position between the new host and the old host, determine the set of transactions to be synchronized from the old host to the new host, and determine the start log sequence number of all transactions in the set of transactions to be synchronized.

[0070] Step 240: Logically decode the set of transactions to be synchronized to obtain structured change data;

[0071] Step 250: Based on the structured change data, trigger the new host to perform log synchronization operation.

[0072] Next, the database log synchronization method mentioned in the embodiments of this application will be described in detail. Please refer to [link / reference]. Figure 7 As shown in the embodiments of this application, the general flow of the database log synchronization method is as follows:

[0073] Step 310: Set the values ​​of each parameter in the log synchronization system through the initialization configuration component 110 in the log synchronization system. If there is no user input, initialize the decoding plugin to the default value; initialize the parameters that require password authentication and forced range decoding to negative or disabled; if there is user input, configure the values ​​of each parameter according to the parameter values ​​entered by the user.

[0074] Step 320: The log location acquisition component in the log synchronization system acquires the write-ahead log directory of the old host; and determines the latest log location of the old host based on the write-ahead log directory of the old host;

[0075] Step 330: The log fork point location component in the log synchronization system determines the log fork position between the new host and the old host and the set of transactions to be synchronized from the old host to the new host based on the latest log position. Specifically, based on the latest log position, the transaction log record segments in the latest write-ahead log file are read sequentially from back to front, and it is confirmed whether the transaction log record segment is recorded in the new host. If it is not recorded, the transaction corresponding to the transaction log record segment is marked as needing decoding. If it is recorded, the position of the transaction log record segment is determined as the log fork position. At the same time as determining the log fork position between the new host and the old host, the set of transactions to be synchronized from the old host to the new host can also be determined.

[0076] Step 340: Based on the log fork position, read the transaction log record segments sequentially backward from the log fork position, and confirm whether the transaction log record segments have been committed to the new host. If not committed, update the start log sequence number of the transaction corresponding to the transaction record segment; until the earliest start log sequence number of all transactions in the transaction set to be synchronized is determined.

[0077] Step 340: The decoding and synchronization transaction component in the log synchronization system performs logical decoding on the set of transactions to be synchronized to obtain structured change data. Specifically, based on the log fork position, the safe restart log sequence number corresponding to the set of transactions to be synchronized is determined. Based on the safe restart log sequence number, the transaction log record segments in the set of transactions to be synchronized are logically copied and changed using a preset decoding strategy to obtain a file in SQL or JSON format as structured change data. When the set of transactions to be synchronized does not have a safe restart log sequence number, the transaction log record segments in the set of transactions to be synchronized are decoded using region decoding to obtain a file in SQL or JSON format as structured change data. Region decoding decodes the write-ahead log file between the start log sequence number and the end log sequence number of the transaction.

[0078] Step 350: Based on the structured change data, trigger the new host to perform log synchronization operation.

[0079] It should be noted that the principle of the database log synchronization method provided in this application embodiment to solve the technical problem is similar to that of the database log synchronization system provided in this application embodiment. Therefore, the implementation of the database log synchronization method provided in this application embodiment can refer to the implementation of the database log synchronization system provided in this application embodiment, and the repeated parts will not be described again.

[0080] After introducing the database log synchronization system and apparatus provided in the embodiments of this application, the electronic equipment provided in the embodiments of this application will be briefly introduced next.

[0081] See Figure 8 As shown, the electronic device 500 provided in this application embodiment includes at least a processor 501, a memory 502, and a computer program stored on the memory 502 and executable on the processor 501. When the processor 501 executes the computer program, it implements the database log synchronization method provided in this application embodiment.

[0082] The electronic device 500 provided in this application embodiment may further include a bus 503 connecting different components (including processor 501 and memory 502). The bus 503 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0083] Memory 502 may include a readable storage medium in the form of volatile memory, such as random access memory (RAM) 5021 and / or cache memory 5022, and may further include read-only memory (ROM) 5023. Memory 502 may also include a program tool 5025 having a set (at least one) of program modules 5024, including but not limited to an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0084] Processor 501 can be a single processing element or a collective term for multiple processing elements. For example, processor 501 can be a central processing unit (CPU) or one or more integrated circuits configured to implement the database log synchronization method provided in the embodiments of this application. Specifically, processor 501 can be a general-purpose processor, including but not limited to CPUs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0085] Electronic device 500 can communicate with one or more external devices 504 (e.g., keyboard, remote control, etc.), and also with one or more devices that enable a user to interact with electronic device 500 (e.g., mobile phone, computer, etc.), and / or with devices that enable electronic device 500 to communicate with one or more other electronic devices 500 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 505. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 506. Figure 8 As shown, network adapter 506 communicates with other modules of electronic device 500 via bus 503. It should be understood that, although... Figure 8 As not shown, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0086] It should be noted that, Figure 8 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0087] The computer-readable storage medium provided in the embodiments of this application is described below. The computer-readable storage medium provided in the embodiments of this application stores computer instructions, which, when executed by a processor, implement the database log synchronization method provided in the embodiments of this application. Specifically, the computer instructions can be built into or installed in the processor, so that the processor can implement the database log synchronization method provided in the embodiments of this application by executing the built-in or installed computer instructions.

[0088] In addition, the database log synchronization method provided in this application embodiment can also be implemented as a computer program product, which includes program code that implements the database log synchronization method provided in this application embodiment when running on a processor.

[0089] The computer program product provided in this application embodiment may employ one or more computer-readable storage media, which may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. Specifically, more specific examples (a non-exhaustive list) of computer-readable storage media include electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0090] The computer program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on electronic devices such as computers. However, the computer program product provided in this application embodiment is not limited thereto. In this application embodiment, the computer-readable storage medium can be any tangible medium that contains or stores program code, which can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0091] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0092] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0093] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0094] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A log synchronization system for a database, characterized by, The log synchronization system is connected with a new host and an old host respectively; the new host is an original asynchronous backup node, and the old host is an original master node; the log synchronization system comprises: An acquisition log position component is configured to determine a latest prewrite log file corresponding to a prewrite log directory of the old host based on the prewrite log directory by comparing string lexicographical order of prewrite log file names; a logical ID and a segment serial number combination value of the latest prewrite log file are the largest; files in the prewrite log directory are sequentially traversed from back to front from the latest prewrite log file to determine a first readable log segment, and the first readable log segment is taken as a starting point; all prewrite log records are traversed from the first readable log segment position to determine a maximum log serial number as a latest log position; a readable log segment satisfies that log file content is not damaged and can be normally parsed, a log file format is expected and can be opened and read; the all prewrite log records are traversed to determine the maximum log serial number as the latest log position, that is, at the first readable log segment position, two pointers are maintained to point to the beginning and end of a current reading record; each time, the beginning position of the reading record is moved to the current record start, and the end position of the reading record is moved to the next record start; the loop is called until the empty pointer is returned; the end position of the reading record saves the end position of the last successfully read record as the maximum log serial number; the latest prewrite log file comprises a plurality of transaction log record segments; A positioning fork point component is configured to determine a log fork position between the new host and the old host and determine a to-be-synchronized transaction set from the old host to the new host based on the latest log position, and determine a start log serial number of all transactions in the to-be-synchronized transaction set; wherein determining the log fork position between the new host and the old host comprises: sequentially reading the transaction log record segments in the latest prewrite log file from back to front based on the latest log position, and confirming whether the transaction log record segments are recorded in the new host; if not, marking the transaction corresponding to the transaction log record segment as needing to be decoded; if yes, determining the position of the transaction log record segment as the log fork position; A decoding synchronization transaction component is configured to logically decode the to-be-synchronized transaction set to obtain structured change data; based on the structured change data, triggering the new host to perform a log synchronization operation.

2. The log synchronization system of a database according to claim 1, wherein, The positioning fork point component is configured to sequentially read the transaction log record segments from the log fork position forward based on the log fork position, and confirm whether the transaction log record segments are submitted to the new host; if not, updating the start log serial number of the transaction corresponding to the transaction log record segment; until the start log serial numbers of all transactions in the to-be-synchronized transaction set are determined.

3. The log synchronization system of a database according to claim 1, wherein, The decoding synchronization transaction component is configured to determine a safe restart log serial number corresponding to the to-be-synchronized transaction set based on the log fork position. Based on the safe restart log sequence number, the transaction log record segments in the to-be-synchronized transaction set are logically copied and pulled changes through a preset decoding strategy, to obtain a sql or json format file as the structured change data.

4. The log synchronization system of a database according to claim 3, wherein, The decoding synchronization transaction component is configured to, when the to-be-synchronized transaction set does not have the safe restart log sequence number, decode the transaction log record segments in the to-be-synchronized transaction set through area decoding, to obtain a sql or json format file as the structured change data; wherein the area decoding is decoding the pre-write log file between the start log sequence number of a transaction and the end log sequence number of the transaction.

5. The log synchronization system of a database according to claim 1, wherein, The initialization configuration component is further configured to initialize the decoding plug-in to a default value; initialize parameters requiring password authentication and forced range decoding to negative or disabled; and obtain parameter values configured by a user for the initialization parameters, and update current values of the initialization parameters based on the parameter values.

6. A log synchronization method of a database, characterized by, The method comprises: obtaining a pre-write log directory of an old host; based on the pre-write log directory of the old host, determining the latest pre-write log file corresponding to the pre-write log directory by comparing the string lexicographical order of the pre-write log file name; wherein the combination value of the logical ID and the segment number of the latest pre-write log file is the largest; traversing the files in the pre-write log directory from the latest pre-write log file to the first readable log segment in sequence, and taking the first readable log segment as the starting point, and traversing all pre-write log records from the first readable log segment position to determine the maximum log sequence number as the latest log position; wherein the readable log segment satisfies that the log file content is not damaged and can be normally parsed, the log file format meets the expectation and can be opened and read; the traversal of all pre-write log records to determine the maximum log sequence number as the latest log position is that, at the first readable log segment position, two pointers are maintained to point to the start and end of the current read record, the start position of the read record is moved to the current record start position each time, the end position of the read record is moved to the next record start position, the loop is called until the empty pointer is returned, and the end position of the read record saves the end position of the last successfully read record as the maximum log sequence number; the latest pre-write log file includes a plurality of transaction log record segments; based on the latest log position, determining the log fork position between the new host and the old host and determining the to-be-synchronized transaction set from the old host to the new host, and determining the start log sequence number of all transactions in the to-be-synchronized transaction set; wherein determining the log fork position between the new host and the old host comprises: based on the latest log position, reading the transaction log record segments in the latest pre-write log file in sequence from back to front, and confirming whether the transaction log record segments are recorded in the new host, if not, marking the transaction corresponding to the transaction log record segment as needing decoding, and if yes, determining the position of the transaction log record segment as the log fork position; logically decoding the set of to-be-synchronized transactions to obtain structured change data; based on the structured change data, triggering the new master to perform a log synchronization operation.

7. An electronic device, comprising: The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the log synchronization method of the database according to claim 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the log synchronization method of the database according to claim 6.

Images