Log synchronization method, device, system, equipment, medium and program product

Abstract

The invention discloses a log synchronization method, device, system and equipment, a medium and a program product, and relates to the technical field of databases. The method comprises the following steps: receiving a to-be-synchronized log packet sent by a main library through a standby library in a database cluster; adding the to-be-synchronized log packet into the asynchronous analysis reconstruction task, and sending a response message to the main library, wherein the response message is used for indicating the main library to write the to-be-synchronized log packet into an online log file of the main library; and executing the asynchronous analysis reconstruction task to obtain a reconstructed log packet of the to-be-synchronized log packet, and performing log replay according to the reconstructed log packet. The standby library adds the received to-be-synchronized log packet into the asynchronous analysis reconstruction task, and the response message is returned to the main library without waiting for the completion of the execution of the asynchronous analysis reconstruction task, so that the response waiting time of the main library is effectively shortened, the transaction submission speed of the main library is accelerated, and the log synchronization efficiency between the main library and the standby library is improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a log synchronization method, device, system, equipment, medium and program product. Background Art

[0002] A data protection system typically consists of a primary database and several backup databases. The backup databases maintain data synchronization with the primary database by replaying logs generated by the primary database and provide read-only services. If the primary database fails, the backup database can take over and continue to provide services, thus meeting user requirements for high database availability.

[0003] During log synchronization between the primary and standby databases, when the primary database commits a transaction, it must send the log package to the standby database. The standby database then parses and reconstructs the log package, adds a replay task, and returns a response message to the primary database. Only then can the primary database write the log package to its online log file, synchronizing the logs between the primary and standby databases. Parsing and reconstruction on the standby database takes time, which, to a certain extent, affects the speed at which the standby database returns response messages, thereby affecting the transaction commit speed of the primary database. Summary of the Invention

[0004] The present invention provides a log synchronization method, apparatus, system, equipment, medium and program product to solve the problem that the parsing and reconstruction time of the standby database is long, resulting in slow response speed of the standby database and transaction submission speed of the primary database, which in turn affects the log synchronization speed between the primary and standby databases.

[0005] In a first aspect, an embodiment of the present invention provides a log synchronization method, which is applied to a standby database in a database cluster, wherein the database cluster also includes a master database. The method includes:

[0006] Receive the log package to be synchronized sent by the master database;

[0007] Add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database, wherein the response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database;

[0008] The asynchronous parsing and reconstruction task is executed to obtain a reconstructed log package of the log package to be synchronized, and log replay is performed according to the reconstructed log package.

[0009] In a second aspect, an embodiment of the present invention provides a log synchronization device, which is applied to a standby database in a database cluster, wherein the database cluster also includes a master database. The device includes:

[0010] A log packet receiving module is used to receive the log packets to be synchronized sent by the master database;

[0011] A response module is used to add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database, wherein the response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database;

[0012] The log synchronization module is used to execute the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and perform log replay according to the reconstructed log package.

[0013] In a third aspect, an embodiment of the present invention provides a log synchronization system, which is applied to a database cluster, wherein the database cluster includes a primary database and at least one secondary database;

[0014] The primary database generates a log package to be synchronized based on the data modification operation, and sends the log package to be synchronized to the secondary database;

[0015] The standby database receives the log package to be synchronized sent by the primary database; adds the log package to be synchronized to the asynchronous parsing and reconstruction task and sends a response message to the primary database; executes the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performs log replay based on the reconstructed log package;

[0016] After receiving the response message, the master database writes the log package to be synchronized into the online log file of the master database.

[0017] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising:

[0018] at least one processor;

[0019] and a memory communicatively coupled to the at least one processor;

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log synchronization method described in any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log synchronization method described in any embodiment of the present invention when executed.

[0022] In a sixth aspect, an embodiment of the present invention provides a computer program product including a computer program, which, when executed by a processor, implements the log synchronization method described in any embodiment of the present invention.

[0023] The technical solution of the embodiment of the present invention is to receive the log package to be synchronized sent by the main database through the backup database in the database cluster; add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the main database, the response message is used to instruct the main database to write the log package to be synchronized into the online log file of the main database; execute the asynchronous parsing and reconstruction task to obtain the reconstructed log package of the log package to be synchronized, and perform log replay according to the reconstructed log package. By adding the received log package to be synchronized to the asynchronous parsing and reconstruction task, the backup database returns a response message to the main database without waiting for the asynchronous parsing and reconstruction task to be completed, effectively shortening the waiting time of the main database for response, speeding up the transaction submission speed of the main database, solving the problem that the parsing and reconstruction time of the backup database is long, resulting in slow response speed of the backup database and slow transaction submission speed of the main database, thereby affecting the log synchronization speed between the main and backup databases, and achieving the beneficial effect of improving the log synchronization efficiency between the main and backup databases.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flow chart of a log synchronization method provided in Example 1 of the present invention;

[0027] Figure 2 A flow chart of a log synchronization method provided in Example 2 of the present invention;

[0028] Figure 3 A schematic diagram of the structure of a log synchronization device provided in Example 3 of the present invention;

[0029] Figure 4 A structural diagram of a log synchronization system provided in Example 4 of the present invention;

[0030] Figure 5 A schematic diagram of the structure of an electronic device for implementing the log synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the words “include” and “have” and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatus.

[0033] A data protection system generally consists of a primary database and several standby databases. The standby databases maintain data synchronization with the primary database by replaying the logs generated by the primary database. The standby databases only provide read-only services. The write-ahead log (REDO log) is used to record the modification operations performed on the data by the primary database, ensuring that committed transactions can be recovered after a system crash or restart, thereby ensuring data persistence and integrity. Each time a data modification generates a REDO log, a new log sequence number (LSN) is used to identify it. The LSN value range is (0 to positive infinity). The LSN value is automatically increased by 1 each time a REDO log is written. One LSN value represents a database modification operation. REDO logs can also be managed in log packages (RPKG). Multiple REDO logs can be stored in a log package body. The primary database sends logs to the standby database in log packages.

[0034] The general log synchronization process between the master and standby databases is as follows: (1) Start transaction TrxA on the master database, perform data modification operations, and the master database generates a REDO log and organizes the REDO log in the format of a log package. (2) Commit the transaction on the master database. When committing, it is necessary to wait for the log package to be written to the master database's online log file. (3) Before writing the log package to the online log file, the master database first sends the log package to the standby database and waits for the standby database's response message. (4) After receiving the log package, the standby database parses and reconstructs the log package, adds it to the replay task system, and then returns a response message to the master database. (5) After receiving the standby database's response message, the master database writes the log package to the master database's online log file. (6) Transaction TrxA is successfully committed. (7) The master database continues to send the next log package.

[0035] The standby database's parsing and reconstruction operations in the log synchronization process described above may include: after receiving the log package sent by the primary database, the standby database sequentially scans each REDO log in the package to determine whether any log types require special processing, obtains the data page address information recorded in the REDO log, initiates a data page read request, and preloads the data pages modified by the REDO log into the buffer. The log package is then reconstructed based on the REDO log and written to the standby database's online log file.

[0036] The analysis and reconstruction operations of the standby database show that these operations affect the speed of returning response messages to a certain extent, which in turn affects the transaction commit speed of the primary database.

[0037] Example 1

[0038] Figure 1 This is a flow chart of a log synchronization method provided in the first embodiment of the present invention. This embodiment is applicable to the log synchronization between the primary database and the backup database in the database cluster. The method can be executed by a log synchronization device. The log synchronization device can be implemented in the form of hardware and / or software. The log synchronization device can be configured in an electronic device, and the backup database in the database cluster is deployed in the electronic device. Figure 1 As shown, the method includes:

[0039] S110: Receive the log package to be synchronized sent by the master database.

[0040] The log package to be synchronized can be understood as the log package sent by the primary database and received by the standby database.

[0041] Log packets are self-describing. Each log packet has a unique serial number (SEQNO). SEQNO can be incremented in the order in which the log packets are generated, for example, by natural numbers or one or more combinations of letters, symbols, and natural numbers. For example, a log packet includes a log header and a log body. The log body can store multiple REDO logs, and the log header records self-describing information (such as the log packet sequence number, total length, minimum LSN, and maximum LSN).

[0042] Specifically, after the primary database modifies data, it generates a REDO log and organizes it into the format of log packages to be synchronized. The primary database commits the transaction and sends logs to the standby database in units of log packages to be synchronized. This allows the standby database to receive the log packages sent by the primary database.

[0043] S120: Add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database. The response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database.

[0044] Asynchronous parsing and reconstruction tasks can be understood as tasks that perform parsing and reconstruction through asynchronous requests without blocking the current program execution. Asynchronous parsing avoids the blocking waits associated with traditional synchronous parsing, allowing the program to process multiple tasks in parallel. For example, asynchronous parsing and reconstruction tasks can be implemented by an asynchronous parsing and reconstruction task system deployed on a standby database. The response message can be understood as the feedback message returned by the standby database to the primary database after adding the log package to be synchronized to the asynchronous parsing and reconstruction task.

[0045] Specifically, after receiving the pending log package from the primary, the standby database adds it to the asynchronous parsing and reconstruction task. This allows the standby database to return a response message to the primary without waiting for the asynchronous parsing and reconstruction task to complete, effectively shortening the time the primary database waits for a response. After receiving the response message, the primary database writes the pending log package to its online log file. Once the primary database successfully commits the transaction, it can immediately continue to send the next pending log package. This speeds up transaction commits on the primary database and improves log synchronization efficiency between the primary and standby databases.

[0046] S130 , executing an asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performing log replay according to the reconstructed log package.

[0047] The reconstructed log package can be understood as a log package after the log package to be synchronized is reconstructed.

[0048] Specifically, after adding the log package to be synchronized to the asynchronous parsing and reconstruction task, the standby database executes the asynchronous parsing and reconstruction task in an asynchronous manner, parses and reconstructs the log package to be synchronized added to the asynchronous parsing and reconstruction task, obtains the reconstructed log package, and replays the logs in the order of the serial numbers of the log packages to be synchronized, so that the logs of the primary database and the standby database are synchronized.

[0049] Exemplarily, the execution process of an asynchronous parsing and reconstruction task may include: scanning each REDO log in the log package to be synchronized, determining whether any log types require special processing, obtaining the data page address information recorded in the REDO log, initiating a data page read request, and preloading the data pages modified by the REDO log into a buffer. The log package is reconstructed based on the REDO log to obtain a reconstructed log package, and the reconstructed log package is written to the online log file of the standby database.

[0050] It's important to note that after sending a response message to the primary database, the standby database can execute asynchronous parsing and reconstruction tasks. This can occur simultaneously with the primary database writing pending log packets to its online log files. If the primary database experiences high traffic, the standby database may accumulate a significant number of pending parsing tasks. Multithreading can be used to execute these tasks. The standby database must ensure that log replay is performed in sequence to ensure correctness.

[0051] Exemplarily, to ensure the correctness of the replay, log replay can be performed according to the order of the log packet sequence numbers of the reconstructed log packets. For example, the reconstructed log packets can be added to the replay task queue according to the order of their log packet sequence numbers, and log replay can be performed according to the replay task queue; or when the reconstructed log packets are added to the replay task queue, replay task numbers can be reassigned to the reconstructed log packets according to the order of their log packet sequence numbers, and log replay can be performed according to the reconstructed log packets corresponding to the replay task numbers in the replay task queue.

[0052] An embodiment of the present invention provides a log synchronization method, which receives a log package to be synchronized sent by a primary database through a backup database in a database cluster; adds the log package to be synchronized to an asynchronous parsing and reconstruction task and sends a response message to the primary database, wherein the response message is used to instruct the primary database to write the log package to be synchronized into the primary database's online log file; executes the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performs log replay based on the reconstructed log package. By adding the received log package to be synchronized to the asynchronous parsing and reconstruction task, the backup database returns a response message to the primary database without waiting for the asynchronous parsing and reconstruction task to be completed, effectively shortening the time the primary database waits for a response and accelerating the transaction submission speed of the primary database, thereby improving the log synchronization efficiency between the primary and backup databases.

[0053] Example 2

[0054] Figure 2 A flowchart of a log synchronization method provided in Example 2 of the present invention. Based on the above embodiment, this embodiment further refines the steps of executing the asynchronous parsing and reconstruction task to obtain the reconstructed log package of the log package to be synchronized, and performing log replay according to the reconstructed log package, including: obtaining the log package to be synchronized from the asynchronous parsing and reconstruction task based on at least one thread, and parsing and reconstructing the log package to be synchronized to obtain the reconstructed log package of the log package to be synchronized; adding the reconstructed log package to the replay task queue in the order of the log package serial numbers of the reconstructed log package; and performing log replay according to the reconstructed log package in the replay task queue.

[0055] like Figure 2 As shown, the method includes:

[0056] S210: Receive the log package to be synchronized sent by the master database.

[0057] S220: Add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database. The response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database.

[0058] S230: Obtain a log package to be synchronized from the asynchronous parsing and reconstruction task based on at least one thread, and parse and reconstruct the log package to be synchronized to obtain a reconstructed log package of the log package to be synchronized.

[0059] The number of threads used to execute the asynchronous parsing and reconstruction task can be configured by the user, and this embodiment does not impose any restrictions on this.

[0060] Specifically, each thread in the standby database obtains a log package to be synchronized from the asynchronous parsing and reconstruction task, parses and reconstructs the log package to be synchronized, and obtains a reconstructed log package.

[0061] When the primary database is under high business pressure, the number of log packages waiting to be synchronized increases as the rate of sending them increases. This means that a certain amount of pending parsing tasks will accumulate in the asynchronous parsing tasks of the standby database. To improve the processing power of the standby database, multiple threads are used to simultaneously perform parsing and reconstruction of different pending log packages, thus avoiding a large backlog of tasks on the standby database.

[0062] S240 : Add the reconstructed log package to the replay task queue according to the order of the log package sequence numbers of the reconstructed log package.

[0063] The replay task queue is understood to be a queue for executing replay tasks. The order of the reconstruction log packets in the replay task queue can represent the order of log replay.

[0064] Specifically, because different threads may process different sizes of log packets to be synchronized and have different processing accuracies, the order in which the reconstructed log packets are obtained may not be exactly the same as the order in which the log packets to be synchronized are sent. However, the sequence numbers of the reconstructed log packets and the corresponding log packets to be synchronized will not change. The backup database must replay the data in the order in which the primary database's log packets are generated to ensure that the data modification order in the backup and primary databases is consistent. Therefore, to ensure the correctness of the replay, after each thread completes the parsing and reconstruction of the log packets to be synchronized and obtains the reconstructed log packets, it will add the reconstructed log packets to the replay task queue in the order of their log packet sequence numbers.

[0065] To ensure that the reconstructed log packets obtained by different threads can be added to the replay task queue in the order of the log packet sequence numbers of the log packets to be synchronized, a waiting and waking mechanism can be introduced when adding the reconstructed log packets to the replay task queue. As an optional embodiment, adding the reconstructed log packets to the replay task queue in the order of their log packet sequence numbers includes:

[0066] S241 . Compare the log packet sequence number of the reconstructed log packet with a sequence number subsequent to the maximum log packet sequence number in the replay task queue.

[0067] Specifically, to ensure that the sequence numbers of the reconstructed log packets added to the replay task queue are continuous and increasing, the sequence number of the reconstructed log packet to be added to the replay task queue should be exactly the sequence number after the largest log packet sequence number in the replay task queue. Therefore, after each thread obtains a reconstructed log packet, it compares the sequence number of the reconstructed log packet with the sequence number after the largest log packet sequence number in the replay task queue to determine whether it should be added to the replay task queue.

[0068] S242: When the log packet sequence number of the reconstructed log packet is equal to the sequence number following the maximum log packet sequence number in the replay task queue, add the reconstructed log packet to the replay task queue.

[0069] Specifically, when the log package sequence number of the reconstructed log package is equal to the next sequence number after the maximum log package sequence number in the replay task queue, it means that the reconstructed log package obtained by this thread is the next reconstructed log package that should be added to the replay task queue, so the reconstructed log package is added to the replay task queue.

[0070] S243. When the log packet sequence number of the reconstructed log packet is not equal to the sequence number after the maximum log packet sequence number in the replay task queue, the reconstructed log packet is set to a waiting for replay state, and after the reconstructed log packet is awakened, the reconstructed log packet is added to the replay task queue.

[0071] Specifically, when the log packet sequence number of the reconstructed log packet is not equal to the next sequence number after the maximum log packet sequence number in the replay task queue (generally, the log packet sequence number of the reconstructed log packet is greater than the next sequence number after the maximum log packet sequence number in the replay task queue), it means that the next reconstructed log packet that should be added to the replay task queue is not the reconstructed log packet obtained by this thread. At this time, the reconstructed log packet can be set to the waiting for replay state. After the reconstructed log packet is awakened by other threads, it means that it is the turn of the reconstructed log packet of this thread to join the replay task queue. At this time, the reconstructed log packet is added to the replay task queue. Although the progress of the reconstruction log is determined differently by each thread, resulting in the order in which the reconstructed log packets are obtained and the order in which the log packets to be synchronized are sent may not be exactly the same, the reconstructed log packets can still be added to the replay task queue according to the order in which the log packets to be synchronized are sent.

[0072] Optionally, in S242 or S243, after adding the reconstruction log package to the replay task queue, the process further includes:

[0073] When a target reconstruction log package in a waiting-for-replay state is obtained, the target reconstruction log package is awakened; wherein the log package sequence number of the target reconstruction log package is a sequence number subsequent to the log package sequence number of the reconstruction log package.

[0074] The target reconstruction log package can be understood as a reconstruction log package that is in a waiting-for-replay state and requires the next log replay.

[0075] Specifically, when each thread is adding a reconstruction log package to the replay task queue, there may be one or more other reconstruction log packages set to the waiting state. In order to ensure that the reconstruction log package in the waiting state can be awakened in time, after each thread adds the reconstruction log package to the replay task queue, it is necessary to query whether there is a next target reconstruction log package that needs to be replayed in the reconstruction log package in the waiting state. When the target reconstruction log package in the waiting state is obtained, the target reconstruction log package is awakened so that the target reconstruction log package can be added to the replay task queue after being awakened.

[0076] It is understood that if the target reconstruction log package in the waiting-for-replay state is not obtained, it means that there is no next reconstruction log package requiring log replay among the reconstruction log packages in the waiting-for-replay state, that is, the next reconstruction log package requiring log replay has not been set to the replay state. At this time, the thread can end the current round of log package parsing and reconstruction tasks.

[0077] Exemplarily, in order to ensure that the waiting and waking-up mechanism of the reconstruction log package can be correctly executed, the way to set the reconstruction log package to the waiting replay state can be to add the reconstruction log package to the waiting replay queue. When the reconstruction log package needs to be added to the replay task queue, the wake-up event is used to wake up the target reconstruction log package in the waiting replay queue, and it is removed from the waiting replay queue and added to the replay task team. The way to set the reconstruction log package to the waiting replay state can also be to set a waiting replay flag value for the reconstruction log package, for example, the waiting replay flag value is set to 1, indicating that it is in the waiting replay state; when the reconstruction log package is woken up, the waiting replay flag value can be set to a default value (such as 0) or the flag value can be deleted, indicating that there is no need to wait.

[0078] S250: Perform log replay according to the reconstructed log package in the replay task queue.

[0079] Specifically, while ensuring that the reconstruction log packages in the replay task queue are arranged in the order of sending the corresponding log packages to be synchronized, the reconstruction log packages in the replay task queue are replayed in sequence, which can ensure the correctness of the log replay of the standby database.

[0080] This embodiment provides a log synchronization method, which receives the log package to be synchronized sent by the main database through the backup database in the database cluster; adds the log package to be synchronized to the asynchronous parsing and reconstruction task and sends a response message to the main database, the response message is used to instruct the main database to write the log package to be synchronized into the online log file of the main database. Based on at least one thread, the log package to be synchronized is obtained from the asynchronous parsing and reconstruction task, and the log package to be synchronized is parsed and reconstructed to obtain a reconstructed log package of the log package to be synchronized; according to the order of the log package sequence number of the reconstructed log package, the reconstructed log package is added to the replay task queue; and log replay is performed based on the reconstructed log package in the replay task queue. By adding the received log package to be synchronized to the asynchronous parsing and reconstruction task, the backup database returns a response message to the main database without waiting for the asynchronous parsing and reconstruction task to be completed. At the same time, the use of multiple threads to execute the parsing and reconstruction task of the backup database improves the parsing and reconstruction speed, effectively shortens the waiting time of the main database for response, speeds up the transaction submission speed of the main database, and thus improves the log synchronization efficiency between the main and backup databases; and the introduction of the waiting and waking mechanism of the reconstructed log package can ensure the correctness of the log replay of the backup database.

[0081] In a specific embodiment, the standby database is configured with three asynchronous parsing and reconstruction threads (the three threads are numbered 0, 1, and 2 respectively). The standby database first receives the log package RPKG1 to be synchronized from the primary database (sequence number SEQNO is 1000). RPKG1 is parsed and reconstructed by the 0th working thread. During this process, the standby database successively receives the log packages RPKG2 (SEQNO is 1001) and RPKG3 (SEQNO is 1002) to be synchronized from the primary database. RPKG2 is parsed by the 1st thread, and RPKG3 is parsed by the 2nd thread. The maximum log package sequence number currently added to the replay queue is 999.

[0082] Since the three log packages have different sizes and parsing progress, they need to be added to the replay task queue in sequence according to SEQNO. The sorting logic is described as follows.

[0083] 1. The first thread first parses and reconstructs the log package RPKG2 to be synchronized with the log package number 1001, attempts to add it to the replay task queue, and finds that the log package numbers are discontinuous (1001 ≠ 999 + 1), so it enters the waiting state.

[0084] 2. The second thread completes the parsing and reconstruction of the log package RPKG3 to be synchronized with the log package number 1002, attempts to join the replay task queue, and finds that the log is discontinuous (1002 ≠ 999 + 1), so it enters the waiting state.

[0085] 3. Thread 0 completes parsing and reconstruction of the log package RPKG1 to be synchronized with the log package number 1000, and successfully adds it to the replay queue (1000=999+1). At this time, the maximum log package number added to the replay queue is updated to 1000.

[0086] 4. After adding 1000 to the replay queue, thread 0 searches for a target reconstruction log package that needs to be awakened. It finds that the reconstruction log packages with sequence numbers 1001 and 1002 are both in a waiting state. It chooses to awaken the reconstruction log package RPKG2 (target reconstruction log package) with sequence number 1001 (because the log package numbers 1001 and 1000 are consecutive).

[0087] 5. After the first thread is awakened, the reconstructed log package RPKG2 with the log package sequence number 1001 is added to the replay queue, and the maximum log package sequence number added to the replay queue is updated to 1001.

[0088] 6. After adding the reconstructed log package RPKG2 with the log package number 1001 to the replay queue, the first thread searches for log packages that need to be awakened. It finds that the reconstructed log package RPKG3 with the log package number 1002 is in a waiting state, and chooses to wake up the reconstructed log package RPKG3 (because the log package numbers 1002 and 1001 are consecutive).

[0089] 7. After the second worker thread is awakened, it adds the log package RPKG3 with the sequence number 1002 to the replay queue, and the maximum log package sequence number added to the replay queue is updated to 1002.

[0090] Example 3

[0091] Figure 3 This is a structural diagram of a log synchronization device provided by the third embodiment of the present invention. Figure 3 As shown, the device includes: a log packet receiving module 310, a response module 320 and a log synchronization module 330; wherein,

[0092] The log packet receiving module 310 is used to receive the log packet to be synchronized sent by the master database;

[0093] A response module 320 is configured to add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database, wherein the response message is configured to instruct the master database to write the log package to be synchronized into the online log file of the master database.

[0094] The log synchronization module 330 is configured to execute the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and perform log replay according to the reconstructed log package.

[0095] This embodiment provides a log synchronization device that receives the log package to be synchronized sent by the main database through the backup database in the database cluster; adds the log package to be synchronized to the asynchronous parsing and reconstruction task and sends a response message to the main database, the response message is used to instruct the main database to write the log package to be synchronized to the main database's online log file; executes the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performs log replay based on the reconstructed log package. By adding the received log package to be synchronized to the asynchronous parsing and reconstruction task, the backup database returns a response message to the main database without waiting for the asynchronous parsing and reconstruction task to be completed, effectively shortening the time the main database waits for a response and speeding up the main database transaction submission speed, thereby improving the log synchronization efficiency between the main and backup databases.

[0096] Optionally, the log synchronization module 330 includes:

[0097] The parsing and reconstruction unit is used to obtain the log package to be synchronized from the asynchronous parsing and reconstruction task based on at least one thread, and parse and reconstruct the log package to be synchronized to obtain a reconstructed log package of the log package to be synchronized.

[0098] Optionally, the log synchronization module 330 includes:

[0099] a replay sorting unit, configured to add the reconstructed log packets to a replay task queue in the order of their log packet sequence numbers;

[0100] The log replay unit is used to perform log replay according to the reconstructed log package in the replay task queue.

[0101] Optionally, the replay sorting unit is specifically configured to:

[0102] Comparing the log packet sequence number of the reconstructed log packet with the next sequence number after the maximum log packet sequence number in the replay task queue;

[0103] When the log packet sequence number of the reconstructed log packet is equal to the next sequence number after the maximum log packet sequence number in the replay task queue, adding the reconstructed log packet to the replay task queue;

[0104] When the log packet sequence number of the reconstructed log packet is not equal to the next sequence number after the maximum log packet sequence number in the replay task queue, the reconstructed log packet is set to a waiting for replay state, and after the reconstructed log packet is awakened, the reconstructed log packet is added to the replay task queue.

[0105] Optionally, the log synchronization module 330 further includes:

[0106] A wake-up unit is used to wake up the target reconstruction log package after adding the reconstruction log package to the replay task queue and obtaining the target reconstruction log package in the waiting replay state; wherein the log package sequence number of the target reconstruction log package is the next sequence number after the log package sequence number of the reconstruction log package.

[0107] The log synchronization device provided by the embodiment of the present invention can execute the log synchronization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] Example 4

[0109] Figure 4 This is a structural diagram of a log synchronization system provided by the fourth embodiment of the present invention. Figure 4 As shown, the log synchronization system is applied to a database cluster, which includes a master database 41 and at least one standby database 42;

[0110] The master database 41 generates a log package to be synchronized based on the data modification operation and sends the log package to be synchronized to the slave database 42;

[0111] The standby database 42 receives the log package to be synchronized sent by the primary database 41; adds the log package to be synchronized to the asynchronous parsing and reconstruction task and sends a response message to the primary database 41; executes the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performs log replay based on the reconstructed log package;

[0112] After receiving the response message, the master library 41 writes the log package to be synchronized into the online log file of the master library 41 .

[0113] Specifically, the main database 41 starts the transaction, performs data modification operations and generates REDO logs, organizes the REDO logs according to the format of the log package to obtain the log package to be synchronized; the main database 41 commits the transaction, sends the log package to be synchronized to the standby database 42, and waits for the response message of the standby database 42. After receiving the log package to be synchronized, the standby database 42 adds the log package to be synchronized to the asynchronous parsing and reconstruction task, and returns a response message to the main database 41 without waiting for the asynchronous parsing and reconstruction task to be completed. After the main database 41 receives the response message from the standby database 42, it writes the log package to be synchronized into the online log file of the main database 41, the transaction is successfully committed, and the next log package is sent. At the same time, the standby database 42 executes the asynchronous parsing and reconstruction task to obtain the reconstructed log package of the log package to be synchronized, and performs log replay according to the reconstructed log package.

[0114] This embodiment provides a log synchronization system, which is applied to a database cluster, and the database cluster includes a main database and at least one backup database; the main database generates a log package to be synchronized based on a data modification operation, and sends the log package to be synchronized to the backup database; the backup database receives the log package to be synchronized sent by the main database; the log package to be synchronized is added to the asynchronous parsing and reconstruction task and a response message is sent to the main database; the asynchronous parsing and reconstruction task is executed to obtain a reconstructed log package of the log package to be synchronized, and log replay is performed based on the reconstructed log package; after receiving the response message, the main database writes the log package to be synchronized to the online log file of the main database. By adding the received log package to be synchronized to the asynchronous parsing and reconstruction task, the backup database returns a response message to the main database without waiting for the asynchronous parsing and reconstruction task to be completed, which effectively shortens the time the main database waits for a response, speeds up the transaction submission speed of the main database, and thus improves the log synchronization efficiency between the main and backup databases.

[0115] Example 5

[0116] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0117] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the log synchronization method.

[0120] In some embodiments, the log synchronization method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log synchronization method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the log synchronization method in any other appropriate manner (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] In some embodiments, the log synchronization method may be implemented as a computer program, which is invisibly included in a computer program product, and the computer program implements the log synchronization method of the present invention when executed by a processor. The computer program product can be understood as a software product that mainly implements its solution through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0127] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0128] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A log synchronization method, characterized in that: Applied to a standby database in a database cluster, the database cluster also including a master database, the method comprising: Receive the log package to be synchronized sent by the master database; Add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database, wherein the response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database; The asynchronous parsing and reconstruction task is executed to obtain a reconstructed log package of the log package to be synchronized, and log replay is performed according to the reconstructed log package.

2. The method according to claim 1, characterized in that The step of executing the asynchronous parsing and reconstruction task to obtain the reconstructed log package of the log package to be synchronized includes: The log package to be synchronized is obtained from the asynchronous parsing and reconstruction task based on at least one thread, and the log package to be synchronized is parsed and reconstructed to obtain a reconstructed log package of the log package to be synchronized.

3. The method according to any one of claims 1-2, characterized in that The log replay according to the reconstructed log package includes: Add the reconstructed log packages to the replay task queue according to the order of the log package serial numbers of the reconstructed log packages; Log replay is performed according to the reconstructed log package in the replay task queue.

4. The method according to claim 3, characterized in that The step of adding the reconstructed log packets to a replay task queue according to the order of the log packet sequence numbers of the reconstructed log packets comprises: Comparing the log packet sequence number of the reconstructed log packet with the next sequence number after the maximum log packet sequence number in the replay task queue; When the log packet sequence number of the reconstructed log packet is equal to the next sequence number after the maximum log packet sequence number in the replay task queue, adding the reconstructed log packet to the replay task queue; When the log packet sequence number of the reconstructed log packet is not equal to the next sequence number after the maximum log packet sequence number in the replay task queue, the reconstructed log packet is set to a waiting for replay state, and after the reconstructed log packet is awakened, the reconstructed log packet is added to the replay task queue.

5. The method according to claim 4, characterized in that After adding the reconstruction log package to the replay task queue, the method further includes: When a target reconstruction log package in a waiting-for-replay state is obtained, the target reconstruction log package is awakened; wherein the log package sequence number of the target reconstruction log package is a sequence number subsequent to the log package sequence number of the reconstruction log package.

6. A log synchronization device, characterized in that: Applied to a standby database in a database cluster, the database cluster also including a master database, the device comprising: A log packet receiving module is used to receive the log packets to be synchronized sent by the master database; A response module is used to add the log package to be synchronized to the asynchronous parsing and reconstruction task and send a response message to the master database, wherein the response message is used to instruct the master database to write the log package to be synchronized into the online log file of the master database; The log synchronization module is used to execute the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and perform log replay according to the reconstructed log package.

7. A log synchronization system, characterized in that: Applied to a database cluster, the database cluster includes a master database and at least one slave database; The primary database generates a log package to be synchronized based on the data modification operation, and sends the log package to be synchronized to the secondary database; The standby database receives the log package to be synchronized sent by the primary database; adds the log package to be synchronized to the asynchronous parsing and reconstruction task and sends a response message to the primary database; Executing the asynchronous parsing and reconstruction task to obtain a reconstructed log package of the log package to be synchronized, and performing log replay according to the reconstructed log package; After receiving the response message, the master database writes the log package to be synchronized into the online log file of the master database.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log synchronization method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log synchronization method according to any one of claims 1 to 5 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the log synchronization method according to any one of claims 1 to 5.

Images