A method and device for ensuring consistency of data in a master and slave database in physical replication

Abstract

The application discloses a method and device for ensuring consistency of master and slave database data in physical replication, which comprises the following steps: controlling the master database to generate a log group about at least one current transaction, and writing log records for processing logical data consistency in the master and slave physical replication when the current transaction starts and commits, so as to form a start log and a commit log different from the log group type at the start end and the end end of the log group; when the slave database encounters the start log during physical replication application of the log group, a logical transaction is started at the same time, and when the slave database encounters the commit log, a logical transaction is committed at the same time. The application solves the problem of "dirty reading" at the slave end during master-slave replication in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a method and device for ensuring data consistency between master and slave databases in physical replication. BACKGROUND

[0002] The synchronization mode of ArkDB master-slave replication includes two kinds. One is logical replication, which is the current mainstream method and is also an officially supported method. However, this method requires that the data of different nodes of the master and slave be physically separated, that is, multiple different data. To ensure data consistency, the same SQL statement needs to be executed independently through logical replication. In order to realize the shared storage feature of the master and slave nodes, another replication method, called physical replication, must be implemented.

[0003] The main goal of physical replication is that the slave database recovers a usable read node according to its current conditions and using existing resources. The existing resources include two parts: the Redo log that is updated in real time and can be continuously obtained from the shared storage, and the B+ tree Page and UNDO log Page that are updated from time to time.

[0004] The process of physical replication of the slave database is to apply the continuously generated Redo log records to the corresponding Page one by one. These Pages are obtained from the shared storage and are not necessarily in the latest state. The Redo log needs to be used to recover to the latest state, so that the Page can always maintain the same state as the master database, that is, to recover a Buffer pool that is the same as the master database. In this way, the slave database can provide read services to users.

[0005] In the creation of a logical transaction, if any SQL modifies the database during execution, it modifies the Page in the Buffer pool. During the modification process, Redo log is generated. The Redo log is physical. After the slave database obtains these Redo logs, it performs Apply operation using these logs. During the Apply process, the Buffer pool of the slave database is modified to recover the Buffer pool of the slave database, so that the Buffer pool of the slave database and the Buffer pool of the master database are as consistent as possible, thereby achieving data consistency of the entire cluster.

[0006] However, in the master-slave replication, the slave database only gets the Redo log generated by the master database, and the logical transaction information cannot be embodied in the Redo log. Thus, the user of the slave database has no transaction characteristics when reading data, because he may see the intermediate state of the transaction being copied, and the data in the intermediate state of a transaction may be incorrect, which leads to the problem of "dirty read" in the slave database. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide a method and device for ensuring the consistency of the data in the master and slave databases in physical replication, aiming to solve the problem of "dirty read" in the slave database in the master-slave replication.

[0008] The embodiment of the present application is implemented as follows:

[0009] A method for ensuring the consistency of the data in the master and slave databases in physical replication, the method comprising:

[0010] controlling the master database to generate a log group about at least one current transaction, and writing log records for processing the logical data consistency in the physical replication of the master and slave databases when the current transaction starts and commits, respectively, to form a start log and a commit log different from the type of the log group at the start end and the end end of the log group;

[0011] when the slave database applies the physical replication of the log group, starting a logical transaction when the start log is encountered, and committing a logical transaction when the commit log is encountered.

[0012] Further, the above method for ensuring the consistency of the data in the master and slave databases in physical replication, wherein the step of applying the physical replication of the log group in the slave database, starting a logical transaction when the start log is encountered, and committing a logical transaction when the commit log is encountered, comprises:

[0013] distributing the start log, the log group and the commit log to threads in the slave database for physical replication application;

[0014] wherein, when the start log is distributed, it is directly executed;

[0015] when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread;

[0016] when the commit log is distributed, the commit log is executed after the execution of the log group is completed.

[0017] Further, the method for ensuring the consistency of the data in the master and slave databases in the physical replication, wherein the step of, when the slave database applies the physical replication to the log group, starting a logical transaction simultaneously when the start log is encountered and committing a logical transaction simultaneously when the commit log is encountered, further comprises:

[0018] creating a new thread for applying the logical log, the thread being responsible for applying the commit log type of log;

[0019] distributing the start log, the log group and the commit log to the threads in the slave database for physical replication application;

[0020] wherein, when the start log is distributed, it is executed directly;

[0021] when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread;

[0022] when the commit log is distributed, the commit log is distributed to the thread for applying the logical log, and the thread determines whether the commit log needs to be distributed according to a preset logic.

[0023] Further, the method for ensuring the consistency of the data in the master and slave databases in the physical replication, wherein the step of, when the logical log application thread determines whether the commit log needs to be distributed according to a preset logic, further comprises:

[0024] obtaining the log sequence number value of the commit log and the minimum log sequence number value applied by each physical replication application thread;

[0025] determining whether the minimum log sequence number value is greater than the log sequence number value of the commit log;

[0026] if not, the commit log is applied;

[0027] if yes, waiting for the minimum log sequence number value to be greater than the log sequence number value of the commit log, and then performing the step of applying the commit log.

[0028] Further, the method for ensuring the consistency of the data in the master and slave databases in the physical replication, wherein the method further comprises:

[0029] When the master library executes an SQL statement, a waiting log is added before the B+ tree modification operation after the completion of the Undo log modification operation, the waiting log records the log sequence value corresponding to the Undo log modification operation;

[0030] When the LSN value that the thread to which the waiting log is distributed has executed is greater than the log sequence value corresponding to the Undo log modification operation, the log of the B+ tree modification operation on the thread where the waiting log is located is continued to be executed.

[0031] Further, the method for ensuring the consistency of the data of the master and slave libraries in physical replication, wherein the step of adding a waiting log before the B+ tree modification operation after the completion of the Undo log modification operation when the master library executes an SQL statement comprises:

[0032] Start executing a certain statement and start executing a certain preceding operation A, and append the log information generated by the preceding operation A to the log sequence, the log being named as Page_ID_A with the Page_ID of the object to be operated as the parallel distribution key;

[0033] Prepare to execute a certain subsequent operation B, and prepare the object ID operated by the operation B, which is named as Page_ID_B;

[0034] Obtain the position information A_LSN after the completion of the execution of the preceding operation A;

[0035] Generate a WAIT_LSN log, the log content including A_LSN and the parallel distribution key information, i.e. Page_ID_B and Page_ID_A, and append the log to the log sequence;

[0036] Start executing the subsequent operation B, i.e. modify the object of Page_ID_B, generate the log information corresponding to the operation B, and append the log information to the log sequence.

[0037] Further, the method for ensuring the consistency of the data of the master and slave libraries in physical replication, wherein the method further comprises:

[0038] Obtain the log sequence generated by the master library, and extract the log information from the log sequence;

[0039] Distribute the log sequence to the corresponding application thread according to the parallel distribution key information;

[0040] If it is a normal log, directly apply;

[0041] If it is a WAIT_LSN log, the wait_for_lsn value recorded in the log is obtained, and the Page_ID_A information is also obtained;

[0042] The thread to which the preceding operation A is dispatched is found through the Page_ID_A value, and the LSN value A_newest_applied_lsn to which the thread is newly applied is obtained;

[0043] The size relationship between A_newest_applied_lsn and wait_for_lsn is judged;

[0044] If wait_for_lsn≤A_newest_applied_lsn, the waiting of the WAIT_LSN log ends, and the thread can apply the next log, i.e. the following operation B;

[0045] If wait_for_lsn>A_newest_applied_lsn, the WAIT_LSN log waits until wait_for_lsn≤A_newest_applied_lsn.

[0046] Another object of the present application is to provide a device for ensuring the consistency of data in a master and a slave in physical replication, characterized in that the device comprises:

[0047] a writing module for controlling the master to generate a log group about at least one current transaction, and writing log records for processing the logical consistency of data in the physical replication of the master and the slave when the current transaction starts and is committed, respectively, to form a start log and a commit log different from the type of the log group at the start end and the end end of the log group;

[0048] an application module for starting a logical transaction when the start log is encountered and committing a logical transaction when the commit log is encountered when the slave applies the physical replication of the log group.

[0049] Another object of the present application is to provide a readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method according to any one of the preceding aspects.

[0050] Another object of the present application is to provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to the preceding aspects when executing the program.

[0051] The application controls the master database to generate a log group about at least one current transaction, and writes log records for processing logical data consistency in the master database and slave database physical replication at the beginning and submission of the current transaction, respectively, to form a start log and a submission log different from the log group type at the beginning and end of the log group; when the slave database encounters the start log during physical replication application of the log group, a logical transaction is started simultaneously, and when the slave database encounters the submission log, a logical transaction is submitted simultaneously. Some logical master database operations are transmitted to the slave database in the form of logs, and the slave database can control the behavior of the slave database according to the needs after obtaining the special logs, thereby ensuring the consistency of the master-slave data. The problem of dirty read in the prior art at the slave database end is solved. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Flow chart of the method for ensuring the consistency of the master-slave database data in the physical replication in the first embodiment of the application;

[0053] Figure 2 Schematic diagram of the master database adding logs in the method for ensuring the consistency of the master-slave database data in the physical replication in an embodiment of the application;

[0054] Figure 3 Schematic diagram of the master-slave database parallel replication in the method for ensuring the consistency of the master-slave database data in the physical replication in an embodiment of the application;

[0055] Figure 4 Schematic diagram of a slave database replication in the method for ensuring the consistency of the master-slave database data in the physical replication in an embodiment of the application;

[0056] Figure 5 Schematic diagram of the master database adding a waiting log in the method for ensuring the consistency of the master-slave database data in the physical replication in an embodiment of the application;

[0057] Figure 6 Structural block diagram of the device for ensuring the consistency of the master-slave database data in the physical replication in the fourth embodiment of the application.

[0058] The following specific embodiments will further illustrate the application in combination with the above drawings. DETAILED DESCRIPTION

[0059] In order to facilitate the understanding of the application, the application will be described more fully below with reference to the accompanying drawings. Several embodiments of the application are given in the drawings. However, the application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the application more thorough and comprehensive.

[0060] It should be noted that when an element is referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. Where an element is referred to as being "connected", "coupled", "attached", or "linked" to another element, it can be directly connected, coupled, attached, or linked to the other element, or intervening elements can be present. As used herein the terms "vertical", "horizontal", "left", "right", and the like are merely used for the purpose of illustration.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0062] The synchronization mode of the master-slave replication of ArkDB database includes two kinds, one is logical replication, which is the current mainstream use method and the official support method, but the premise of this method is that the data of the master-slave different nodes is physically separated, which is multiple different data, in order to ensure data consistency, it is necessary to execute the same SQL statement independently through logical replication. In order to realize the shared storage characteristics of the master-slave nodes, another replication method must be implemented, which is called physical replication compared with logical replication.

[0063] The ArkDB database of shared storage architecture has one write node and N read nodes. The data changes generated when the write node writes data include three parts:

[0064] Log: including Redo Log and Binlog. Binlog is used in the logical replication mentioned above, because Binlog generally records SQL statements, which can be executed independently in each node. Redo log is used in physical replication, because the recording form of Redo log is the write information of the underlying physical page (Page), such as it records a piece of data with specified length and content written on a certain Page, so it is called physical. Among the two kinds of logs, only Redo log will be written into shared storage to do physical replication, and Binlog is used for logical replication, so it will not be written into shared storage. Redo log will continuously write the latest log into shared storage, so that the slave library can apply more timely, in order to maintain the consistency with the master library state.

[0065] B+Tree Buffer pool Pages: This type of Page is used to store the real data in a table, and is organized in a B+ tree structure. These pages are constantly modified in the master database, and the modification process generates Redo logs, which are used for physical replication between the master and slave. After modification, the Page is a cache object in the Buffer pool, so it is not immediately flushed to the storage layer (shared storage). It is only flushed when needed.

[0066] UNDO Log Buffer pool pages. This type of Page is used to store rollback records, i.e. the historical version of a record in a table after it has been modified, commonly known as a rollback record. The corresponding Page is called a rollback Page. These Pages are also Buffer pool objects, and their flushing mechanism is the same as that of the B+ tree Pages.

[0067] In the process of writing new data to the master, only Redo Log is seen by read-only nodes in real time, while other data may not be seen at all, as Buffer pool data is used for caching to improve performance. Only when the Redo Log is written and the Log file space is insufficient, a checkpoint is needed, and the Buffer pool is flushed in order of LSN value from small to large. After being written to shared storage, it can be seen. However, the Pages seen are not necessarily the current updated or written data Pages, so only Redo Log can be used by slave nodes.

[0068] LSN (log sequence number) is the value that increases by 1 for each byte of log generated during log generation. Therefore, each byte of Redo log corresponds to an LSN. Both the master and the slave use LSN to distinguish different states, including the master's newest state, newest_log_lsn, checkpoint_lsn, the slave's newest applied LSN value, newest_applied_lsn, etc. A Page also has its LSN state, which represents the maximum redo log lsn value generated when modifying the Page, represented by newest_page_lsn. LSN values can be compared for size. For the same object, a larger LSN value indicates an updated state.

[0069] The primary goal of physical replication is for the slave database to restore a usable read node based on its current conditions and existing resources. These existing resources consist of two parts:

[0070] 1. The constantly updated redo log can be continuously retrieved from shared storage.

[0071] 2. The B+ tree page and UNDO log page are updated from time to time.

[0072] The physical replication process from the slave database involves continuously applying redo log records, one by one, to the corresponding pages. These pages are retrieved from shared storage and may not be in the latest state. The redo log needs to be used to restore them to the latest state so that the pages always remain consistent with the master database. In other words, a buffer pool identical to that of the master database is restored, allowing the slave database to provide read services to users.

[0073] Redo logs are generated sequentially and stored in log files, appended continuously. Therefore, the content is logically sequential, composed of individual log records, each consisting of several bytes. Each record is defined by the ArkDB kernel, and different records are related to each other, or sequentially. A series of consecutive log records form a log group. A group consists of logs generated by a specific physical transaction, written to the file sequentially. A group includes several log records of different types, but the beginning of each log group signifies the start of the physical transaction, and the end of the log group signifies the commit operation of that physical transaction.

[0074] The redo log is the carrier used to update data during physical replication. The smallest complete unit for applying is the group mentioned above. That is to say, during the replication process, either all statements in this group are applied, or none of them are applied. The log group is said to be the smallest unit because above this unit, there is a larger unit, namely the logical transaction. This is a fundamental concept in databases. It ensures MVCC and data integrity. It is a higher-level business concept in the database. Since it is a transaction, it means that the SQL statements executed in this transaction are either all executed and committed, and the data is successfully modified, or none of them are executed.

[0075] To create a logical transaction, a start transaction needs to be executed when the SQL statement is executed in the master, and a commit transaction needs to be executed after the execution is completed, all the statements executed between the start transaction and the commit transaction belong to this transaction, so these statements have all the characteristics of a transaction, one of the most obvious properties is the isolation level, that is, if the isolation level is set to Read committed during the execution of the transaction, other users cannot see any data modification made by the executing transaction, that is, other users cannot see the intermediate state of a transaction, that is, they see the state before the transaction is committed or the state after the transaction is committed, such a feature relies on the start and commit operations of the transaction executed in the master.

[0076] Since the transaction here is logical, any SQL executed in the process will modify the Page in the Buffer pool, and the Redo log will be generated during the modification process, the Redo log is physical, and after the slave obtains the Redo log, the Apply operation will be performed, and the modification of the Buffer pool of the slave during the Apply process is to restore the Buffer pool of the slave, so that the Buffer pool of the slave and the Buffer pool of the master are as consistent as possible, thereby achieving the data consistency of the entire cluster.

[0077] However, when the master-slave replication is performed, the slave only obtains the Redo log generated by the master, and the information of the logical transaction cannot be reflected in the Redo log, so the user of the slave does not have the transaction characteristics when reading data, because he may see the intermediate state of the transaction being replicated, because the data of the intermediate state of a transaction may be wrong, which leads to the "dirty read" problem on the slave side.

[0078] The following will be described in detail in combination with specific embodiments and drawings how to solve the problem of the "dirty read" problem on the slave side during physical replication.

[0079] Embodiment One

[0080] Please refer to Figure 1 , which shows a method for ensuring the consistency of data of the master and the slave in the physical replication according to the first embodiment of the present application, the method comprises steps S10-S11.

[0081] Step S10, the control master library generates a log group about at least one current transaction, and writes log records for processing logical data consistency in the master library and slave library physical replication when the current transaction starts and commits, respectively, to form a start log and a commit log different from the log group type at the start end and the end end of the log group.

[0082] Wherein, the log (Redo log) generated by the modification of the master library data is sequentially generated when generated, and then stored in the log file, which is always appended to the rear, so the content logic is sequential, composed of individual log records, a record is composed of several bytes, each record is defined by the kernel of ArkDB, and there is an association between different records or before and after. A plurality of continuous log records form a log group, a group is a log generated by a certain physical transaction, and then written to the file in a continuous manner, a group includes a plurality of different types of log records, but the beginning of each log group means the start operation of the physical transaction, and the end of the log group means the commit operation of the physical transaction.

[0083] Specifically, in order to ensure that the slave library also has transaction characteristics, transaction information needs to be added to the log when the master library generates the log, that is, a log record is written when the transaction starts, and a log record is written when the transaction commits. The two records are special log types, different from other types of logs, which are used to process logical data consistency in master-slave physical replication, not physical data consistency, which are start log (starttransaction) and commit log (commit transaction), wherein, as shown in Figure 2 Each log group includes different log records, such as (log-2, log-3, log-4, log-5), (log-8, log-9, log-10), a start log and a commit log are added at the start and end of each log group.

[0084] Exemplarily, when the log type is increased, the following steps can be performed to increase the log type of the master library.

[0085] 1. The master library starts to execute a certain SQL statement;

[0086] 2. The master library triggers an event of adding a new log type;

[0087] 3. The master library creates a new Redo log record, and the type of the record is the triggered type;

[0088] 4. The master library writes the new Redo log into the log file.

[0089] Step S11, when the start log is encountered during the physical replication application of the log group from the library, a logical transaction is started at the same time, and when the commit log is encountered, a logical transaction is committed at the same time.

[0090] Specifically, with the two types of logs, the start log and the commit log, when the physical replication application is performed, a logical transaction can be started at the same time when the start log of the log is encountered, and a logical transaction can be committed at the same time when the commit log of the log is encountered. The log group between the two logs has visibility due to the control of the transaction mechanism, that is, before the commit, the modifications during the period are invisible to other users, so that the expected goal is achieved, and the significance of the logical transaction exists.

[0091] It can be understood that by using this method of increasing the type of log, the consistency of the master-slave data can be solved, and the original logical master library operation is transmitted to the slave library in the form of a log. After the slave library obtains these special logs, the behavior of the slave library can be controlled according to the needs, and the consistency of the master-slave data is ensured.

[0092] In summary, the method for ensuring the consistency of the master-slave data in the physical replication in the above-mentioned embodiments of the present application controls the master library to generate a log group about at least one current transaction, and writes log records for processing the logical data consistency in the physical replication of the master library and the slave library at the start and the commit of the current transaction, respectively, to form a start log and a commit log different from the type of the log group at the start end and the end end of the log group. When the slave library applies the physical replication to the log group, a logical transaction is started at the same time when the start log is encountered, and a logical transaction is committed at the same time when the commit log is encountered. The original logical master library operation is transmitted to the slave library in the form of a log. After the slave library obtains these special logs, the behavior of the slave library can be controlled according to the needs, and the consistency of the master-slave data is ensured. The problem of "dirty read" in the prior art at the slave end is solved.

[0093] Embodiment two

[0094] The embodiment of the present application also proposes a method for ensuring the consistency of the master-slave data in the physical replication. The method for ensuring the consistency of the master-slave data in the physical replication proposed in the embodiment is different from the method for ensuring the consistency of the master-slave data in the physical replication proposed in the first embodiment of the present application in that:

[0095] Step S11 includes:

[0096] The start log, the log group, and the commit log are distributed to threads in the slave library for physical replication application;

[0097] wherein, when the start log is distributed, it is executed directly;

[0098] when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread;

[0099] when the commit log is distributed, the commit log is executed after the execution of the log group is completed.

[0100] wherein, after the two types of transaction log types are added, the data "dirty read" problem of the slave is solved, but this is only well solved when the physical replication of the slave has only one thread, and the physical replication of ArkDB itself is parallel, that is, there are multiple threads that can be dynamically set, and in the case of parallel application of logs and parallel replication, all logs that need to be applied are distributed to multiple threads according to a certain rule, and the log records are distributed according to the original order, one by one. Since this is physical replication, in order to ensure that the same physical page is modified multiple times and still ensure the original modification order, the physical page identity (Page ID) recorded in the log is used for distribution, so the same physical page will be distributed to the same thread, so even if it is parallel, the final consistency of the data can be ensured.

[0101] The premise of such distribution is that each log record is physical and contains a necessary information, Page ID. However, the start log and commit log types added above to solve the data visibility problem are logical and do not have physical properties, so there is no Page ID, so the above distribution logic cannot be used to distribute it. At the same time, the start log must be executed before all the logs it contains, and the commit log must be executed after these logs, so in this embodiment, as shown in Figure 3 wherein, when the start log is distributed, it is executed directly; when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread; when the commit log is distributed, the commit log is executed after the execution of the log group is completed.

[0102] However, because all physical records in the log will definitely belong to a logical transaction, if the parallel is changed back to serial every time a start log and a commit log are encountered, the performance will not be much better than that of serial. Considering the characteristics of the start log and the commit log, the logs contained therein need to meet two conditions during application:

[0103] 1. These logs must be executed between this pair of logical logs;

[0104] 2. It should not have a significant impact on the performance of parallel applications.

[0105] Therefore, in some preferred embodiments of the present invention, step S11 further includes:

[0106] Create a new thread for the logical logging application, which is responsible for applying the logs of the committed log type;

[0107] The start log, log group, and commit log are distributed to the threads in the slave database for physical replication.

[0108] Specifically, when the start log is distributed, it is executed directly.

[0109] When the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and physical pages with the same identity are distributed to the same thread;

[0110] When the commit log is distributed, it is distributed to the thread of the logical log application, and the thread of the logical log application determines whether the commit log needs to be distributed according to preset logic.

[0111] Among them, such as Figure 4 As shown, a new thread for logical log application is created. This thread is only responsible for applying commit logs. When distributing each redo log record, if it is a normal physical log, it can be distributed normally. If it is a start log, it is applied directly because this is the starting point of the logical transaction. Applying it at this time ensures that the redo log application operations contained in it are after the application operations of this log. If it is a commit log, it is distributed to the logical log application thread. Whether this thread applies it or not is its own logic. Continue to distribute subsequent log records.

[0112] Specifically, the thread logic for the logical log application is as follows:

[0113] Because each Redo log has a log sequence value (LSN), from the LSN point of view, for a logical transaction, the LSN value of a commit log is greater than the LSN value of the Redo log contained therein, and the LSN of a start type log is less than the LSN value of the Redo log contained therein; the thread of the logical log application obtains the LSN (commit_transaction_lsn) value of each log, and then obtains the latest LSN value applied by each physical replication Apply thread, takes the minimum value, that is, newest_applied_lsn, and then compares it with commit_transaction_lsn; if commit_transaction_lsn≤ newest_applied_lsn, it means that the physical log before the commit transaction log, that is, all the Redo logs contained in the logical transaction, has been applied, at this time the log can be applied, which means that the logical transaction can be committed; if commit_transaction_lsn> newest_applied_lsn, it means that among the logs contained therein, there are logs that have not been applied, at this time it does not apply, that is, the logical transaction cannot be committed, and it can wait, because the Apply is still continuing, and newest_applied_lsn is constantly increasing, so it only needs to wait a little to satisfy commit_transaction_lsn≤ newest_applied_lsn.

[0114] The above logical loop reciprocates, which can guarantee both data consistency and parallel Apply of physical replication.

[0115] For example, refer to Figure 4 The commit transaction of the logical type log is all distributed to the logical thread, and the application time point thereof is after the application time point of the corresponding Redo log, Commit1 is applied after Redo log (2, 3, 4, 5), and Commit2 is applied after Redo log (8, 9, 10), which can meet the requirements described above, and it can also be seen that other normal type Redo logs are applied in parallel, and the performance is not affected.

[0116] In addition, the implementation process of the embodiment is clearly described above by using the logical transaction, which is only an example and does not constitute a limitation on the embodiment, and in practice, similar pairs of logical logs are not only this pair, but also other types of logical logs.

[0117] In summary, the method for ensuring the consistency of the data of the master and slave databases in the physical replication in the above embodiment of the application forms a log group about at least one current transaction by controlling the master database, and writes log records for processing the logical data consistency in the physical replication of the master and slave databases at the start and submission of the current transaction, to form start logs and submission logs different from the log group type at the start end and end end of the log group; when the slave database encounters the start logs during the physical replication and application of the log group, a logical transaction is started at the same time, and when the slave database encounters the submission logs, a logical transaction is submitted at the same time. Some logical master database operations are transmitted to the slave database in the form of logs, and the slave database can control the behavior of the slave database according to the needs after obtaining the special logs, thereby ensuring the consistency of the master and slave data. The problem of "dirty read" at the slave database end in the prior art is solved.

[0118] In addition, the "bottleneck" problem encountered in the serialization parallelization is easily solved by using different threads to perform the Apply, which ensures the performance after parallelization and also ensures the consistency effect of the master and slave data.

[0119] Embodiment three

[0120] The embodiment of the application also provides a method for ensuring the consistency of the data of the master and slave databases in the physical replication, and the method for ensuring the consistency of the data of the master and slave databases in the physical replication provided by the embodiment is different from the method for ensuring the consistency of the data of the master and slave databases in the physical replication provided by the embodiment one of the application in that:

[0121] The method further includes:

[0122] When the master database executes an SQL statement, after completing an Undo log modification operation and before a B+ tree modification operation, a waiting log is added, and the waiting log records a log sequence value corresponding to the Undo log modification operation;

[0123] When an LSN value executed by a thread to which the waiting log is distributed is greater than the log sequence value corresponding to the Undo log modification operation, a log of the B+ tree modification operation on the thread on which the waiting log is located is continuously executed.

[0124] Wherein, in the main library to execute a SQL statement, usually have an impact on multiple data modules, just take the delete statement to delete a data (a logical transaction), will produce 3 data modification, the 3 data are:

[0125] 1. Stored in Undo log Page: this information must be written first, this is the deleted record corresponding to the UNDO Log, to record the data before deletion, there are two purposes, if the logical transaction is eventually rolled back, find this record, cancel the delete mark on the record, thus restoring the original data, if the transaction is eventually committed, find this record, and then make the record has been marked for deletion of the real deletion. Regardless of which operation, the key information recorded in the undo log is used;

[0126] 2. Stored in B+ tree Page: write Undo log Page after the above, can write this, this is the modification of the record itself, that is, the real work of the SQL, that is, delete a data, as described above, delete is only in the record marked for deletion;

[0127] 3. Stored in Redo log file: Redo log generated when modifying the above two parts of data, which is also a data.

[0128] The three data described above are actually strongly dependent, and the Undo log must be written first to modify the B+ tree Page, because in the access process of the slave, for each record, there may be a corresponding historical version, if the Redo log of the B+ tree operation is applied first, the record is deleted, the slave will find the corresponding historical version, but the Undo log corresponding to the Redo log has not been applied, that is, the Undo log does not exist, so this will cause data reading error, so the slave needs to ensure the original order when applying.

[0129] But in the Redo log generated by the master, there is no information to help the slave ensure the original order when applying. At the same time, as mentioned above, the three data stored in the three locations are completely independent, and after parallel physical replication distribution, it is more difficult to ensure their order, because some threads execute fast and some threads execute slow.

[0130] Therefore, when the master generates the log, some auxiliary information is added to help the slave ensure the correctness of the application when parallel.

[0131] Specifically, during the slave's Apply operation, it's crucial to ensure that the Apply operation for B+ tree modifications can only be executed after its corresponding Undo log Apply operation. Both operations generate Redo logs with corresponding LSN values. Therefore, the solution is to add a WAIT_LSN wait log after the Undo log modification operation and before the B+ tree modification operation. This log records the LSN value (wait_for_lsn) corresponding to the completion of the preceding operation. This log is also physical, and its recorded Page ID is on the same page as the log for the subsequent B+ tree modification operation, ensuring their order. The WAIT_LSN log is used for waiting; the waiting event is the completion of the preceding operation, i.e., the LSN value reached by the thread it was assigned to. If the LSN value is greater than wait_for_lsn, it indicates that the WAIT_LSN log wait is complete, and the thread containing the WAIT_LSN log can then continue execution. Following this is the log for the B+ tree modification operation. At this point, due to the Undo log... Since the modification operation log has already been applied, the log for this B+ tree modification operation can also be applied with confidence.

[0132] For example, such as Figure 5 As shown, blue icons represent logs of type WAIT_LSN. The first diagram represents the initial log sequence, listing three groups of logs in the following order: 1->2, 3->4, 5->6. That is, 1 must be applied before 2, 3 before 4, and 5 before 6. The order within these three groups is not important. A log of type WAIT_LSN, w-log-2, is added between PageIDs 1 and 2. This type of log waits for Log-1 to be applied. Only after Log-1 is applied can WAIT_LSN be considered applied, and then the next log entry can be executed. The Page_ID recorded in the WAIT_LSN log is the same as the Page_ID of the logs that follow it. This ensures they are dispatched to the same thread to control their execution order. The actual order of each group is indicated by the red dashed line. As you can see, after adding the WAIT_LSN log, the order of each group is guaranteed. The order between groups does not affect the result. This will depend on different logs and the specific execution of the thread.

[0133] More specifically, the steps to add a new log to the primary database include:

[0134] 1. Begin executing a certain statement;

[0135] 2. Start to execute a pre-operation A, which will generate a log, and then append the log to the log sequence. The log is named Page_ID_A, with the Page_ID of the object as the parallel dispatch Key;

[0136] 3. Prepare to execute a post-operation B, and prepare the object ID, Page_ID, of the operation B, named Page_ID_B information;

[0137] 4. Get the position information A_LSN after the pre-operation A is executed;

[0138] 5. Generate a WAIT_LSN log, which includes A_LSN, the parallel dispatch Key information, Page_ID_B, and Page_ID_A, and then append the log to the log sequence;

[0139] 6. Start to execute the post-operation B, i.e. modify the object of Page_ID_B, which will generate the log information of operation B, and then append the log to the log sequence. The log is named Page_ID_B, with the Page_ID of the object as the parallel dispatch Key;

[0140] 7. Continue to execute the remaining operations.

[0141] The steps of applying the log from the library include:

[0142] 1. The library gets the log sequence generated by the master library;

[0143] 2. Extract the log information from the log sequence;

[0144] 3. Distribute the log sequence according to the parallel dispatch Key information, and hand it over to the Apply thread;

[0145] 4. If it is a normal log, then directly apply it;

[0146] 5. If it is a WAIT_LSN log, then get the wait_for_lsn value recorded in the log, and get the Page_ID_A information;

[0147] 6. Find the thread to which the pre-operation A is dispatched through the Page_ID_A value, and get the LSN value A_newest_applied_lsn to which the thread is newly applied;

[0148] 7. Judge the size relationship between A_newest_applied_lsn and wait_for_lsn.

[0149] 8. If wait_for_lsn <= A_newest_applied_lsn, the waiting of WAIT_LSN is over, the thread can apply the next log, i.e. the post-operation B;

[0150] 9. If wait_for_lsn > A_newest_applied_lsn, the WAIT_LSN sleeps for several milliseconds, and goes to step 8.

[0151] Go to step 4, apply the next log.

[0152] In summary, the method for ensuring consistency of data in the master and slave databases in the physical replication in the above embodiments of the application controls the master database to generate a log group about at least one current transaction, and writes log records for processing logical data consistency in the physical replication of the master and slave databases at the start and submission of the current transaction, so as to form start logs and submission logs different from the type of the log group at the start end and end end of the log group; when the slave database applies the physical replication of the log group, a logical transaction is started when the start log is encountered, and a logical transaction is submitted when the submission log is encountered. Some logical operations of the master database are transmitted to the slave database in the form of logs, and the slave database can control the behavior of the slave database according to the needs after obtaining the special logs, thereby ensuring consistency of data in the master and slave databases. The problem of "dirty reading" at the slave database end in the prior art is solved.

[0153] In addition, the problem of key sequence dependence is solved by adding the waiting log in the log.

[0154] Embodiment Four

[0155] Referring to Figure 6 , a device for ensuring consistency of data in the master and slave databases in the physical replication is shown, the device comprises:

[0156] The writing module 100 is configured to control the master database to generate a log group about at least one current transaction, and write log records for processing logical data consistency in the physical replication of the master and slave databases at the start and submission of the current transaction, so as to form start logs and submission logs different from the type of the log group at the start end and end end of the log group.

[0157] The application module 200 is configured to start a logical transaction when the start log is encountered, and submit a logical transaction when the submission log is encountered when the slave database applies the physical replication of the log group.

[0158] The functions or operation steps realized when the above modules are executed are substantially the same as the method embodiments, and will not be described herein again.

[0159] Embodiment five

[0160] Another aspect of the present application further provides a readable storage medium, which stores a computer program, and the program realizes the steps of the method according to any one of the above embodiments one to three when executed by a processor.

[0161] Embodiment six

[0162] Another aspect of the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor realizes the steps of the method according to any one of the above embodiments one to three when executing the program.

[0163] The technical features of the above embodiments can be combined in any manner. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0164] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for realizing the logic function, which can be specifically embodied in any computer readable storage medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from an instruction execution system, device or apparatus. For the present specification, the "computer readable storage medium" can be any device that can contain, store, communicate, propagate or transport a program for use by or in conjunction with an instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0165] More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable storage medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion, interpretation or processing, if necessary, in other suitable manner, and then stored in a computer memory.

[0166] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, the various steps or methods can be implemented, in part, or in whole, in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, the implementation can be with any one or a combination of the following technologies, which are all well-known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals; an application specific integrated circuit having appropriate combinational logic gates; a programmable gate array (PGA); a field programmable gate array (FPGA), and the like.

[0167] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0168] The above-described embodiments only express several implementation manners of the application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the application, which are all within the protection scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A method for ensuring consistency of data between a master and a slave in physical replication, characterized in that, The method comprises: controlling a master database to generate a log group about at least one current transaction, and writing log records for processing logical data consistency in the master database and slave database physical replication respectively when the current transaction starts and commits, to form a start log and a commit log different from the log group type at the start end and the end end of the log group; when the slave database physically replicates and applies the log group, starting a logical transaction when the start log is encountered, and committing a logical transaction when the commit log is encountered; the step of starting a logical transaction when the start log is encountered and committing a logical transaction when the commit log is encountered when the slave database physically replicates and applies the log group comprises: distributing the start log, the log group, and the commit log to threads in the slave database for physical replication and application; when the start log is distributed, it is directly executed; when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread; when the commit log is distributed, the commit log is executed after the execution of the log group is completed; the step of starting a logical transaction when the start log is encountered and committing a logical transaction when the commit log is encountered when the slave database physically replicates and applies the log group further comprises: a logical log application thread is newly created, and the logical log application thread is responsible for applying the commit log type log; the start log, the log group, and the commit log are distributed to threads in the slave database for physical replication and application; when the start log is distributed, it is directly executed; when the log group is distributed, the physical pages contained in the log group are distributed to multiple threads for parallel replication, and the physical pages with the same identity are distributed to the same thread; when the commit log is distributed, the commit log is distributed to the logical log application thread, and whether the commit log needs to be distributed is determined by the logical log application thread according to a preset logic.

2. The method for ensuring the data consistency between the master and slave databases in physical replication according to claim 1, wherein, the step of determining whether the commit log needs to be distributed by the logical log application thread according to a preset logic comprises: obtaining a log sequence number value of the commit log, and obtaining a minimum log sequence number value newly applied by each physical replication and application thread; determining whether the minimum log sequence number value is greater than the log sequence number value of the commit log; if not, the commit log is applied; if yes, the step of applying the commit log is executed after the minimum log sequence number value is greater than the log sequence number value of the commit log.

3. The method for ensuring the data consistency between the master and slave databases in physical replication according to claim 2, characterized in that, the method further comprises: When the master library executes an SQL statement, a waiting log is added before the B+ tree modification operation after the completion of the Undo log modification operation, the waiting log recording the log sequence value corresponding to the Undo log modification operation; When the LSN value to which the thread to which the waiting log is distributed has executed is greater than the log sequence value corresponding to the Undo log modification operation, the B+ tree modification operation log on the thread to which the waiting log is located is continued to be executed.

4. The method for ensuring the data consistency between the master and slave databases in physical replication according to claim 3, characterized in that, The step of adding a waiting log before the B+ tree modification operation after the completion of the Undo log modification operation when the master library executes an SQL statement comprises: Start executing a certain statement and start executing a certain pre-operation A, and append the log information generated by the pre-operation A to the log sequence, the log being named as Page_ID_A with the Page_ID of the object to be operated as the parallel distribution Key; Prepare to execute a certain post-operation B, and prepare the object ID operated by the operation B, named as Page_ID_B; Obtain the position information A_LSN after the completion of the execution of the pre-operation A; Generate a WAIT_LSN log, the log content including A_LSN and the parallel distribution Key information, namely Page_ID_B and Page_ID_A, and append the log to the log sequence; Start executing the post-operation B, that is, modify the object of Page_ID_B, generate the log information corresponding to the operation B, and append the log to the log sequence.

5. The method for ensuring the data consistency between the master and slave databases in physical replication according to claim 4, characterized in that, The method further comprises: Obtain the log sequence generated by the master library, and extract the log information from the log sequence; Distribute the log sequence to the corresponding application thread according to the parallel distribution Key information; If it is a normal log, directly apply it; If it is a WAIT_LSN log, obtain the wait_for_lsn value recorded in the log, and obtain the Page_ID_A information; Find the thread to which the pre-operation A is distributed through the Page_ID_A value, and obtain the LSN value A_newest_applied_lsn to which the thread is newly applied; Determine the size relationship between A_newest_applied_lsn and wait_for_lsn; If wait_for_lsn≤A_newest_applied_lsn, the waiting of the WAIT_LSN log ends, and the thread can apply the next log, namely the post-operation B; If wait_for_lsn>A_newest_applied_lsn, the WAIT_LSN log waits until wait_for_lsn≤A_newest_applied_lsn.

6. An apparatus for ensuring data consistency between master and slave databases during physical replication, characterized in that, The device for implementing the method for ensuring the consistency of data in the master and slave libraries in physical replication according to any one of claims 1 to 5 comprises: a writing module, configured to control the master database to generate a log group about at least one current transaction, and write log records for logically processing data consistency in the master database and the slave database physical replication when the current transaction starts and commits, respectively, to form a start log and a commit log different from the log group type at the start end and the end end of the log group; an application module, configured to, when the slave database physically replicates and applies the log group, start a logical transaction when the start log is encountered, and commit a logical transaction when the commit log is encountered.

7. A readable storage medium, having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

8. An electronic device, comprising: A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method in any one of claims 1 to 5 when executing the program.

Images