Data synchronization and query method and device for master and slave databases

Abstract

The invention discloses a data synchronization and query method and device for master and slave databases, and relates to the field of data processing. According to the method, the to-be-synchronized data in the master database is quickly written into the memory database, and the memory database performs data synchronization on the slave database before the to-be-synchronized data is updated to the slave database. Based on the data synchronization method, when the slave database copies the updated data of the master database, the memory database also stores the updated data which are not copied in time by the slave database, and in the query process, the data which are not synchronized in time in the memory database are preferentially read, so that data missing is avoided, and the query efficiency is improved. And under the condition that the target data is not queried in the memory database, the slave database is queried again, and the data in the slave database is updated data at the moment, so that the problem that the query result is consistent with that of the master database does not exist.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data synchronization and query method and device for a master-slave database. Background Art

[0002] The master-slave database architecture is a widely used database deployment model. It usually includes a master database and one or more slave databases. The master-slave replication data synchronization mechanism is used between each database to replicate data changes on the master database to another or more slave databases to achieve data consistency among multiple databases.

[0003] When the data of the master and slave databases are consistent, the slave database is usually responsible for reading the master database to share the load of the master database. However, during the master-slave replication process, there is data written to the master database that has not been fully synchronized to the slave database, resulting in the data results obtained by the slave database in response to the read operation being inconsistent with the master database. Summary of the invention

[0004] In view of the above problems, the present application provides a data synchronization and query method and device for a master-slave database to achieve the purpose of keeping the data reading result consistent with the data in the master database. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data synchronization method for a master-slave database, which is applied to a server, wherein the server comprises: a master database, a memory database, and at least one slave database;

[0006] The data synchronization method of the master-slave database comprises:

[0007] In response to an update operation on the master database, at least one piece of to-be-synchronized data corresponding to the update operation is sequentially written into the memory database, wherein the to-be-synchronized data is data in the master database waiting to be synchronized to the slave database;

[0008] Determine the expiration time of each piece of data to be synchronized in the memory database according to the write time of each piece of data to be synchronized and the master-slave synchronization delay time, wherein the master-slave synchronization delay time is the time it takes to copy a piece of data in the master database to the slave database;

[0009] The data to be synchronized stored in the memory database are synchronized to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

[0010] In a possible implementation, the memory database includes: a first memory block and a second memory block, wherein the first memory block and the second memory block are respectively used to store the data to be synchronized;

[0011] The step of writing the at least one piece of to-be-synchronized data corresponding to the update operation into the memory database in sequence includes:

[0012] Setting the first memory block as a main memory block;

[0013] Writing the data to be synchronized into the main memory block in sequence;

[0014] Determining whether the storage capacity of the first memory block is less than a preset storage capacity;

[0015] When the storage capacity of the first memory block is not less than the preset storage capacity, the data to be synchronized in the second memory block is migrated to the Bloom filter, and the second memory block is set as the main memory block, and the step of writing the data to be synchronized into the main memory block in sequence is executed.

[0016] In a possible implementation, migrating the to-be-synchronized data in the second memory block to a Bloom filter includes:

[0017] Migrate the data to be synchronized in the first memory block to a Bloom filter, and obtain a Bloom filter list corresponding to each piece of the data to be synchronized, each of the Bloom filter lists corresponding to a list time, and the list time is the time when the data to be synchronized is migrated to the Bloom filter;

[0018] Determine the time difference between the list time corresponding to each of the Bloom filter lists and the current time;

[0019] The Bloom filter list whose time difference is greater than the master-slave synchronization delay is deleted.

[0020] In a possible implementation, the method further includes:

[0021] When the current time is the expiration time corresponding to any one of the to-be-synchronized data in the main memory block, the to-be-synchronized data corresponding to the current time in the main memory block is cleared.

[0022] In a possible implementation, the process of determining the master-slave synchronization delay time includes:

[0023] Determine the data synchronization progress between the master database and the slave database corresponding to the current detection cycle according to the preset detection cycle;

[0024] Determine the first master-slave synchronization delay duration corresponding to the current detection cycle according to the data synchronization progress corresponding to the current detection cycle;

[0025] Get the second master-slave synchronization delay duration of the last detection cycle;

[0026] Determine whether a time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold;

[0027] If the time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold, the first master-slave synchronization delay time is determined as the master-slave synchronization delay time.

[0028] A second aspect of the present application provides a data query method for a master-slave database, which is applied to the data synchronization method for the master-slave database. The data query method for the master-slave database includes:

[0029] In response to a query request for target data in the main database, query whether the target data is stored in the memory database;

[0030] In a case where the target data is stored in the memory database, reading the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request;

[0031] In the case that the target data is not stored in the memory database, the slave database is queried whether the target data is stored. If the slave database stores the target data, the synchronization data of the master database corresponding to the target data stored in the slave database is read as the query result of the query request.

[0032] In a possible implementation, querying whether the target data is stored in the memory database includes:

[0033] querying in sequence whether the target data is stored in the first memory block and the second memory block;

[0034] When the target data is not stored in the first memory block and the second memory block, it is queried whether the target data is stored in the Bloom filter list.

[0035] In a possible implementation, the method further includes:

[0036] If the slave database does not store the target data, it is determined that the query result of the query request is a query failure.

[0037] A third aspect of the present application provides a data synchronization device for a master-slave database, which is applied to a server, wherein the server comprises: a master database, a memory database, and at least one slave database;

[0038] The data synchronization device of the master-slave database comprises:

[0039] a data writing unit, configured to, in response to an update operation on the master database, write at least one piece of to-be-synchronized data corresponding to the update operation into the memory database in sequence, wherein the to-be-synchronized data is data in the master database waiting to be synchronized to the slave database;

[0040] An expiration time determination unit, used to determine the expiration time of each piece of data to be synchronized in the memory database according to the write time of each piece of data to be synchronized and the master-slave synchronization delay time, wherein the master-slave synchronization delay time is the time it takes to copy a piece of data in the master database to the slave database;

[0041] A data synchronization unit is used to synchronize the data to be synchronized stored in the memory database to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

[0042] A fourth aspect of the present application provides a data query device for a master-slave database, which is applied to the data synchronization device for the master-slave database. The data query device for the master-slave database includes:

[0043] A memory data query unit, configured to query whether the target data is stored in the memory database in response to a query request for the target data in the main database;

[0044] A query result determining first unit is used to read the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request when the target data is stored in the memory database;

[0045] The query result determines the second unit, which is used to query whether the target data is stored in the slave database when the target data is not stored in the memory database. If the target data is stored in the slave database, the synchronization data of the master database corresponding to the target data stored in the slave database is read as the query result of the query request.

[0046] The data synchronization method of the master-slave database provided in the embodiment of the present application utilizes the fast reading and writing characteristics of the memory database to quickly write the updated data in the master database, that is, the data to be synchronized to the slave database, into the memory database. Before the data to be synchronized is updated to the slave database, the memory database is used as a transfer station for the data to be synchronized, and the memory database synchronizes the data to the slave database. Based on this data synchronization method, while the slave database copies the updated data of the master database, the memory database also stores the updated data that the slave database failed to copy in time, so that during the query process, the updated data that failed to be synchronized in time can be queried, providing a query channel to ensure the consistency of the data query results.

[0047] Based on this, when performing data queries, since the slave database may not have completely copied all the data to be synchronized, if the slave database is read directly, there is still a problem that the reading result is inconsistent with the main database. Therefore, this application gives priority to reading the data that has not been synchronized in time in the memory database to avoid missing data. If the target data is not found in the memory database, the slave database is queried again. At this time, the data in the slave database are all updated data, so there is no problem of the query result being consistent with the main database. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A server architecture diagram is provided for implementing an embodiment of the present application;

[0050] Figure 2 A flowchart of a method for implementing data synchronization between a master and a slave database provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a flow chart of a method for implementing a master-slave database data query provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of a data synchronization device for a master-slave database provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a data query device for a master-slave database provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The master-slave database architecture can be used for hot standby of data, that is, when the master database fails, it can quickly switch to the slave database so that the system business can run normally. Therefore, it is necessary to ensure the data consistency between the master and slave databases through database master-slave replication. Among them, the process of database master-slave replication can be understood as follows: after the master server performs the data modification operation, it transmits the modified content to the slave server in the form of binary logs. The slave server reads and parses the binary logs to copy the data changes of the master server to its own database, thereby achieving data consistency with the master server. When the data of the master and slave databases are consistent, in order to reduce the load of the master database, the slave database usually responds to the read operation.

[0056] In the existing master-slave replication process, in order to improve the speed of responding to the front-end write instructions, the master database usually feeds back the front-end an indication of synchronization completion after transmitting the binary log, but in fact the slave database has not yet read and parsed the binary log, that is, the data synchronization operation of the slave database has not actually been completed. If the front-end issues a read operation at this time, the slave database cannot provide the synchronized data, and the read result of the read operation is likely to be old data, resulting in inconsistency between the read result and the data of the master database.

[0057] In order to solve the above technical problems, the present application embodiment provides a data synchronization method for a master-slave database, which is applied to a server, referring to Figure 1 , an embodiment of the present application provides a server architecture diagram, wherein the server includes: a master database, a memory database and at least one slave database.

[0058] Among them, the relationship between the master database and the slave database is the same as the conventional master-slave database architecture. One or more slave databases are used for hot standby of the master database, responding to any data changes in the master database, and synchronizing any data changes in the master database from the slave database to ensure that the data in the master and slave databases remain consistent.

[0059] The difference is that in the embodiment of the present application, an in-memory database is inserted between the master database and the slave database as a transit station for data in the master database to be synchronized to the slave database. When the front end writes the write operation of the data edit of the master database to the master database, the master database writes the same write operation to the in-memory database almost at the same time, and the in-memory database records each write operation.

[0060] Further, according to the structure of the above server, refer to Figure 2 , a flow chart of a method for implementing data synchronization of a master-slave database provided in an embodiment of the present application, which illustrates the steps of the method for implementing data synchronization of a master-slave database, may specifically include the following steps:

[0061] Step S110, in response to the update operation on the main database, at least one piece of to-be-synchronized data corresponding to the update operation is sequentially written into the in-memory database.

[0062] The embodiments of the present application take into account that the synchronization of the master-slave database is affected by factors such as replication delay and data consistency requirements, and the master-slave data synchronization rate is slow. Compared with the slave database, the in-memory database has a faster data access speed and reduces I / O operations between disks. Therefore, before the master-slave database is synchronized, the data in the master database waiting to be synchronized to the slave database, such as data modification, deletion, update and other write operations, are written into the in-memory database, so that the updated data content can be saved as soon as possible, providing a query channel for subsequent data query and update data.

[0063] Among them, the update operation on the main database can be understood as the modification, deletion, replacement, update and other operations written to the main database. Optionally, the primary key in the memory database is set to the primary key of the data table, and the value is x. For the addition and modification operations, x is the value of other fields in the data table; for the deletion operation, x is the deletion flag.

[0064] Step S120, determining the expiration time of each piece of data to be synchronized in the memory database according to the writing time of each piece of data to be synchronized and the master-slave synchronization delay time.

[0065] Step S130, synchronizing the data to be synchronized stored in the memory database to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

[0066] It is understandable that the memory database has a storage capacity limit, and for the embodiment of the present application, the data stored in the memory database will eventually be synchronized to the slave database, so the synchronized data is useless data for the memory database. Therefore, the expiration time of each piece of data to be synchronized in the memory database can be determined based on the duration of data synchronization. The expiration time can indicate the time when the data is synchronized to the slave database. According to the expiration time of each piece of data to be synchronized, the memory database deletes the expired data to be synchronized in an orderly manner to make room for memory space for the new data to be synchronized waiting to be written.

[0067] Furthermore, the result of adding the writing time of the data to be synchronized to the master-slave synchronization delay is used as the expiration time, wherein the time for the write operation to write to the memory database is almost the same as the time for writing to the master database, and the time required for the content database to synchronize data with the slave database can be determined by referring to the difference between the time for the slave database to complete the write operation of the master database and the time for the write operation to write to the master database, which is the master-slave synchronization delay. Due to factors such as the data access speed of the memory database and the reduction of I / O operations, the data synchronization time between the memory database and the slave database will be correspondingly shorter than the master-slave replication time. Therefore, the expiration time determined based on the master-slave synchronization delay time and the write time is not shorter than the master-slave synchronization delay time.

[0068] In one possible implementation, the process of determining the master-slave synchronization delay time includes: determining the data synchronization progress between the master database and the slave database corresponding to the current detection cycle based on a preset detection cycle; determining the first master-slave synchronization delay time corresponding to the current detection cycle based on the data synchronization progress corresponding to the current detection cycle; obtaining the second master-slave synchronization delay time of the previous detection cycle; judging whether the time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold; if the time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than the preset threshold, determining the first master-slave synchronization delay time as the master-slave synchronization delay time.

[0069] The replication process of the master-slave database is affected by factors such as data size, database transaction processing capacity, disk response, etc., which results in the inability to accurately quantify the replication time / delay time of the master-slave database. In addition, since the embodiment of the present application is synchronization between the in-memory database and the slave database, the replication delay time between the master-slave database is initially used, which has a certain error, and the expiration time obtained has a certain time margin, which may affect the data processing speed of the in-memory database.

[0070] Based on this, the embodiment of the present application initiates the detection of the synchronization progress of the master-slave database at regular intervals. The update progress of the master-slave database and the synchronization delay duration of the master-slave database can be detected by comparing the amount of updated data in the master database and the amount of updated data in the slave database, or the update time difference between the master database and the slave database when updating the same data. Specifically, mysql runs show slave status to obtain the different delay times of the master and slave: Seconds_behind_Master, and obtains the master-slave synchronization delay time.

[0071] It is understandable that there is a small difference between the master-slave synchronization delay time obtained from each detection, which can be ignored. However, if the difference between the master-slave synchronization delay time of two detections exceeds the threshold, it may be necessary to correct the master-slave synchronization delay time to increase the expiration time of each piece of data to be synchronized in the memory database, so as to avoid the data to be synchronized being cleared from the memory database too early, resulting in the failure to synchronize the data from the slave database, or the data to be synchronized still remaining in the memory database after synchronization, resulting in the inability to write new update operations in the memory database in time, thereby reducing the master-slave synchronization efficiency.

[0072] Based on this, if the difference between the master-slave synchronization delay time of the two detections exceeds the threshold, the master-slave synchronization delay time obtained in this detection will be used as the master-slave synchronization delay time used by the subsequent memory database to determine the expiration time. Optionally, the master-slave synchronization delay time of this detection and the sum of the threshold value can also be used as the master-slave synchronization delay time used by the subsequent memory database to determine the expiration time. It can be understood that the threshold value set in the embodiment of the present application is a relatively reasonable time error value. If it exceeds the threshold value, it may also be caused by accidental factors such as a large amount of updated data. Therefore, in order to avoid the delay caused by accidental factors to the synchronization delay of all data, the embodiment of the present application uses a more reasonable threshold value and the sum of the master-slave synchronization delay time of this time as the master-slave synchronization delay time of step S120.

[0073] Based on this, within the time period corresponding to the expiration time, the data to be synchronized corresponding to the expiration time must be able to be synchronized to the slave database. In order to avoid the situation where some data to be synchronized is deleted before being synchronized to the slave database due to timing confusion, the real-time internal slave database of this application synchronizes data to the slave database according to the timing order between the expiration times when synchronizing data to the slave database, so as to avoid missing data to be synchronized, improve the data accuracy of the slave database, and improve the data consistency between the slave database and the master database.

[0074] Further, refer to Figure 3 , a flow chart of a method for implementing a master-slave database data query provided by an embodiment of the present application, which illustrates a method for implementing a master-slave database data query applied to the master-slave database data synchronization method, may specifically include the following steps:

[0075] Step S210, in response to a query request for target data in the main database, query whether the target data is stored in the memory database. If yes, execute step S240; if not, execute step S220.

[0076] Step S220, query whether the target data is stored in the slave database, if so, execute step S230; if not, determine that the query result of the query request is a query failure.

[0077] It is understandable that if the slave database fails to copy the to-be-synchronized data corresponding to the target data, it proves that the expiration time of the to-be-synchronized data has not yet arrived, and the data corresponding to the target data that is consistent with the master database still remains in the memory database. If the target data is queried in the slave database first, it is likely that the old data of the target data in the slave database will be used as the query result, and the query result will be inconsistent with the master database.

[0078] Based on the above considerations, the embodiment of the present application gives priority to querying the memory database, and gives priority to obtaining the latest data corresponding to the target data from the memory database. If the target data does not exist in the memory database, it proves that the data to be synchronized corresponding to the target data has been copied, and the data consistent with the master database can be obtained from the slave database. Alternatively, if it is unclear whether the master database has updated the target data, the memory database is queried first, which can exclude the situation where the target data has been updated. Even if the old data corresponding to the target data is found in the slave database, it is the old data consistent with the master database, which improves the accuracy and reliability of the data query results.

[0079] Step S230, reading the synchronization data of the master database corresponding to the target data stored in the slave database as the query result of the query request.

[0080] Step S240, reading the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request,

[0081] It is understandable that no matter the target data is queried in the memory database or the slave database, the synchronized data corresponding to the target data that has been queried and the master database are returned to the front end as the query result of the query request. If the data corresponding to the target data is not queried in the memory database or the slave database, it proves that the data does not exist in the master database either, and the query result corresponding to the query request is a query failure.

[0082] In summary, the data synchronization method of the master-slave database provided in the embodiment of the present application utilizes the fast reading and writing characteristics of the memory database to quickly write the updated data in the master database, that is, the data to be synchronized to the slave database, into the memory database. Before the data to be synchronized is updated to the slave database, the memory database is used as a transfer station for the data to be synchronized, and the memory database synchronizes the data to the slave database. Based on this data synchronization method, while the slave database copies the updated data of the master database, the memory database also stores the updated data that the slave database failed to copy in time, so that during the query process, the updated data that failed to be synchronized in time can be queried, providing a query channel to ensure the consistency of the data query results.

[0083] Based on this, when performing data queries, since the slave database may not have completely copied all the data to be synchronized, if the slave database is read directly, there is still a problem that the reading result is inconsistent with the main database. Therefore, this application gives priority to reading the data that has not been synchronized in time in the memory database to avoid missing data. If the target data is not found in the memory database, the slave database is queried again. At this time, the data in the slave database are all updated data, so there is no problem of the query result being consistent with the main database.

[0084] Next, according to the following embodiments, other possible implementations of the above-mentioned master-slave database data synchronization method and master-slave database data query method are described.

[0085] In a possible implementation, the memory database includes: a first memory block and a second memory block, the first memory block and the second memory block are respectively used to store data to be synchronized; at least one piece of data to be synchronized corresponding to the update operation is written into the memory database in sequence, including: setting the first memory block as the main memory block; writing the data to be synchronized into the main memory block in sequence; judging whether the storage capacity of the first memory block is less than the preset storage capacity; if the storage capacity of the first memory block is not less than the preset storage capacity, migrating the data to be synchronized in the second memory block to the Bloom filter, setting the second memory block as the main memory block, and executing the step of writing the data to be synchronized into the main memory block in sequence.

[0086] Since the memory of an in-memory database cannot grow infinitely, it is necessary to use the memory space reasonably so that update operations can be written to the in-memory database in a timely manner. Figure 1 In the embodiment of the present application, the memory database is divided into a first memory block A and a second memory block B, the total memory of the two memory blocks is not greater than the maximum memory of the memory database, and each memory block is respectively set with a maximum storage capacity.

[0087] In an embodiment of the present application, first, a main memory block currently used to write data is determined from two memory blocks. For example, the main memory block is determined to be the first memory block A, and the first memory block A is used to receive update operations written by the main database. At the same time, when the current time is the expiration time corresponding to any piece of data to be synchronized in the main memory block, the data to be synchronized corresponding to the current time in the main memory block is cleared.

[0088] It is understandable that if the interval between expiration times is small, the memory database may frequently execute data clearing operations, resulting in frequent downtime of the memory database operation. Therefore, the data in the memory block may be cleared at intervals of a certain period of time. For example, the data to be synchronized in the memory database may be cleared at intervals of 4 master-slave synchronization delays.

[0089] If the storage capacity of the first memory block A is scanned and it is found that its current storage capacity is less than its corresponding maximum storage capacity, it proves that the current first memory block A still has room for write update operations. If the current storage capacity is not less than its corresponding maximum storage capacity, it proves that the first memory block A has no room for write update operations, and continued writing may cause memory failure. Therefore, it is necessary to switch the second memory block B as the main memory block to receive writes from the main database. At the same time, in order to avoid the second memory block B reaching its maximum storage capacity when writing data, when switching back to the first memory block A, the first memory block A has not yet freed up storage space, resulting in no storage space for the memory database, making it impossible to synchronize the data to be synchronized and unable to be queried.

[0090] Therefore, the embodiment of the present application switches the second memory block B as the main memory block, compresses the data to be synchronized in the first memory block A and transfers it to the Bloom filter, occupying less memory space of the memory database, freeing up the memory in the first memory block A, and providing storage space for the subsequently written data to be synchronized, so that the write operation is not interrupted.

[0091] Specifically, the process of migrating the data to be synchronized in the second memory block to the Bloom filter may include: migrating the data to be synchronized in the first memory block to the Bloom filter, obtaining a Bloom filter list corresponding to each piece of data to be synchronized, each Bloom filter list corresponding to a list time, and the list time is the time when the data to be synchronized is migrated to the Bloom filter; determining the time difference between the list time corresponding to each Bloom filter list and the current time; and deleting the Bloom filter list whose time difference is greater than the master-slave synchronization delay.

[0092] The data to be synchronized in the first memory block is transferred to a new Bloom filter. Since the migration or insertion time of each data to be synchronized in the Bloom filter is different, the latest insertion time of all data is used as the insertion time of the Bloom filter. For example, there are 5 data in the first memory database. The 5 data are inserted into the Bloom filter in sequence to generate a corresponding Bloom filter list. The list time is the time when the 5th data is inserted into the Bloom filter in sequence.

[0093] However, since there is a lot of data to be synchronized, and each time the main memory block is switched, a new Bloom filter needs to be re-established, that is, a new Bloom filter list is generated. The data in the Bloom expirer list generated previously may have been synchronized to the slave database, and still remain in the memory database, which will still cause the memory database to have no remaining space to receive the written data to be synchronized, resulting in synchronization interruption.

[0094] Therefore, in the embodiment of the present application, the difference between the list time of the Bloom filter list and the current time is greater than the master-slave synchronization delay time as a deletion condition. The storage time of the last inserted data to be synchronized in the Bloom filter list that meets the deletion condition must be greater than the master-slave synchronization delay time in the memory database. Then the last inserted data to be synchronized can be considered to have been synchronized to the slave database. If the last inserted data to be synchronized has been synchronized, the data inserted before must also be synchronized. Therefore, the data in the Bloom filter list that meets the deletion condition are all synchronized data and can be deleted from the memory database. It can be understood that the data in the Bloom filter list that does not meet the deletion condition are all unsynchronized data.

[0095] Based on this, query whether the target data is stored in the memory database, including: querying whether the target data is stored in the first memory block and the second memory block in turn; if the target data is not stored in the first memory block and the second memory block, query whether the target data is stored in the Bloom filter list.

[0096] When executing step S210, the first memory block and the second memory block in the memory database can be queried in sequence. When the target data is not stored in the first memory block and the second memory block, the Bloom filter list is queried, which contains data that is not synchronized. Based on the above synchronization and query method, the memory database can store as much data to be synchronized as possible to meet the requirement that the query results are consistent with the main database.

[0097] The following describes a data synchronization device for a master-slave database provided in an embodiment of the present application. The data synchronization device for a master-slave database described below and the data synchronization method for a master-slave database described above can refer to each other.

[0098] First, combine Figure 4 , a data synchronization device for a master-slave database applied to a server is introduced, wherein the server includes: a master database, a memory database and at least one slave database. Figure 4 As shown, the data synchronization device of the master-slave database may include:

[0099] The data writing unit 1100 is used to write at least one piece of to-be-synchronized data corresponding to the update operation into the memory database in response to the update operation on the master database, wherein the to-be-synchronized data is data in the master database waiting to be synchronized to the slave database;

[0100] An expiration time determination unit 1200 is used to determine the expiration time of each piece of data to be synchronized in the memory database according to the write time of each piece of data to be synchronized and the master-slave synchronization delay time, wherein the master-slave synchronization delay time is the time it takes to copy a piece of data in the master database to the slave database;

[0101] The data synchronization unit 1300 is used to synchronize the data to be synchronized stored in the memory database to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

[0102] In a possible implementation, the memory database includes: a first memory block and a second memory block, wherein the first memory block and the second memory block are respectively used to store the data to be synchronized; and the data writing unit 1100 includes:

[0103] A main memory setting subunit, used for setting the first memory block as a main memory block;

[0104] A data writing subunit, used for writing the to-be-synchronized data into the main memory block in sequence;

[0105] A storage capacity determination subunit, used to determine whether the storage capacity of the first memory block is less than a preset storage capacity;

[0106] The main memory block switching sub-unit is used to migrate the data to be synchronized in the second memory block to the Bloom filter when the judgment result of the storage capacity judgment sub-unit is no, and set the second memory block as the main memory block, and execute the step of writing the data to be synchronized into the main memory block in sequence.

[0107] In a possible implementation, the main memory block switching subunit includes:

[0108] a data migration subunit, configured to migrate the data to be synchronized in the first memory block to a Bloom filter, and obtain a Bloom filter list corresponding to each piece of the data to be synchronized, each of the Bloom filter lists corresponding to a list time, and the list time is the time when the data to be synchronized is migrated to the Bloom filter;

[0109] A list time difference determination subunit, used to determine the time difference between the list time corresponding to each of the Bloom filter lists and the current time;

[0110] The list deleting subunit is used to delete the Bloom filter list whose time difference is greater than the master-slave synchronization delay time.

[0111] In a possible implementation, the method further includes:

[0112] The memory block data clearing subunit is used to clear the to-be-synchronized data corresponding to the current time in the main memory block when the current time is the expiration time corresponding to any one of the to-be-synchronized data in the main memory block.

[0113] In a possible implementation, the functional unit for determining the master-slave synchronization delay time includes:

[0114] A progress detection unit, used to determine the data synchronization progress between the master database and the slave database corresponding to the current detection cycle according to a preset detection cycle;

[0115] Determine the first master-slave synchronization delay duration corresponding to the current detection cycle according to the data synchronization progress corresponding to the current detection cycle;

[0116] A duration acquisition unit, used to acquire the second master-slave synchronization delay duration of the last detection cycle;

[0117] A threshold judgment unit, used to judge whether the time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold;

[0118] The master-slave synchronization delay duration determining unit is configured to determine the first master-slave synchronization delay duration as the master-slave synchronization delay duration if the judgment result of the threshold judgment unit is yes.

[0119] Combination Figure 5 , a data query device for a master-slave database running on the basis of a data synchronization device for a master-slave database is introduced, such as Figure 5 As shown, the data query device of the master-slave database may include:

[0120] The memory data query unit 2100 is used to query whether the target data is stored in the memory database in response to a query request for the target data in the main database;

[0121] The query result determining first unit 2200 is configured to read the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request when the target data is stored in the memory database;

[0122] The query result determines the second unit 2300, which is used to query whether the target data is stored in the slave database when the target data is not stored in the memory database. If the target data is stored in the slave database, the synchronization data of the master database corresponding to the target data stored in the slave database is read as the query result of the query request.

[0123] In a possible implementation, the memory data query unit 2100 includes:

[0124] A memory block query subunit, used to query in sequence whether the target data is stored in the first memory block and the second memory block;

[0125] The Bloom filter list query subunit is used to query whether the target data is stored in the Bloom filter list when the target data is not stored in the first memory block and the second memory block.

[0126] In a possible implementation, the method further includes:

[0127] The query result determination third unit is used to determine that the query result of the query request is a query failure if the slave database does not store the target data.

[0128] In summary, the embodiment of the present application utilizes the fast read and write characteristics of the memory database to quickly write the updated data in the master database, that is, the data to be synchronized to the slave database, into the memory database. Before the data to be synchronized is updated to the slave database, the memory database is used as a transfer station for the data to be synchronized, and the memory database synchronizes the data to the slave database. Based on this data synchronization method, while the slave database copies the updated data of the master database, the memory database also stores the updated data that the slave database failed to copy in time, so that during the query process, the updated data that failed to be synchronized in time can be queried, providing a query channel to ensure the consistency of the data query results.

[0129] Based on this, when performing data queries, since the slave database may not have completely copied all the data to be synchronized, if the slave database is read directly, there is still a problem that the reading result is inconsistent with the main database. Therefore, this application gives priority to reading the data that has not been synchronized in time in the memory database to avoid missing data. If the target data is not found in the memory database, the slave database is queried again. At this time, the data in the slave database are all updated data, so there is no problem of the query result being consistent with the main database.

[0130] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the data in the database of the present disclosure may involve the type, scope of use, usage scenarios, etc. of personal information, and the user's authorization should be informed in an appropriate manner in accordance with relevant laws and regulations.

[0131] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method for a master-slave database, characterized in that: Applied to a server, the server comprising: a master database, a memory database and at least one slave database; The data synchronization method of the master-slave database comprises: In response to an update operation on the master database, at least one piece of to-be-synchronized data corresponding to the update operation is sequentially written into the memory database, wherein the to-be-synchronized data is data in the master database waiting to be synchronized to the slave database; Determine the expiration time of each piece of data to be synchronized in the memory database according to the write time of each piece of data to be synchronized and the master-slave synchronization delay time, wherein the master-slave synchronization delay time is the time it takes to copy a piece of data in the master database to the slave database; The data to be synchronized stored in the memory database are synchronized to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

2. The data synchronization method of the master-slave database according to claim 1, characterized in that: The memory database comprises: a first memory block and a second memory block, wherein the first memory block and the second memory block are respectively used to store the data to be synchronized; The step of writing the at least one piece of to-be-synchronized data corresponding to the update operation into the memory database in sequence includes: Setting the first memory block as a main memory block; Writing the data to be synchronized into the main memory block in sequence; Determining whether the storage capacity of the first memory block is less than a preset storage capacity; When the storage capacity of the first memory block is not less than the preset storage capacity, the data to be synchronized in the second memory block is migrated to the Bloom filter, and the second memory block is set as the main memory block, and the step of writing the data to be synchronized into the main memory block in sequence is executed.

3. The data synchronization method of the master-slave database according to claim 2, characterized in that: The step of migrating the to-be-synchronized data in the second memory block to a Bloom filter includes: Migrate the data to be synchronized in the first memory block to a Bloom filter, and obtain a Bloom filter list corresponding to each piece of the data to be synchronized, each of the Bloom filter lists corresponding to a list time, and the list time is the time when the data to be synchronized is migrated to the Bloom filter; Determine the time difference between the list time corresponding to each of the Bloom filter lists and the current time; The Bloom filter list whose time difference is greater than the master-slave synchronization delay is deleted.

4. The data synchronization method of the master-slave database according to claim 2, characterized in that: Also includes: When the current time is the expiration time corresponding to any one of the to-be-synchronized data in the main memory block, the to-be-synchronized data corresponding to the current time in the main memory block is cleared.

5. The data synchronization method of the master-slave database according to claim 1, characterized in that: The process of determining the master-slave synchronization delay time includes: Determine the data synchronization progress between the master database and the slave database corresponding to the current detection cycle according to the preset detection cycle; Determine the first master-slave synchronization delay duration corresponding to the current detection cycle according to the data synchronization progress corresponding to the current detection cycle; Get the second master-slave synchronization delay duration of the last detection cycle; Determine whether a time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold; If the time difference between the first master-slave synchronization delay time and the second master-slave synchronization delay time is greater than a preset threshold, the first master-slave synchronization delay time is determined as the master-slave synchronization delay time.

6. A data query method for a master-slave database, characterized in that: The data synchronization method for the master-slave database according to any one of claims 1 to 5, wherein the data query method for the master-slave database comprises: In response to a query request for target data in the main database, query whether the target data is stored in the memory database; In a case where the target data is stored in the memory database, reading the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request; In the case that the target data is not stored in the memory database, the slave database is queried whether the target data is stored. If the slave database stores the target data, the synchronization data of the master database corresponding to the target data stored in the slave database is read as the query result of the query request.

7. The data query method of the master-slave database according to claim 6, characterized in that: The querying whether the target data is stored in the memory database includes: querying in sequence whether the target data is stored in the first memory block and the second memory block; When the target data is not stored in the first memory block and the second memory block, it is queried whether the target data is stored in the Bloom filter list.

8. The data query method of the master-slave database according to any one of claims 6-7, characterized in that: Also includes: If the slave database does not store the target data, it is determined that the query result of the query request is a query failure.

9. A data synchronization device for a master-slave database, characterized in that: Applied to a server, the server comprising: a master database, a memory database and at least one slave database; The data synchronization device of the master-slave database comprises: a data writing unit, configured to, in response to an update operation on the master database, write at least one piece of to-be-synchronized data corresponding to the update operation into the memory database in sequence, wherein the to-be-synchronized data is data in the master database waiting to be synchronized to the slave database; An expiration time determination unit, used to determine the expiration time of each piece of data to be synchronized in the memory database according to the write time of each piece of data to be synchronized and the master-slave synchronization delay time, wherein the master-slave synchronization delay time is the time it takes to copy a piece of data in the master database to the slave database; A data synchronization unit is used to synchronize the data to be synchronized stored in the memory database to the slave database in sequence before the expiration time corresponding to the data to be synchronized.

10. A data query device for a master-slave database, characterized in that: The data synchronization device for the master-slave database according to claim 9, wherein the data query device for the master-slave database comprises: A memory data query unit, configured to query whether the target data is stored in the memory database in response to a query request for the target data in the main database; A query result determining first unit is used to read the synchronization data of the main database corresponding to the target data in the memory database as the query result of the query request when the target data is stored in the memory database; The query result determines the second unit, which is used to query whether the target data is stored in the slave database when the target data is not stored in the memory database. If the target data is stored in the slave database, the synchronization data of the master database corresponding to the target data stored in the slave database is read as the query result of the query request.

Images