Data synchronization method, device, equipment, medium and product

Abstract

The embodiments of this specification provide a data synchronization method and device. The data synchronization method includes: obtaining a set of pre-write log data recorded for data processing on a database master node; performing a read operation on each pre-write log data in the pre-write log data set based on multiple parallel read threads; performing a parsing operation on each pre-write log data based on multiple parallel parsing threads; and applying each pre-write log data to a database slave node based on multiple parallel application threads according to the generation time of each pre-write log data, to obtain a database slave node after data synchronization. In the case of reading, parsing and applying operations on different pre-write log data, the reading thread, parsing thread and application thread run in parallel. This solution can improve the efficiency of data synchronization processing based on parallelization and pipeline processing between reading operations, parsing operations and application operations.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method. The present invention also relates to a data synchronization device, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] The data synchronization between the master and slave nodes of the database is realized based on the pre-write log data as the carrier. We call this synchronization mechanism a physical replication mechanism. In the existing physical replication mechanism, the data reading operation, data parsing operation, and data application operation involved in data synchronization processing are processed in a serial manner based on the same thread. In the serial processing process, only one of the data reading operation, data parsing operation, and data application operation can be performed at the same time, and the other two operations can only wait. Therefore, the existing serialized physical replication mechanism cannot perform data reading operations, data parsing operations, and data application operations at the same time.

[0003] Based on this, how to provide a physical replication mechanism with excellent performance to improve the efficiency of data synchronization has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data synchronization method. One or more embodiments of this specification also relate to a data synchronization device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data synchronization method is provided, including:

[0006] Obtain a set of write-ahead log data recorded for data processing on the database master node;

[0007] Performing a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data;

[0008] Performing a parsing operation on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data;

[0009] According to the generation time of each write-ahead log data in the write-ahead log data set, application operations are performed on each of the parsed write-ahead log data based on multiple application threads, and each of the parsed write-ahead log data is synchronized to a database slave node, wherein the multiple reading threads, the multiple parsing threads, and the multiple application threads are all threads running in parallel, and when reading, parsing, and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread, and the application thread run in parallel.

[0010] According to a second aspect of an embodiment of this specification, a data synchronization device is provided, including:

[0011] An acquisition module is configured to acquire a set of pre-write log data recorded for data processing performed on a database master node;

[0012] A reading module is configured to perform a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data;

[0013] A parsing module is configured to perform a parsing operation on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data;

[0014] The application module is configured to perform application operations on each of the parsed write-ahead log data based on multiple application threads according to the generation time of each write-ahead log data in the write-ahead log data set, and synchronize each of the parsed write-ahead log data to the database slave node, wherein the multiple reading threads, the multiple parsing threads and the multiple application threads are all threads running in parallel, and when reading, parsing and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread and the application thread run in parallel.

[0015] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0016] Memory and processor;

[0017] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned data synchronization method are implemented.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0020] An embodiment of the present specification realizes that, in the process of realizing data synchronization between master and slave nodes of a database based on pre-write log data, each pre-write log data is synchronized to the database slave node according to the generation time of the pre-write log data, which can ensure the timeliness of data synchronization for the database slave node, and the data reading operation, data parsing operation and data application operation involved in the data synchronization process are based on different threads to complete the operation, so that the data reading operation, data parsing operation and data application operation can be performed at the same time. Therefore, on the basis of ensuring the timeliness of data synchronization, the data reading operation, data parsing operation and data application operation can be parallelized and pipelined, thereby improving the efficiency of data synchronization for the database slave node. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a data synchronization method provided by an embodiment of this specification;

[0022] Figure 2 It is a flowchart of allocating each buffer area to each parsing thread provided by an embodiment of this specification;

[0023] Figure 3 It is a flowchart of a parsing operation for a parsing thread queue provided by an embodiment of this specification;

[0024] Figure 4 This is a flow chart of data synchronization for a database slave node based on each RedoLog provided in an embodiment of this specification;

[0025] Figure 5 It is a structural diagram of a data synchronization device provided by an embodiment of this specification;

[0026] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0027] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0028] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] First, the terms involved in one or more embodiments of this specification are explained.

[0032] Write-ahead log data: RedoLog, is used to record the modification of the physical pages of the database. It stores the specific modification operations on the data pages. It is usually generated before the transaction is committed and is mainly used for rapid recovery after abnormal database failures.

[0033] Physical replication: The specific physical operations that the database system needs to perform to restore the database state. These operations are for physical storage and are usually related to the actual storage location and structure of the data.

[0034] Apply: Apply, the operation of modifying the data page according to RedoLog.

[0035] Data Definition Language: (DDL) is used to create, modify, or delete database objects.

[0036] Cache area: Redo Buf, mainly refers to the buffer used to store pre-write log data.

[0037] In this specification, a data synchronization method is provided. This specification also relates to a data synchronization device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0038] See also Figure 1 , Figure 1 A flow chart of a data synchronization method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0039] Step 102: Obtain a set of write-ahead log data recorded during data processing on a database master node.

[0040] In the embodiments of this specification, when the user performs data processing on the database master node, the pre-write log data (RedoLog) will be recorded in the pre-write log file accordingly, wherein the database can be a relational database (MySQL). The format in which the RedoLog is recorded in the pre-write log file can be a binary data format, and each RedoLog in the pre-write log file can be recorded in sequence from early to late according to the generation time of the RedoLog. Among them, data processing can include any one of data addition processing, data deletion processing and data update processing for the database master node, or data processing can also include updating operations on the database structure for the database master node, such as: deleting operations on data tables, creating operations on data tables, or adding and subtracting columns, creating new indexes, etc. for data tables in the database.

[0041] In an embodiment of the present specification, a write-ahead log data set may be data obtained from a shared storage of a database master node. The method of obtaining a write-ahead log data set may include: periodically obtaining a write-ahead log data set from a shared storage, wherein the write-ahead log data set may include all write-ahead log data or part of the write-ahead log data in the shared storage. Specifically, if after the synchronization of the database slave node data based on the write-ahead log data in the shared storage is completed, the shared database automatically deletes the stored write-ahead log data, then the obtained write-ahead log data set may be all the write-ahead log data stored in the shared storage; if after the synchronization of the database slave node data based on the write-ahead log data in the shared storage is completed, the shared database does not automatically delete the stored write-ahead log data, then the obtained write-ahead log data set may be the write-ahead log data for which the database slave node data synchronization has not been performed.

[0042] Step 104: performing a read operation on each write-ahead log data in the write-ahead log data set based on multiple read threads to obtain each read write-ahead log data.

[0043] In the embodiment of this specification, the reading thread may be a thread in a database slave node. The method of configuring the number of reading threads may include: dynamically adjusting the number of configured reading threads according to the data volume of the write-ahead log data set.

[0044] In the embodiment of this specification, the earlier the generation time of RedoLog in the pre-write log file, the earlier the storage position of RedoLog can be. Multiple reading threads can be distributed in a queue, and each reading thread in the queue can read RedoLog from the pre-write log file in parallel in a front-to-back order.

[0045] Optionally, the method for a reading thread to read the pre-written log data may include: any reading thread may read the RedoLog at a preset data size (batch size) each time, and the read data may be a binary data stream. The order of reading the RedoLog may be read from front to back according to the storage order of the RedoLog.

[0046] In the embodiments of this specification, any RedoLog is provided with a log sequence number (LogSequenceNumber, LSN), and LSN can be used to indicate the storage location information of the RedoLog in the pre-written log file. The earlier the generation time of the RedoLog is, the smaller the LSN of the RedoLog can be. Therefore, the generation time sequence of each RedoLog can be indicated according to the LSN sequence of each RedoLog.

[0047] Optionally, each reading thread may read the RedoLog into the memory of the database slave node. The method in which each reading thread caches the RedoLog into the memory of the database slave node may include: for an idle first buffer area (Redo Buf) in the memory, each reading thread caches the read RedoLog in parallel into the first Redo Buf, and when the first Redo Buf is filled, each reading thread caches the read RedoLog in parallel into an idle second Redo Buf.

[0048] After caching the RedoLog in the first Redo Buf and the second Redo Buf, the first Redo Buf and the second Redo Buf can be arranged in a queue in ascending order according to the LSN of the RedoLog, wherein the LSN of the RedoLog in the Redo Buf located in the front of the queue is smaller than the LSN of the RedoLog in the Redo Buf located in the back of the queue. For any RedoBuf, the RedoLogs cached in the Redo Buf can be arranged in ascending order according to the LSN of the RedoLog.

[0049] Step 106: performing a parsing operation on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data.

[0050] In the embodiment of the specification, the parsing thread may be a thread in a database slave node. The manner of configuring the number of parsing threads may include: dynamically adjusting the number of configured parsing threads according to the amount of write-ahead log data to be parsed.

[0051] In the embodiment of the present specification, after the parsing thread performs a parsing operation on RedoLog, the type of RedoLog can be parsed. For RedoLog, the parsed types may include: any one of the Parallel type, Before type and Serial type. Among them, the Parallel type can represent the type of physical modification involving the data page (page), such as the RedoLog recorded for the physical modification of data addition, deletion and update of the data page. The Before type can represent the type that needs to be applied (apply) in advance before the physical modification of the Page, such as the RedoLog recorded for logical events such as the start and submission of the transaction, or the global timestamp change of the transaction. The Serial type can represent the type that needs to be forced to synchronize, such as the RedoLog recorded for the update of the database structure.

[0052] Step 108: According to the generation time of each write-ahead log data in the write-ahead log data set, application operations are performed on each of the parsed write-ahead log data based on multiple application threads, and each of the parsed write-ahead log data is synchronized to the database slave node, wherein the multiple reading threads, the multiple parsing threads, and the multiple application threads are all threads running in parallel, and when reading, parsing, and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread, and the application thread run in parallel.

[0053] Optionally, according to the generation time of each pre-write log data in the pre-write log data set, applying the parsed pre-write log data based on multiple application threads may include: according to the LSN of each RedoLog, in the order of LSN from small to large, applying the parsed RedoLog based on the application thread. Among them, the order of LSN of each RedoLog from small to large can reflect the order of generation time of each RedoLog from early to late.

[0054] In the embodiments of the present specification, application threads and data pages in the database may have a corresponding relationship, and each application thread is responsible for data application operations on at least one data page.

[0055] Optionally, the process of applying the operation to each RedoLog may include: for each parsed RedoLog, generating an application address list (Recv_addr list) for each RedoLog according to specific information of the RedoLog, traversing the Recv_addr list, and applying each RedoLog to a corresponding data page. The specific information may include relevant information for indicating the location where the RedoLog is applied.

[0056] Optionally, for any RedoLog, the process of applying the RedoLog to the data page may include: determining the data page corresponding to the RedoLog according to the application address information of the RedoLog; determining the corresponding application thread based on the correspondence between the application thread and the data page, and applying the RedoLog to the corresponding data page based on the determined application thread.

[0057] It should be noted that, for the Recv_addr list generated for each RedoLog, the Recv_addr list can be saved in a hash table at the application thread. With this setting, all relevant RedoLog information for the page can be queried through the hash table.

[0058] In the embodiments of this specification, the multiple reading threads may include reading thread 1, reading thread 2... reading thread n, etc., the multiple parsing threads may include parsing thread 1, parsing thread 2... parsing thread n, etc., and the multiple application threads may include application thread 1, application thread 2... application thread n, etc. The multiple reading threads, the multiple parsing threads, and the multiple application threads are all threads running in parallel, which may include: taking thread 1, reading thread 2... reading thread n as threads running in parallel, parsing thread 1, parsing thread 2... parsing thread n as threads running in parallel, and application thread 1, application thread 2... application thread n as threads running in parallel. In the case of performing reading, parsing and applying operations on different pre-write log data, the reading thread, parsing thread and application thread run in parallel, which may include: if at time t, the reading thread a performs a reading operation on the pre-write log data a, the parsing thread b performs a parsing operation on the pre-write log data b, and the application thread c performs a parsing operation on the pre-write log data c, then the reading thread a, the parsing thread b and the application thread c are threads running in parallel with each other, wherein the reading thread a is any one of the multiple reading threads, the parsing thread b is any one of the multiple parsing threads, and the application thread c is any one of the multiple application threads.

[0059] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification can be interchanged according to actual needs, or some steps can be omitted or deleted.

[0060] Figure 1 In the method, in the process of applying the pre-write log data to the database slave node to achieve data synchronization between the database master node and the database slave node, the read operation, the parsing operation and the application operation can be run in parallel based on different threads, so as to solve the problem of other operations waiting in the process of running any operation, so as to improve the efficiency of data synchronization processing. In the process of applying each RedoLog to the database slave node, each RedoLog is applied to the database slave node in the order from the earliest to the latest according to the generation time of each RedoLog, so as to achieve the improvement of the efficiency of data synchronization processing, and also maintain the timeliness of each RedoLog application process, avoid the risk of data conflict in the process of data synchronization, so as to improve the timeliness of data synchronization for the database slave node.

[0061] based on Figure 1 The method, the embodiments of this specification also provide some specific implementation methods of the method, which are described below.

[0062] In order to improve the rational utilization of resources, the number of read threads can be dynamically configured.

[0063] Optionally, performing a read operation on each write-ahead log data in the write-ahead log data set based on multiple read threads to obtain each write-ahead log data after reading may include: determining the total amount of data of each write-ahead log data contained in the write-ahead log data set; based on the total amount of data, determining the number of threads configured for the multiple read threads; and performing a read operation on each write-ahead log data in the write-ahead log data set based on the number of read threads to obtain each write-ahead log data after reading.

[0064] In the embodiment of the present specification, the total amount of data of each write-ahead log data may include any one of the number of the write-ahead log data and the storage space occupied by each write-ahead log data.

[0065] Optionally, the method of determining the number of threads of the reading thread based on the total amount of data may include: based on a pre-established correspondence table between the total amount of data and the number of threads, after determining the total amount of data, determining the number of threads of the reading thread by querying the correspondence table. Alternatively, a preset threshold may be pre-set, and after determining the total amount of data, determining the number of threads of the reading thread by comparing the total amount of data with the preset threshold.

[0066] In the embodiments of the present specification, the number of threads of the configured reading thread can be dynamically adjusted based on the total amount of data in the write-ahead log data set. In this way, when the total amount of data in the write-ahead log data set is large, the number of threads of the reading thread can be increased to improve the rate of reading the write-ahead log data set. When the total amount of data in the write-ahead log data set is small, the number of threads of the reading thread can be reduced to achieve reasonable use of system resources.

[0067] In the embodiments of this specification, a specific embodiment of determining the number of reading threads based on the total amount of data is also proposed.

[0068] Optionally, determining the number of threads configured for the multiple read threads based on the total amount of data may include: obtaining a unit read data amount used to reflect the reading performance of a read thread; calculating a quotient between the total amount of data and the unit read data amount; and determining the number of threads configured for the multiple read threads based on the quotient.

[0069] In the embodiment of the present specification, the unit read data volume of the reading thread may include any one of the amount of data read by the reading thread at a single time, or the amount of data read by the reading thread in a unit time.

[0070] Optionally, a method for obtaining the unit read data volume of the reading thread may include: obtaining the total amount of read data of a preset number of times based on the historical read data of the reading thread, and calculating the unit read data volume of the reading thread based on the total amount of read data and the preset number of times; or obtaining the total amount of read data within a preset time period, and calculating the unit read data volume of the reading thread based on the total amount of read data and the preset time period.

[0071] Optionally, the method of calculating the quotient between the total amount of data and the unit read data amount may include: calculating the quotient between the total amount of data and the unit read data amount based on the "round-by-1 method"; or calculating the quotient between the total amount of data and the unit read data amount based on the "truncation method".

[0072] In the embodiments of this specification, the quotient between the total amount of data and the unit read data amount can be directly determined as the number of threads of the read thread. Alternatively, the quotient between the total amount of data and the unit read data amount can be used as a parameter to determine the number of threads of the read thread, and the number of threads of the read thread can be reasonably adjusted according to the range of the quotient.

[0073] In the embodiment of the present specification, in the process of determining the number of read threads based on the total amount of data in the write-ahead log data set, the read performance of a single read thread is also considered, thereby improving the accuracy of determining the number of read threads.

[0074] In the embodiments of this specification, a specific embodiment of performing a read operation on each write-ahead log data based on a dynamically determined preset number of read threads is also proposed.

[0075] Optionally, the reading thread based on the number of threads performs a reading operation on each write-ahead log data in the write-ahead log data set to obtain each write-ahead log data after reading, which may include: sorting each write-ahead log data based on the generation time of each write-ahead log data to obtain a target sequence; for each write-ahead log data, determining a corresponding relationship reflecting that each write-ahead log data is read by a target reading thread according to a result of a modulo operation between a sequence number corresponding to the position information of each write-ahead log data in the target sequence and the number of threads; and based on the corresponding relationship, performing a reading operation on each write-ahead log data in the write-ahead log data set to obtain each write-ahead log data after reading.

[0076] Optionally, sorting each write-ahead log data based on the generation time of the write-ahead log data to obtain the target sequence may include: sorting each write-ahead log data in ascending order based on the LSN of the write-ahead log data to obtain the target sequence.

[0077] In an embodiment of the present specification, for any write-ahead log data in a write-ahead log data set, the position information (sequencing number) of the write-ahead log data in the target sequence is determined, a modulo operation is performed on the sequence number and the number of threads of the reading thread, and based on a numerical value corresponding to a calculation result of the modulo operation, a target reading thread at the position of the numerical value is queried in the reading thread queue, a corresponding relationship between the write-ahead log data and the target reading thread is established, and the target reading thread is instructed to perform a read operation on the write-ahead log data to obtain the read write-ahead log data.

[0078] The aforementioned any one of the write-ahead log data may be any one of the write-ahead log data in the write-ahead log data set, and for each of the write-ahead log data in the write-ahead log data set, a reading thread responsible for reading the write-ahead log data may be determined in the aforementioned manner.

[0079] In the embodiment of the present specification, each write-ahead log data in the write-ahead log data set is allocated to each read thread according to the above-mentioned modulo method, so that the read tasks of each read thread can be balanced to improve the read performance of the read thread.

[0080] In the process of each parsing thread parsing each write-ahead log data, in order to improve the parsing performance of the parsing thread, each write-ahead log data may also be allocated to each parsing thread by balancing the parsing task amount.

[0081] Optionally, performing parsing operations on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data may include: allocating each of the read write-ahead log data to each of the parsing threads based on preset rules, wherein the preset rules are used to reflect that the write-ahead log data obtained by each of the parsing threads are complete write-ahead log data; performing parsing operations on each of the read write-ahead log data based on each of the parsing threads to obtain each of the parsed write-ahead log data.

[0082] Optionally, based on preset rules, the method of allocating the read write-ahead log data to each parsing thread may include: allocating each complete write-ahead log data to each parsing thread based on the modulo operation result between the sorting number of each write-ahead log data and the number of threads of the parsing thread; or, based on the quotient between the number of each write-ahead log data and the number of threads of the parsing thread, selecting a corresponding number of continuous complete write-ahead log data to allocate to each parsing thread in turn; if there is any remaining write-ahead log data after allocating a corresponding number of continuous complete write-ahead log data to each parsing thread, the remaining write-ahead log data is randomly allocated to some of the parsing threads.

[0083] In the embodiment of the present specification, after each of the read write-ahead log data is distributed to each parsing thread based on a preset rule, the write-ahead log data acquired by each parsing thread can be complete write-ahead log data.

[0084] In order to facilitate the sequential application of each piece of parsed write-ahead log data, this specification also proposes a specific embodiment for allocating each piece of read write-ahead log data to each parsing thread.

[0085] Optionally, each of the read pre-write log data is cached in a plurality of cache areas distributed in a first queue. In the first queue, if the first cache area is located before the second cache area, the generation time of the pre-write log data included in the first cache area is earlier than the generation time of the pre-write log data included in the second cache area. The allocating the read write-ahead log data to the parsing threads based on the preset rule may include: for any of the cache areas in the first queue, according to the splitting position in any of the cache areas, based on the parsing coordination thread used to coordinate the multiple parsing threads, dividing the data in any of the cache areas into a first cache data block and a second cache data block, wherein the first cache data block includes at least one complete write-ahead log data, and the splitting position is used to split out the complete write-ahead log data; based on the parsing coordination thread, allocating the first cache data block to the first parsing thread in each of the parsing threads; for an adjacent cache area located behind any of the cache areas and adjacent to any of the cache areas, according to the splitting position in the adjacent cache area, based on the parsing coordination thread, dividing the data in the adjacent cache area into a third cache data block and a fourth cache data block, wherein the third cache data block includes at least one complete write-ahead log data; based on the parsing coordination thread, allocating the second cache data block and the third cache data block to the second parsing thread in each of the parsing threads.

[0086] In an embodiment of the present specification, the cache area may be a log cache area (Redo Buf) in the database slave node memory for caching log data. Each cache area is sorted in ascending order according to the LSN of the pre-write log data to obtain a first queue. In the first queue, the LSN of the pre-write log data in the cache area at the front end of the first queue is smaller than the LSN of the pre-write log data in the cache area at the back end of the first queue. This reflects that the generation time of the pre-write log data in the cache area at the front end of the first queue is earlier than the generation time of the pre-write log data in the cache area at the back end of the first queue. For each cache area, each pre-write log data cached in the cache area is distributed in queues in ascending order according to the LSN.

[0087] Figure 2 1 is a flow chart of allocating each buffer area to each parsing thread provided by the embodiment of this specification. Figure 2 The process shown includes steps 202 to 216 .

[0088] Step 202: For the first queue, in the order of LSN from small to large, a first buffer area is selected from the first queue based on the parsing coordination thread.

[0089] In the embodiments of this specification, the parsing coordination thread (Parse_coordinator) may be a core component responsible for coordinating and scheduling parsing tasks, and the parsing thread (Parse_worker) may be a work unit that specifically performs parsing operations. Parse_coordinator can ensure that tasks are performed as planned and coordinate multiple Parse_workers to collaborate efficiently to ensure the correctness and efficiency of data parsing.

[0090] Step 204: Determine whether there is a splitting point in the first buffer area, and obtain a first determination result, wherein the splitting point is used to split out the complete write-ahead log data.

[0091] In the embodiments of the present specification, the splitting site may be an mtr boundary, and the mtr boundary may be expressed as the boundary of a write-ahead log record (redo record). Since a write-ahead log data may be understood as log data for a Page, the mtr boundary may be understood as an atomic change boundary of the content of a Page. Through mtr boundary splitting, not only the integrity of the redo record of a Page may be guaranteed, but also the integrity of the redorecords of all Pages may be guaranteed within the range of an mtr boundary split. The data set split by the mtr boundary may be understood as a complete redo record set for physical modifications to a group of Pages. The data set may include multiple complete redo records for physical modifications to multiple Pages, or may include a complete redo record for physical modifications to a Page.

[0092] In the embodiments of the present specification, the write-ahead log data cached in the buffer area can be a data stream in binary format. Since the storage space of the buffer area is limited, for a binary format write-ahead log data, part of the data may be stored in one buffer area, and the remaining part of the data may be stored in another buffer area. Therefore, incomplete write-ahead log data may be stored in the buffer area. The split point can be the connection point between the complete write-ahead log data at the end and the incomplete write-ahead log data in the binary data chain stored in the buffer area. The split point can be used to separate the incomplete write-ahead log data at the end of the binary data chain from the complete write-ahead log data at the front.

[0093] Step 206: If the first judgment result indicates that the splitting site exists in the first cache area, the data in the first cache area is split into a first cache data block and a second cache data block based on the parsing coordination thread.

[0094] In the embodiment of the present specification, the first cache data block includes at least one complete write-ahead log data, and the second cache data block includes incomplete write-ahead log data.

[0095] Step 208: Allocate the first cache data block to the idle first parsing thread based on the parsing coordination thread, and add the first parsing thread to the parsing thread queue.

[0096] In the embodiment of the present specification, after an idle parsing thread is assigned a parsing task, the parsing thread can be added to a parsing thread queue, and the parsing threads in the parsing thread queue are arranged in order from front to back according to the time sequence of joining.

[0097] Step 210: For the first queue, select the second buffer area from the first queue based on the parsing coordination thread in the order of LSN from small to large.

[0098] Step 212: If the splitting site exists in the second cache area, the data in the second cache area is split into a third cache data block and a fourth cache data block based on the parsing coordination thread.

[0099] In the embodiment of the present specification, the third cache data block includes at least one complete write-ahead log data, and the fourth cache data block includes incomplete write-ahead log data.

[0100] Step 214: Allocate the second cache data block and the third cache data block to an idle second parsing thread based on the parsing coordination thread, and add the second parsing thread to a parsing thread queue.

[0101] In the embodiment of the present specification, after the data in the first buffer area is allocated, the first buffer area may be set as a free buffer area to cache subsequent write-ahead log data.

[0102] Step 216: Loop according to the above steps until the buffer areas in the first queue are all allocated or there are no idle parsing threads.

[0103] It should be noted that if no write-ahead log data is cached in the second cache area, it means that there is no data in the second cache area that needs to be synchronized to the database slave node, the second cache data block is allocated to the idle second parsing thread, and the second parsing thread is added to the parsing thread queue.

[0104] Optionally, based on a preset rule, allocating the read pre-write log data to the parsing threads may also include: if any of the cache areas does not contain the split site, and the adjacent cache area contains the split site, then based on the parsing coordination thread, allocating any of the cache areas and the third cache data block to the third parsing thread among the parsing threads.

[0105] In the embodiments of this specification, Figure 2 If step 206 indicates that the split point does not exist in the first cache area, and step 212 indicates that the split point exists in the second cache area, the write-ahead log data in the first cache area and the data in the third cache data block are allocated to the first parsing thread.

[0106] In the embodiment of this specification, if the split point does not exist in the first cache area, it can be reflected that the data stored in the first cache area is incomplete write-ahead log data, and part of the incomplete write-ahead log data may be stored in the second cache area located after the first cache area. Therefore, the data in the first cache area and the third cache data block split from the second cache area are allocated to the first parsing thread together, so that the data obtained by the first parsing thread can be complete write-ahead log data.

[0107] In the embodiments of the present specification, the binary data streams allocated to each parsing thread are complete pre-write log data, so that each pre-write log data parsed by each parsing thread is complete pre-write log data, so that when the subsequent application thread applies the data parsed by the parsing thread, the complete pre-write log data can be applied to the database slave node, so as to improve the data integrity of data synchronization for the database slave node.

[0108] In the embodiments of this specification, a specific embodiment in which a parsing thread performs a parsing operation on each piece of pre-written log data to be parsed is also proposed.

[0109] Optionally, for any of the parsing threads, each of the write-ahead log data to be parsed that is allocated to any of the parsing threads is distributed in a second queue according to the generation time; the parsing operation on each of the read write-ahead log data based on each of the parsing threads to obtain each parsed write-ahead log data may include: for any of the parsing threads, traversing each of the write-ahead log data to be parsed in the second queue; if a specific type of write-ahead log data exists in each of the write-ahead log data to be parsed, parsing the write-ahead log data to be parsed in the second queue that is located before the specific type of write-ahead log data into a first data set, the specific type of write-ahead log data at least including log data for updating a database structure; parsing the specific type of write-ahead log data into a second data set; parsing the write-ahead log data to be parsed in the second queue that is located after the specific type of write-ahead log data into a third data set.

[0110] Optionally, each of the write-ahead log data to be parsed is distributed in the second queue according to the generation time, which may include: each of the write-ahead log data to be parsed is distributed in the second queue in ascending order according to the LSN of the write-ahead log data. The earlier the generation time of the write-ahead log data, the smaller the LSN of the write-ahead log data.

[0111] In the embodiment of the present specification, each pre-write log data to be parsed assigned to any of the parsing threads may be distributed in the second queue. The process of any of the parsing threads parsing each pre-write log data to be parsed in the second queue may include: for the second queue, traversing each pre-write log data to be parsed in order from front to back; dividing each pre-write log data to be parsed into each data set (LogBatch) according to the type of the pre-write log data to be parsed.

[0112] Optionally, according to the type of the write-ahead log data to be parsed, each log data to be parsed is divided into each data set, which may include: if there is a specific type of write-ahead log data in each write-ahead log data to be parsed, then each write-ahead log data to be parsed in the second queue that is located before the specific type of write-ahead log data is parsed into the first LogBatch, the specific type of write-ahead log data is parsed into the second LogBatch, and each write-ahead log data to be parsed in the second queue that is located after the specific type of write-ahead log data is parsed into the third LogBatch. Wherein, the specific type of write-ahead log data may at least include Data Definition Language (DDL), which may be used to create, modify, and delete database objects, and the database objects may include metadata such as database (Database) and table (Table).

[0113] In the embodiment of this specification, the first LogBatch, the second LogBatch and the third LogBatch can be distributed in queues in ascending order according to the LSN of the pre-write log data. This allows subsequent application threads to apply data according to the queues sorted by the first LogBatch, the second LogBatch and the third LogBatch. This can improve the time sequence of application of each pre-write log data, thereby improving the effectiveness of data synchronization for the database slave node.

[0114] In the embodiments of this specification, a specific embodiment of performing application operations on each pre-written log data to be applied parsed by the parsing thread is also proposed.

[0115] Optionally, each of the parsing threads is distributed in a third queue according to the generation time of the write-ahead log data to be parsed. According to the generation time of each write-ahead log data in the write-ahead log data set, based on multiple application threads, the application operation is performed on each of the parsed write-ahead log data, and each of the parsed write-ahead log data is synchronized to the database slave node, including: according to the generation time, based on the application coordination thread used to coordinate the multiple application threads, traversing each of the parsing threads in the third queue; for any of the traversed parsing threads, applying the write-ahead log data parsed by any of the parsing threads to the database slave node based on the multiple application threads, and obtaining the database slave node after data synchronization.

[0116] In the embodiments of this specification, the application coordination thread (Apply_coordinator) can be responsible for coordinating and scheduling work to ensure that all application threads (Apply_worker) operate in the correct order and dependencies. Apply_coordinator usually manages the coordination tasks in the workflow, transaction and processing process to ensure the consistency of the system. Apply_worker can be responsible for executing specific data application or transaction operations. Each Apply_worker can process the tasks assigned by Apply_coordinator to complete the actual data modification or change operation.

[0117] In the embodiment of the specification, after each parsing thread is assigned a parsing task, the parsing threads can be sorted according to the generation time of the pre-write log data for parsing, in the order of generation time from early to late, to obtain a third queue. Specifically, the parsing threads can be sorted according to the LSN of the pre-write log data, in the order of LSN from small to large, to obtain the third queue.

[0118] Optionally, according to the generation time, traversing each of the parsing threads in the third queue based on the application coordination thread used to coordinate the multiple application threads may include: according to the LSN of the pre-write log data to be applied, traversing each of the parsing threads in the third queue based on the application coordination thread in ascending order of LSN.

[0119] In the embodiment of this specification, for each parsing thread traversed, each pre-write log data parsed by the parsing thread is applied to the database slave node by using an application thread to obtain a database slave node after data synchronization.

[0120] In an embodiment of the present specification, the application thread can apply the various write-ahead log data parsed by the parsing thread in the order of LSN from small to large according to the LSN of the write-ahead log data, thereby improving the timing of the application of each write-ahead log data and improving the effectiveness of data synchronization for the database slave node.

[0121] In the embodiments of this specification, a specific embodiment of performing application operations on each of the pre-written log data to be applied parsed by one of the parsing threads is also proposed.

[0122] Optionally, each data set generated by parsing any of the parsing threads is distributed in a fourth queue according to the generation time of the write-ahead log data to be applied; for any traversed parsing thread, applying the write-ahead log data parsed by any of the parsing threads to the database slave node based on the multiple application threads to obtain the database slave node after data synchronization may include: traversing each data set in the fourth queue based on the application coordination thread according to the generation time; for any traversed data set, applying each write-ahead log data in any of the data sets to the database slave node based on the multiple application threads to obtain the database slave node after data synchronization.

[0123] In the embodiment of the present specification, after the parsing thread performs the parsing operation on each pre-write log data to be parsed, each data set can be generated according to the type of the pre-write log data. According to the generation time of the pre-write log data contained in the data set, each data set can be sorted in the order of generation time from early to late to obtain a fourth queue. Specifically, according to the LSN of the pre-write log data, each data set can be sorted in the order of LSN from small to large to obtain a fourth queue.

[0124] Optionally, traversing each of the data sets in the fourth queue based on the application coordination thread according to the generation time may include: traversing each of the data sets in the fourth queue in ascending order of LSN according to the LSN of the pre-write log data to be applied.

[0125] In the embodiment of the present specification, for each traversed data set, each write-ahead log data in the data set is applied to the database slave node by using an application thread to obtain a database slave node after data synchronization.

[0126] In an embodiment of the present specification, the application thread can apply each write-ahead log data contained in the data set in the order of LSN from small to large according to the LSN of the write-ahead log data, thereby improving the timing of application of each write-ahead log data and improving the effectiveness of data synchronization for the database slave node.

[0127] In the embodiments of this specification, a specific embodiment of performing application operation on each to-be-applied write-ahead log data in one of the data sets generated by the parsing thread is also proposed.

[0128] Optionally, for any of the traversed data sets, applying each write-ahead log data in any of the data sets to a database slave node based on the multiple application threads to obtain a database slave node after data synchronization may include: determining whether any of the data sets contains the specific type of write-ahead log data to obtain a determination result; if the determination result indicates that any of the data sets does not contain the specific type of write-ahead log data, applying each first type of write-ahead log data in any of the data sets to a database slave node based on the application coordination thread, wherein the first type of write-ahead log data includes write-ahead log data recorded by executing logical transactions; applying each second type of write-ahead log data in any of the data sets to a database slave node based on the multiple application threads to obtain a database slave node after data synchronization, wherein the second type of write-ahead log data includes write-ahead log data recorded by performing physical modification operations on data pages; if the determination result indicates that any of the data sets contains a specific type of write-ahead log data, applying the specific type of write-ahead log data to a database slave node based on the application coordination thread to obtain a database slave node after data synchronization.

[0129] Figure 3 1 is a flow chart of performing a parsing operation on a parsing thread queue provided by an embodiment of this specification. Figure 3 The process shown includes steps 302 to 316 .

[0130] Step 302: For the third queue formed by the parsing threads, according to the LSN of the pre-write log data, in order from small to large, a first parsing thread is selected based on the application coordination thread.

[0131] Step 304: For the fourth queue formed by the LogBatchs included in the first parsing thread, the first LogBatch is selected based on the application coordination thread in the order from small to large according to the LSN of the pre-written log data.

[0132] Step 306: Determine whether there is a specific type of write-ahead log data in the first LogBatch.

[0133] Step 308: If the specific type of write-ahead log data does not exist in the first LogBatch, each first type of write-ahead log data in the first LogBatch is applied to the database slave node based on the application coordination thread.

[0134] In the embodiment of this specification, the first type of write-ahead log data may include write-ahead log data recorded by logical events of executing transactions. For example, the logical events of transactions may include the start and commit of transactions, or the change of global timestamp of transactions.

[0135] In the embodiments of the present specification, the first type of write-ahead log data may include logical log content. In order to avoid destroying the logical structure of each logical log, a thread is required to apply each first type of write-ahead log data to the database slave node. In an application batch, when the application coordination thread performs application operations for each first type of write-ahead log data, it must complete the application of each first type of write-ahead log data before switching to subsequent operations. This can reduce the number of switching times of the application coordination thread and improve the efficiency of applying each first type of write-ahead log data.

[0136] Step 310: Apply each second type of write-ahead log data in the first LogBatch to the database slave node based on the multiple application threads to obtain a database slave node after data synchronization.

[0137] In the embodiments of the present specification, the second type of write-ahead log data may include write-ahead log data recorded by physical modification operations on data pages. After the application of the first type of write-ahead log data is completed, the application operation can be performed on the second type of write-ahead log data. In this way, after the logical log data is synchronized with the database slave node so that the operating environment of the database slave node meets the preset logical requirements, the physical log data is synchronized with the standby database, which can improve the smoothness of data synchronization with the database slave node.

[0138] In the embodiment of the present specification, each of the multiple application threads may have a corresponding relationship with at least one data page in the database slave node. According to the information related to the data table and the data page carried by the pre-write log data, and according to the corresponding relationship between the application thread and the data page, the pre-write log data is allocated to the corresponding application thread, and the pre-write log data is applied to the corresponding data page based on the corresponding application thread, and the modification operation of the physical content of the data page is completed, and the database slave node after data synchronization is obtained.

[0139] Optionally, establishing a correspondence between application threads and data pages may include: obtaining data table ID information (space_id) and data page number information (page_no) corresponding to each data page, and calculating a hash value for the target data page based on space_id and page_no, where the hash value may be a sequence number set for the target data page. Calculating a modulo operation result between the hash value and the number of application threads, where each application thread is distributed in a queue, querying the application thread at the sequence number corresponding to the value from the application thread queue according to the value corresponding to the modulo operation result, and establishing a correspondence between the application thread at the sequence number and the target data page.

[0140] The target data page may be any data page in the database slave node, and a corresponding relationship between each data page in the database slave node and the application thread may be established in the above manner.

[0141] In the embodiments of this specification, each application thread and each data page have the above-mentioned corresponding relationship, so that each application thread can perform application operations on a fixed number of data pages, so that the task load of each application thread can be balanced to improve the performance of the application thread in performing application operations. Each application thread can be run in parallel, so that the efficiency of the application thread in performing application operations can be improved.

[0142] Step 312: If there is a specific type of write-ahead log data in the first LogBatch, the specific type of write-ahead log data is applied to the database slave node based on the application coordination thread.

[0143] In the embodiment of this specification, if there is a specific type of write-ahead log data in the first LogBatch, then a piece of write-ahead log data may be stored in the LogBatch. The specific type of write-ahead log data may include the write-ahead log data recorded in the forced synchronization operation on the database, such as DDL.

[0144] In the embodiment of the present specification, when applying the DDL, it is necessary to update the database structure of the database slave node. Since it is necessary to update the structure of the database slave node, all the first write-ahead log data that occurred before the DDL must be applied in advance, and the second write-ahead log data that occurred after the DDL must not be applied. The reason for this setting is that if the application result of the DDL is to delete an old data page from the database slave node, before the DDL application is completed, the first write-ahead log data that occurred before the DDL is applied. In the scenario where the first write-ahead log data is a physical modification of the deleted old data page, a data application conflict event will occur in the data synchronization process. Alternatively, if the application result of the DDL is to create a new data page for the database slave node, before the DDL application is completed, the second write-ahead log data that occurred after the DDL is applied. In the scenario where the second write-ahead log data is a physical modification of the newly created new data page, a data application conflict event will occur in the data synchronization process. Therefore, applying each write-ahead log data in the order of the third queue and the fourth queue can avoid conflict events in the data application, thereby improving the timing of application of each write-ahead log data and improving the effectiveness of data synchronization for the database slave node.

[0145] Step 314: cyclically process each LogBatch in the fourth queue according to the processing method for the first LogBatch.

[0146] Step 316: cyclically process each parsing thread in the third queue in accordance with the processing method for the first parsing thread.

[0147] In the embodiment of the present specification, after each write-ahead log data parsed by the parsing thread is applied, the parsing thread can be set as an idle parsing thread to receive subsequent write-ahead log data to be parsed.

[0148] Figure 4 This is a flow chart of data synchronization for a database slave node based on each RedoLog provided in an embodiment of this specification. Figure 4 The process shown includes steps 402 to 408 .

[0149] Step 402: The reading thread reads each RedoLog into each Redo Buf, and each Redo Buf is arranged in ascending order of the LSN of the RedoLog.

[0150] Step 404: the RedoLogs in each Redo Buf are allocated to each parsing thread in ascending order of LSN. After each parsing thread is allocated a RedoLog to be parsed, a parsing thread queue is formed in ascending order of LSN.

[0151] Step 406: any parsing thread parses each RedoLog to be parsed into each LogBatch according to the type of the RedoLog to be parsed, and each LogBatch forms a LogBatch queue in ascending order of LSN.

[0152] Step 408: The application thread traverses each parsing thread in the parsing thread queue and each LogBatch in the LogBatch queue in the order of LSN from small to large, executes the application operation for each LogBatch, and obtains the database slave node after data synchronization.

[0153] In the embodiments of this specification, since the parsing threads assigned with the parsing task have the above-mentioned parsing thread queue, and each LogBatch generated by each parsing thread has the above-mentioned LogBatch queue. Therefore, in the application stage, through the parsing thread queue and the LogBatch queue, each RedoLog can be applied to the database slave node in the order of LSN from small to large. Therefore, in the parsing stage, when the DDL is parsed, there is no need to wait for the DDL to be applied before performing parsing operations on each RedoLog after the DDL, so as to avoid the risk of data synchronization delay caused by waiting for the application of the DDL in the parsing stage, thereby improving the timeliness of data synchronization for the database slave node.

[0154] In the embodiments of this specification, based on the parallelization and pipelined processing between the reading thread, the parsing thread and the application thread, the rate of data synchronization for the database slave node can be improved, and each RedoLog can be applied to the database slave node in sequence according to the LSN from small to large, so as to improve the timing of data synchronization for the database slave node, thereby avoiding data conflict events during the data synchronization process, and further making the parallelization and pipelined processing between the above-mentioned reading thread, the parsing thread and the application thread an effective synchronization processing process, so as to improve the timeliness of data synchronization for the database slave node.

[0155] Corresponding to the above method embodiment, this specification also provides a data synchronization device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of a data synchronization device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0156] The acquisition module 502 is configured to acquire a set of write-ahead log data recorded during data processing performed on a database master node.

[0157] The reading module 504 is configured to perform a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data.

[0158] The parsing module 506 is configured to perform a parsing operation on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data.

[0159] The application module 508 is configured to perform application operations on each of the parsed write-ahead log data based on multiple application threads according to the generation time of each write-ahead log data in the write-ahead log data set, and synchronize each of the parsed write-ahead log data to the database slave node, wherein the multiple reading threads, the multiple parsing threads and the multiple application threads are all threads running in parallel, and when reading, parsing and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread and the application thread run in parallel.

[0160] Optionally, the reading module 504 may include:

[0161] The first determining unit is configured to determine a total amount of data of each write-ahead log data included in the write-ahead log data set.

[0162] The second determining unit is configured to determine the number of threads configured for the multiple reading threads based on the total amount of data.

[0163] The reading unit is configured to perform a reading operation on each write-ahead log data in the write-ahead log data set based on the reading threads of the number of threads to obtain each write-ahead log data after reading.

[0164] Optionally, the second determining unit may include:

[0165] The acquisition subunit is configured to acquire a unit read data amount for reflecting the read performance of a read thread.

[0166] The calculation subunit is configured to calculate a quotient between the total amount of data and the unit read data amount.

[0167] The first determining subunit is configured to determine the number of threads configured for the multiple reading threads based on the quotient.

[0168] Optionally, the reading unit may include:

[0169] The sorting subunit is configured to sort the respective write-ahead log data based on the generation time of the respective write-ahead log data to obtain a target sequence.

[0170] The second determining subunit is configured to determine, for any of the write-ahead log data, a corresponding relationship reflecting that any of the write-ahead log data is read by a target reading thread according to a result of a modulo operation between a sequence number corresponding to the position information of any of the write-ahead log data in the target sequence and the number of threads.

[0171] The reading subunit is configured to perform a reading operation on any of the write-ahead log data based on the corresponding relationship to obtain the read write-ahead log data.

[0172] Optionally, the parsing module 506 may include:

[0173] The allocation unit is configured to allocate the read write-ahead log data to the parsing threads based on a preset rule, wherein the preset rule is used to reflect that the write-ahead log data acquired by the parsing threads are all complete write-ahead log data.

[0174] The parsing unit is configured to perform a parsing operation on each of the read write-ahead log data based on each of the parsing threads to obtain each of the parsed write-ahead log data.

[0175] Optionally, each of the read pre-write log data is cached in a plurality of cache areas distributed in a first queue. In the first queue, if the first cache area is located before the second cache area, the generation time of the pre-write log data included in the first cache area is earlier than the generation time of the pre-write log data included in the second cache area.

[0176] The allocation unit may include:

[0177] The first segmentation unit is configured to, for any of the cache areas in the first queue, divide the data in any of the cache areas into a first cache data block and a second cache data block according to a segmentation point in any of the cache areas and based on a parsing coordination thread for coordinating the multiple parsing threads, wherein the first cache data block includes at least one complete pre-write log data, and the segmentation point is used to divide out the complete pre-write log data.

[0178] The first allocating subunit is configured to allocate the first cache data block to a first parsing thread among the parsing threads based on the parsing coordination thread.

[0179] The second splitting subunit is configured to, for an adjacent cache area located behind any of the cache areas and adjacent to any of the cache areas, split the data in the adjacent cache area into a third cache data block and a fourth cache data block based on the parsing coordination thread according to the splitting location in the adjacent cache area, wherein the third cache data block includes at least one complete pre-write log data.

[0180] The second allocating subunit is configured to allocate the second cache data block and the third cache data block to a second parsing thread among the parsing threads based on the parsing coordination thread.

[0181] Optionally, the allocation unit may further include:

[0182] The third allocation subunit is configured to allocate any one of the cache areas and the third cache data block to the third parsing thread among the parsing threads based on the parsing coordination thread if the split site is not included in any one of the cache areas and the split site is included in the adjacent cache area.

[0183] Optionally, for any of the parsing threads, each to-be-parsed write-ahead log data allocated to the any of the parsing threads is distributed in a second queue according to the generation time.

[0184] Optionally, the parsing unit may include:

[0185] The first traversal subunit is configured to traverse each of the to-be-parsed write-ahead log data in the second queue for any of the parsing threads.

[0186] The first parsing sub-unit is configured to parse the write-ahead log data to be parsed in the second queue that is located before the specific type of write-ahead log data into a first data set if there is a specific type of write-ahead log data among the various write-ahead log data to be parsed, wherein the specific type of write-ahead log data at least includes log data for updating a database structure.

[0187] The second parsing subunit is configured to parse the specific type of write-ahead log data into a second data set.

[0188] The third parsing subunit is configured to parse the to-be-parsed write-ahead log data located after the specific type of write-ahead log data in the second queue into a third data set.

[0189] Optionally, the parsing threads are distributed in a third queue according to the generation time of the write-ahead log data to be parsed.

[0190] The application module 508 may include:

[0191] a traversal unit configured to traverse each of the parsing threads in the third queue based on an application coordination thread for coordinating the multiple application threads according to the generation time;

[0192] The application unit is configured to apply the pre-written log data parsed by any of the traversed parsing threads to the database slave node based on the multiple application threads to obtain the database slave node after data synchronization.

[0193] Optionally, each data set generated by parsing of any of the parsing threads is distributed in a fourth queue according to the generation time of the write-ahead log data to be applied.

[0194] The application unit may include:

[0195] The second traversal subunit is configured to traverse each of the data sets in the fourth queue based on the application coordination thread according to the generation time.

[0196] The application subunit is configured to apply each pre-write log data in any of the traversed data sets to the database slave node based on the multiple application threads to obtain the database slave node after data synchronization.

[0197] Optionally, the application subunit is specifically configured as follows:

[0198] It is determined whether any of the data sets contains the specific type of write-ahead log data to obtain a determination result.

[0199] If the judgment result indicates that the specific type of write-ahead log data is not included in any of the data sets, then based on the application coordination thread, each first type of write-ahead log data in any of the data sets is applied to the database slave node, and the first type of write-ahead log data includes the write-ahead log data recorded by the logical events of executing the transaction.

[0200] Based on the multiple application threads, each second type of write-ahead log data in any of the data sets is applied to the database slave node to obtain the database slave node after data synchronization, wherein the second type of write-ahead log data includes the write-ahead log data recorded by the physical modification operation on the data page.

[0201] If the judgment result indicates that any of the data sets contains a specific type of write-ahead log data, the specific type of write-ahead log data is applied to the database slave node based on the application coordination thread to obtain the database slave node after data synchronization.

[0202] The data synchronization device provided in this specification can improve the rate of data synchronization for database slave nodes based on the parallelization and pipelined processing between the reading thread, the parsing thread and the application thread, and can also make each RedoLog applied to the database slave node in sequence from small to large LSN, so as to improve the timing of data synchronization for the database slave node, thereby avoiding data conflict events during the data synchronization process, and further making the parallelization and pipelined processing between the above-mentioned reading thread, the parsing thread and the application thread an effective synchronization processing process.

[0203] The above is a schematic scheme of a data synchronization device of this embodiment. It should be noted that the technical scheme of the data synchronization device and the technical scheme of the above data synchronization method belong to the same concept, and the details not described in detail in the technical scheme of the data synchronization device can be referred to the description of the technical scheme of the above data synchronization method.

[0204] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0205] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0206] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0207] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0208] The processor 620 is used to execute the following computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by the processor.

[0209] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the data synchronization method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data synchronization method embodiment.

[0210] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0211] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the data synchronization method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data synchronization method embodiment.

[0212] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0213] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data synchronization method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the above data synchronization method.

[0214] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0215] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0216] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0217] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0218] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data synchronization method, comprising: Obtain a set of write-ahead log data recorded for data processing on the database master node; Performing a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data; Based on multiple parsing threads, a parsing operation is performed on each of the read write-ahead log data to obtain each parsed write-ahead log data; the parsing operation is used to parse the specific type of write-ahead log data used to update the database structure into a second data set, parse the write-ahead log data located before the specific type of write-ahead log data into a first data set, and parse the write-ahead log data located after the specific type of write-ahead log data into a third data set, and each data set generated by the parsing of any of the parsing threads is distributed in a queue according to the generation time of the write-ahead log data; According to the generation time of each write-ahead log data in the write-ahead log data set, application operations are performed on each of the parsed write-ahead log data based on multiple application threads, and each of the parsed write-ahead log data is synchronized to a database slave node, wherein the multiple reading threads, the multiple parsing threads, and the multiple application threads are all threads running in parallel, and when reading, parsing, and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread, and the application thread run in parallel.

2. The method according to claim 1, wherein the step of performing a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data comprises: Determining the total amount of data of each write-ahead log data included in the write-ahead log data set; Based on the total amount of data, determining the number of threads configured for the multiple reading threads; The reading threads based on the number of threads perform a reading operation on each write-ahead log data in the write-ahead log data set to obtain each read write-ahead log data.

3. The method according to claim 2, wherein determining the number of threads configured for the plurality of reading threads based on the total amount of data comprises: Get the unit read data volume used to reflect the read performance of a read thread; Calculating a quotient between the total amount of data and the unit read data amount; Based on the quotient, the number of threads configured for the plurality of reading threads is determined.

4. The method according to claim 2, wherein the reading thread based on the number of threads performs a reading operation on each of the write-ahead log data in the write-ahead log data set to obtain each of the read write-ahead log data, comprising: sorting the respective write-ahead log data based on the generation time of the respective write-ahead log data to obtain a target sequence; For any of the write-ahead log data, determining a corresponding relationship reflecting that any of the write-ahead log data is read by a target reading thread according to a modulo operation result between a sequence number corresponding to the position information of any of the write-ahead log data in the target sequence and the number of threads; Based on the corresponding relationship, a read operation is performed on any of the write-ahead log data to obtain the read write-ahead log data.

5. The method according to claim 1, wherein the parsing operation is performed on each of the read write-ahead log data based on multiple parsing threads to obtain each of the parsed write-ahead log data, comprising: Based on a preset rule, each of the read write-ahead log data is distributed to each of the parsing threads, wherein the preset rule is used to reflect that the write-ahead log data obtained by each of the parsing threads are all complete write-ahead log data; Based on each of the parsing threads, a parsing operation is performed on each of the read write-ahead log data to obtain each of the parsed write-ahead log data.

6. The method of claim 5, wherein each of the read write-ahead log data is cached in a plurality of cache areas distributed in a first queue, and in the first queue, if the first cache area is located before the second cache area, the generation time of the write-ahead log data included in the first cache area is earlier than the generation time of the write-ahead log data included in the second cache area; The method of allocating the read write-ahead log data to the parsing threads based on a preset rule includes: For any of the cache areas in the first queue, according to a split point in any of the cache areas, based on a parsing coordination thread for coordinating the multiple parsing threads, data in any of the cache areas is split into a first cache data block and a second cache data block, the first cache data block includes at least one complete write-ahead log data, and the split point is used to split out the complete write-ahead log data; Based on the parsing coordination thread, allocating the first cache data block to a first parsing thread among the parsing threads; For an adjacent cache area located behind any of the cache areas and adjacent to any of the cache areas, according to the splitting position in the adjacent cache area, based on the parsing coordination thread, data in the adjacent cache area is split into a third cache data block and a fourth cache data block, wherein the third cache data block includes at least one complete pre-write log data; Based on the parsing coordination thread, the second cache data block and the third cache data block are allocated to the second parsing thread among the parsing threads.

7. The method according to claim 6, wherein the step of allocating the read write-ahead log data to the parsing threads based on a preset rule further comprises: If any of the cache areas does not contain the split site, and the adjacent cache area contains the split site, then based on the parsing coordination thread, any of the cache areas and the third cache data block are allocated to the third parsing thread among the parsing threads.

8. The method of claim 5, wherein each of the parsing threads is distributed in a third queue according to the generation time of the write-ahead log data to be parsed; The step of performing application operations on each of the parsed write-ahead log data based on multiple application threads according to the generation time of each of the write-ahead log data in the write-ahead log data set, and synchronizing each of the parsed write-ahead log data to a database slave node includes: According to the generation time, traverse each of the parsing threads in the third queue based on an application coordination thread for coordinating the multiple application threads; For any of the traversed parsing threads, the write-ahead log data parsed by any of the parsing threads is applied to a database slave node based on the multiple application threads to obtain a database slave node after data synchronization.

9. The method according to claim 8, wherein each data set generated by parsing by any of the parsing threads is distributed in a fourth queue according to the generation time of the write-ahead log data to be applied; The method of applying the pre-written log data parsed by any of the parsing threads to the database slave node based on the multiple application threads for any of the parsing threads to obtain the database slave node after data synchronization includes: According to the generation time, traverse each of the data sets in the fourth queue based on the application coordination thread; For any of the traversed data sets, each write-ahead log data in any of the data sets is applied to a database slave node based on the multiple application threads to obtain a database slave node after data synchronization.

10. The method according to claim 9, wherein for any of the traversed data sets, based on the multiple application threads, each write-ahead log data in any of the data sets is applied to the database slave node to obtain the database slave node after data synchronization, comprising: Determine whether any of the data sets contains the specific type of write-ahead log data, and obtain a determination result; If the judgment result indicates that any of the data sets does not contain the specific type of write-ahead log data, then applying each first type of write-ahead log data in any of the data sets to the database slave node based on the application coordination thread, where the first type of write-ahead log data includes the write-ahead log data recorded by the logical event of executing the transaction; Applying each second type of write-ahead log data in any of the data sets to a database slave node based on the multiple application threads to obtain a database slave node after data synchronization, wherein the second type of write-ahead log data includes write-ahead log data recorded by a physical modification operation performed on a data page; If the judgment result indicates that any of the data sets contains a specific type of write-ahead log data, the specific type of write-ahead log data is applied to the database slave node based on the application coordination thread to obtain the database slave node after data synchronization.

11. A data synchronization device, comprising: An acquisition module is configured to acquire a set of pre-write log data recorded for data processing performed on a database master node; A reading module is configured to perform a reading operation on each write-ahead log data in the write-ahead log data set based on multiple reading threads to obtain each read write-ahead log data; A parsing module is configured to perform a parsing operation on each of the read write-ahead log data based on multiple parsing threads to obtain each parsed write-ahead log data; the parsing operation is used to parse the specific type of write-ahead log data used to update the database structure into a second data set, parse the write-ahead log data located before the specific type of write-ahead log data into a first data set, and parse the write-ahead log data located after the specific type of write-ahead log data into a third data set, and each data set generated by the parsing of any of the parsing threads is distributed in a queue according to the generation time of the write-ahead log data; The application module is configured to perform application operations on each of the parsed write-ahead log data based on multiple application threads according to the generation time of each write-ahead log data in the write-ahead log data set, and synchronize each of the parsed write-ahead log data to the database slave node, wherein the multiple reading threads, the multiple parsing threads and the multiple application threads are all threads running in parallel, and when reading, parsing and applying operations are performed on different write-ahead log data, the reading thread, the parsing thread and the application thread run in parallel.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the data synchronization method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 10.

Images