Data synchronization method and apparatus, electronic device, and storage medium

Abstract

The application discloses a data synchronization method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a transaction identifier corresponding to each transaction, and a number corresponding to each sub-statement after each transaction is divided into sub-statements according to an execution order; and synchronizing the sub-statements located on different data nodes to the outside in sequence according to the transaction identifier and the number. The scheme provided by the application can keep the distributed data synchronized to the outside in sequence, and avoid synchronization failure of the distributed data due to the execution order.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, and in particular to a data synchronization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of computer technology and internet finance, the financial industry has experienced explosive growth in data volume and user scale. Traditional single-machine relational databases can no longer meet the needs of storing and processing the massive amounts of data. Therefore, more and more companies are using distributed databases to replace outdated single-machine relational databases. With the popularization of distributed databases, distributed data synchronization technology has also become a hot research topic.

[0003] For distributed databases, synchronization failures or errors due to incorrect execution order are very common during external synchronization. For example, an UPDATE statement is split into DELETE and INSERT statements. To ensure successful data updates, DELETE must be executed first, followed by INSERT. While ensuring the execution order of statements is straightforward in a single-machine database, it becomes exceptionally difficult in a distributed database. For instance, with cross-shard UPDATE statements, since DELETE and INSERT reside on two separate shards, the execution order cannot be guaranteed. It's possible for INSERT to occur before DELETE, leading to failed external synchronization. Summary of the Invention

[0004] To address the technical problem of external synchronization failure caused by data order issues in distributed databases, embodiments of the present invention provide a data synchronization method, apparatus, electronic device, and storage medium.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a data synchronization method, the method comprising:

[0007] Obtain the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order;

[0008] Sub-statements located on different data nodes are synchronized externally in sequence according to the transaction identifier and the number.

[0009] In one embodiment, before obtaining the transaction identifier corresponding to each transaction, the method further includes:

[0010] Obtain the transaction identifier for each transaction from the management node;

[0011] The transaction is divided into sub-statements according to the execution order, and the sub-statements are numbered according to the execution order.

[0012] The transaction identifier, the sub-statement, and the corresponding number of the sub-statement are sent to the data node corresponding to the sub-statement.

[0013] In one embodiment, after sending the transaction identifier, the sub-statement, and the number corresponding to the sub-statement to the data node corresponding to the sub-statement, the method further includes:

[0014] Write the transaction identifier to disk into the binary log;

[0015] The process of obtaining the transaction identifier corresponding to each transaction includes:

[0016] Parse the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction.

[0017] In one embodiment, writing the transaction identifier to disk into the binary log includes:

[0018] Add an internal column to the binary log table.

[0019] The transaction identifier is stored in the internal column.

[0020] In one embodiment, parsing the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction includes:

[0021] Parse the transaction identifier from the binary log;

[0022] Get the list of active transaction identifiers;

[0023] Based on the parsed transaction identifier and the list of active transaction identifiers, inactive transaction identifiers are determined;

[0024] The inactive transaction identifier is identified as the transaction identifier to be synchronized externally.

[0025] In one embodiment, the step of sequentially synchronizing sub-statements located on different data nodes according to the transaction identifier and the number includes:

[0026] For data nodes that perform the same transaction identifier synchronization task, each data node synchronizes the sub-statements corresponding to each number in sequence according to the number.

[0027] After synchronizing the sub-statement corresponding to a number, determine whether other data nodes have also synchronized the sub-statement corresponding to the same number.

[0028] After confirming that other data nodes have also synchronized the sub-statements corresponding to the specified number, continue to synchronize the sub-statements corresponding to the next specified number in sequence.

[0029] In one embodiment, after synchronizing a sub-statement corresponding to a number, the method further includes:

[0030] Use an array to store the number of times each sub-statement corresponding to a number needs to be synchronized;

[0031] When the number of times the statement corresponding to each number in the array needs to be synchronized is zero, it is determined that the sub-statement corresponding to that number has been synchronized by this data node.

[0032] This invention also provides a data synchronization device, the device comprising:

[0033] The acquisition module is used to acquire the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order;

[0034] The synchronization module is used to synchronize sub-statements located on different data nodes in sequence according to the transaction identifier and the number.

[0035] This invention also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; wherein,

[0036] When the processor is used to run a computer program, it performs the steps of any of the methods described above.

[0037] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0038] The data synchronization method, apparatus, electronic device, and storage medium provided in this invention obtain a transaction identifier corresponding to each transaction, and a number corresponding to each sub-statement after the transaction is divided into sub-statements according to the execution order; and synchronizes the sub-statements located on different data nodes externally in sequence according to the transaction identifier and the number. The solution provided by this invention can ensure that distributed data is synchronized externally in sequence, avoiding synchronization failure of distributed data due to execution order issues. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the data synchronization method according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the distributed database system architecture according to an embodiment of the present invention;

[0041] Figure 3This is a schematic diagram of the data synchronization device according to an embodiment of the present invention;

[0042] Figure 4 This is an internal structural diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0044] This invention provides a data synchronization method, such as... Figure 1 As shown, the method includes:

[0045] Step 101: Obtain the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order;

[0046] Step 102: Synchronize the sub-statements located on different data nodes in sequence according to the transaction identifier and the number.

[0047] This embodiment can be applied to distributed databases to synchronize data within the distributed database.

[0048] In this embodiment, within a distributed database, the Global Transaction Manager Node (GTM) and the intermediate component Computer Node (CN) synchronize distributed data externally based on the globally unique transaction ID (GTID) generated by the GTM (i.e., the transaction identifier in this embodiment) and the sub-statement numbers within the transaction. This method can be used to solve the ordering problem that exists when synchronizing distributed data externally.

[0049] In one embodiment, before obtaining the transaction identifier corresponding to each transaction, the method further includes:

[0050] Obtain the transaction identifier for each transaction from the management node;

[0051] The transaction is divided into sub-statements according to the execution order, and the sub-statements are numbered according to the execution order.

[0052] The transaction identifier, the sub-statement, and the corresponding number of the sub-statement are sent to the data node corresponding to the sub-statement.

[0053] In this embodiment, for statements passing through the CN node, a unique transaction identifier is obtained from the GTM node (i.e., the management node), and the sub-statements within the transaction are numbered.

[0054] See Figure 2 , Figure 2 This is a partial structural diagram of the distributed database in this embodiment. It includes CN nodes, DN nodes, and GTM nodes. At the start of a transaction, the CN node requests a GTID from the GTM node, then numbers the different statements within the transaction according to their execution order, and distributes the GTID and the statements carrying the numbering order to the DN nodes of different shards.

[0055] In one embodiment, after sending the transaction identifier, the sub-statement, and the number corresponding to the sub-statement to the data node corresponding to the sub-statement, the method further includes:

[0056] Write the transaction identifier to disk into the binary log;

[0057] The process of obtaining the transaction identifier corresponding to each transaction includes:

[0058] Parse the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction.

[0059] In this embodiment, a serial number is included when the Data Node (DN) is issued and is written to disk in the binary log. When data synchronization is required, the corresponding serial number is parsed from the binary log and then sorted.

[0060] Here, sorting can be divided into two dimensions: transaction order and statement order within a transaction.

[0061] In one embodiment, writing the transaction identifier to disk into the binary log includes:

[0062] Add an internal column to the binary log table.

[0063] The transaction identifier is stored in the internal column.

[0064] In this embodiment, GTID values ​​can be stored by adding an internal column to each table.

[0065] In addition, the execution order number of statements can be distributed to the DN node via comments.

[0066] In one embodiment, parsing the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction includes:

[0067] Parse the transaction identifier from the binary log;

[0068] Get the list of active transaction identifiers;

[0069] Based on the parsed transaction identifier and the list of active transaction identifiers, inactive transaction identifiers are determined;

[0070] The inactive transaction identifier is identified as the transaction identifier to be synchronized externally.

[0071] In this embodiment, when synchronizing data externally, a single thread is used to compare the GTIDs parsed from the binary log with the list of active GTIDs. Here, a GTID requested by a transaction is considered active from the time it is requested until it is released; all active GTIDs form the active GTID list. This thread also identifies currently inactive data. Since the active GTID list is updated every five minutes, the data in this thread is also updated every five minutes. After obtaining the inactive data, the thread, based on the GTIDs, sequentially distributes different transactions to different threads to operate on the DNs on different shards, thus ensuring the correct transaction execution order.

[0072] In one embodiment, the step of sequentially synchronizing sub-statements located on different data nodes according to the transaction identifier and the number includes:

[0073] For data nodes that perform the same transaction identifier synchronization task, each data node synchronizes the sub-statements corresponding to each number in sequence according to the number.

[0074] After synchronizing the sub-statement corresponding to a number, determine whether other data nodes have also synchronized the sub-statement corresponding to the same number.

[0075] After confirming that other data nodes have also synchronized the sub-statements corresponding to the specified number, continue to synchronize the sub-statements corresponding to the next specified number in sequence.

[0076] Here, an array can also be used to store the number of times the sub-statement corresponding to each number needs to be synchronized; when the number of times the statement corresponding to each number needs to be synchronized is zero, it is determined that the sub-statement corresponding to the number has been synchronized by this data node.

[0077] In this embodiment, the thread assigned to execute the task can be responsible only for the DN in its respective shard. For example, if there are three DNs on three shards, then three threads will operate on the DNs on their respective shards. This ensures that the execution order of statements within a single DN is correct, because a single DN is similar to a single-machine database.

[0078] Each thread can have an array to count the number of times each statement with a specific number needs to be executed within its respective slice. For example, if two `delete` statements and one `insert` statement need to be executed on a slice, then `delete` is numbered 0 and `insert` is numbered 1. Therefore, the value of the array in this thread should be [2, 1]. If there are three slices, DN1, DN2, and DN3, the value of the array in the thread on slice DN2 is [1, 2] (here, the array index represents the statement number, and the number represents the number of statements to be executed for that number). This array indicates that there is one statement with number 0 to be executed and two statements with number 1 to be executed on this slice (0 indicates the highest priority and is executed first). At this moment, the thread can execute statement number 0 because there are no statements with a higher priority than 0. After each statement is executed, the number of times that statement is to be executed in the array is decremented by 1, so after executing the statement, the value of the array will be updated to [0, 2].

[0079] At this moment, the thread cannot continue executing statement number 1 because it needs to confirm with all other threads whether they have completed executing all statements numbered 0. The confirmation method is as follows: the thread responsible for operating DN2 retrieves the values ​​of the statistical arrays within the threads operating DN1 and DN3, and checks if the value at position 0 of the array is 0. If all values ​​are 0, the thread responsible for operating DN2 can begin executing statement number 1. If any value is not 0, DN2 will notify this thread to issue a notification when the value at position 0 of the array becomes 0, and DN2 will then enter a waiting state. This method ensures that statements on different slices can be executed in the prescribed order.

[0080] The distributed data outward synchronization processing technology provided in this embodiment can guarantee the order of statement execution during distributed data outward synchronization by providing a unique identifier for each transaction and using a specially designed sorting algorithm.

[0081] Using the method described in this embodiment, external data synchronization will not affect the normal use of the database or the online operation of business. Furthermore, this method ensures that data synchronization errors will not occur due to incorrect statement execution order during the synchronization process.

[0082] The data synchronization method provided in this invention obtains the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after the transaction is divided into sub-statements according to the execution order; and synchronizes the sub-statements located on different data nodes externally in sequence according to the transaction identifier and the number. The solution provided by this invention can ensure that distributed data is synchronized externally in sequence, avoiding synchronization failure of distributed data due to execution order issues.

[0083] To implement the method of the embodiments of the present invention, the embodiments of the present invention also provide a data synchronization device, such as... Figure 3 As shown, the data synchronization device 300 includes: an acquisition module 301 and a synchronization module 302; wherein,

[0084] The acquisition module 301 is used to acquire the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order;

[0085] The synchronization module 302 is used to synchronize sub-statements located on different data nodes in sequence according to the transaction identifier and the number.

[0086] In practical applications, the acquisition module 301 and the synchronization module 302 can be implemented by the processor in the data synchronization device.

[0087] It should be noted that the above-described apparatus, when executed, is only illustrated by the division of the program modules described above. In actual applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the terminal can be divided into different program modules to complete all or part of the processing described above. Furthermore, the apparatus and method embodiments described above belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0088] To implement the method of the embodiments of the present invention, the present invention also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the above-described method.

[0089] Based on the hardware implementation of the above-described program modules, and in order to implement the method of this embodiment of the invention, this embodiment also provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements the method of any of the above embodiments. The display screen A04 can be a liquid crystal display or an electronic ink display. The input device A05 can be a touch layer covering the display screen, a button, trackball, or touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0090] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0091] The device provided in the embodiments of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the method of any of the above embodiments.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It is understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or both. Specifically, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0100] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0101] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data synchronization method, characterized in that, The method includes: Obtain the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order; Sub-statements located on different data nodes are synchronized externally in sequence according to the transaction identifier and the number; The method further includes: obtaining the transaction identifier of each transaction from the management node; dividing the transaction into sub-statements according to the execution order, and numbering the sub-statements according to the execution order; and sending the transaction identifier, the sub-statement, and the number corresponding to the sub-statement to the data node corresponding to the sub-statement. Wherein, after the sub-statement and its corresponding number are sent to the data node corresponding to the sub-statement, the method further includes: writing the transaction identifier to disk into the binary log; obtaining the transaction identifier corresponding to each transaction includes: parsing the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction; The step of parsing transaction identifiers from binary logs to obtain the transaction identifier corresponding to each transaction includes: parsing transaction identifiers from binary logs; obtaining a list of active transaction identifiers; determining inactive transaction identifiers based on the parsed transaction identifiers and the list of active transaction identifiers; and identifying the inactive transaction identifiers as transaction identifiers to be synchronized externally.

2. The method according to claim 1, characterized in that, The step of writing the transaction identifier to disk in the binary log includes: Add an internal column to the binary log table. The transaction identifier is stored in the internal column.

3. The method according to claim 1, characterized in that, The step of synchronizing sub-statements located on different data nodes sequentially according to the transaction identifier and the number includes: For data nodes that perform the same transaction identifier synchronization task, each data node synchronizes the sub-statements corresponding to each number in sequence according to the number. After synchronizing the sub-statement corresponding to a number, determine whether other data nodes have also synchronized the sub-statement corresponding to the same number. After confirming that other data nodes have also synchronized the sub-statements corresponding to the specified number, continue to synchronize the sub-statements corresponding to the next specified number in sequence.

4. The method according to claim 1, characterized in that, After synchronizing a sub-statement corresponding to a number, the method further includes: Use an array to store the number of times each sub-statement corresponding to a number needs to be synchronized; When the number of times the statement corresponding to each number in the array needs to be synchronized is zero, it is determined that the sub-statement corresponding to that number has been synchronized by this data node.

5. A data synchronization device, characterized in that, The device includes: The acquisition module is used to acquire the transaction identifier corresponding to each transaction, and the number corresponding to each sub-statement after each transaction is divided into sub-statements according to the execution order; The synchronization module is used to synchronize sub-statements located on different data nodes in sequence according to the transaction identifier and the number. The device is further configured to: obtain the transaction identifier of each transaction from the management node; divide the transaction into sub-statements according to the execution order, and number the sub-statements according to the execution order; and send the transaction identifier, the sub-statement, and the number corresponding to the sub-statement to the data node corresponding to the sub-statement. The process of sending the sub-statement and its corresponding number to the data node corresponding to the sub-statement further includes: writing the transaction identifier to disk into the binary log; obtaining the transaction identifier corresponding to each transaction includes: parsing the transaction identifier from the binary log to obtain the transaction identifier corresponding to each transaction. The step of parsing transaction identifiers from binary logs to obtain the transaction identifier corresponding to each transaction includes: parsing transaction identifiers from binary logs; obtaining a list of active transaction identifiers; determining inactive transaction identifiers based on the parsed transaction identifiers and the list of active transaction identifiers; and identifying the inactive transaction identifiers as transaction identifiers to be synchronized externally.

6. An electronic device, characterized in that, include: A processor and memory for storing computer programs that can run on the processor; wherein, When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 4.

7. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images