Data synchronization method, apparatus, medium, and storage device

Abstract

The present disclosure provides a data synchronization method, which can be applied to the technical field of big data and the technical field of financial technology. The data synchronization method comprises the following steps: in response to receiving a pre-commit request sent by a storage engine of a columnar database, parsing a transaction identifier from the pre-commit request, wherein the transaction identifier is used to represent a target operation of a second client on first data; synchronizing the target operation on the first data in the first client to obtain second data; sending confirmation information representing successful execution of the target operation to the storage engine, so that the storage engine generates a commit request after receiving the confirmation information; and in response to receiving the commit request sent by the storage engine, storing the second data in the first client. The present disclosure also provides a data synchronization device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the fields of big data technology and financial technology technology, and more specifically to a data synchronization method, apparatus, device, medium and program product. Background Technology

[0002] With the continuous development of big data technology, the demand for data quality and data storage is increasing. Data is typically stored in a database, and operations such as modification and querying are performed by controlling the database engine.

[0003] In related technologies, to meet processing efficiency requirements, database engines can perform concurrent operations on data to improve processing efficiency. However, when a database engine modifies the same data through multiple concurrent operations, data inconsistency issues can arise. Furthermore, due to the limited capabilities of database engines in fault recovery and transaction processing, the availability and stability of the system are affected. Summary of the Invention

[0004] In view of the above problems, this disclosure provides data synchronization methods, apparatus, devices, media and program products.

[0005] According to a first aspect of this disclosure, a data synchronization method is provided, applied to a first client, comprising: in response to receiving a pre-commit request sent by a storage engine of a columnar database, parsing a transaction identifier from the pre-commit request, wherein the transaction identifier is used to characterize a target operation of a second client on first data; synchronizing the target operation on the first data in the first client to obtain second data; sending an acknowledgment message indicating successful execution of the target operation to the storage engine, so that the storage engine generates a commit request after receiving the acknowledgment message; and in response to receiving a commit request sent by the storage engine, storing the second data in the first client.

[0006] According to an embodiment of this disclosure, the method of synchronizing a target operation on the first data in the first client to obtain second data includes: obtaining the first data from a first address; performing the target operation on the first data to obtain second data; and storing the second data in a second address, wherein the second address is different from the first address where the first data is stored.

[0007] According to embodiments of this disclosure, the above-mentioned response to receiving a submission request sent by the storage engine, storing the second data in the first client, includes: obtaining the second data from the second address and storing the second data in the first address.

[0008] According to embodiments of this disclosure, the target operation includes adding or deleting; the method further includes: determining first state information of the first data before adding or deleting target data in the first data; determining second state information of the second data after adding or deleting target data in the first data; determining an integrity result for the target data based on the first state information and the second state data; and generating confirmation information when the integrity result indicates that the target data is completely stored in the first client or the target data is completely deleted from the first client.

[0009] According to a second aspect of this disclosure, a data synchronization method is provided, applied to the storage engine of a columnar storage database, comprising: in response to receiving a transaction request from a second client, generating a transaction identifier corresponding to the transaction request, wherein the transaction request is used to synchronize a target operation on first data on M first clients other than the second client, where M is a positive integer; generating a pre-commit request based on the transaction identifier of the transaction request; sending the pre-commit request to each of the first clients so that each of the first clients synchronizes the target operation on the first data to obtain second data; and in response to receiving confirmation information from the M first clients, sending a commit request to each of the first clients so that the second data is stored on each of the first clients, wherein the confirmation information is used to indicate that the target operation was successfully executed.

[0010] According to embodiments of this disclosure, the method further includes: in response to receiving fault information sent by the m-th first client, sending the submission request to each of the first clients other than the m-th first client, so as to store the second data in each of the first clients other than the m-th first client, where m is a positive integer less than M; in response to receiving fault recovery information sent by the m-th first client, determining the fault duration of the m-th first client; if it is determined that the fault duration does not exceed a predetermined duration, obtaining N transaction identifiers added during the fault of the m-th first client, where N is a positive integer; and synchronizing the operations corresponding to the N newly added transaction identifiers in the m-th first client.

[0011] According to embodiments of this disclosure, the method further includes: in response to detecting that the reception time of the confirmation information by the m-th first client exceeds a predetermined time, or receiving execution failure information sent by the m-th first client, sending an operation cancellation request to each of the first clients other than the m-th first client, so that each of the first clients other than the m-th first client cancels the target operation on the first data; and in response to receiving operation cancellation success information from (M-1) of the first clients other than the m-th first client, returning synchronization failure information to the second client.

[0012] A third aspect of this disclosure provides a data synchronization apparatus applied to a first client, comprising: a parsing module, configured to, in response to receiving a pre-commit request sent by a storage engine of a columnar database, parse a transaction identifier from the pre-commit request, wherein the transaction identifier is used to characterize a target operation of a second client on first data; a synchronization module, configured to synchronize the target operation on the first data in the first client to obtain second data; an acknowledgment module, configured to send acknowledgment information to the storage engine characterizing the successful execution of the target operation, so that the storage engine generates a commit request after receiving the acknowledgment information; and a storage module, configured to, in response to receiving a commit request sent by the storage engine, store the second data in the first client.

[0013] A fourth aspect of this disclosure provides a data synchronization apparatus applied to a storage engine of a columnar storage database, comprising: a first generation module, configured to generate a transaction identifier corresponding to a transaction request in response to receiving a transaction request from a second client, wherein the transaction request is used to synchronize a target operation on first data on M first clients other than the second client, where M is a positive integer; a second generation module, configured to generate a pre-commit request based on the transaction identifier of the transaction request; a first request sending module, configured to send the pre-commit request to each of the first clients so that each of the first clients synchronizes the target operation on the first data to obtain second data; and a second request sending module, configured to send a commit request to each of the first clients in response to receiving confirmation information from the M first clients so that the second data is stored on each of the first clients, wherein the confirmation information is used to indicate that the target operation was successfully executed.

[0014] A fifth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data synchronization method described above.

[0015] A sixth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the data synchronization method described above.

[0016] The seventh aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data synchronization method.

[0017] According to embodiments of this disclosure, when the second client performs a target operation on the first data, a pre-commit request is sent to the first client to achieve synchronization of the target operation within the distributed system; the second data is obtained by performing a target operation on the first data, but the second data is not actually committed, to prevent inconsistencies in the first data among multiple first clients due to failure of some first clients in executing the target operation; by sending confirmation information to the storage engine, the storage engine judges the execution of the target operation by multiple first clients, and only sends a commit request when it is determined that multiple first clients have successfully executed the target operation, and the first client stores the second data only upon receiving the commit request, thus ensuring data consistency. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 This illustration schematically depicts an application scenario of the data synchronization method according to embodiments of the present disclosure;

[0020] Figure 2 A flowchart illustrating a data synchronization method applied to a first client according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 3 A flowchart illustrating the generation of confirmation information according to an embodiment of this disclosure is shown schematically;

[0022] Figure 4 A flowchart illustrating a data synchronization method for a storage engine applied to a columnar storage database according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 5 A flowchart illustrating data synchronization controlled by a storage engine according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 6 This schematically illustrates a structural block diagram of a data synchronization device applied to a first client according to an embodiment of the present disclosure;

[0025] Figure 7This schematically illustrates a structural block diagram of a data synchronization apparatus for a storage engine applied to a columnar storage database according to an embodiment of the present disclosure; and

[0026] Figure 8 A block diagram of an electronic device suitable for a data synchronization method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0031] In related technologies, columnar databases such as ClickHouse are widely used in big data analytics scenarios. The ReplicatedMergeTree storage engine for columnar databases is a high-performance, highly available distributed storage engine with automatic data replication and sharding capabilities. However, when using ReplicatedMergeTree to modify the same data through multiple concurrent operations, data inconsistency issues arise, failing to meet the requirements for strong consistency guarantees. Furthermore, ReplicatedMergeTree has limited capabilities in fault recovery and transaction processing, impacting system availability and stability.

[0032] Embodiments of this disclosure provide a data synchronization method applied to a first client, comprising: in response to receiving a pre-commit request sent by a storage engine of a columnar database, parsing a transaction identifier from the pre-commit request, wherein the transaction identifier is used to characterize a target operation of a second client on first data; synchronizing the target operation on the first data in the first client to obtain second data; sending confirmation information indicating successful execution of the target operation to the storage engine, so that the storage engine generates a commit request after receiving the confirmation information; and storing the second data in the first client in response to receiving a commit request sent by the storage engine.

[0033] Figure 1 The illustration depicts an application scenario of the data synchronization method according to an embodiment of the present disclosure.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminals 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the data synchronization method provided in this embodiment can generally be executed by server 105. Correspondingly, the data synchronization device provided in this embodiment can generally be located in server 105. The data synchronization method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data synchronization device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] For example, a user can operate through one of terminals 101, 102, and 103 to initiate a pre-commit request to server 105. In response to the pre-commit request, the server parses the transaction identifier from the pre-commit request; synchronizes the target operation on the first data on other terminal devices besides the user-operated terminal device to obtain the second data; if the target operation is determined to have been successfully executed, it sends an acknowledgment message indicating successful execution of the target operation; and in response to receiving a commit request, it stores the second data on other terminal devices besides the user-operated terminal device.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] The following will be based on Figure 1 The described scene, through Figures 2-5 The data synchronization method of the disclosed embodiments will be described in detail.

[0042] Figure 2 A flowchart illustrating a data synchronization method applied to a first client according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2 As shown, the method includes operations S210 to S240.

[0044] In operation S210, in response to receiving a pre-commit request sent by the storage engine of the columnar database, the transaction identifier is parsed from the pre-commit request.

[0045] According to embodiments of this disclosure, a transaction identifier is used to characterize a target operation of a second client on the first data. The transaction identifier can take various forms, including code, statements, numbers, strings, tables, or any combination thereof. The transaction identifier includes at least the database and table information of the first data, which includes the location of the database and table corresponding to the first data. Data partitioning methods can include field-based conditional partitioning, vertical partitioning, range partitioning, etc., and are not limited thereto.

[0046] According to embodiments of this disclosure, the users of the second client and the first client may be the same or different, but both the first and second clients store the same first data. The first client may include multiple clients, forming a distributed system with the second client. The target operation may be a modification operation on the first data initiated by the second client, typically an operation performed on data stored in the distributed system. The target operation needs to succeed or fail simultaneously in the distributed system to ensure data consistency.

[0047] According to embodiments of this disclosure, the pre-commit request is sent by the columnar database storage engine and may include a target operation on the first data. When a user performs a target operation on the first data through a second client, the second client sends a modification request to the columnar storage engine to synchronize the target operation. In response to receiving the modification request for the first data from the second client, the columnar database storage engine sends a pre-commit request to the first client.

[0048] According to an embodiment of this disclosure, in response to receiving a pre-commit request sent by the storage engine of the columnar database, the first client parses the transaction identifier from the pre-commit request and determines the target operation of the second client on the first data.

[0049] In operation S220, the target operation on the first data is synchronized in the first client to obtain the second data.

[0050] According to embodiments of this disclosure, the second data can be obtained by the first client performing a target operation on the first data. Since the first client and the second client belong to the same distributed system and both contain the first data, when the second client performs a target operation on the first data, in order to ensure data consistency, the first client also needs to synchronize its target operation and perform the target operation on the first data. It should be noted that at this time, it is not necessary to submit the second data; it is only necessary to obtain the second data.

[0051] According to an embodiment of this disclosure, the first client obtains a transaction identifier by parsing the received pre-commit request, and performs a target operation on the first data based on the transaction identifier to obtain the second data.

[0052] In operation S230, an acknowledgment message indicating the successful execution of the target operation is sent to the storage engine, so that the storage engine can generate a commit request after receiving the acknowledgment message.

[0053] According to embodiments of this disclosure, the confirmation information indicates that the first client has successfully executed the target operation on the first data and obtained the second data. Upon determining that the target operation was successfully executed, the first client sends confirmation information to the storage engine.

[0054] According to embodiments of this disclosure, a submission request is sent by the storage engine to the first client, which can be used to instruct the first client to submit the second data. Upon receiving confirmation messages from all first clients in the distributed system, the storage engine determines that all first clients have successfully executed the target operation and obtained the second data, and that the conditions for submitting the second data are met. At this point, it can send a submission request to all first clients to perform data synchronization.

[0055] In operation S240, in response to receiving a commit request sent by the storage engine, the second data is stored on the first client.

[0056] According to embodiments of this disclosure, upon receiving a commit request from the storage engine, the first client determines that it can commit the second data, stores the second data, and achieves data synchronization with the second client and other first clients. Furthermore, during the storage of the second data, a distributed lock can be used for control to prevent other concurrent transactions from affecting the integrity of the second data.

[0057] According to embodiments of this disclosure, when the second client performs a target operation on the first data, a pre-commit request is sent to the first client to achieve synchronization of the target operation within the distributed system; the second data is obtained by performing a target operation on the first data, but the second data is not actually committed, to prevent inconsistencies in the first data among multiple first clients due to failure of some first clients in executing the target operation; by sending confirmation information to the storage engine, the storage engine judges the execution of the target operation by multiple first clients, and only sends a commit request when it is determined that multiple first clients have successfully executed the target operation, and the first client stores the second data only upon receiving the commit request, thus ensuring data consistency.

[0058] According to an embodiment of this disclosure, synchronously performing a target operation on first data in a first client to obtain second data includes: obtaining first data from a first address; performing a target operation on the first data to obtain second data; and storing the second data at a second address, wherein the second address is different from the first address where the first data is stored.

[0059] According to embodiments of this disclosure, the first address can be the location where the first data is stored, the second data can be the location where the second data is stored, and the locations pointed to by the first address and the second address can both be located in the terminal where the first client is located, but the locations pointed to by the first address and the second address are different.

[0060] According to the embodiments of this disclosure, since the process of performing a target operation on the first data to obtain the second data is a pre-submission process, the generated second data cannot be submitted. Instead, the generated second data needs to be stored in a second address that is different from the first address to avoid conflicts.

[0061] According to embodiments of this disclosure, a first client can parse the first address of the first data from the pre-commit request and obtain the first data from the first address using the transaction identifier of the first data. The obtained first data can be written to a second address, and a target operation can be performed on the first data in the second address to obtain the second data, without needing to perform any operation on the first data in the first address.

[0062] According to embodiments of this disclosure, by storing the second data at a second address different from the first address, the impact on the first data at the first address is reduced, thus ensuring data consistency.

[0063] According to embodiments of this disclosure, in response to receiving a commit request sent by the storage engine, storing second data on a first client includes: retrieving the second data from a second address and storing the second data to a first address.

[0064] According to embodiments of this disclosure, when the first client receives a commit request, the second data can be committed. Specifically, the first client obtains the second data from the second address, obtains the first address through the transaction identifier, and writes the second data to the first address.

[0065] According to embodiments of this disclosure, after the second data is written to the first address, the second data in the second address can be cleared, saving storage space.

[0066] According to embodiments of this disclosure, by writing the second data to the first address, the first data is synchronously updated, thus ensuring data consistency.

[0067] Figure 3 A flowchart illustrating the generation of confirmation information according to an embodiment of this disclosure is shown schematically.

[0068] like Figure 3 As shown, generating confirmation information includes operations S310 to S340.

[0069] Before adding or deleting target data in the first data in operation S310, the first status information of the first data is determined.

[0070] According to embodiments of this disclosure, the target data can be the target operation that adds or deletes the first data. For example, the first data can be data stored in a first data table, so when the second client modifies the field value of the third row of the first column, the field value of the third row of the first column is the target data.

[0071] According to embodiments of this disclosure, the first state information is used to characterize the state information of the first data in the first address, including but not limited to the number of rows, columns, total data volume, MD5 (Message-Digest Algorithm 5) value, and field values ​​of the first data.

[0072] According to embodiments of this disclosure, the storage state of the first data before the target data is added or deleted is determined by determining the first state information of the first data at the first address. For example, data fragments of each column in the first data can be obtained first, then the MD5 value of each row in each data fragment can be calculated, and finally the MD5 values ​​of each data fragment in the first data can be sorted to obtain the first state information.

[0073] According to embodiments of this disclosure, the first state information may further include version information of the first data in the first address to support multi-version concurrent control.

[0074] After adding or deleting target data in the first data in operation S320, the second status information of the second data is determined.

[0075] According to embodiments of this disclosure, the second state information is used to characterize the state information after the first client writes the second data to the first address, including but not limited to the number of rows, columns, total data volume, MD5 value, and field values ​​of the second data.

[0076] According to embodiments of this disclosure, the storage state after adding or deleting target data in the first data is determined by determining the second state information of the second data at the first address. For example, data fragments of each column in the second data can be obtained first, then the MD5 value of each row in each data fragment can be calculated, and finally the MD5 values ​​of each data fragment in the second data can be sorted to obtain the second state information.

[0077] According to embodiments of this disclosure, the second state information may further include version information of the first data in the first address to support multi-version concurrent control.

[0078] In operation S330, the integrity result for the target data is determined based on the first state information and the second state information.

[0079] According to embodiments of this disclosure, the integrity result can be determined by comparing the first state information with the second data to identify the completeness of the second data. For example, when the target operation is to add target data, the number of rows of the second data represented by the second state information is greater than the number of rows of the first data represented by the first state information. When the target operation is to delete target data, the number of columns of the second data represented by the second state information is greater than the number of columns of the first data represented by the first state information.

[0080] According to embodiments of this disclosure, it can also be determined that the data in the first data, excluding the target data, has not changed by comparing the MD5 value of the first data with the MD5 value of the second data one by one.

[0081] In operation S340, the integrity result for the target data is determined based on the first state information and the second state information.

[0082] According to an embodiment of this disclosure, when the target operation is to add target data, the integrity result determines that the target data has been stored in the first client, indicating that the second data submission is complete, and a confirmation message is sent to the database engine.

[0083] According to an embodiment of this disclosure, when the target operation is to delete target data, the integrity result determines that the target data has been deleted from the first client, indicating that the second data submission is complete, and a confirmation message is sent to the database engine.

[0084] According to embodiments of this disclosure, after the second data is stored at the first address, the first state information and the second state information determine the integrity result of the target data, thereby ensuring the integrity of the data.

[0085] Figure 4 A flowchart illustrating a data synchronization method for a storage engine applied to a columnar storage database according to an embodiment of the present disclosure is shown.

[0086] like Figure 4 As shown, the data synchronization method for the storage engine applied to columnar storage databases includes operations S410 to S440.

[0087] In operation S410, in response to receiving a transaction request from the second client, a transaction identifier corresponding to the transaction request is generated, wherein the transaction request is used to synchronize the target operation on the first data on M first clients other than the second client, where M is a positive integer.

[0088] According to embodiments of this disclosure, the transaction request can be a request issued by a second client for performing a target operation on the first data. After the second client performs the target operation on the first data, in order to ensure operation synchronization, a transaction request needs to be issued to the storage engine of the columnar storage database, so that the storage engine of the columnar storage database controls the first client to perform the target operation on the first data.

[0089] According to embodiments of this disclosure, the transaction identifier is generated by the storage engine of the columnar storage database and includes at least the database and table information of the first data, including the location of the database and table corresponding to the first data. The transaction identifier may also include version information to support multi-version concurrency control. In addition, the transaction identifier may also include time information (such as month information, date information, etc.) or status information (such as whether null rollback is allowed, partial state, etc.) of the first data.

[0090] According to embodiments of this disclosure, multiple first clients and second clients form a distributed system. Therefore, when a second client performs a target operation on the first data, the first clients need to synchronize the target operation.

[0091] In operation S420, a pre-commit request is generated based on the transaction identifier of the transaction request.

[0092] According to embodiments of this disclosure, multiple sub-identifiers can be obtained by parsing the information in the transaction identifier. Based on the format of the pre-commit request, the multiple sub-identifiers are combined to obtain the pre-commit request.

[0093] According to embodiments of this disclosure, the pre-commit request may also include a global transaction number assigned to the target operation, which facilitates client querying and storage engine tracking of the transaction's lifecycle.

[0094] In operation S430, a pre-commit request is sent to each first client so that each first client can synchronize the target operation for the first data and obtain the second data.

[0095] According to embodiments of this disclosure, a pre-submission request is sent to each first client to achieve the target operation of synchronizing the first client with the second client. Each first client stores first data.

[0096] According to an embodiment of this disclosure, after receiving a pre-commit request, the first client obtains a transaction identifier by parsing the pre-commit request, obtains the first address of the first data through the transaction identifier, performs a target operation on the first data, and obtains the second data.

[0097] In operation S440, in response to receiving confirmation information from M first clients, a submit request is sent to each first client so that second data can be stored on each first client. The confirmation information is used to indicate that the target operation was successfully executed.

[0098] According to embodiments of this disclosure, it is ensured that all M first clients perform target operations on the first data to obtain the second data, and a submission request is sent to each first client to submit the second data.

[0099] According to embodiments of this disclosure, the target operation is synchronized to the first client by generating a pre-commit request based on the transaction request from the second client. Data consistency and atomicity are guaranteed by sending a commit request after receiving confirmation from the first client.

[0100] According to embodiments of this disclosure, the method further includes: in response to receiving fault information sent by the m-th first client, sending a commit request to each first client other than the m-th first client, so as to store second data in each first client other than the m-th first client, where m is a positive integer less than M; in response to receiving fault recovery information sent by the m-th first client, determining the fault duration of the m-th first client; if the fault duration is determined not to exceed a predetermined duration, obtaining N transaction identifiers added during the fault of the m-th first client, where N is a positive integer; and synchronizing the operations corresponding to the N newly added transaction identifiers in the m-th first client.

[0101] According to embodiments of this disclosure, the fault information can be sent by a first client to indicate a failure of the first client. If the first client fails to execute the target operation, it can send fault information to the storage engine, and record the fault information in the transaction log to indicate that subsequent operations cannot be performed. Upon receiving the fault information, the storage engine will not send a commit request to the first client that sent the fault information, thus preventing the first client from storing erroneous data and affecting data consistency.

[0102] According to embodiments of this disclosure, the fault information can be information sent by the first client for fault recovery. The fault duration can be the time from the occurrence of the fault to its recovery by the first client. When the first client recovers to a normal state, it can send fault recovery information to the storage engine. The storage engine obtains the identifier of the first client based on the fault recovery information, traverses the transaction identifiers in the transaction log to obtain the fault information, and obtains the time of the fault occurrence through the fault information. The fault duration is determined based on the fault occurrence time and fault recovery time.

[0103] According to embodiments of this disclosure, the predetermined duration can be a threshold value for the fault duration. If the fault duration is too long, there will be too many transactions that need to be recovered, which would be meaningless. Therefore, if the fault duration does not exceed the predetermined duration, transactions are recovered.

[0104] According to embodiments of this disclosure, transaction identifiers added during the first client failure can be queried through the transaction log, and operations can be performed based on the transaction identifiers to achieve data recovery.

[0105] According to embodiments of this disclosure, data inconsistency caused by the failure of the first client is avoided by synchronizing the operation of the first client that is faulty.

[0106] According to embodiments of this disclosure, the method further includes, in response to detecting that the reception time of the confirmation information by the m-th first client exceeds a predetermined time, or receiving execution failure information sent by the m-th first client, sending an operation cancellation request to each first client other than the m-th first client, so that each first client other than the m-th first client cancels the target operation on the first data; and in response to receiving operation cancellation success information from (M-1) first clients other than the m-th first client, returning synchronization failure information to the second client.

[0107] According to embodiments of this disclosure, the reception duration can be the time from when the storage engine sends a pre-commit request to when it receives confirmation information. The predetermined duration can be a threshold value for the reception duration. When the first client malfunctions and is unable to send fault information to the storage engine, the storage engine can determine the duration based on the reception duration. If the reception duration exceeds the predetermined duration, the storage engine will send an operation cancellation request to the first client.

[0108] According to embodiments of this disclosure, if the storage engine determines that a faulty first client exists, it sends an operation cancellation request to all other first clients besides the faulty one. Upon receiving the operation cancellation request, a non-faulty first client immediately stops its current operation and, if it successfully stops the operation, sends a cancellation success message to the storage engine. Upon receiving the cancellation success message, the storage engine returns a synchronization failure message to the second client.

[0109] According to embodiments of this disclosure, in the event of a fault in a first client, the operation of the first client that is not faulty is stopped to ensure data consistency.

[0110] Figure 5 A flowchart illustrating data synchronization controlled by a storage engine according to an embodiment of the present disclosure is shown schematically.

[0111] like Figure 5As shown, data synchronization is controlled by the storage engine, including operations S510 to S570.

[0112] When operating S510, a transaction request is issued.

[0113] During the operation of S520, a transaction identifier is generated.

[0114] When operating S530, send a pre-commit request.

[0115] When operating S540, send a confirmation message.

[0116] When operating the S550, a submit request is sent.

[0117] When operating the S560, a submission success request is sent.

[0118] When operating S570, send a transaction commit success message.

[0119] According to an embodiment of this disclosure, when a user adds a row of data to Table 1 through a second client, the second database sends a transaction request to the columnar database storage engine to add a row of data to Table 1. The columnar database receives the transaction request from the second client, generates a transaction identifier for the transaction, and sends it to the second client. Simultaneously, the columnar database storage engine sends pre-commit requests to multiple first clients associated with Table 1, including first client A, first client B, etc. Upon receiving the pre-commit request, each first client adds a row of data to Table 1 stored at a first address and stores it as a temporary table at a second address different from the first address. Then, it sends confirmation information to the columnar database storage engine. After receiving confirmation information from all first clients, the columnar database engine sends commit requests to all multiple first clients associated with Table 1. Upon receiving the commit request, each first client stores the temporary table stored at the second address back to the first address. After successful storage, it sends a commit success request to the columnar database storage engine. After receiving the commit success requests from all multiple first clients associated with Table 1, the columnar database storage engine sends transaction commit success information to the second client.

[0120] Figure 6 A schematic block diagram of a data synchronization device applied to a first client according to an embodiment of the present disclosure is shown.

[0121] like Figure 6 As shown, the data synchronization device 600 applied to the first client in this embodiment includes a parsing module 610, a synchronization module 620, a confirmation module 630, and a storage module 640.

[0122] The parsing module 610 is configured to, in response to receiving a pre-commit request sent by the storage engine of the columnar database, parse a transaction identifier from the pre-commit request, wherein the transaction identifier is used to characterize the target operation of the second client on the first data. In one embodiment, the parsing module 610 may be used to perform the operation S210 described above, which will not be repeated here.

[0123] The synchronization module 620 is used to synchronize the target operation on the first data in the first client to obtain the second data. In one embodiment, the synchronization module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0124] The confirmation module 630 is used to send confirmation information indicating the successful execution of the target operation to the storage engine, so that the storage engine can generate a commit request after receiving the confirmation information. In one embodiment, the confirmation module 630 can be used to execute the operation S230 described above, which will not be repeated here.

[0125] Storage module 640 is configured to store second data on the first client in response to receiving a commit request from the storage engine. In one embodiment, storage module 640 may be used to perform the operation S230 described above, which will not be repeated here.

[0126] According to embodiments of this disclosure, the synchronization module 620 further includes an acquisition submodule, an execution submodule, and a first storage submodule.

[0127] The `get` submodule is used to retrieve the first data from the first address.

[0128] The execution submodule is used to perform the target operation on the first data to obtain the second data.

[0129] The first storage submodule is used to store the second data at a second address, wherein the second address is different from the first address where the first data is stored.

[0130] According to embodiments of this disclosure, storage module 640 further includes a second storage submodule.

[0131] The second storage submodule is used to retrieve the second data from the second address and store the second data in the first address.

[0132] According to embodiments of this disclosure, the data synchronization device 600 applied to the first client further includes a first determining module, a second determining module, a result determining module, and an information generating module.

[0133] The first determining module is used to determine the first status information of the first data before adding or deleting target data in the first data.

[0134] The second determining module is used to determine the second status information of the second data after adding or deleting target data in the first data.

[0135] The result determination module is used to determine the integrity result for the target data based on the first state information and the second state data.

[0136] The information generation module is used to generate confirmation information when the integrity result characterization target data is completely stored in the first client or completely deleted from the first client.

[0137] According to embodiments of this disclosure, the data synchronization device 600 applied to a first client further includes a request sending module, a duration determination module, an identifier acquisition module, and an operation synchronization module.

[0138] The request sending module is used to send a submit request to each first client other than the m-th first client in response to receiving a fault information sent by the m-th first client, so as to store the second data in each first client other than the m-th first client, where m is a positive integer less than M.

[0139] The duration determination module is used to determine the fault duration of the m-th first client in response to receiving fault recovery information sent by the m-th first client.

[0140] The identifier acquisition module is used to acquire N transaction identifiers added during the m-th first client failure, where N is a positive integer, provided that the failure duration has not exceeded the predetermined duration.

[0141] The operation synchronization module is used to synchronize the operations corresponding to the N newly added transaction identifiers in the m-th first client.

[0142] According to embodiments of this disclosure, the data synchronization device 600 applied to a first client further includes an operation cancellation module and a failure return module.

[0143] The operation cancellation module is used to send an operation cancellation request to each first client other than the m-th first client in response to detecting that the reception time of the confirmation information of the m-th first client exceeds a predetermined time, or to receiving the execution failure information sent by the m-th first client, so that each first client other than the m-th first client cancels the target operation on the first data.

[0144] The failure return module is used to return synchronization failure information to the second client in response to receiving operation cancellation success information from (M-1) first clients other than the m-th first client.

[0145] According to embodiments of this disclosure, any multiple modules among the parsing module 610, synchronization module 620, confirmation module 630, and storage module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0146] According to embodiments of this disclosure, at least one of the parsing module 610, synchronization module 620, confirmation module 630, and storage module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the parsing module 610, synchronization module 620, confirmation module 630, and storage module 640 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0147] Figure 7 A schematic block diagram of a data synchronization apparatus for a storage engine applied to a columnar storage database according to an embodiment of the present disclosure is shown.

[0148] like Figure 7 As shown, the data synchronization device 700 applied to the storage engine of the columnar storage database includes a first generation module 710, a second generation module 720, a first request sending module 730, and a second request sending module 740.

[0149] The first generation module 710 is configured to generate a transaction identifier corresponding to the transaction request in response to receiving a transaction request from the second client. The transaction request is used to synchronize a target operation on the first data across M first clients (excluding the second client), where M is a positive integer. In one embodiment, the first generation module 710 can be used to execute the operation S410 described above, which will not be repeated here.

[0150] The second generation module 720 is used to generate a pre-commit request based on the transaction identifier of the transaction request. In one embodiment, the second generation module 720 can be used to perform the operation S420 described above, which will not be repeated here.

[0151] The first request sending module 730 is used to send a pre-commit request to each first client so that each first client can synchronously perform a target operation on the first data to obtain the second data. In one embodiment, the first request sending module 730 can be used to perform the operation S430 described above, which will not be repeated here.

[0152] The second request sending module 740 is configured to send a submission request to each of the M first clients in response to receiving confirmation information from each first client, so as to store the second data on each first client. The confirmation information is used to indicate that the target operation was successfully executed. In one embodiment, the second request sending module 740 can be used to perform the operation S440 described above, which will not be repeated here.

[0153] Figure 8 A block diagram of an electronic device suitable for a data synchronization method according to an embodiment of the present disclosure is shown schematically.

[0154] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0155] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0156] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0157] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0158] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0159] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0160] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0161] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0162] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0163] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0166] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this disclosure. It should be understood that the above descriptions are merely specific embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A data synchronization method, applied to a first client, the method comprising: In response to receiving a pre-commit request sent by the storage engine of the columnar database, a transaction identifier is parsed from the pre-commit request, wherein the transaction identifier is used to characterize the target operation of the second client on the first data; In the first client, a target operation on the first data is synchronized to obtain the second data; Send a confirmation message indicating the successful execution of the target operation to the storage engine, so that the storage engine generates a commit request upon receiving the confirmation message; and In response to receiving a commit request from the storage engine, the second data is stored in the first client; The step of synchronizing a target operation on the first data in the first client to obtain the second data includes: Retrieve the first data from the first address; Perform the target operation on the first data to obtain the second data; and The second data is stored at a second address, wherein the second address is different from the first address where the first data is stored; The step of storing the second data in the first client in response to receiving a commit request from the storage engine includes: The second data is obtained from the second address and stored in the first address.

2. The method according to claim 1, wherein, The target operation includes adding or deleting; the method also includes: Before adding or deleting target data in the first data, determine the first state information of the first data; After adding or deleting target data in the first data, determine the second status information of the second data; Based on the first state information and the second state information, determine the integrity result for the target data; and The confirmation information is generated when the integrity result indicates that the target data is completely stored in the first client or the target data is completely deleted from the first client.

3. A data synchronization method applied to the storage engine of a columnar storage database, the method comprising: In response to receiving a transaction request from a second client, a transaction identifier corresponding to the transaction request is generated, wherein the transaction request is used to synchronize a target operation on the first data on M first clients other than the second client, where M is a positive integer; A pre-commit request is generated based on the transaction identifier of the transaction request; Send the pre-commit request to each of the first clients so that each of the first clients can synchronously perform a target operation on the first data to obtain the second data; and In response to receiving confirmation messages from M of the first clients, a submission request is sent to each of the first clients so that the second data can be stored on each of the first clients. The confirmation messages are used to indicate that the target operation was successfully executed. The first clients acquire and store the second data in the following manner: In response to receiving a pre-commit request sent by the storage engine of the columnar database, a transaction identifier is parsed from the pre-commit request, wherein the transaction identifier is used to characterize the target operation of the second client on the first data; Obtain the first data from the first address; perform the target operation on the first data to obtain the second data; store the second data in the second address, wherein the second address is different from the first address where the first data is stored; Send a confirmation message indicating that the target operation was successfully executed to the storage engine, so that the storage engine generates a submission request after receiving the confirmation message; In response to receiving a commit request from the storage engine, the second data is retrieved from the second address and stored in the first address.

4. The method according to claim 3, further comprising: In response to receiving a fault message sent by the m-th first client, the commit request is sent to each of the first clients other than the m-th first client, so that the second data is stored in each of the first clients other than the m-th first client, where m is a positive integer less than M; In response to receiving fault recovery information sent by the m-th first client, the fault duration of the m-th first client is determined; If it is determined that the fault duration does not exceed the predetermined duration, obtain N transaction identifiers added during the m-th first client fault, where N is a positive integer; as well as The operations corresponding to the newly added N transaction identifiers are synchronized in the m-th first client.

5. The method according to claim 3, further comprising: In response to detecting that the reception time of the confirmation information by the m-th first client exceeds a predetermined time, or receiving the execution failure information sent by the m-th first client, an operation cancellation request is sent to each of the first clients other than the m-th first client, so that each of the first clients other than the m-th first client cancels the target operation on the first data; as well as In response to receiving operation cancellation success messages from (M-1) first clients other than the m-th first client, a synchronization failure message is returned to the second client.

6. A data synchronization device, applied to a first client, the device comprising: The parsing module is used to respond to a pre-commit request sent by the storage engine of the columnar database, and to parse a transaction identifier from the pre-commit request, wherein the transaction identifier is used to characterize the target operation of the second client on the first data; The synchronization module is used to synchronize the target operation on the first data in the first client to obtain the second data; The confirmation module is used to send confirmation information indicating the successful execution of the target operation to the storage engine, so that the storage engine generates a commit request after receiving the confirmation information; and A storage module is configured to store the second data in the first client in response to receiving a commit request from the storage engine; The synchronization module is further configured to: obtain the first data from the first address; perform the target operation on the first data to obtain the second data; and store the second data in the second address, wherein the second address is different from the first address where the first data is stored. The storage module is further configured to: obtain the second data from the second address and store the second data in the first address.

7. A data synchronization device applied to a storage engine of a columnar storage database, the device comprising: The first generation module is configured to generate a transaction identifier corresponding to the transaction request in response to receiving a transaction request from the second client, wherein the transaction request is used to synchronize a target operation on the first data on M first clients other than the second client, where M is a positive integer; The second generation module is used to generate a pre-commit request based on the transaction identifier of the transaction request; A first request sending module is configured to send the pre-submission request to each of the first clients, so that each of the first clients can synchronously perform a target operation on the first data to obtain the second data; and The second request sending module is configured to send a submission request to each of the first clients in response to receiving confirmation information from M of the first clients, so as to store the second data in each of the first clients, wherein the confirmation information is used to indicate that the target operation was successfully executed; The first client obtains and stores the second data in the following manner: In response to receiving a pre-commit request sent by the storage engine of the columnar database, a transaction identifier is parsed from the pre-commit request, wherein the transaction identifier is used to characterize the target operation of the second client on the first data; Obtain the first data from the first address; perform the target operation on the first data to obtain the second data; store the second data in the second address, wherein the second address is different from the first address where the first data is stored; Send a confirmation message indicating that the target operation was successfully executed to the storage engine, so that the storage engine generates a submission request after receiving the confirmation message; In response to receiving a commit request from the storage engine, the second data is retrieved from the second address and stored in the first address.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images