Data synchronization method and apparatus, electronic device, and readable storage medium

Abstract

Provided in the implementation of the present application is a data synchronization method, device, electronic equipment and readable storage medium. In response to a data update operation on a source data set in a distributed database, the method takes an execution timestamp of the data update operation as an operation identifier of an update log and writes the operation identifier into a source log file of the source data set. The source log file records the update log, and the update log is used to record the data update operation. The update log in the source log file is sequentially synchronized to a specified directory of a target data set to be synchronized in the distributed database according to the operation identifier, so as to obtain a target log file. The data stored in the source data set is associated with the data stored in the target data set. The data in the target data set is updated based on the target log file. In this way, the synchronization can be performed according to the operation identifier of the update log, so that the synchronization process is consistent with the execution order of each operation, and the accuracy and consistency of data synchronization are ensured.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data synchronization method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the gradual development of computer technology, more and more users are using various service providers. At the same time, the amount of data that these service providers need to process has also increased significantly. In order to improve the efficiency of data processing, distributed databases are often used to distribute large amounts of data into various sub-databases or sub-tables to obtain multiple datasets.

[0003] In this scenario, to ensure that any dataset can respond to user queries, data synchronization of the distributed database is necessary. Therefore, ensuring the accuracy of data synchronization becomes a critical technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a data synchronization method, apparatus, electronic device, and readable storage medium to at least solve the aforementioned problems. The specific technical solution is as follows:

[0005] In a first aspect of this invention, a data synchronization method is provided, which can be applied to a server cluster. This method may include:

[0006] In response to a data update operation on a source dataset in a distributed database, the execution timestamp of the data update operation is used as the operation identifier of the update log and written to the source log file of the source dataset; the source log file records the update log, which is used to record the data update operation;

[0007] The update logs in the source log file are sequentially synchronized to the specified directory of the target dataset to be synchronized in the distributed database according to the operation identifier, so as to obtain the target log file; the data stored in the source dataset is related to the data stored in the target dataset;

[0008] The data in the target dataset is updated based on the target log file.

[0009] In a second aspect of the present invention, a data synchronization device is provided, which can be applied to a server cluster. The device may include:

[0010] The timestamp writing module is used to respond to a data update operation on the source dataset in the distributed database by using the execution timestamp of the data update operation as the operation identifier of the update log and writing it to the source log file of the source dataset; the source log file records the update log, which is used to record the data update operation;

[0011] The synchronization module is used to sequentially synchronize the update logs in the source log file to a specified directory of the target dataset to be synchronized in the distributed database according to the operation identifier, so as to obtain the target log file; the data stored in the source dataset is related to the data stored in the target dataset;

[0012] The update module is used to update the data in the target dataset based on the target log file.

[0013] A third aspect of the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0014] Memory, used to store computer programs;

[0015] The processor, when executing a program stored in memory, performs any of the data synchronization methods described above.

[0016] In a fourth aspect of the invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the data synchronization methods described above.

[0017] In a fifth aspect of the invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data synchronization methods described above.

[0018] In response to a data update operation on a source dataset in a distributed database, this embodiment of the invention uses the execution timestamp of the data update operation as an operation identifier for the update log and writes it to the source log file of the source dataset. The source log file records the update log, which is used to record the data update operation. The update logs in the source log file are sequentially synchronized to a designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, to obtain a target log file. The data stored in the source dataset is associated with the data stored in the target dataset. The data in the target dataset is updated based on the target log file. Thus, by using the execution timestamp of the data update operation as the operation identifier for the update log, and sequentially synchronizing the update logs in the source log file to the designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, the target dataset can obtain a target log file recorded according to the operation identifier. This ensures that the log synchronization process is consistent with the execution order of each data update operation. Furthermore, updating the data in the target dataset according to the target log file ensures that the data update process is consistent with each data update operation, thereby guaranteeing the accuracy and consistency of data synchronization. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a flowchart of the steps of a data synchronization method provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a scenario provided by an embodiment of the present invention;

[0022] Figure 3 This is a flowchart of another data synchronization method provided in an embodiment of the present invention;

[0023] Figure 4 This is a structural block diagram of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0025] Figure 1 This is a flowchart illustrating the steps of a data synchronization method provided in an embodiment of the present invention, which can be applied to server clusters, such as... Figure 1 As shown, the method may include the following steps:

[0026] Step 101: In response to a data update operation on the source dataset in the distributed database, the execution timestamp of the data update operation is used as the operation identifier of the update log and written to the source log file of the source dataset; the source log file records the update log, which is used to record the data update operation.

[0027] The aforementioned distributed database refers to a database in a distributed scenario. Specifically, it can involve splitting data from a single database into multiple sub-databases or sub-tables through database sharding or table partitioning, resulting in multiple datasets. Correspondingly, the aforementioned server cluster can contain several server nodes, each storing different datasets. Understandably, the data in the datasets stored on each server node is interconnected.

[0028] The source dataset mentioned above refers to the dataset where data update operations occur. Each data update operation refers to an operation that modifies the data, which can correspond to a transaction or event. These data update operations can be recorded in a log, resulting in an update log entry. Specifically, the data update operation can be an operation to add or remove a specific data item from the source dataset. It can be triggered upon receiving a data update command, which can include the object to be updated, the updated attribute, and the update amount, depending on the specific circumstances.

[0029] Accordingly, the update log can include the update object, update attributes, and update amount of the data update operation. Furthermore, this embodiment of the invention can be applied to a relational database management system (MySQL), and thus the update log can record the MySQL statement corresponding to the execution of this data update operation. The source log file refers to the file that records the update log of the source dataset, typically a binary file (which can be a binlog).

[0030] The execution timestamp mentioned above refers to the timestamp when the data update operation occurs. Specifically, a global clock server can be used as the timestamp server, and all servers in the server cluster can generate corresponding timestamps through the timestamp server as needed to obtain the aforementioned execution timestamp. Furthermore, the accuracy of timestamps is usually greater than the millisecond level, so the execution timestamp of each operation is often unique. Therefore, in this embodiment of the invention, the execution timestamp of the data update operation can be used as the unique identifier of this data update operation, that is, the global transaction identifier. At the same time, the execution timestamp can also provide the time when this operation occurred.

[0031] Step 102: According to the operation identifier, the update logs in the source log file are sequentially synchronized to the specified directory of the target dataset to be synchronized in the distributed database to obtain the target log file; the data stored in the source dataset is related to the data stored in the target dataset.

[0032] The target dataset mentioned above refers to the dataset to be synchronized, which corresponds to the source dataset. It can be any dataset in the distributed database other than the source dataset, and can be one or multiple datasets, depending on actual needs. It's understood that the target dataset and the source dataset are related, but they are located on different server nodes.

[0033] The specified directory mentioned above refers to the specified directory under the server where the target dataset is located. It can be a dedicated directory on the server where the target dataset is located, used to record the logs of the target dataset. It can be set according to actual needs. Specifically, the above synchronization can be to synchronize each update log to the specified directory in sequence according to the operation identifier of the update log recorded in the source log file, so as to obtain the target log file, which can also be called the relay log file.

[0034] Understandably, since the update logs in the source log file all use execution timestamps as operation identifiers, in this embodiment of the invention, the time sequence of the corresponding operation can be identified according to each operation identifier, and then synchronization can be performed sequentially according to the execution time of each log, so that the order of the synchronized update logs in the target log file corresponds to the execution time sequence of each operation.

[0035] Furthermore, when the server or network fails, synchronization may be interrupted. In this embodiment of the invention, the execution timestamp of the update log being synchronized at the time of the interruption can be determined based on the operation identifier of the current update log in the target log file. Then, the synchronization operation can continue according to the operation identifier, thereby avoiding data loss.

[0036] Step 103: Update the data in the target dataset based on the target log file.

[0037] Since the update log is used to record data update operations, this embodiment of the invention can update the target dataset according to the update log in the target log file after obtaining the target log file. Specifically, the data in the target dataset can be made to perform the operations indicated by the update log, thereby achieving data replay and completing the update.

[0038] Specifically, when updating data in the target dataset, the data can be updated sequentially according to the operation identifiers in the update log of the target log file, that is, according to the execution timestamps. This ensures that the order of data updates is consistent with the operation identifiers in the update log, thereby guaranteeing that the data update process is consistent with the actual time sequence of the data update operations.

[0039] In summary, this embodiment of the invention, in response to a data update operation on a source dataset in a distributed database, uses the execution timestamp of the data update operation as the operation identifier for the update log and writes it to the source log file of the source dataset. The source log file records the update log, which is used to record the data update operation. The update logs in the source log file are sequentially synchronized to a designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, to obtain a target log file. The data stored in the source dataset is associated with the data stored in the target dataset. The data in the target dataset is updated based on the target log file. Thus, by using the execution timestamp of the data update operation as the operation identifier for the update log, and sequentially synchronizing the update logs in the source log file to the designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, the target dataset can obtain a target log file recorded according to the operation identifier. This ensures that the log synchronization process is consistent with the execution order of each data update operation. Furthermore, updating the data in the target dataset according to the target log file ensures that the data update process is consistent with each data update operation, thereby guaranteeing the accuracy and consistency of data synchronization.

[0040] Optionally, embodiments of the present invention may specifically include the following steps:

[0041] Step 201: Create a thread pool containing multiple threads, and write the update logs in the target log file to a preset message queue based on the multiple threads.

[0042] The aforementioned thread pool can contain multiple empty threads. The number of threads in a thread pool can be set arbitrarily; understandably, more threads result in higher write efficiency. The preset message queue can be a Kafka queue or a RubyMQ queue. Understandably, message queues can serve to smooth out peak loads and decouple processes. Specifically, the operation of writing update logs to the message queue can involve reading update logs from the target log file and asynchronously writing them to the preset message queue using multiple threads in the thread pool, so that the message queue stores update log content with a global transaction identifier.

[0043] Step 202: Read the update log in the preset message queue to create a query index based on the update log.

[0044] Step 203: Upon receiving a query instruction from the query object for the distributed database, perform a query based on the query index.

[0045] The operation of reading update logs can be implemented using a pre-defined data storage tool. For example, the message queue can be consumed using a high-performance slice data storage tool (StarRocks). After obtaining the update logs from the target log file, a query index can be created using StarRocks' built-in structure. The query index can contain the specific data updated by each update log. In other words, the query index can be used to provide a historical query data table, and each index can point to update data at different historical times.

[0046] Furthermore, the aforementioned query object can be the entity that initiates the query request to the distributed database, or more specifically, the client used by the query entity. Specifically, upon receiving a query instruction from the query object to the distributed database, the query can be performed by finding the corresponding query data from the query index based on the query time indicated in the query instruction and the query object's location.

[0047] In this embodiment of the invention, a thread pool containing multiple threads is created, and update logs from the target log file are written to a preset message queue based on these threads. The update logs in the preset message queue are read to create a query index. Upon receiving a query instruction from a query object for the distributed database, a query is performed based on the query index. Thus, by using a thread pool and a message queue, log reading efficiency can be improved. Furthermore, by creating a query index based on the update logs, queries can be provided to the query object through the index, improving the efficiency of subsequent queries.

[0048] Optionally, the data update operation is used to instruct the updating of data on the target attribute of a first object in the source dataset; the updating of data in the target dataset based on the target log file includes:

[0049] Step 301: Obtain the target data corresponding to the target attribute of the target object in the target dataset; the target object is an object in the target dataset that contains the first object.

[0050] Here, the first object refers to an object in the source dataset, and the target attribute refers to any attribute corresponding to the first object in the source dataset. Any object in the source dataset can correspond to data with different attributes, and correspondingly, any attribute can correspond to data from different objects. For example, the first object could be account A, and the target attribute could be the account balance; therefore, the data update operation could be updating the account balance of account A.

[0051] Furthermore, the aforementioned target object refers to the object in the target dataset that contains the first object. Correspondingly, the target data corresponding to the target attribute of the aforementioned target object refers to the data value of the object containing the first object under the target attribute. Specifically, the target dataset can be traversed row by row with reference to the target attribute until the object containing the first object is found.

[0052] For example, taking the first object as account A, the target object can be account A + account B, and the target data can be the account balances corresponding to account A + account B.

[0053] Step 302: Update the target data sequentially based on the update logs in the target log file, according to the order represented by the operation identifiers of each update log in the target log file.

[0054] Specifically, the order represented by the above operation identifier refers to the execution time order of each update log. It can be understood that when a data update operation occurs in the source dataset, the data update is also performed sequentially according to the execution time of each operation. Therefore, in order to ensure that the update order of the target data in the target dataset is consistent with that of the source dataset, in this embodiment of the invention, the target data in the target dataset can also be updated sequentially according to the order represented by the operation identifier.

[0055] In this embodiment of the invention, the data update operation is used to instruct the updating of the target attribute data of a first object in the source dataset; by obtaining the target data corresponding to the target attribute of the target object in the target dataset; the target object is an object in the target dataset that contains the first object; and updating the target data sequentially based on the update logs in the target log file according to the order represented by the operation identifiers of each update log in the target log file. Thus, by updating the target data in the target dataset according to the order of the operation identifiers, the update order of the target data in the target dataset can be kept consistent with that of the source dataset, further improving the consistency of data synchronization.

[0056] Optionally, the server cluster includes real-time nodes, which are used to acquire real-time data; the operation of performing a query based on the query index upon receiving a query instruction from a query object for the distributed database can specifically include the following steps in this embodiment of the invention:

[0057] Step 401: Obtain the timestamp data carried in the query command based on the management node in the server cluster.

[0058] Step 402: Based on the fact that the timestamp data indicates that the query instruction is used to query real-time data, the management node forwards the query instruction to the real-time node.

[0059] Step 403: The real-time node obtains the data queried by the query instruction based on the query index as the first query result, and returns the first query result to the management node.

[0060] Step 404: The management node outputs the first query result to the query object.

[0061] Regarding steps 401 to 404 above, Figure 2 This is a schematic diagram of a scenario provided by an embodiment of the present invention, such as... Figure 2 As shown, the servers in the server cluster can be called nodes, and different nodes can be used to provide different services or functions.

[0062] The aforementioned management node, also known as the proxy node, can be used to obtain data query instructions. When a client initiates a data query request, the management node can capture the request and read the query object and query time indicated by the request. Accordingly, the timestamp data carried in the query instruction in step 401 can be understood as the query time.

[0063] The timestamp data carried in the query command can represent the time period of the data that the query object wants to retrieve. Specifically, the timestamp data in the query command can be compared with the operation identifier of the most recent update log. If the time represented by the timestamp data is before the operation identifier, it indicates that the query command is used to query historical data. Conversely, if the time represented by the timestamp data is after the operation identifier, it indicates that the query command is used to query real-time data.

[0064] Specifically, the aforementioned real-time nodes can be dedicated to querying real-time data. When the management node determines that a query instruction is for real-time data, it can forward the instruction to the real-time node. The real-time node then retrieves the real-time data as the first query result using the query index and returns it to the management node. Furthermore, the management node can output the first query result to the query object. This can be achieved by returning the first query result to the client corresponding to the query instruction.

[0065] In this embodiment of the invention, the server cluster includes real-time nodes, which are used to acquire real-time data. The query instruction carries timestamp data obtained from the management node in the server cluster. Based on the timestamp data indicating that the query instruction is for querying real-time data, the management node forwards the query instruction to the real-time nodes. The real-time nodes retrieve the data indicated by the query instruction based on the query index as a first query result and return the first query result to the management node. The management node then outputs the first query result to the queried object. Thus, by setting up real-time and management nodes, when a query instruction for real-time data is received, data querying can be performed through interaction between the management node and the real-time nodes, eliminating the need to traverse the nodes containing each dataset, thereby improving query efficiency.

[0066] Optionally, the server cluster further includes historical nodes, which are used to acquire historical data. Specifically, embodiments of the present invention may include the following steps:

[0067] Step 501: Based on the fact that the timestamp data indicates that the query instruction is used to query historical data, the management node forwards the query instruction to the historical node.

[0068] Step 502: The historical node obtains the historical data corresponding to the timestamp based on the query index, uses it as the second query result, and returns the second query result to the management node.

[0069] Step 503: The management node outputs the second query result to the query object.

[0070] Specifically, such as Figure 2 As shown, historical nodes can interact with management nodes. When the query command is to query historical data, the historical node can receive the query command forwarded by the management node and obtain the queried historical data.

[0071] Specifically, the aforementioned real-time nodes can obtain real-time data through the index. After each data update operation, the real-time nodes can obtain the latest real-time data and send the historical data to the distributed file system for storage. The distributed file system can be a server node used to store historical data.

[0072] Furthermore, such as Figure 2 As shown, after receiving a query command forwarded by the management node, the historical node can initiate a query request to the distributed file system based on the timestamp data of the query command. This prompts the distributed file system to distribute the historical data indicated by the query command to the historical node. The historical node then returns the received historical data as a second query result to the management node. Furthermore, the management node can output the second query result to the query object. Specifically, the management node can return the second query result to the client corresponding to the query command to achieve query result output.

[0073] In this embodiment of the invention, the server cluster further includes historical nodes, which are used to acquire historical data. When the management node forwards a query instruction to the historical node based on the timestamp data indicating that the query instruction is for querying historical data, the historical node retrieves the historical data corresponding to the timestamp based on the query index, using it as a second query result, and returns the second query result to the management node. The management node then outputs the second query result to the query object. Thus, by setting historical nodes, when a query instruction for querying historical data is received, data querying can be achieved through interaction between the management node and the historical node, eliminating the need to traverse the nodes containing each dataset, thereby improving query efficiency.

[0074] Optionally, the operation of synchronizing the update logs in the source log file to the specified directory of the target dataset to be synchronized in the distributed database according to the operation identifier may specifically include the following steps in this embodiment of the invention:

[0075] Step 601: Determine the execution order of the update operations recorded in each update log based on the operation identifier of each update log in the source log file.

[0076] Step 602: Copy the update logs from the source log files to the specified directory of the target dataset in the order of execution.

[0077] Specifically, in this embodiment of the invention, the execution order of each update operation can be determined according to the operation identifier of each update log in the aforementioned source log file. Specifically, since the operation identifier is the execution timestamp of each update operation, this embodiment of the invention can sort the operation identifiers in ascending order to obtain the aforementioned execution order. A larger timestamp indicates that the event corresponding to that timestamp is later in the sequence.

[0078] Furthermore, after obtaining the execution order, the update logs in the source log file can be copied sequentially to the specified directory according to the above execution order, thereby ensuring that the log content in the obtained target log file is also arranged in the execution order.

[0079] In this embodiment of the invention, the execution order of the update operations recorded in each update log is determined based on the operation identifier of each update log in the source log file. Following this execution order, the update logs in the source log file are sequentially copied to a designated directory of the target dataset. Thus, by using the operation identifier of each update log to determine the execution order of each update operation, copying the update logs in the source log file according to this execution order ensures that the log content in the resulting target log file is also arranged in the execution order, thereby maintaining consistency between the target log file and the source log file and improving the consistency of data synchronization.

[0080] Optionally, before the operation of sequentially synchronizing the update logs in the source log file to the designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, the embodiments of the present invention may further include the following steps:

[0081] Step 701: Obtain the metadata of the source log file.

[0082] Step 702: Back up the metadata to the specified location.

[0083] Before determining the execution order of the update operations recorded in each of the aforementioned update logs, this embodiment of the invention may further include the following steps:

[0084] Step 703: If the source log file is not successfully read, reread the source log file based on the metadata at the specified location.

[0085] Regarding steps 701 to 703 above, the aforementioned metadata refers to the attribute data of the source log file, which may include the name, size, and other data of the source log file. The aforementioned specified location refers to a storage location that is not easily lost, which may be a storage module such as a hard disk or disk drive.

[0086] Specifically, if the server where the source log file is located fails or the network is interrupted, there may be situations where the source log file cannot be read successfully. Therefore, this embodiment of the invention can follow the Write Ahead Log (WAL) principle to back up the metadata of the source log file, so that when the server where the source log file is located fails and the source log file cannot be read, the source log file can be restored based on the backed-up metadata.

[0087] In this embodiment of the invention, the metadata of the source log file is obtained; the metadata is backed up to a designated location; and if the source log file cannot be successfully read, it is reread based on the metadata at the designated location. Thus, by backing up the metadata of the source log file to a designated location, effective data backup can be achieved and data loss can be avoided when the source log file cannot be read successfully, allowing for rereading based on the metadata at the designated location.

[0088] Figure 3 This is a flowchart of another data synchronization method provided in an embodiment of the present invention, as follows: Figure 3 As shown, it may include:

[0089] Step 211: Follow the write-ahead principle of WAL to back up the binlog file name of the source dataset, and record the execution timestamp as a transaction identifier in the binlog.

[0090] The binlog mentioned above is the source log file in the preceding steps. The transaction identifier refers to the operation identifier, and the execution timestamp can be used as the global transaction identifier.

[0091] Step 212: Place the relay log from the binlog of the source dataset into the directory of the target synchronization set, and record the log name as the log name of the source dataset to complete the log synchronization.

[0092] The target synchronization set mentioned above refers to the target dataset. This step refers to synchronizing the log content in the binlog log to the specified directory of the target dataset to obtain the target log file, which is the relay log of the binlog log.

[0093] Step 213: Use the high-performance slice data storage tool StarRocks to obtain incremental logs in a distributed scenario.

[0094] In the above distributed scenario, incremental logs refer to the log content in the target log file.

[0095] Step 214: After obtaining the incremental log content, use Kafka message queue as middleware to store the incremental log information, and build an index using StarRocks' built-in structure.

[0096] Step 215: Utilize the established incremental log index to accelerate data retrieval.

[0097] As can be seen, compared with the existing methods that do not adopt distributed data synchronization optimization and only use the binlog mechanism to achieve synchronization, the data synchronization method proposed in this embodiment of the invention can ensure data consistency and completeness in the database in a distributed scenario, optimize the data synchronization logic, and consume very little memory.

[0098] In terms of functionality, this method, based on the binlog mechanism synchronization, completes the log preprocessing, log storage, and log consumption transformation scheme for transaction identifier recording, satisfying the need for accelerated data query while fulfilling the original functions. Regarding data synchronization, in the event of a source dataset failure, worker machine downtime, service unavailability, or network failure, a timestamped transaction identifier serves as a fallback, ensuring a valid database connection, preventing transaction loss, and ensuring no duplicate or missing data awaiting synchronization. Furthermore, users are unaware of this in the cloud. Once the source dataset and the host machine are back to normal operation, data synchronization can continue, thus guaranteeing data synchronization security.

[0099] Furthermore, in terms of performance, this embodiment of the invention is compared with a method that does not use message queues for storage optimization, does not use distributed synchronization optimization, and only uses binlog mechanism for synchronization. Under the condition of a single customer's cloud connection, 300,000 read (get) and write (set) operations were performed on data with value sizes of 1KB and 10KB respectively, and the following test results table (unit: ms) was obtained:

[0100] plan Order Value 1k Value 10k binlog only get 0.82 1.34 This plan get 0.19 0.36 binlog only set 0.91 1.69 This plan set 0.24 0.49

[0101] It can be seen that, regardless of the operation method used, the solution proposed in this embodiment of the invention has significantly better performance than the solution that does not use message queues for storage optimization, does not use distributed synchronization optimization, and only uses the binlog mechanism, with a speed improvement of about 4 times.

[0102] Figure 4 This is a structural block diagram of a data synchronization device 80 provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device may include:

[0103] The timestamp writing module 801 is used to respond to a data update operation on the source dataset in the distributed database by using the execution timestamp of the data update operation as the operation identifier of the update log and writing it to the source log file of the source dataset; the source log file records the update log, and the update log is used to record the data update operation;

[0104] The synchronization module 802 is used to sequentially synchronize the update logs in the source log file to a specified directory of the target dataset to be synchronized in the distributed database according to the operation identifier, so as to obtain the target log file; the data stored in the source dataset is related to the data stored in the target dataset;

[0105] The update module 803 is used to update the data in the target dataset based on the target log file.

[0106] Optionally, the data update operation is used to instruct the data of the target attribute of the first object in the source dataset to be updated; the update module is specifically used for:

[0107] Obtain the target data corresponding to the target attribute of the target object in the target dataset; the target object is an object in the target dataset that contains the first object.

[0108] The target data is updated sequentially based on the update logs in the target log file, according to the order represented by the operation identifiers of each update log in the target log file.

[0109] Optionally, the server cluster includes real-time nodes, which are used to acquire real-time data; the query module is specifically used for:

[0110] The timestamp data carried in the query command is obtained based on the management node in the server cluster;

[0111] Based on the fact that the timestamp data indicates that the query instruction is used to query real-time data, the management node forwards the query instruction to the real-time node;

[0112] The real-time node obtains the data queried by the query instruction based on the query index as the first query result, and returns the first query result to the management node;

[0113] The management node outputs the first query result to the query object.

[0114] Optionally, the server cluster further includes historical nodes for acquiring historical data; the device 80 further includes:

[0115] The instruction forwarding module is used to forward the query instruction to the historical node based on the fact that the timestamp data indicates that the query instruction is used to query historical data.

[0116] The result query module is used to obtain historical data corresponding to the timestamp based on the query index by the historical node, use it as the second query result, and return the second query result to the management node;

[0117] The result output module is used by the management node to output the second query result to the query object.

[0118] Optionally, the synchronization module is specifically used for:

[0119] Based on the operation identifier of each update log in the source log file, determine the execution order of the update operations recorded in each update log;

[0120] In accordance with the execution order, the update logs in the source log files are copied sequentially to the specified directory of the target dataset.

[0121] Optionally, the device 80 further includes:

[0122] The metadata acquisition module is used to acquire the metadata of the source log file before the update log in the source log file is sequentially synchronized to the specified directory of the target dataset to be synchronized in the distributed database according to the operation identifier.

[0123] The backup module is used to back up the metadata to a specified location;

[0124] The reading module is used to reread the source log file based on the metadata at the specified location before determining the execution order of the update operations recorded in each update log, in the case where the source log file cannot be read successfully.

[0125] In summary, in response to a data update operation on a source dataset in a distributed database, this embodiment of the invention uses the execution timestamp of the data update operation as the operation identifier of the update log and writes it to the source log file of the source dataset. The source log file records the update log, which is used to record the data update operation. The update log in the source log file is sequentially synchronized to a designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, to obtain a target log file. The data stored in the source dataset is associated with the data stored in the target dataset. The data in the target dataset is updated based on the target log file. Thus, by using the execution timestamp of the data update operation as the operation identifier of the update log, and sequentially synchronizing the update log in the source log file to the designated directory of the target dataset to be synchronized in the distributed database according to the operation identifier, the target dataset can obtain a target log file recorded according to the operation identifier. This ensures that the log synchronization process is consistent with the execution order of each data update operation. Furthermore, updating the data in the target dataset according to the target log file ensures that the data update process is consistent with each data update operation, thereby guaranteeing the accuracy and consistency of data synchronization.

[0126] In another embodiment of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0127] Memory, used to store computer programs;

[0128] The processor, when executing a program stored in memory, performs any of the data synchronization methods described above.

[0129] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the data synchronization methods described in the above embodiments.

[0130] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data synchronization methods described in the above embodiments.

[0131] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0132] It should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data synchronization method applied to a server cluster, characterized in that, The method comprises: In response to a data update operation on a source data set in a distributed database, an execution timestamp of the data update operation is taken as an operation identifier of an update log and written into a source log file of the source data set; the source log file records the update log used to record the data update operation; The update log in the source log file is sequentially synchronized into a specified directory of a target data set to be synchronized in the distributed database according to the operation identifier, so as to obtain a target log file; the data stored in the source data set is associated with the data stored in the target data set; Data in the target data set is updated based on the target log file.

2. The method of claim 1, wherein, The method further comprises: A thread pool comprising a plurality of threads is created, and the update log in the target log file is written into a preset message queue based on the plurality of threads; The update log in the preset message queue is read to create a query index based on the update log; In a case where a query instruction of a query object to the distributed database is received, a query is performed based on the query index.

3. The method of claim 1, wherein, The data update operation is used to indicate that data of a target attribute of a first object in the source data set is updated; the updating of data in the target data set based on the target log file comprises: Target data corresponding to the target attribute of a target object in the target data set is obtained; the target object is an object in the target data set containing the first object; The target data is sequentially updated based on the update log in the target log file according to an order represented by the operation identifier of each update log in the target log file.

4. The method of claim 2, wherein, The server cluster comprises a real-time node used to obtain real-time data; The performing of the query based on the query index in a case where the query instruction of the query object to the distributed database is received comprises: Timestamp data carried in the query instruction is obtained based on a management node in the server cluster; In a case where the timestamp data represents that the query instruction is used to query real-time data, the query instruction is forwarded to the real-time node based on the management node; Data indicated by the query instruction is obtained as a first query result based on the query index by the real-time node, and the first query result is returned to the management node; The first query result is output to the query object by the management node.

5. The method of claim 4, wherein, The server cluster further comprises a historical node used to obtain historical data; the method further comprises: In a case where the timestamp data represents that the query instruction is used to query historical data, the query instruction is forwarded to the historical node based on the management node; Historical data corresponding to the timestamp is obtained as a second query result based on the query index by the historical node, and the second query result is returned to the management node; The second query result is output to the query object by the management node.

6. The method of claim 1, wherein, The method comprises the following steps: According to the operation identifier, the execution order of the update operation recorded by each update log in the source log file is determined. According to the execution order, the update log in the source log file is sequentially copied to the specified directory of the target data set.

7. The method of claim 6, wherein, Before the update log in the source log file is sequentially synchronized to the specified directory of the target data set in the distributed database according to the operation identifier, the method further comprises the following steps: Obtain the metadata of the source log file; Backup the metadata to a specified location; Before determining the execution order of the update operation recorded by each update log, the method further comprises the following steps: In the case where the source log file is not successfully read, the source log file is re-read based on the metadata of the specified location.

8. A data synchronization apparatus applied to a server cluster, characterized in that, The device comprises: A timestamp writing module is configured to, in response to a data update operation on a source data set in a distributed database, write an execution timestamp of the data update operation as an operation identifier of an update log into a source log file of the source data set; the source log file records the update log, and the update log is used to record the data update operation. A synchronization module is configured to sequentially synchronize the update log in the source log file to a specified directory of a target data set to be synchronized in the distributed database according to the operation identifier, so as to obtain a target log file; the data stored in the source data set is associated with the data stored in the target data set. An update module is configured to update the data in the target data set based on the target log file.

9. The apparatus of claim 8, wherein, The device further comprises: A writing module is configured to create a thread pool comprising a plurality of threads, and write the update log in the target log file into a preset message queue based on the plurality of threads; An index creating module is configured to read the update log in the preset message queue, so as to create a query index based on the update log; A query module is configured to, in the case where a query instruction of a query object to the distributed database is received, perform a query based on the query index.

10. The apparatus of claim 8, wherein, The data update operation is used to indicate that the data of a target attribute of a first object in the source data set is updated; the update module is specifically configured to: Obtain target data corresponding to the target attribute of a target object in the target data set; the target object is an object in the target data set comprising the first object; According to the order represented by the operation identifier of each update log in the target log file, sequentially update the target data based on the update log in the target log file.

11. The apparatus of claim 9, wherein, The server cluster comprises a real-time node configured to obtain real-time data; the query module is specifically configured to: Obtain timestamp data carried in the query instruction based on a management node in the server cluster; forwarding the query instruction to the real-time node based on that the management node determines that the timestamp data indicates that the query instruction is used to query real-time data; obtaining, by the real-time node, data indicated by the query instruction as a first query result based on the query index, and returning the first query result to the management node; outputting, by the management node, the first query result to the query object.

12. The apparatus of claim 11, wherein, The server cluster further comprises a history node configured to obtain historical data; the device further comprises: an instruction forwarding module configured to forward the query instruction to the history node based on that the management node determines that the timestamp data indicates that the query instruction is used to query historical data; a result querying module configured to obtain, by the history node, historical data corresponding to the timestamp as a second query result based on the query index, and return the second query result to the management node; a result outputting module configured to output, by the management node, the second query result to the query object.

13. The apparatus of claim 8, wherein, The synchronization module is specifically configured to: determine an execution order of update operations recorded by each of the update logs in the source log file according to the operation identifiers of the update logs; copy the update logs in the source log file to the specified directory of the target data set in the execution order.

14. An electronic device, comprising: The server comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method in any one of claims 1-7.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1-7.

Images