Data synchronization method and system

Abstract

The embodiment of the present specification provides a data synchronization method and system, which obtains a target data processing operation by acquiring and analyzing operation log records of a source database; determines a source database table corresponding to the target data processing operation, and determines a target to-be-executed queue group according to the source database table; in the case that a target database table corresponding to the source database table contains a target column, determines a target to-be-executed queue from the target to-be-executed queue group, and puts the target data processing operation into the target to-be-executed queue; in the case that it is determined that the target to-be-executed queue satisfies a preset batch accumulation condition, determines a to-be-executed operation group according to at least one target data processing operation in the target to-be-executed queue, so as to execute at least one target data processing operation in the to-be-executed operation group in the target database; and through a batch accumulation mode of database table and target column levels, high-performance and low-disturbance data synchronization is realized in the target database.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data synchronization method. One or more embodiments of this specification also relate to a data synchronization system, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] When synchronizing data changes from the source database to the analytical database by relying on the incremental log records corresponding to the source database, the write time of the analytical database does not increase significantly with the amount of data when writing a certain range of data at once. Therefore, the data processing operations corresponding to the incremental log records can be appropriately batch-processed to obtain higher write efficiency.

[0003] While batching data processing operations by database table can accumulate a large volume of data writes, to ensure data consistency before and after synchronization, each database table corresponds to a write thread. This can lead to the write thread of a particular database table becoming a system bottleneck when a large number of data processing operations are performed on that table, resulting in synchronization delays. On the other hand, batching data processing operations by merge key results in a batch containing data with the same merge key, making the amount of data written in a single instance relatively small. Given that the time taken to write large and small batches of data is similar within a certain range for analytical databases, this batching method can negatively impact the performance of analytical databases. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data synchronization method. One or more embodiments of this specification also relate to a data synchronization system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical defects in the prior art where unreasonable batching methods lead to synchronization delays in the target database or affect the performance of the target database.

[0005] According to a first aspect of the embodiments of this specification, a data synchronization method is provided, comprising: Obtain and parse the operation log records of the source database to obtain the target data processing operations; Determine the source database table corresponding to the target data processing operation, and determine the target queue group to be executed based on the source database table; If the target database table corresponding to the source database table contains a target column, a target execution queue is determined from the target execution queue group, and the target data processing operation is placed into the target execution queue, wherein the target database table is a database table in the target database; If the target queue to be executed is determined to meet the preset batching conditions, an operation group to be executed is determined based on at least one target data processing operation in the target queue to be executed, so as to execute at least one target data processing operation in the operation group to be executed in the target database.

[0006] According to a second aspect of the embodiments of this specification, a data synchronization system is provided, including an acquisition and parsing unit, a group allocation unit, a queue allocation unit, and an execution determination unit, wherein... The acquisition and parsing unit is used to acquire and parse the operation log records of the source database to obtain the target data processing operation; The group allocation unit is used to determine the source database table corresponding to the target data processing operation, and to determine the target queue group to be executed based on the source database table; The queue allocation unit is used to determine the target queue to be executed from the target queue group when the target database table corresponding to the source database table contains the target column, and to put the target data processing operation into the target queue to be executed, wherein the target database table is a database table in the target database; The execution determination unit is used to determine an operation group to be executed based on at least one target data processing operation in the target queue to be executed, when it is determined that the target queue to be executed meets the preset batching conditions, so as to execute at least one target data processing operation in the target database.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data synchronization method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described data synchronization method.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data synchronization method.

[0010] One embodiment of this specification provides a data synchronization method that obtains and parses the operation log records of the source database to obtain the target data processing operation. By executing the target data processing operation in the target database, the target database can undergo the same data changes as the source database, thereby achieving data synchronization between the source and target databases. Specifically, when the target data processing operation needs to be batch-processed and then written to the target database, a target execution queue group is determined based on the source database table corresponding to the target data processing operation. The source database table corresponds to the target database table in the target database. If it is determined that the target database table contains a target column, the target column is determined from the target execution queue group. The system identifies a queue to be executed, placing target data processing operations into appropriate queues. If the queue meets preset batching conditions, a group of operations to be executed is determined based on at least one target data processing operation in the queue. This enables batching of data for at least one target data processing operation in the queue. When target data processing operations are allocated to the queue based on database tables and target columns, this hierarchical batching method, using database tables and target columns, allows for batching of as many data processing operations as possible while maintaining data synchronization consistency, achieving high-performance, low-disturbance data synchronization in the target database. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating a data synchronization method provided in one embodiment of this specification. Figure 2 This is a flowchart illustrating a data synchronization method provided in one embodiment of this specification; Figure 3 This is a schematic diagram illustrating the processing procedure of a data synchronization method for analytical databases provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a data synchronization system provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0017] Incremental Log: The operation log of the source database, which records all modification operations on the data stored in the database in a rolling manner, including the complete content after data is written (corresponding to data insertion operation), before and after data is updated (corresponding to data update operation), and before data is deleted (corresponding to data deletion operation).

[0018] Merge key: A column specified by the user in an analytical database table. For data with the same merge key, the database will retain the latest data according to the order in which they were written.

[0019] Data retrieval job: Responsible for retrieving operation logs from the source database. Data replay job: Responsible for replaying the parsed operation logs in the target database.

[0020] This specification provides a data synchronization method, and also relates to a data synchronization system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0021] See Figure 1 , Figure 1 A schematic diagram of a data synchronization method provided according to an embodiment of this specification is shown. Specifically, the data synchronization method is applied to database system 104. End device 102 is used to send a data processing operation request to database system 104. Database system 104 includes source data and target database. The data processing operation request is a data processing operation request for the source database. When it is determined that the source database has received the data processing operation request, the data processing operation request is executed to obtain the data processing result and send the data processing result response to end device 102.

[0022] After the source database completes the data processing operation request, a corresponding operation log record will be generated in the incremental log. To achieve data synchronization between the source and target databases, the operation log record of the source database is obtained and parsed to obtain the target data processing operation. The source database table corresponding to the target data processing operation is determined, and a target queue group to be executed is determined based on the source database table. If the target database table corresponding to the source database table contains the target column, a target queue to be executed is determined from the target queue group, and the target data processing operation is placed into the target queue to be executed. The target database table is a database table in the target database. If the target queue to be executed meets the preset batching conditions, an operation group to be executed is determined based on at least one target data processing operation in the target queue to be executed, so that at least one target data processing operation in the operation group to be executed is executed in the target database.

[0023] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a Real-Time Communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0024] The data synchronization method provided in this specification's embodiments obtains and parses the operation log records of the source database to obtain the target data processing operation. By executing the target data processing operation in the target database, the target database can undergo the same data changes as the source database, thereby achieving data synchronization between the source and target databases. Specifically, when the target data processing operation needs to be batch-processed and then written to the target database, a target execution queue group is determined based on the source database table corresponding to the target data processing operation. The source database table corresponds to the target database table in the target database. If it is determined that the target database table contains the target column, it is determined from the target execution queue group. The target pending queue is used to place target data processing operations into appropriate pending queues. When the target pending queue meets the preset batching conditions, the pending operation group is determined based on at least one target data processing operation in the target pending queue. This achieves data batching for at least one target data processing operation in the target pending queue. When the target data processing operations are allocated to the target pending queue based on database tables and target columns, the hierarchical data batching method based on database tables and target columns can batch data for as many data processing operations as possible while ensuring data synchronization consistency, achieving high-performance and low-disturbance data synchronization in the target database.

[0025] See Figure 2 , Figure 2 A flowchart of a data synchronization method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0026] Step 202: Obtain and parse the operation log records of the source database to obtain the target data processing operations.

[0027] In this specification, the source database can be understood as the original database from which the data to be synchronized originates, serving as the data source for synchronization. This source database is capable of read and write operations; data changes (addition, deletion, and updates) occur in the source database, and these changes are then synchronized to the target database. The target database can be understood as the database that receives synchronized data from the source database; its data content is synchronized from the source database. The target database achieves data synchronization with the source database by performing the same data changes as the source database. In this embodiment, an analytical database is used as an example to illustrate the data synchronization method in detail.

[0028] Specifically, taking data synchronization for data backup as an example, when implementing data backup between a source database and a target database, it is first necessary to ensure that the data structure of the target database is compatible with the data structure of the source database. Then, the backup data is migrated from the source database to the target database. During the data migration process, the source database remains running and continues to receive and execute data processing operations on the backup data. It can also write multiple operation log records to the log file based on the executed data processing operations. Therefore, based on the migration of the backup data from the source database to the target database, incremental data migration can continue based on the operation log records. During the incremental data migration process, the source database continues to run, so that the operation log records in the log file are updated in real time. The data synchronization in the embodiments of this specification refers to incremental data synchronization (corresponding to incremental data migration).

[0029] In practical applications, operation log records can be understood as operation log records in incremental logs. Specifically, incremental logs can be real-time updated online database transaction logs, such as the database binary log MySQL binlog, the transaction log of the relational database SQL Server, the redo log of the relational database, etc., without being limited here.

[0030] The operation log records include, but are not limited to, timestamps, operation types, and related data information. Operation types can include data insertion (writing), data deletion, and data update. Since data reading or querying operations do not change the data, the operation type here may not include data querying or data reading operations. The data information includes the name of the source database table corresponding to the data processing operation recorded in the operation log, and the complete data content before and after the data processing operation. For example, if the data processing operation is a data insertion operation, the data information includes the complete row data after the data is written; if the data processing operation is a data update operation, the data information includes the complete row data before and after the data is written; and if the data processing operation is a data deletion operation, the data information includes the complete row data before the data is deleted.

[0031] For example, the operation log record corresponding to a data processing operation is as follows: Type: UPDATE_ROWS_EVENT (Operation type: Row update) Table: test.orders (Table name: test.orders) Timestamp: 2023-10-01 14:30:00 (Timestamp: 2024-10-1 14:30:00) Before: {order_id=2001, status='paid', amount=100.00} (Previously: {order ID=2001, status='paid', amount=100.00}) After: {order_id=2001, status='shipped', amount=100.00} (After modification: {order_id=2001, status='shipped', amount=100.00}) When a data synchronization event is triggered, the data retrieval job retrieves the operation log records from the source database, parses them to obtain the target data processing operation corresponding to the operation log record, and the target data processing operation can be understood as the data processing operation to be executed in the target database. The target database achieves data changes consistent with the source database by executing the target data processing operation, thereby achieving data synchronization with the source database.

[0032] It should be noted that the incremental logs of the source database can be monitored in real time, thereby triggering a data synchronization event whenever a new operation log record is added to the incremental logs, and the data retrieval job retrieves the operation log record from the source database; alternatively, a preset time interval can be set according to actual conditions, and the incremental logs will retrieve the operation log records added within the preset time interval after each preset time interval; of course, the data synchronization event can also be triggered by sending a data synchronization command as needed, which is not limited here. In the embodiments of this specification, taking the real-time synchronization between the source database and the target database as an example, in the implementation of this specification, the data retrieval job will continuously retrieve operation log records from the source database in the order in which the operation log records are generated.

[0033] Step 204: Determine the source database table corresponding to the target data processing operation, and determine the target queue group to be executed based on the source database table.

[0034] In this context, each target queue group corresponds to a source database table, and the target data processing operations corresponding to the source database table will be assigned to the target queue group corresponding to the source database table.

[0035] Specifically, when the operation log records involve the source database table for the operation, the target data processing operation obtained by parsing the operation log records will correspond to a source database table. This allows us to determine the target queue group to be executed for the target data processing operation, and subsequently assign the target data processing operation to the queue to be executed in the target queue group.

[0036] In one or more embodiments of this specification, when data synchronization is performed, the table structures of the source database and the target database are similar, that is, the source database table corresponds to the target database table in the target database. Therefore, when the source database table corresponds to a target queue group to be executed, the target database table corresponding to the source database table also corresponds to the target queue group to be executed. At this time, the target queue group to be executed can be accurately determined based on the source identifier of the source database table and / or the target identifier of the target database table. Specific implementation methods are described below: The step of determining the target queue group to be executed based on the source database table includes: The target database table in the target database is determined based on the source database table, wherein there is a corresponding relationship between the source database table and the target database table; The target queue group to be executed is determined based on the source identifier of the source database table and / or the target identifier of the target database table.

[0037] Specifically, the source identifier of the source database table is a data used to uniquely identify the source database table. The source database table can be determined based on this source identifier, for example, the source identifier can be the source table name of the source database table; similarly, the target identifier of the target database table is a data used to uniquely identify the source database table. The target database table can be determined based on this target identifier, for example, the target identifier can be the target table name of the target database table.

[0038] In practice, the target database table in the target database can be determined based on the source database table. That is, if the source database table undergoes data changes, the target database table in the target database will undergo the same data changes as the source database table.

[0039] When the source database table corresponds to the target database table and the target queue group to be executed, the target database table corresponds to the target queue group to be executed. Therefore, the target queue group to be executed can be determined based on the source database table or the target database table. When the target queue group to be executed is determined based on the source database table and / or the target database table, the target queue group to be executed is accurately determined based on the source identifier of the source database table and / or the target identifier of the target database table.

[0040] The data synchronization method provided in the embodiments of this specification, when the source database table corresponds to the target database table and the source database table corresponds to the target queue group to be executed, there is also a correspondence between the target database table and the target queue group to be executed. Therefore, the target queue group to be executed can be accurately determined based on the source identifier of the source database table and / or the target identifier of the target database table.

[0041] Step 206: If the target database table corresponding to the source database table contains the target column, determine the target execution queue from the target execution queue group, and put the target data processing operation into the target execution queue, wherein the target database table is a database table in the target database.

[0042] In this context, the target column can be understood as a merge key, such as the primary key of the target database; the target queue to be executed can be understood as the queue to which the target data processing operations will be assigned; in the embodiments of this specification, different queues to be executed are used for concurrent processing, that is, data processing operations in different queues to be executed can be written to the target database concurrently.

[0043] Specifically, before data synchronization, the table structure of the target database table can be obtained, thereby identifying the target column among multiple columns in the target database table. Multiple execution queues can be set in the target execution queue group, and the concurrency of writing can be controlled by setting the number of execution queues in the execution queue group. However, if the target database table does not have a target column, it is usually a scenario for recording transactional information, which is not suitable for concurrent writing. Therefore, setting one execution queue in the target execution queue group is sufficient.

[0044] That is, based on different target queue groups, data processing operations between different target database tables can be processed concurrently, and based on different target queues in the target queue group, data processing operations on the same target database table but different target columns can also be processed concurrently.

[0045] In practice, if the target database table contains a target column, the target execution queue is determined from multiple execution queues in the target execution queue group, thereby enabling the target data processing operation to be placed into the target execution queue.

[0046] In one or more embodiments of this specification, since the operation log records contain complete data content before and after the data processing operation, the target value of the target column corresponding to the target data processing operation can be determined based on the operation log records. Therefore, by hashing the target value, the target queue to be executed corresponding to the target data processing operation can be determined. Specific implementation methods are described below: When the target database table corresponding to the source database table contains a target column, determining the target queue to be executed from the target queue group includes: If the target database table corresponding to the source database table contains a target column, the target value corresponding to the target data processing operation is determined according to the operation log record, wherein the target value is the value of the target column in the target data processing operation; The target queue to be executed is determined from multiple queues to be executed in the target queue group by hashing the target value.

[0047] If the target database table corresponding to the source database table does not contain the target column, the target data processing operation is placed in one of the execution queues contained in the target execution queue group.

[0048] Specifically, as described in the above embodiments, the data information corresponding to the operation log records may include data from a complete row. Therefore, if a target column exists in the target database table, the value of the target column to be processed by the target data processing operation can be determined based on the operation log records.

[0049] For example, if the target column in the target database table is order_id (order number), and the operation log contains a complete row of data with "order_id=2001,status='paid',amount=100.00", then the target value corresponding to the target data processing operation can be determined to be 2001.

[0050] In practical applications, when a target column exists in the target database table, multiple execution queues can be set in the target execution queue group. For example, the target execution queue group can contain three execution queues. When the target value is in the range of 1-1000, a hash algorithm is used to calculate the target value, and the hash result is mapped to the first execution queue, thus allocating the target data processing operation to the first execution queue of the target execution queue group. When the target value is in the range of 1001-2000, a hash algorithm is used to calculate the target value, and the hash result is mapped to the second execution queue, thus allocating the target data processing operation to the second execution queue of the target execution queue group. When the target value is in the range of 2001-3000, a hash algorithm is used to calculate the target value, and the hash result is mapped to the third execution queue, thus allocating the target data processing operation to the third execution queue of the target execution queue group.

[0051] Data processing operations on different target columns within the same target database table can also be performed concurrently, based on different target queues in the target queue group.

[0052] For example, if three target data processing operations are obtained (first data processing operation, second data processing operation, and third data processing operation), it is determined that the first data processing operation and the second data processing operation correspond to the first source database table, while the third data processing operation corresponds to the second source database table. The first source database table corresponds to the first target queue group to be executed, and the second source database table corresponds to the second target queue group to be executed.

[0053] If the first target database table corresponding to the first source database table contains a target column, and the target value corresponding to the first data processing operation is 1005 and the target value corresponding to the second data processing operation is 2009, then the first data processing operation will be assigned to the second queue of the first target queue group, and the second data processing operation will be assigned to the third queue of the first target queue group. If the second target database table corresponding to the second source database table does not contain a target column, then the second target queue group will contain one queue (for example, the fourth queue), and the third data processing operation will be assigned to the fourth queue.

[0054] It should be noted that, for data with the same merge key, the target database will retain the latest data according to the order of writing. Here, the latest data can be understood as the latest version of the data. For example, if the data corresponding to "order_id=2001" above is updated twice, the first update will obtain the first version of the data, and the second update will obtain the second version of the data based on the first version of the data. At this time, the target database will retain the latest second version of the data.

[0055] The data synchronization method provided in this specification, by hashing the target value, can place data processing operations that process the same target column value into the same execution queue. Data processing operations in the same execution queue will be written to the target database in chronological order, while data processing operations in different execution queues will be written to the target database concurrently. That is, through hashing, it is ensured that data processing operations that process the same target column value will not be assigned to different execution queues. Therefore, it avoids the problem of data inconsistency caused by concurrent execution of data processing operations in different execution queues when data processing operations that process the same target column value are assigned to different execution queues, thus ensuring the consistency of data synchronization.

[0056] In one or more embodiments of this specification, the initial data processing operation corresponding to the source database can be obtained by parsing the operation log records. The initial data processing operation is then split and judged to determine whether it meets preset splitting conditions. Based on the judgment result, the target data processing operation is determined based on the initial data processing operation. Specific implementation methods are as follows: The process of acquiring and parsing the operation log records of the source database to obtain the target data processing operations includes: Obtain the operation log records from the source database, parse and process the operation log records to obtain the initial data processing operations; The initial data processing operation is broken down and judged, the judgment result is determined, and the target data processing operation is determined based on the judgment result.

[0057] The initial data processing operation can be understood as the data processing operation performed by the source database.

[0058] Specifically, when the operation log records of the source database are obtained, the operation log records are parsed and transformed into explicit and operable initial data processing operations. That is, the source database realizes the data changes corresponding to the operation log records by executing the initial data processing operations.

[0059] The initial data processing operation is split and determined, that is, whether the initial data processing operation meets the preset splitting conditions and can be split. Based on the result of splitting or not splitting, the target data processing operation is determined. In practical applications, if the result is not to split, the initial data processing operation is determined as the target data processing operation; if the result is to split, the initial data processing operation is split into the target data processing operation.

[0060] The data synchronization method provided in the embodiments of this specification determines the initial data processing operation based on the operation log record, splits and judges the initial data processing operation, and determines the target data processing operation based on the judgment result. In this way, when the target data processing operation is allocated to a queue, it can provide data support for accurate queue allocation.

[0061] In one or more embodiments of this specification, a determination is made as to whether the initial data processing operation should be split based on the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation, thereby obtaining an accurate determination result. Specific implementation methods are described below: The step of splitting and determining the initial data processing operation to obtain the determination result includes: Based on the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation, the initial data processing operation is split and judged to determine the judgment result. The data information corresponding to the initial data processing operation is similar to the data information mentioned above. The data information includes, but is not limited to, the source database tables involved in the initial data processing operation and the complete data content before and after the initial data processing operation.

[0062] Specifically, if the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation meet the preset splitting conditions, it is determined that the initial data processing operation needs to be split, thereby obtaining a splitting determination result; if the operation type of the initial data processing operation and / or the data information corresponding to the initial data processing operation do not meet the preset splitting conditions, it is determined that the initial data processing operation does not need to be split, thereby obtaining a non-split determination result.

[0063] In practice, since the target queue group to be executed needs to be determined based on the source database table, and the target queue to be executed needs to be determined from the target queue group if the target database table corresponding to the source database table contains the target column, if the initial data processing operation is a data update operation involving updating the target column value, this initial data processing operation will involve the target column value before the update and the target column value after the update. Therefore, there is a risk that the determined target queue to be executed may be inaccurate. That is, the target queue to be executed determined from the target queue group may be different based on the original target column value (the target column value before the update) or the updated target column value (the target column value after the update). For example, when the target column is the order number column, the initial data processing operation is to update the first order number to the second order number. The first order number corresponds to the first target queue to be executed in the target queue group, and the second order number corresponds to the second target queue to be executed in the target queue group. This may result in the same initial data processing operation being assigned to different queues to be executed.

[0064] Therefore, to ensure the accuracy of queue allocation, the data update operation involving the updating of the target column value is split into the data deletion operation of the original target column value and the data insertion operation of the updated target column value. Based on the original target column value involved in the data deletion operation, the data deletion operation is allocated to the first target queue corresponding to the original target column value, and based on the updated target column value involved in the data insertion operation, the data insertion operation is allocated to the second target queue corresponding to the updated target column value.

[0065] Therefore, in one or more embodiments of this specification, the preset splitting condition can be understood as the condition that the operation type of the initial data processing operation is a data update operation, and the target column is determined to be updated based on the data information; that is, when the operation type of the initial data processing operation is a data update operation, and the target column is determined to be updated based on the data information corresponding to the initial data processing operation, the determination result is splitting. The specific implementation method is as follows: The step of splitting and determining the initial data processing operation based on the operation type and the corresponding data information of the initial data processing operation, and determining the determination result, includes: If the operation type of the initial data processing operation is determined to be a data update operation, and the target column is determined to have a value update based on the data information corresponding to the initial data processing operation, then the determination result is determined to be splitting. Otherwise, the determination result is determined to be no splitting.

[0066] Specifically, the splitting determination is based on the operation type of the initial data processing operation, whether the target database table has a merge key, and whether the value of the merge key has been updated. If the initial data processing operation is a data update operation, the target database table has a merge key, and the value of the merge key has been updated, then the data update operation is split into a data deletion operation and a data insertion operation in sequence. In other cases, the integrity of the initial data processing operation is maintained and no splitting is performed.

[0067] In practical applications, when the initial data processing operation is a data update operation, it is further determined whether the target column has been updated. Since the data information includes, but is not limited to, the source database table involved in the initial data processing operation and the complete data content before and after the initial data processing operation, the source database table can be identified based on the data information. Then, the corresponding target database table can be determined. If the target database table contains a merge key, based on the complete data content before and after the initial data processing operation in the data information, it is determined whether the data update operation updates the value of the merge key. If so, the initial data processing operation is then split. Alternatively, if the target database table is determined to contain a merge key, it can be further determined whether the initial data processing operation is a data update operation and whether the data update operation involves updating the value of the merge key; this is not limited here.

[0068] The data synchronization method provided in this specification determines whether to split the initial data processing operation based on the operation type of the initial data processing operation, whether the target database table has a merge key, and whether the value of the merge key has been updated. This accurately determines the target data processing operation. Based on the allocation of the queue to be executed for the target data processing operation, the accuracy of the queue allocation can be guaranteed.

[0069] In one or more embodiments of this specification, determining the target data processing operation based on the determination result includes: If the determination result is split, the initial data processing operation is split to obtain the target data processing operation; If the determination result is no splitting, the initial data processing operation is determined as the target data processing operation.

[0070] Specifically, when the determination result is splitting, the initial data processing operation is split to obtain the target data processing operation. When the initial data processing operation is a data update operation and splitting is performed, the data update operation is split into sequential data deletion operation and data insertion operation. For specific implementation, please refer to the above embodiment, which will not be repeated here.

[0071] If the determination result is no splitting, then the initial data processing operation is directly identified as the target data processing operation. In current practical applications, the initial data processing operation is an operation on the source database table, while the target data processing operation is the operation that the target database will perform. To ensure that the target database can correctly execute the target data processing operation, the initial data processing operation can be transformed to obtain the target data processing operation. For example, if the source database table to be processed by the initial data processing operation is orders_1, and the target database table corresponding to the source database table orders_1 is C_orders_1, then the table names of the source database table involved in the initial data processing operation need to be converted to the corresponding table names of the target database table.

[0072] The data synchronization method provided in the embodiments of this specification, when the determination result is split, obtains sequential data deletion and data insertion operations by splitting the initial data processing operation, and when the target data processing operation is determined based on the initial data processing operation, it can perform reasonable transformation processing on the initial data processing operation to ensure that the target database can correctly execute the target data processing operation.

[0073] Step 208: If the target queue to be executed meets the preset batching conditions, determine the operation group to be executed based on at least one target data processing operation in the target queue to be executed, so as to execute at least one target data processing operation in the operation group to be executed in the target database.

[0074] The pending operation group can be understood as a batch of data processing operations obtained through data batching.

[0075] Specifically, when the target queue to be executed meets the preset batching conditions, at least one target data processing operation in the target queue to be executed is identified as a group of operations to be executed and taken out from the target queue to be executed. Subsequently, at least one target data processing operation in the group of operations to be executed is executed sequentially in the target database, thereby realizing data synchronization between the source database and the target database.

[0076] In one or more embodiments of this specification, the insertion time of the target data processing operation at the target position in the target queue is checked. If the time interval between the insertion time and the current time exceeds a preset time interval, it is determined that the target queue meets the preset batching conditions. Specific implementation methods are as follows: The step of determining that the target queue to be executed meets the preset batching conditions includes: Determine the target data processing operation at the target position in the target queue to be executed, and determine the time when the target data processing operation at the target position is placed into the target queue to be executed; If the time interval between the insertion time and the current time exceeds a preset time interval, the target queue to be executed is determined to meet the preset batching conditions.

[0077] The target data processing operation at the target location is the first target data processing operation placed in the current target queue to be executed.

[0078] In specific implementation, taking the target position as the head of the queue as an example, since the target data processing operation in the current target queue will be taken out when the target queue meets the preset batching conditions, there is no target data processing operation in the target queue at this time. However, as the operation log records are continuously obtained, new target data processing operations will be assigned to the currently empty target queue. At this time, the first target data processing operation put into the current target queue is placed at the head of the target queue, and the time of putting the target data processing operation is determined.

[0079] In this embodiment of the specification, a timeout concurrency determiner is provided. The timeout concurrency determiner sets a time interval (i.e., a preset time interval). The timeout concurrency determiner continuously scans the target data processing operations at the head of the target queue to be executed, and determines whether the time interval between the insertion time and the current time exceeds the preset time interval. If it exceeds the preset time interval, it is determined that the target queue to be executed has reached the batching time, that is, the target queue to be executed meets the preset batching condition, thereby batching the target data processing operations in the target queue to be executed into a group of operations to be executed.

[0080] In practical applications, the preset batching condition can also be the number of target data processing operations contained in the target queue to be executed. That is, if the target queue to be executed contains a preset number of target data processing operations, it is determined that the target queue to be executed meets the preset batching condition. In other words, the preset batching condition can be set according to the actual situation and is not limited here.

[0081] The data synchronization method provided in this specification can achieve controllable batching synchronization of data based on a preset time interval by determining whether the time interval between the insertion time and the current time exceeds a preset time interval and whether the target queue to be executed meets the preset batching conditions.

[0082] In one or more embodiments of this specification, when an operation group to be executed is determined from a target queue to be executed, the operation group to be executed will be placed into a concurrent execution queue. This concurrent execution queue contains operation groups to be executed from different target queues to be executed. Since data processing operations within the same target queue to be executed cannot be executed concurrently, the operation group currently retrieved from the target queue to be executed is placed into the concurrent execution queue only when no operation group to be executed from the target queue is present in the concurrent execution queue. Specific implementation methods are described below: After determining the group of operations to be executed based on at least one target data processing operation in the target queue to be executed, the method further includes: If the target queue does not contain a group of operations to be executed, the group of operations to be executed is placed in the concurrent execution queue, wherein the concurrent execution queue is used to enable the target database to retrieve the group of operations to be executed.

[0083] Specifically, the target database will obtain the operation groups to be executed from the concurrent execution queue. In the case of different execution queues, there are different operation groups to be executed in the concurrent execution queue. The target database can concurrently execute the data processing operations in these different operation groups to be executed.

[0084] Specifically, as described in the above embodiments, when the target queue to be executed meets the preset batching conditions, the batch-obtained batch of operations to be executed is placed into the concurrent execution queue. If the batch of operations to be executed has not yet been completed and the target queue to be executed meets the preset batching conditions again, a new batch of operations to be executed is obtained. At this time, since the batch of operations to be executed corresponding to the target queue to be executed in the concurrent execution queue has not yet been completed, the new batch of operations to be executed cannot be placed into the concurrent execution queue. This is because the data processing operations of the same target queue to be executed should be executed sequentially and cannot be executed concurrently. Therefore, after the batch of operations to be executed in the concurrent execution queue is completed and removed, the new batch of operations to be executed is added to the concurrent execution queue.

[0085] The data synchronization method provided in this specification, when there is no corresponding pending operation group in the concurrent execution queue, adds the pending operation group corresponding to the target pending execution queue obtained in the newly batch to the concurrent execution queue, thereby avoiding concurrent execution of different pending operation groups in the same target pending execution queue and causing data inconsistency.

[0086] In one or more embodiments of this specification, when there are groups of operations to be executed from different queues in the concurrent execution queue, a thread can be set up for each queue. This thread is used to retrieve the group of operations to be executed from the corresponding queue and execute the data processing operations in that group. By concurrently executing multiple threads in the target database, the processing efficiency of data synchronization is improved. Specific implementation methods are described below: The step of performing at least one target data processing operation from the group of operations to be executed in the target database includes: When the concurrent execution queue contains at least two target queues corresponding to operation groups to be executed, at least two target threads corresponding to the at least two target queues to be executed are concurrently executed in the target database, wherein one target queue corresponds to one target thread, and the target thread is used to execute at least one target data processing operation in the operation group to be executed.

[0087] In practical applications, when there are groups of pending operations from different pending queues in the concurrent execution queue, the different groups of pending operations are assigned to threads in the corresponding pending queues for processing. By using concurrent processing threads, data processing operations in different groups of pending operations can be executed concurrently in the target database.

[0088] The data synchronization method provided in this specification ensures data synchronization consistency when a group of pending operations is allocated as a whole to the corresponding thread in the queue for execution. In this case, the data processing operations in the group of pending operations will not be split but executed sequentially, thus ensuring data synchronization consistency. Furthermore, by concurrently processing the batch of pending operations, higher write efficiency can be achieved in the target database.

[0089] The data synchronization method provided in the embodiments of this specification offers a hierarchical batching algorithm based on a database table-merge key. Specifically, it determines the target queue group corresponding to the target data processing operation based on the database table, and determines the target queue from the target queue group based on the merge key of the target database table. With the ability to set the number of queues in the queue group and a preset time interval, it achieves data synchronization with controllable concurrency and batching time. Furthermore, by pre-splitting the initial data processing operation before queue allocation, it ensures that the target queue is allocated based on the accurate target data processing operation, guaranteeing the consistency of data synchronization when applying the hierarchical batching algorithm.

[0090] Specifically, when analytical databases write a certain range of data at once, the write time does not increase significantly with the data volume. Therefore, appropriate batch writing of incremental data can achieve high write efficiency. When the incremental data flow is large, a concurrent write strategy needs to be adopted on the basis of batch writing. This brings two challenges: firstly, when the analytical database writes frequently, there will be significant pressure on merging multiple versions of data, affecting database performance, and the concurrency of writes needs to be controlled; secondly, in order to ensure the consistency of data synchronization, a mechanism is needed to determine whether data can be concurrently replayed.

[0091] Current data batching methods are somewhat limited. Typical methods include batching by database / table name and batching by primary key (a type of merge key) value. In scenarios involving real-time data synchronization to analytical databases, while batching by database / table name can achieve larger batch sizes, it limits the concurrency of writing to a single table. To ensure data consistency before and after synchronization, a single thread is required for writing, which can lead to performance bottlenecks and synchronization delays under high traffic. Batching by primary key value can result in overly scattered data. Although concurrency is high, the batch sizes are small. Since analytical databases take similar time to write large and small batches within a certain range, this strategy can also lead to synchronization delays. Furthermore, high-concurrency writes to the same table can put significant data merging pressure on the analytical database, impacting database performance.

[0092] To address the above problems, this specification provides a data synchronization method for analytical databases. See also... Figure 3 , Figure 3 This diagram illustrates the processing steps of a data synchronization method for analytical databases provided in one embodiment of this specification.

[0093] Specifically, when performing data processing operations on the source database, an operation log record is generated in the source database's log file for the specific data processing operation; the data retrieval job can continuously retrieve and parse the operation log records from the source database in the order in which the operation log records were generated.

[0094] Once the operation log records are obtained, the following log splitting-batch algorithm is used to process the operation log records to obtain a group of operations to be executed in the concurrently executable queue.

[0095] In practice, the initial data processing operation is obtained by parsing the operation log records. A splitting decision-maker determines whether the initial data processing operation can be split, thereby identifying the target data processing operation. Specifically, the splitting decision-maker determines the splitting based on the operation type of the initial data processing operation, whether the corresponding target database table has a merge key (i.e., the target column in the above embodiment), and whether the merge key value has been updated. If the initial data processing operation is a data update operation, the target database table has a merge key, and the merge key value has been updated, then the data update operation is split into a sequential data deletion operation and a data insertion operation; that is, the data deletion operation and the data insertion operation are the target data processing operation. In other cases, the integrity of the initial data processing operation is maintained and it is not split, thus identifying the initial data processing operation as the target data processing operation. It should be noted that since data synchronization is achieved between the source database and the target database, a target database table exists in the target database corresponding to the source database table. This source database table is the database table in the source database that the initial data processing operation targets.

[0096] In practical applications, although this is a breakdown of the initial data processing operation, in reality, when the operation log records are obtained, the processing is based on the operation log records. For example, when the initial data processing operation is broken down into a data deletion operation and a data insertion operation, the operation log records are actually broken down to obtain the first operation log record (corresponding to the data deletion operation) and the second operation log record (corresponding to the data insertion operation). The data processing operation is obtained by parsing from the operation log records.

[0097] The hierarchical queue allocator allocates target data processing operations to corresponding execution queues based on the target database table's table type (categorized by the presence or absence of a merge key), table name, and merge key value. Specifically, when the hierarchical queue allocator receives the target data processing operation after processing by the splitting determiner, it calculates a hash value based on the source database table's table name to determine the group of the execution queue corresponding to the target data processing operation (i.e., the target execution queue group in the above embodiment).

[0098] If the target database table has a merge key, there will be multiple queues in the target execution queue group. The specific queue is determined by hashing the merge key value corresponding to the target data processing operation within the target execution queue group. If the target database table does not have a merge key, it is typically used for scenarios such as recording log information, which are not suitable for concurrent writes and must maintain the same write order as the source database. Therefore, the target execution queue group will contain only one queue. It should be noted that before data synchronization, the table structure of the target database can be obtained to determine if a merge key exists. If a merge key exists, multiple queues can be set in the execution queue group. When data from different queues can be executed concurrently by data replay jobs, the concurrency of writes can be controlled by setting the number of queues in the execution queue group, achieving hierarchical data batching and concurrency control based on the database table and merge key.

[0099] For example, such as Figure 3 As shown, Table 1 in the target database has a merge key, so the queue group to be executed for Table 1 includes two data queues (i.e., queues to be executed): data queue a and data queue b. For Tables 2 and 3, which do not have a merge key, Table 2 corresponds to one data queue and Table 3 corresponds to one data queue.

[0100] The timeout concurrency checker continuously checks the data processing operation at the head of each queue (i.e., the first data processing operation to enter the queue in the current queue) and whether the queue has reached the batching time (i.e., whether the queue meets the preset batching conditions). For example, if the target queue meets the preset batching conditions, the system determines and retrieves the group of operations to be executed based on at least one target data processing operation in the target queue. After retrieval, it checks whether there are any previous batches of operations to be executed from the target queue in the concurrent execution queue (i.e., the concurrent execution queue in the above embodiment). If so, it waits for these previous batches of operations to be replayed and removed before adding a new batch of operations to be executed in sequence to the concurrent execution queue for replay by the data replay job. A single group of operations to be executed will not be split among multiple data replay jobs.

[0101] It should be noted that, similar to the above, although the description here refers to the processing of target data processing operations, that is, adding target data processing operations to the queue to be executed, in actual applications, the processing of target data processing operations actually refers to the processing of target operation log records (including split operation log records). That is, the group of operations to be executed contains at least one target operation log record. When the target analytical database obtains the target operation log record, the target operation log record is parsed to obtain the target data processing operation, and the data synchronization with the source database is achieved by executing the target data processing operation.

[0102] The data synchronization method provided in the embodiments of this specification can pre-split data by designing and implementing a splitting decision-maker, realize hierarchical batching capability of database table-merge key through a hierarchical queue allocator, and write data into the target analytical database in a way that controls the concurrency and batching time, thereby achieving high-performance and low-disturbance real-time data synchronization while ensuring data consistency.

[0103] Corresponding to the above method embodiments, this specification also provides data synchronization system embodiments. Figure 4 A schematic diagram of the structure of a data synchronization system provided in one embodiment of this specification is shown. Figure 4 As shown, the system includes an acquisition and parsing unit 402, a group allocation unit 404, a queue allocation unit 406, and an execution determination unit 408, wherein... The acquisition and parsing unit 402 is used to acquire and parse the operation log records of the source database to obtain the target data processing operation; The group allocation unit 404 is used to determine the source database table corresponding to the target data processing operation, and to determine the target queue group to be executed based on the source database table. The queue allocation unit 406 is used to determine the target queue to be executed from the target queue group when the target database table corresponding to the source database table contains the target column, and to put the target data processing operation into the target queue to be executed, wherein the target database table is a database table in the target database; The execution determination unit 408 is used to determine an operation group to be executed based on at least one target data processing operation in the target queue to be executed when the target queue to be executed meets the preset batching conditions, so as to execute at least one target data processing operation in the target database.

[0104] Optionally, the acquisition and parsing unit 402 includes a data acquisition module and a splitting determiner, wherein, The data acquisition module is used to acquire the operation log records of the source database, parse and process the operation log records, and obtain the initial data processing operation. The splitting decision-maker is used to split the initial data processing operation, determine the decision result, and determine the target data processing operation based on the decision result. Optionally, the splitting determiner is further configured to: Based on the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation, the initial data processing operation is split and judged to determine the judgment result.

[0105] Optionally, the splitting determiner is further configured to: If the operation type of the initial data processing operation is determined to be a data update operation, and the target column is determined to have a value update based on the data information corresponding to the initial data processing operation, then the determination result is determined to be splitting. Otherwise, the determination result is determined to be no splitting.

[0106] Optionally, the splitting determiner is further configured to: If the determination result is split, the initial data processing operation is split to obtain the target data processing operation; If the determination result is no splitting, the initial data processing operation is determined as the target data processing operation.

[0107] Optionally, the group allocation unit 404 is further configured to: The target database table in the target database is determined based on the source database table, wherein there is a corresponding relationship between the source database table and the target database table; The target queue group to be executed is determined based on the source identifier of the source database table and / or the target identifier of the target database table.

[0108] Optionally, the queue allocation unit 406 is further configured to: If the target database table corresponding to the source database table contains a target column, the target value corresponding to the target data processing operation is determined according to the operation log record, wherein the target value is the value of the target column in the target data processing operation; The target queue to be executed is determined from multiple queues to be executed in the target queue group by hashing the target value.

[0109] Optionally, the queue allocation unit 406 is further configured to: If the target database table corresponding to the source database table does not contain the target column, the target data processing operation is placed in one of the execution queues contained in the target execution queue group.

[0110] Optionally, the execution determination unit 408 includes a concurrency determiner, wherein, The concurrency determiner is used to determine the target data processing operation at the target position in the target queue to be executed, and to determine the time when the target data processing operation at the target position is placed into the target queue to be executed. If the time interval between the placement time and the current time exceeds a preset time interval, the target queue to be executed is determined to meet the preset batching condition.

[0111] Optionally, the execution determination unit 408 is further configured to: If the target queue does not contain a group of operations to be executed, the group of operations to be executed is placed in the concurrent execution queue, wherein the concurrent execution queue is used to enable the target database to retrieve the group of operations to be executed.

[0112] Optionally, the execution determination unit 408 is further configured to: When the concurrent execution queue contains at least two target queues corresponding to operation groups to be executed, at least two target threads corresponding to the at least two target queues to be executed are concurrently executed in the target database, wherein one target queue corresponds to one target thread, and the target thread is used to execute at least one target data processing operation in the operation group to be executed.

[0113] This specification provides a data synchronization system in one embodiment. By acquiring and parsing the operation log records of the source database, it obtains the target data processing operation. By executing this target data processing operation in the target database, it can cause the same data changes in the target database as in the source database, thereby achieving data synchronization between the source and target databases. Specifically, when the target data processing operation needs to be batch-processed and then written to the target database, a target execution queue group is determined based on the source database table corresponding to the target data processing operation. The source database table corresponds to the target database table in the target database. If it is determined that the target database table contains the target column, the target execution queue group is selected... A target queue for execution is defined, and target data processing operations are placed into the appropriate queue. If the target queue for execution meets the preset batching conditions, a group of operations to be executed is determined based on at least one target data processing operation in the target queue. This enables data batching for at least one target data processing operation in the target queue. When target data processing operations are allocated to the target queue based on database tables and target columns, the hierarchical data batching method based on database tables and target columns can batch data for as many data processing operations as possible while ensuring data synchronization consistency, achieving high-performance and low-disturbance data synchronization in the target database.

[0114] The above is an illustrative scheme of a data synchronization system according to this embodiment. It should be noted that the technical solution of this data synchronization system and the technical solution of the data synchronization method described above belong to the same concept. For details not described in detail in the technical solution of the data synchronization system, please refer to the description of the technical solution of the data synchronization method described above.

[0115] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0116] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0117] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0118] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0119] The processor 520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data synchronization method.

[0120] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the data synchronization method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the data synchronization method embodiments.

[0121] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data synchronization method.

[0122] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are described simply because they are substantially similar to the data synchronization method embodiments; relevant parts can be referred to the descriptions of the data synchronization method embodiments.

[0123] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data synchronization method.

[0124] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data synchronization method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data synchronization method described above.

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0129] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data synchronization method, comprising: Obtain and parse the operation log records of the source database to obtain the target data processing operations; Determine the source database table corresponding to the target data processing operation, and determine the target queue group to be executed based on the source database table; If the target database table corresponding to the source database table contains a target column, a target execution queue is determined from the target execution queue group, and the target data processing operation is placed into the target execution queue, wherein the target database table is a database table in the target database; If the target queue to be executed is determined to meet the preset batching conditions, an operation group to be executed is determined based on at least one target data processing operation in the target queue to be executed, so as to execute at least one target data processing operation in the operation group to be executed in the target database.

2. The data synchronization method according to claim 1, wherein obtaining and parsing the operation log records of the source database to obtain the target data processing operation includes: Obtain the operation log records from the source database, parse and process the operation log records to obtain the initial data processing operations; The initial data processing operation is broken down and judged, the judgment result is determined, and the target data processing operation is determined based on the judgment result.

3. The data synchronization method according to claim 2, wherein the step of splitting and determining the initial data processing operation to determine the determination result includes: Based on the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation, the initial data processing operation is split and judged to determine the judgment result.

4. The data synchronization method according to claim 3, wherein the step of splitting and determining the initial data processing operation based on the operation type of the initial data processing operation and the data information corresponding to the initial data processing operation, and determining the determination result, includes: If the operation type of the initial data processing operation is determined to be a data update operation, and the target column is determined to have a value update based on the data information corresponding to the initial data processing operation, then the determination result is determined to be splitting. Otherwise, the determination result is determined to be no splitting.

5. The data synchronization method according to any one of claims 2-4, wherein determining the target data processing operation based on the determination result includes: If the determination result is split, the initial data processing operation is split to obtain the target data processing operation; If the determination result is no splitting, the initial data processing operation is determined as the target data processing operation.

6. The data synchronization method according to claim 1 or 2, wherein determining the target queue group to be executed based on the source database table includes: The target database table in the target database is determined based on the source database table, wherein there is a corresponding relationship between the source database table and the target database table; The target queue group to be executed is determined based on the source identifier of the source database table and / or the target identifier of the target database table.

7. The data synchronization method according to claim 1, wherein determining the target queue to be executed from the target queue group when the target database table corresponding to the source database table contains a target column includes: If the target database table corresponding to the source database table contains a target column, the target value corresponding to the target data processing operation is determined according to the operation log record, wherein the target value is the value of the target column in the target data processing operation; The target queue to be executed is determined from multiple queues to be executed in the target queue group by hashing the target value.

8. The data synchronization method according to claim 1 or 7, further comprising: If the target database table corresponding to the source database table does not contain the target column, the target data processing operation is placed in one of the execution queues contained in the target execution queue group.

9. The data synchronization method according to any one of claims 1-4 and 7, wherein determining that the target queue to be executed meets the preset batching conditions includes: Determine the target data processing operation at the target position in the target queue to be executed, and determine the time when the target data processing operation at the target position is placed into the target queue to be executed; If the time interval between the insertion time and the current time exceeds a preset time interval, the target queue to be executed is determined to meet the preset batching conditions.

10. The data synchronization method according to any one of claims 1-4 and 7, further comprising, after determining the group of operations to be executed based on at least one target data processing operation in the target queue to be executed: If the target queue does not contain a group of operations to be executed, the group of operations to be executed is placed in the concurrent execution queue, wherein the concurrent execution queue is used to enable the target database to retrieve the group of operations to be executed.

11. The data synchronization method according to claim 10, wherein executing at least one target data processing operation from the group of operations to be executed in the target database comprises: When the concurrent execution queue contains at least two target queues corresponding to operation groups to be executed, at least two target threads corresponding to the at least two target queues to be executed are concurrently executed in the target database, wherein one target queue to be executed corresponds to one target thread, and the target thread is used to execute at least one target data processing operation in the operation group to be executed.

12. A data synchronization system, comprising an acquisition and parsing unit, a group allocation unit, a queue allocation unit, and an execution determination unit, wherein, The acquisition and parsing unit is used to acquire and parse the operation log records of the source database to obtain the target data processing operation; The group allocation unit is used to determine the source database table corresponding to the target data processing operation, and to determine the target queue group to be executed based on the source database table; The queue allocation unit is used to determine the target queue to be executed from the target queue group when the target database table corresponding to the source database table contains the target column, and to put the target data processing operation into the target queue to be executed, wherein the target database table is a database table in the target database; The execution determination unit is used to determine an operation group to be executed based on at least one target data processing operation in the target queue to be executed, when it is determined that the target queue to be executed meets the preset batching conditions, so as to execute at least one target data processing operation in the target database.

13. The data synchronization system according to claim 12, wherein the acquisition and parsing unit comprises a data acquisition module and a splitting determiner, wherein, The data acquisition module is used to acquire the operation log records of the source database, parse and process the operation log records, and obtain the initial data processing operation. The splitting decision-maker is used to split the initial data processing operation, determine the decision result, and determine the target data processing operation based on the decision result.

14. The data synchronization system according to claim 12 or 13, wherein the execution determination unit includes a concurrency determiner, wherein, The concurrency determiner is used to determine the target data processing operation at the target position in the target queue to be executed, and to determine the time when the target data processing operation at the target position is placed into the target queue to be executed. If the time interval between the placement time and the current time exceeds a preset time interval, the target queue to be executed is determined to meet the preset batching condition.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data synchronization method according to any one of claims 1 to 11.

16. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data synchronization method according to any one of claims 1 to 11.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data synchronization method according to any one of claims 1 to 11.

Images